2024-03-30T00:28:45.524 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-30T00:28:45.525 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254 -- /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254/orig.config.yaml 2024-03-30T00:28:45.579 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T00:28:45.678 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254 branch: wip-rishabh-testing-20240327.051042-reef description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} email: null first_in_suite: false job_id: '7629254' kernel: client: branch: testing flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-rishabh-testing-20240327.051042-reef ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.dir_split install: ceph: flavor: default sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240327.051042-reef sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f owner: scheduled_rishabh@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8052 sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f sleep_before_teardown: 0 subset: 497/512 suite: fs suite_branch: wip-rishabh-testing-20240327.051042-reef suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA6XbZFTDc54XT+aHs3Y2K72HMrW5XLMk+7FAiTHcVxVgTja3iUR1A0aKnE602FFJd1crdGwHYrks5TRnOyHJ+8= smithi130.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMX4g0uFsn++8qv8DwFoauiwYRSbKZAeu6KCLqrAg8f6uAXA+IM9hl1MkMM1NlTwd9IdHWv/V9wA/DbTjf80MPY= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNwPJjYQmBQoAIcUGr6zYEui9LBk9TBV9mpoOZxEWpIAvCfb6aY5bd5GXH+KO6A/5kNuWWClpHLNZKdGzdie1Jw= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - workunit: clients: all: - suites/blogbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-29_18:05:24 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-30T00:28:45.678 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa; will attempt to use it 2024-03-30T00:28:45.679 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks 2024-03-30T00:28:45.679 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-30T00:28:45.680 INFO:teuthology.task.internal:Checking packages... 2024-03-30T00:28:45.704 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '470100a4ccbcf7769052bf09a5413881ccf1b97f' 2024-03-30T00:28:45.705 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T00:28:45.705 INFO:teuthology.packaging:ref: None 2024-03-30T00:28:45.705 INFO:teuthology.packaging:tag: None 2024-03-30T00:28:45.705 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240327.051042-reef 2024-03-30T00:28:45.705 INFO:teuthology.packaging:sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T00:28:45.705 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-rishabh-testing-20240327.051042-reef 2024-03-30T00:28:45.920 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1018.g470100a4 2024-03-30T00:28:45.922 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-30T00:28:45.928 INFO:teuthology.task.internal:no buildpackages task found 2024-03-30T00:28:45.928 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-30T00:28:45.934 INFO:teuthology.task.internal:Saving configuration 2024-03-30T00:28:45.953 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-30T00:28:45.959 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T00:28:45.979 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 00:23:46.500069', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA6XbZFTDc54XT+aHs3Y2K72HMrW5XLMk+7FAiTHcVxVgTja3iUR1A0aKnE602FFJd1crdGwHYrks5TRnOyHJ+8='} 2024-03-30T00:28:46.000 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 00:23:46.498528', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMX4g0uFsn++8qv8DwFoauiwYRSbKZAeu6KCLqrAg8f6uAXA+IM9hl1MkMM1NlTwd9IdHWv/V9wA/DbTjf80MPY='} 2024-03-30T00:28:46.021 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 00:23:46.501407', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNwPJjYQmBQoAIcUGr6zYEui9LBk9TBV9mpoOZxEWpIAvCfb6aY5bd5GXH+KO6A/5kNuWWClpHLNZKdGzdie1Jw='} 2024-03-30T00:28:46.021 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-30T00:28:46.026 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-30T00:28:46.026 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-30T00:28:46.026 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-30T00:28:46.026 INFO:teuthology.run_tasks:Running task console_log... 2024-03-30T00:28:46.123 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4058584160>, signals=[15]) 2024-03-30T00:28:46.123 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-30T00:28:46.130 INFO:teuthology.task.internal:Opening connections... 2024-03-30T00:28:46.130 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-03-30T00:28:46.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T00:28:46.206 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2024-03-30T00:28:46.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T00:28:46.280 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-03-30T00:28:46.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T00:28:46.355 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-30T00:28:46.361 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-03-30T00:28:46.382 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-03-30T00:28:46.382 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-03-30T00:28:46.441 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="Red Hat Enterprise Linux" 2024-03-30T00:28:46.441 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="8.6 (Ootpa)" 2024-03-30T00:28:46.441 INFO:teuthology.orchestra.run.smithi116.stdout:ID="rhel" 2024-03-30T00:28:46.441 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="fedora" 2024-03-30T00:28:46.441 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="8.6" 2024-03-30T00:28:46.441 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el8" 2024-03-30T00:28:46.441 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-30T00:28:46.441 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-03-30T00:28:46.441 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-30T00:28:46.442 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://www.redhat.com/" 2024-03-30T00:28:46.442 INFO:teuthology.orchestra.run.smithi116.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-30T00:28:46.442 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T00:28:46.442 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:28:46.442 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-30T00:28:46.442 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-30T00:28:46.442 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-30T00:28:46.442 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-30T00:28:46.443 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-03-30T00:28:46.472 DEBUG:teuthology.orchestra.run.smithi130:> uname -m 2024-03-30T00:28:46.494 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2024-03-30T00:28:46.495 DEBUG:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2024-03-30T00:28:46.553 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="Red Hat Enterprise Linux" 2024-03-30T00:28:46.553 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="8.6 (Ootpa)" 2024-03-30T00:28:46.553 INFO:teuthology.orchestra.run.smithi130.stdout:ID="rhel" 2024-03-30T00:28:46.553 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="fedora" 2024-03-30T00:28:46.554 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="8.6" 2024-03-30T00:28:46.554 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el8" 2024-03-30T00:28:46.554 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-30T00:28:46.554 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31" 2024-03-30T00:28:46.554 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-30T00:28:46.554 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://www.redhat.com/" 2024-03-30T00:28:46.554 INFO:teuthology.orchestra.run.smithi130.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-30T00:28:46.554 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T00:28:46.554 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-30T00:28:46.554 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-30T00:28:46.554 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-30T00:28:46.554 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-30T00:28:46.554 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-30T00:28:46.555 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2024-03-30T00:28:46.580 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-03-30T00:28:46.602 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-03-30T00:28:46.602 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-03-30T00:28:46.661 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="Red Hat Enterprise Linux" 2024-03-30T00:28:46.661 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="8.6 (Ootpa)" 2024-03-30T00:28:46.661 INFO:teuthology.orchestra.run.smithi181.stdout:ID="rhel" 2024-03-30T00:28:46.661 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="fedora" 2024-03-30T00:28:46.661 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="8.6" 2024-03-30T00:28:46.661 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el8" 2024-03-30T00:28:46.661 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-30T00:28:46.662 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-03-30T00:28:46.662 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-30T00:28:46.662 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://www.redhat.com/" 2024-03-30T00:28:46.662 INFO:teuthology.orchestra.run.smithi181.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-30T00:28:46.662 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T00:28:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:28:46.662 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-30T00:28:46.662 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-30T00:28:46.662 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-30T00:28:46.662 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-30T00:28:46.662 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-03-30T00:28:46.680 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-30T00:28:46.692 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-30T00:28:46.697 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-30T00:28:46.698 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-03-30T00:28:46.700 DEBUG:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2024-03-30T00:28:46.702 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-03-30T00:28:46.720 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-30T00:28:46.753 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-30T00:28:46.753 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-03-30T00:28:46.759 DEBUG:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2024-03-30T00:28:46.761 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-03-30T00:28:46.791 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-30T00:28:46.863 INFO:teuthology.run_tasks:Running task kernel... 2024-03-30T00:28:46.879 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-30T00:28:46.879 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-30T00:28:46.879 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-30T00:28:46.880 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T00:28:46.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T00:28:46.899 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2024-03-30T00:28:47.108 DEBUG:teuthology.task.kernel:sha1 for client.0 is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-30T00:28:47.108 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-30T00:28:47.108 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-30T00:28:47.128 INFO:teuthology.orchestra.run.smithi116.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-03-30T00:28:47.128 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-513.9.1.el8_9.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-30T00:28:47.128 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-30T00:28:47.129 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum upgrade -y linux-firmware 2024-03-30T00:28:48.650 INFO:teuthology.orchestra.run.smithi116.stdout:Updating Subscription Management repositories. 2024-03-30T00:28:48.650 INFO:teuthology.orchestra.run.smithi116.stdout:Unable to read consumer identity 2024-03-30T00:28:48.863 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:28:48.863 INFO:teuthology.orchestra.run.smithi116.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-03-30T00:28:48.863 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:28:50.597 INFO:teuthology.orchestra.run.smithi116.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-03-30T00:28:52.597 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 16 MB 00:01 2024-03-30T00:28:56.248 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 468 kB/s | 24 kB 00:00 2024-03-30T00:28:57.061 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-30T00:28:57.062 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-03-30T00:28:57.063 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-30T00:28:57.193 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-30T00:28:57.193 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-30T00:28:57.344 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ 2024-03-30T00:28:57.344 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ --input-file=- 2024-03-30T00:28:57.347 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi116.front.sepia.ceph.com to complete... 2024-03-30T00:29:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30 00:29:11 URL:https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [401601944/401601944] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-30T00:29:11.026 INFO:teuthology.orchestra.run.smithi116.stderr:FINISHED --2024-03-30 00:29:11-- 2024-03-30T00:29:11.026 INFO:teuthology.orchestra.run.smithi116.stderr:Total wall clock time: 14s 2024-03-30T00:29:11.026 INFO:teuthology.orchestra.run.smithi116.stderr:Downloaded: 1 files, 383M in 13s (28.6 MB/s) 2024-03-30T00:29:11.029 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-30T00:29:11.029 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-30T00:29:12.863 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying... ######################################## 2024-03-30T00:29:12.954 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing... ######################################## 2024-03-30T00:29:13.128 INFO:teuthology.orchestra.run.smithi116.stdout:Updating / installing... 2024-03-30T00:29:53.616 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-30T00:29:57.832 INFO:teuthology.orchestra.run.smithi116.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T00:30:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T00:32:28.878 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-30T00:32:28.878 DEBUG:teuthology.orchestra.run.smithi116:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-30T00:32:28.958 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-30T00:32:28.958 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-30T00:32:28.958 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-30T00:32:28.958 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-30T00:32:28.958 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-30T00:32:28.958 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-30T00:32:28.958 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-30T00:32:28.958 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-30T00:32:28.958 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-30T00:32:28.958 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-30T00:32:28.959 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-30T00:32:28.959 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-30T00:32:28.959 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-30T00:32:28.959 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-30T00:32:28.959 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-30T00:32:28.959 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-30T00:32:28.959 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-30T00:32:28.959 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-30T00:32:28.959 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-30T00:32:28.959 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-30T00:32:28.959 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-30T00:32:28.959 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-30T00:32:28.959 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-30T00:32:28.959 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-30T00:32:28.960 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-30T00:32:28.960 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-30T00:32:28.960 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-30T00:32:28.960 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-30T00:32:28.960 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-30T00:32:28.960 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-30T00:32:28.961 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-30T00:32:28.961 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-30T00:32:28.961 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-30T00:32:28.961 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-30T00:32:28.961 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-30T00:32:28.961 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-30T00:32:28.961 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-30T00:32:28.961 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-30T00:32:28.961 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-30T00:32:28.961 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-30T00:32:28.961 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-30T00:32:28.962 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-30T00:32:28.962 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-30T00:32:28.962 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-30T00:32:28.962 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-30T00:32:28.962 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-30T00:32:28.962 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-30T00:32:28.962 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-30T00:32:28.962 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-30T00:32:28.962 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-30T00:32:28.962 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-30T00:32:28.962 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-30T00:32:28.962 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-30T00:32:28.962 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-30T00:32:28.963 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-30T00:32:28.964 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-30T00:32:28.964 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-30T00:32:28.964 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-30T00:32:28.964 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-30T00:32:28.964 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-30T00:32:28.964 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-30T00:32:28.964 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-30T00:32:28.964 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-30T00:32:28.964 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-30T00:32:28.964 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-30T00:32:28.964 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-30T00:32:28.964 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-30T00:32:28.964 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-30T00:32:28.964 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-30T00:32:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-30T00:32:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-30T00:32:28.967 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-30T00:32:28.968 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-30T00:32:28.968 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-30T00:32:28.968 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-30T00:32:28.968 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-30T00:32:28.968 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-30T00:32:28.968 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-30T00:32:28.968 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-30T00:32:28.968 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-30T00:32:28.968 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-30T00:32:28.968 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-30T00:32:28.968 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-30T00:32:28.968 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-30T00:32:28.968 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-30T00:32:28.968 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-30T00:32:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-30T00:32:28.970 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-30T00:32:28.971 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-30T00:32:28.971 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-30T00:32:28.971 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-30T00:32:28.971 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-30T00:32:28.971 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-30T00:32:28.971 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-30T00:32:28.971 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-30T00:32:28.971 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-30T00:32:28.971 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-30T00:32:28.971 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-30T00:32:28.971 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-30T00:32:28.971 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-30T00:32:28.971 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-30T00:32:28.971 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-30T00:32:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-30T00:32:28.973 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-30T00:32:28.973 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-30T00:32:28.973 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-30T00:32:28.973 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-30T00:32:28.973 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-30T00:32:28.973 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-30T00:32:28.973 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-30T00:32:28.973 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-30T00:32:28.973 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-30T00:32:28.973 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-30T00:32:28.973 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-30T00:32:28.973 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-30T00:32:28.973 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-30T00:32:28.973 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-30T00:32:28.974 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-30T00:32:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-30T00:32:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-30T00:32:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-30T00:32:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-30T00:32:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-30T00:32:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-30T00:32:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-30T00:32:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-30T00:32:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-30T00:32:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-30T00:32:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-30T00:32:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-30T00:32:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-30T00:32:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-30T00:32:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-30T00:32:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-30T00:32:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-30T00:32:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-30T00:32:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-30T00:32:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-30T00:32:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-30T00:32:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-30T00:32:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-30T00:32:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-30T00:32:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-30T00:32:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-30T00:32:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-30T00:32:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-30T00:32:28.977 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-30T00:32:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-30T00:32:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-30T00:32:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-30T00:32:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-30T00:32:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-30T00:32:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-30T00:32:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-30T00:32:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-30T00:32:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-30T00:32:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-30T00:32:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-30T00:32:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-30T00:32:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-30T00:32:28.978 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-30T00:32:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-30T00:32:28.980 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-30T00:32:28.981 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-30T00:32:28.982 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-30T00:32:28.983 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-30T00:32:28.984 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-30T00:32:28.984 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-30T00:32:28.984 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-30T00:32:28.984 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-30T00:32:28.984 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-30T00:32:28.984 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-30T00:32:28.984 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-30T00:32:28.984 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-30T00:32:28.984 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-30T00:32:28.984 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-30T00:32:28.984 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-30T00:32:28.984 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-30T00:32:28.984 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-30T00:32:28.984 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-30T00:32:28.985 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-30T00:32:28.986 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-30T00:32:28.986 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-30T00:32:28.986 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-30T00:32:28.986 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-30T00:32:28.986 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-30T00:32:28.986 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-30T00:32:28.986 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-30T00:32:28.986 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-30T00:32:28.986 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-30T00:32:28.986 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-30T00:32:28.986 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-30T00:32:28.986 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-30T00:32:28.986 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-30T00:32:28.986 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-30T00:32:28.987 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-30T00:32:28.988 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-30T00:32:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-30T00:32:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-30T00:32:28.991 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-30T00:32:28.992 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-30T00:32:28.993 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-30T00:32:28.993 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-30T00:32:28.993 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-30T00:32:28.993 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-30T00:32:28.993 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-30T00:32:28.993 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-30T00:32:28.993 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-30T00:32:28.993 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-30T00:32:28.993 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-30T00:32:28.993 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-30T00:32:28.993 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-30T00:32:28.993 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-30T00:32:28.993 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-30T00:32:28.993 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-30T00:32:28.994 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-30T00:32:28.994 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-30T00:32:28.994 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-30T00:32:28.994 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-30T00:32:28.994 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-30T00:32:28.994 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-30T00:32:28.994 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-30T00:32:28.994 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-30T00:32:28.994 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-30T00:32:28.994 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-30T00:32:28.994 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-30T00:32:28.994 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-30T00:32:28.994 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-30T00:32:28.994 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-30T00:32:28.995 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-30T00:32:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-30T00:32:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-30T00:32:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-30T00:32:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-30T00:32:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-30T00:32:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-30T00:32:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-30T00:32:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-30T00:32:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-30T00:32:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-30T00:32:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-30T00:32:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-30T00:32:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-30T00:32:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-30T00:32:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-30T00:32:28.998 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-30T00:32:28.999 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-30T00:32:29.000 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-30T00:32:29.001 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-30T00:32:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-30T00:32:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-30T00:32:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-30T00:32:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-30T00:32:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-30T00:32:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-30T00:32:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-30T00:32:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-30T00:32:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-30T00:32:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-30T00:32:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-30T00:32:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-30T00:32:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-30T00:32:29.002 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-30T00:32:29.003 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-30T00:32:29.004 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-30T00:32:29.005 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-30T00:32:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-30T00:32:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-30T00:32:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-30T00:32:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-30T00:32:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-30T00:32:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-30T00:32:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-30T00:32:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-30T00:32:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-30T00:32:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-30T00:32:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-30T00:32:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-30T00:32:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-30T00:32:29.006 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-30T00:32:29.007 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-30T00:32:29.008 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-30T00:32:29.009 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-30T00:32:29.010 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-30T00:32:29.010 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-30T00:32:29.010 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-30T00:32:29.010 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-30T00:32:29.010 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-30T00:32:29.010 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-30T00:32:29.010 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-30T00:32:29.010 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-30T00:32:29.010 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-30T00:32:29.010 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-30T00:32:29.010 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-30T00:32:29.010 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-30T00:32:29.010 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-30T00:32:29.010 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-30T00:32:29.011 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-30T00:32:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-30T00:32:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-30T00:32:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-30T00:32:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-30T00:32:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-30T00:32:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-30T00:32:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-30T00:32:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-30T00:32:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-30T00:32:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-30T00:32:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-30T00:32:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-30T00:32:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-30T00:32:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-30T00:32:29.013 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-30T00:32:29.014 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-30T00:32:29.015 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-30T00:32:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-30T00:32:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-30T00:32:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-30T00:32:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-30T00:32:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-30T00:32:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-30T00:32:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-30T00:32:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-30T00:32:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-30T00:32:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-30T00:32:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-30T00:32:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-30T00:32:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-30T00:32:29.016 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-30T00:32:29.017 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-30T00:32:29.018 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-30T00:32:29.019 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-30T00:32:29.020 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-30T00:32:29.021 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-30T00:32:29.022 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-30T00:32:29.023 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-30T00:32:29.024 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-30T00:32:29.025 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-30T00:32:29.026 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-30T00:32:29.027 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-30T00:32:29.028 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-30T00:32:29.028 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-30T00:32:29.028 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-30T00:32:29.028 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-30T00:32:29.028 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-30T00:32:29.028 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-30T00:32:29.028 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-30T00:32:29.028 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-30T00:32:29.028 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-30T00:32:29.028 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-30T00:32:29.028 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-30T00:32:29.028 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-30T00:32:29.028 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-30T00:32:29.028 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-30T00:32:29.029 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-30T00:32:29.030 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-30T00:32:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-30T00:32:29.032 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-30T00:32:29.032 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-30T00:32:29.032 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-30T00:32:29.032 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-30T00:32:29.032 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-30T00:32:29.032 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-30T00:32:29.032 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-30T00:32:29.032 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-30T00:32:29.032 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-30T00:32:29.032 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-30T00:32:29.032 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-30T00:32:29.032 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-30T00:32:29.032 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-30T00:32:29.032 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-30T00:32:29.033 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-30T00:32:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-30T00:32:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-30T00:32:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-30T00:32:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-30T00:32:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-30T00:32:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-30T00:32:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-30T00:32:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-30T00:32:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-30T00:32:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-30T00:32:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-30T00:32:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-30T00:32:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-30T00:32:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-30T00:32:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-30T00:32:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-30T00:32:29.037 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-30T00:32:29.038 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-30T00:32:29.038 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-30T00:32:29.038 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-30T00:32:29.038 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-30T00:32:29.038 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-30T00:32:29.038 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-30T00:32:29.038 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-30T00:32:29.038 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-30T00:32:29.038 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-30T00:32:29.038 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-30T00:32:29.038 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-30T00:32:29.038 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-30T00:32:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-30T00:32:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-30T00:32:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-30T00:32:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-30T00:32:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-30T00:32:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-30T00:32:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-30T00:32:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-30T00:32:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-30T00:32:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-30T00:32:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-30T00:32:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-30T00:32:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-30T00:32:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-30T00:32:29.040 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-30T00:32:29.041 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-30T00:32:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-30T00:32:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-30T00:32:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-30T00:32:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-30T00:32:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-30T00:32:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-30T00:32:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-30T00:32:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-30T00:32:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-30T00:32:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-30T00:32:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-30T00:32:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-30T00:32:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-30T00:32:29.042 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-30T00:32:29.043 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-30T00:32:29.044 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-30T00:32:29.045 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-30T00:32:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-30T00:32:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-30T00:32:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-30T00:32:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-30T00:32:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-30T00:32:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-30T00:32:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-30T00:32:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-30T00:32:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-30T00:32:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-30T00:32:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-30T00:32:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-30T00:32:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-30T00:32:29.046 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-30T00:32:29.047 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-30T00:32:29.048 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-30T00:32:29.049 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-30T00:32:29.050 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-30T00:32:29.051 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-30T00:32:29.052 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-30T00:32:29.053 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-30T00:32:29.054 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-30T00:32:29.054 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-30T00:32:29.054 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-30T00:32:29.054 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-30T00:32:29.054 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-30T00:32:29.054 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-30T00:32:29.054 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-30T00:32:29.054 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-30T00:32:29.054 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-30T00:32:29.054 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-30T00:32:29.054 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-30T00:32:29.054 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-30T00:32:29.054 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-30T00:32:29.054 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-30T00:32:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-30T00:32:29.056 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-30T00:32:29.057 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-30T00:32:29.057 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-30T00:32:29.057 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-30T00:32:29.057 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-30T00:32:29.057 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-30T00:32:29.057 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-30T00:32:29.057 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-30T00:32:29.057 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-30T00:32:29.057 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-30T00:32:29.057 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-30T00:32:29.057 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-30T00:32:29.057 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-30T00:32:29.057 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-30T00:32:29.057 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-30T00:32:29.058 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-30T00:32:29.058 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-30T00:32:29.058 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-30T00:32:29.058 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-30T00:32:29.058 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-30T00:32:29.058 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-30T00:32:29.058 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-30T00:32:29.058 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-30T00:32:29.058 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-30T00:32:29.058 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-30T00:32:29.058 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-30T00:32:29.058 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-30T00:32:29.058 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-30T00:32:29.058 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-30T00:32:29.059 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-30T00:32:29.060 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-30T00:32:29.060 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-30T00:32:29.060 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-30T00:32:29.060 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-30T00:32:29.060 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-30T00:32:29.060 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-30T00:32:29.060 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-30T00:32:29.060 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-30T00:32:29.060 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-30T00:32:29.060 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-30T00:32:29.060 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-30T00:32:29.060 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-30T00:32:29.060 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-30T00:32:29.060 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-30T00:32:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-30T00:32:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-30T00:32:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-30T00:32:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-30T00:32:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-30T00:32:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-30T00:32:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-30T00:32:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-30T00:32:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-30T00:32:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-30T00:32:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-30T00:32:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-30T00:32:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-30T00:32:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-30T00:32:29.062 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-30T00:32:29.063 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-30T00:32:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-30T00:32:29.065 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-30T00:32:29.066 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-30T00:32:29.066 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-30T00:32:29.066 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-30T00:32:29.066 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-30T00:32:29.066 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-30T00:32:29.066 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-30T00:32:29.066 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-30T00:32:29.066 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-30T00:32:29.066 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-30T00:32:29.066 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-30T00:32:29.066 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-30T00:32:29.066 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-30T00:32:29.066 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-30T00:32:29.066 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-30T00:32:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-30T00:32:29.068 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-30T00:32:29.069 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-30T00:32:29.070 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-30T00:32:29.070 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-30T00:32:29.070 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-30T00:32:29.070 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-30T00:32:29.070 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-30T00:32:29.070 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-30T00:32:29.070 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-30T00:32:29.070 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-30T00:32:29.070 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-30T00:32:29.070 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-30T00:32:29.070 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-30T00:32:29.070 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-30T00:32:29.070 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-30T00:32:29.070 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-30T00:32:29.071 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-30T00:32:29.072 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-30T00:32:29.073 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-30T00:32:29.074 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-30T00:32:29.074 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-30T00:32:29.074 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-30T00:32:29.074 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-30T00:32:29.074 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-30T00:32:29.074 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-30T00:32:29.074 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-30T00:32:29.074 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-30T00:32:29.074 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-30T00:32:29.074 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-30T00:32:29.074 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-30T00:32:29.074 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-30T00:32:29.074 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-30T00:32:29.074 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-30T00:32:29.075 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-30T00:32:29.076 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-30T00:32:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-30T00:32:29.078 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-30T00:32:29.078 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-30T00:32:29.078 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-30T00:32:29.078 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-30T00:32:29.078 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-30T00:32:29.078 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-30T00:32:29.078 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-30T00:32:29.078 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-30T00:32:29.078 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-30T00:32:29.078 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-30T00:32:29.078 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-30T00:32:29.078 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-30T00:32:29.078 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-30T00:32:29.078 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-30T00:32:29.079 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-30T00:32:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-30T00:32:29.081 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-30T00:32:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-30T00:32:29.083 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-30T00:32:29.084 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-30T00:32:29.085 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-30T00:32:29.086 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-30T00:32:29.087 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-30T00:32:29.088 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-30T00:32:29.089 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-30T00:32:29.090 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-30T00:32:29.090 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-30T00:32:29.090 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-30T00:32:29.090 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-30T00:32:29.090 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-30T00:32:29.090 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-30T00:32:29.090 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-30T00:32:29.090 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-30T00:32:29.094 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-30T00:32:29.094 DEBUG:teuthology.orchestra.run.smithi116:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-30T00:32:29.121 INFO:teuthology.orchestra.run.smithi116.stdout:postinstall scriptlet (using /bin/sh): 2024-03-30T00:32:29.121 INFO:teuthology.orchestra.run.smithi116.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-30T00:32:29.121 INFO:teuthology.orchestra.run.smithi116.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-30T00:32:29.122 INFO:teuthology.orchestra.run.smithi116.stdout:fi 2024-03-30T00:32:29.122 INFO:teuthology.orchestra.run.smithi116.stdout:for file in vmlinuz System.map config; do 2024-03-30T00:32:29.122 INFO:teuthology.orchestra.run.smithi116.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-30T00:32:29.122 INFO:teuthology.orchestra.run.smithi116.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-30T00:32:29.122 INFO:teuthology.orchestra.run.smithi116.stdout: fi 2024-03-30T00:32:29.122 INFO:teuthology.orchestra.run.smithi116.stdout:done 2024-03-30T00:32:29.122 INFO:teuthology.orchestra.run.smithi116.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-30T00:32:29.122 INFO:teuthology.orchestra.run.smithi116.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-30T00:32:29.122 INFO:teuthology.orchestra.run.smithi116.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-30T00:32:29.122 INFO:teuthology.orchestra.run.smithi116.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-30T00:32:29.122 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-30T00:32:29.122 INFO:teuthology.orchestra.run.smithi116.stdout:fi 2024-03-30T00:32:29.122 INFO:teuthology.orchestra.run.smithi116.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-30T00:32:29.122 INFO:teuthology.orchestra.run.smithi116.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-30T00:32:29.123 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-30T00:32:29.123 INFO:teuthology.orchestra.run.smithi116.stdout:fi 2024-03-30T00:32:29.123 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2024-03-30T00:32:29.192 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2024-03-30T00:32:29.192 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2024-03-30T00:32:29.192 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.02 2024-03-30T00:32:29.192 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 123.el8 2024-03-30T00:32:29.192 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2024-03-30T00:32:29.192 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Thu 14 Dec 2023 12:39:42 AM UTC 2024-03-30T00:32:29.192 INFO:teuthology.orchestra.run.smithi116.stdout:Group : System Environment/Base 2024-03-30T00:32:29.192 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 9265450 2024-03-30T00:32:29.192 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2024-03-30T00:32:29.192 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout:Relocations : (not relocatable) 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : Red Hat, Inc. 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : Red Hat, Inc. 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2024-03-30T00:32:29.193 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:32:29.194 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T00:32:29.194 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T00:32:29.195 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 6.8.0-g1df0c7e8a235 2024-03-30T00:32:29.195 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T00:32:29.952 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-03-30T00:32:32.219 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-03-30T00:32:32.221 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-03-30T00:32:32.239 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.2KOfkqtmI7 2024-03-30T00:32:32.239 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2KOfkqtmI7 2024-03-30T00:32:32.312 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.2KOfkqtmI7 2024-03-30T00:32:32.455 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.2KOfkqtmI7 is 6KB 2024-03-30T00:32:32.504 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.2KOfkqtmI7 2024-03-30T00:32:32.520 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T00:32:32.590 INFO:teuthology.orchestra.run.smithi116.stdout:2db68f91c8f90721a6596efe657a5324-0-rescue.conf 2024-03-30T00:32:32.590 INFO:teuthology.orchestra.run.smithi116.stdout:2db68f91c8f90721a6596efe657a5324-6.8.0-g1df0c7e8a235.conf 2024-03-30T00:32:32.590 INFO:teuthology.orchestra.run.smithi116.stdout:963f8b6c99184de696879b3f7da9ecce-0-rescue.conf 2024-03-30T00:32:32.590 INFO:teuthology.orchestra.run.smithi116.stdout:963f8b6c99184de696879b3f7da9ecce-4.18.0-372.9.1.el8.x86_64.conf 2024-03-30T00:32:32.590 INFO:teuthology.orchestra.run.smithi116.stdout:963f8b6c99184de696879b3f7da9ecce-4.18.0-513.9.1.el8_9.x86_64.conf 2024-03-30T00:32:32.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 2db68f91c8f90721a6596efe657a5324-6.8.0-g1df0c7e8a235 2024-03-30T00:32:32.738 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-03-30T00:33:02.741 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-30T00:33:02.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-30T00:33:02.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T00:33:21.182 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-30T00:33:30.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-30T00:33:30.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T00:33:33.246 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-30T00:33:45.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-30T00:33:45.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T00:34:17.091 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-30T00:34:17.500 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-30T00:34:17.500 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g1df0c7e8a235"... 2024-03-30T00:34:17.500 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-30T00:34:17.541 INFO:teuthology.orchestra.run.smithi116.stdout:6.8.0-g1df0c7e8a235 2024-03-30T00:34:17.541 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-30T00:34:17.541 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T00:34:17.541 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2024-03-30T00:34:18.542 DEBUG:teuthology.parallel:result is None 2024-03-30T00:34:18.542 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-30T00:34:18.554 INFO:teuthology.task.internal:Creating test directory... 2024-03-30T00:34:18.554 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T00:34:18.559 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T00:34:18.562 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T00:34:18.601 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-30T00:34:18.608 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-30T00:34:18.665 INFO:teuthology.task.internal:Creating archive directory... 2024-03-30T00:34:18.666 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T00:34:18.670 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T00:34:18.672 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T00:34:18.725 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-30T00:34:18.732 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-30T00:34:18.732 DEBUG:teuthology.orchestra.run.smithi130:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T00:34:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T00:34:18.764 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T00:34:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T00:34:18.783 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T00:34:18.787 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T00:34:18.806 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T00:34:18.859 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T00:34:18.879 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T00:34:18.880 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T00:34:18.894 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T00:34:18.936 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T00:34:18.990 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T00:34:18.994 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-30T00:34:19.000 INFO:teuthology.task.internal:Configuring sudo... 2024-03-30T00:34:19.000 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T00:34:19.004 DEBUG:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T00:34:19.006 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T00:34:19.065 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-30T00:34:19.074 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-30T00:34:19.074 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T00:34:19.077 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T00:34:19.079 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T00:34:19.114 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T00:34:19.216 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T00:34:19.308 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T00:34:19.373 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T00:34:19.457 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:34:19.457 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T00:34:19.520 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T00:34:19.568 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T00:34:19.622 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T00:34:19.700 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T00:34:19.750 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-30T00:34:19.750 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T00:34:19.817 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T00:34:19.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T00:34:19.918 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T00:34:19.999 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T00:34:20.049 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-30T00:34:20.049 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T00:34:20.118 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-03-30T00:34:20.122 DEBUG:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2024-03-30T00:34:20.124 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-03-30T00:34:20.176 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T00:34:20.233 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T00:34:20.233 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T00:34:20.608 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-30T00:34:20.616 INFO:teuthology.task.internal:Starting timer... 2024-03-30T00:34:20.616 INFO:teuthology.run_tasks:Running task pcp... 2024-03-30T00:34:20.710 INFO:teuthology.run_tasks:Running task selinux... 2024-03-30T00:34:20.739 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-30T00:34:20.739 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-03-30T00:34:20.917 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-03-30T00:34:20.921 DEBUG:teuthology.orchestra.run.smithi130:> sudo service auditd rotate 2024-03-30T00:34:21.019 INFO:teuthology.orchestra.run.smithi130.stdout:Rotating logs: 2024-03-30T00:34:21.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-03-30T00:34:21.085 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-03-30T00:34:21.087 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-30T00:34:21.087 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-03-30T00:34:21.137 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-03-30T00:34:21.137 DEBUG:teuthology.orchestra.run.smithi130:> /usr/sbin/getenforce 2024-03-30T00:34:21.159 INFO:teuthology.orchestra.run.smithi130.stdout:Permissive 2024-03-30T00:34:21.159 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-03-30T00:34:21.183 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-03-30T00:34:21.183 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi130.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-03-30T00:34:21.183 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T00:34:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T00:34:21.254 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T00:34:21.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T00:34:21.282 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T00:34:21.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T00:34:21.310 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-30T00:34:21.310 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-03-30T00:34:21.374 DEBUG:teuthology.orchestra.run.smithi130:> sudo /usr/sbin/setenforce permissive 2024-03-30T00:34:21.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-03-30T00:34:21.427 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-30T00:34:21.437 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-30T00:34:21.443 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-30T00:34:21.569 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-30T00:34:21.584 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-30T00:34:21.585 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com,smithi130.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-30T00:45:01.874 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi130.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-03-30T00:45:01.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-30T00:45:01.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T00:45:02.003 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-30T00:45:02.114 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-30T00:45:02.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-30T00:45:02.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T00:45:02.210 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-03-30T00:45:02.281 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-30T00:45:02.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-03-30T00:45:02.282 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T00:45:02.367 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-03-30T00:45:02.438 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-03-30T00:45:02.439 INFO:teuthology.run_tasks:Running task clock... 2024-03-30T00:45:02.452 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-30T00:45:02.453 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T00:45:02.453 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T00:45:02.457 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T00:45:02.457 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T00:45:02.459 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T00:45:02.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T00:45:02.492 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T00:45:02.508 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T00:45:02.514 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T00:45:02.523 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T00:45:02.530 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T00:45:02.555 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T00:45:02.573 INFO:teuthology.orchestra.run.smithi130.stderr:sudo: ntpd: command not found 2024-03-30T00:45:02.586 INFO:teuthology.orchestra.run.smithi130.stdout:506 Cannot talk to daemon 2024-03-30T00:45:02.599 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-03-30T00:45:02.600 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T00:45:02.612 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-03-30T00:45:02.615 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T00:45:02.627 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T00:45:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T00:45:02.646 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-03-30T00:45:02.674 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-03-30T00:45:02.690 INFO:teuthology.orchestra.run.smithi130.stderr:bash: ntpq: command not found 2024-03-30T00:45:02.694 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T00:45:02.694 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-03-30T00:45:02.694 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T00:45:02.694 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T00:45:02.694 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T00:45:02.694 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T00:45:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T00:45:02.709 INFO:teuthology.orchestra.run.smithi181.stderr:bash: ntpq: command not found 2024-03-30T00:45:02.712 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T00:45:02.713 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-03-30T00:45:02.713 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T00:45:02.713 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T00:45:02.713 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T00:45:02.713 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T00:45:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T00:45:02.840 INFO:teuthology.orchestra.run.smithi116.stderr:bash: ntpq: command not found 2024-03-30T00:45:02.847 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T00:45:02.847 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-30T00:45:02.847 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T00:45:02.847 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T00:45:02.847 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T00:45:02.847 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T00:45:02.849 INFO:teuthology.run_tasks:Running task pexec... 2024-03-30T00:45:02.859 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-30T00:45:02.860 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi116.front.sepia.ceph.com 2024-03-30T00:45:02.860 DEBUG:teuthology.orchestra.run.smithi116:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-30T00:45:02.860 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi130.front.sepia.ceph.com 2024-03-30T00:45:02.860 DEBUG:teuthology.orchestra.run.smithi130:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-30T00:45:02.861 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi181.front.sepia.ceph.com 2024-03-30T00:45:02.861 DEBUG:teuthology.orchestra.run.smithi181:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-30T00:45:03.236 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2024-03-30T00:45:03.238 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-03-30T00:45:03.339 INFO:teuthology.orchestra.run.smithi116.stdout:Updating Subscription Management repositories. 2024-03-30T00:45:05.076 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 55 kB/s | 4.1 kB 00:00 2024-03-30T00:45:05.106 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 51 kB/s | 4.1 kB 00:00 2024-03-30T00:45:05.309 INFO:teuthology.orchestra.run.smithi116.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-03-30T00:45:05.432 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-03-30T00:45:05.446 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 kB/s | 4.5 kB 00:00 2024-03-30T00:45:05.753 INFO:teuthology.orchestra.run.smithi116.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-03-30T00:45:08.550 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-30T00:45:08.551 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-30T00:45:08.552 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2024-03-30T00:45:08.552 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-30T00:45:08.552 INFO:teuthology.orchestra.run.smithi130.stdout:Resetting modules: 2024-03-30T00:45:08.552 INFO:teuthology.orchestra.run.smithi130.stdout: container-tools 2024-03-30T00:45:08.552 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-30T00:45:08.552 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-30T00:45:08.552 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-30T00:45:08.552 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-30T00:45:08.556 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-30T00:45:08.557 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-30T00:45:08.558 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-03-30T00:45:08.558 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-30T00:45:08.558 INFO:teuthology.orchestra.run.smithi181.stdout:Resetting modules: 2024-03-30T00:45:08.558 INFO:teuthology.orchestra.run.smithi181.stdout: container-tools 2024-03-30T00:45:08.558 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:45:08.558 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-30T00:45:08.558 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-30T00:45:08.558 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:45:09.023 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-30T00:45:09.024 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-30T00:45:09.025 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-30T00:45:09.025 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-30T00:45:09.025 INFO:teuthology.orchestra.run.smithi116.stdout:Resetting modules: 2024-03-30T00:45:09.025 INFO:teuthology.orchestra.run.smithi116.stdout: container-tools 2024-03-30T00:45:09.025 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:45:09.025 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-30T00:45:09.025 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-30T00:45:09.025 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:45:22.388 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-30T00:45:22.892 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-03-30T00:45:23.187 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-30T00:45:23.667 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2024-03-30T00:45:24.611 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 44 kB/s | 4.1 kB 00:00 2024-03-30T00:45:24.633 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-30T00:45:25.057 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-03-30T00:45:25.247 INFO:teuthology.orchestra.run.smithi116.stdout:Updating Subscription Management repositories. 2024-03-30T00:45:25.374 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-03-30T00:45:25.710 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2024-03-30T00:45:27.190 INFO:teuthology.orchestra.run.smithi116.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 22 kB/s | 4.1 kB 00:00 2024-03-30T00:45:27.654 INFO:teuthology.orchestra.run.smithi116.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 25 kB/s | 4.5 kB 00:00 2024-03-30T00:45:28.144 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-30T00:45:28.306 INFO:teuthology.orchestra.run.smithi181.stdout:========================================================================================================== 2024-03-30T00:45:28.306 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-03-30T00:45:28.306 INFO:teuthology.orchestra.run.smithi181.stdout:========================================================================================================== 2024-03-30T00:45:28.306 INFO:teuthology.orchestra.run.smithi181.stdout:Installing group/module packages: 2024-03-30T00:45:28.306 INFO:teuthology.orchestra.run.smithi181.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-03-30T00:45:28.306 INFO:teuthology.orchestra.run.smithi181.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-03-30T00:45:28.306 INFO:teuthology.orchestra.run.smithi181.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout:Installing module profiles: 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout: container-tools/common 2024-03-30T00:45:28.307 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:45:28.308 INFO:teuthology.orchestra.run.smithi181.stdout:Enabling module streams: 2024-03-30T00:45:28.308 INFO:teuthology.orchestra.run.smithi181.stdout: container-tools rhel8 2024-03-30T00:45:28.308 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:45:28.308 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-30T00:45:28.308 INFO:teuthology.orchestra.run.smithi181.stdout:========================================================================================================== 2024-03-30T00:45:28.308 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-03-30T00:45:28.308 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:45:28.309 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 26 M 2024-03-30T00:45:28.309 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 87 M 2024-03-30T00:45:28.309 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-30T00:45:28.492 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): python3-pyxdg-0.25-16.el8.noarch.rpm 512 kB/s | 94 kB 00:00 2024-03-30T00:45:28.518 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-pytoml-0.1.14-5.git7dea353.el8. 119 kB/s | 25 kB 00:00 2024-03-30T00:45:28.543 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): criu-libs-3.18-4.module+el8.9.0+20326+3 168 kB/s | 39 kB 00:00 2024-03-30T00:45:28.635 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 288 kB/s | 41 kB 00:00 2024-03-30T00:45:28.678 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 6.3 MB/s | 1.0 MB 00:00 2024-03-30T00:45:28.794 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 4.6 MB/s | 739 kB 00:00 2024-03-30T00:45:28.836 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 1.5 MB/s | 239 kB 00:00 2024-03-30T00:45:28.953 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): buildah-1.31.3-3.module+el8.9.0+21243+a 22 MB/s | 8.8 MB 00:00 2024-03-30T00:45:28.994 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-podman-4.6.0-2.module+el8.9.0+2 968 kB/s | 154 kB 00:00 2024-03-30T00:45:29.016 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-30T00:45:29.078 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): netavark-1.7.0-2.module+el8.9.0+21243+ 13 MB/s | 3.7 MB 00:00 2024-03-30T00:45:29.177 INFO:teuthology.orchestra.run.smithi130.stdout:========================================================================================================== 2024-03-30T00:45:29.177 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-03-30T00:45:29.177 INFO:teuthology.orchestra.run.smithi130.stdout:========================================================================================================== 2024-03-30T00:45:29.177 INFO:teuthology.orchestra.run.smithi130.stdout:Installing group/module packages: 2024-03-30T00:45:29.177 INFO:teuthology.orchestra.run.smithi130.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-03-30T00:45:29.177 INFO:teuthology.orchestra.run.smithi130.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-03-30T00:45:29.178 INFO:teuthology.orchestra.run.smithi130.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-03-30T00:45:29.178 INFO:teuthology.orchestra.run.smithi130.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-03-30T00:45:29.178 INFO:teuthology.orchestra.run.smithi130.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-30T00:45:29.178 INFO:teuthology.orchestra.run.smithi130.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-03-30T00:45:29.178 INFO:teuthology.orchestra.run.smithi130.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-03-30T00:45:29.178 INFO:teuthology.orchestra.run.smithi130.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-03-30T00:45:29.178 INFO:teuthology.orchestra.run.smithi130.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-03-30T00:45:29.179 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-30T00:45:29.179 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-03-30T00:45:29.179 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-03-30T00:45:29.179 INFO:teuthology.orchestra.run.smithi130.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-03-30T00:45:29.179 INFO:teuthology.orchestra.run.smithi130.stdout:Installing weak dependencies: 2024-03-30T00:45:29.179 INFO:teuthology.orchestra.run.smithi130.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-30T00:45:29.179 INFO:teuthology.orchestra.run.smithi130.stdout:Installing module profiles: 2024-03-30T00:45:29.179 INFO:teuthology.orchestra.run.smithi130.stdout: container-tools/common 2024-03-30T00:45:29.179 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-30T00:45:29.179 INFO:teuthology.orchestra.run.smithi130.stdout:Enabling module streams: 2024-03-30T00:45:29.180 INFO:teuthology.orchestra.run.smithi130.stdout: container-tools rhel8 2024-03-30T00:45:29.180 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-30T00:45:29.180 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-30T00:45:29.180 INFO:teuthology.orchestra.run.smithi130.stdout:========================================================================================================== 2024-03-30T00:45:29.180 INFO:teuthology.orchestra.run.smithi130.stdout:Install 13 Packages 2024-03-30T00:45:29.180 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-30T00:45:29.181 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 14 MB/s | 2.5 MB 00:00 2024-03-30T00:45:29.181 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 26 M 2024-03-30T00:45:29.181 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 87 M 2024-03-30T00:45:29.181 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-30T00:45:29.203 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+21243+a5 396 kB/s | 49 kB 00:00 2024-03-30T00:45:29.312 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 23 MB/s | 8.1 MB 00:00 2024-03-30T00:45:29.313 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-30T00:45:29.313 INFO:teuthology.orchestra.run.smithi181.stdout:Total 25 MB/s | 26 MB 00:01 2024-03-30T00:45:29.335 INFO:teuthology.orchestra.run.smithi130.stdout:(1/13): criu-libs-3.18-4.module+el8.9.0+20326+3 255 kB/s | 39 kB 00:00 2024-03-30T00:45:29.360 INFO:teuthology.orchestra.run.smithi130.stdout:(2/13): python3-pytoml-0.1.14-5.git7dea353.el8. 138 kB/s | 25 kB 00:00 2024-03-30T00:45:29.415 INFO:teuthology.orchestra.run.smithi130.stdout:(3/13): python3-pyxdg-0.25-16.el8.noarch.rpm 400 kB/s | 94 kB 00:00 2024-03-30T00:45:29.477 INFO:teuthology.orchestra.run.smithi130.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 288 kB/s | 41 kB 00:00 2024-03-30T00:45:29.527 INFO:teuthology.orchestra.run.smithi130.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 6.0 MB/s | 1.0 MB 00:00 2024-03-30T00:45:29.561 INFO:teuthology.orchestra.run.smithi130.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 8.7 MB/s | 739 kB 00:00 2024-03-30T00:45:29.594 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-30T00:45:29.611 INFO:teuthology.orchestra.run.smithi130.stdout:(7/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 2.8 MB/s | 239 kB 00:00 2024-03-30T00:45:29.616 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-30T00:45:29.616 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-30T00:45:29.786 INFO:teuthology.orchestra.run.smithi130.stdout:(8/13): buildah-1.31.3-3.module+el8.9.0+21243+a 24 MB/s | 8.8 MB 00:00 2024-03-30T00:45:29.811 INFO:teuthology.orchestra.run.smithi130.stdout:(9/13): python3-podman-4.6.0-2.module+el8.9.0+2 767 kB/s | 154 kB 00:00 2024-03-30T00:45:29.844 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-30T00:45:29.853 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-30T00:45:29.937 INFO:teuthology.orchestra.run.smithi130.stdout:(10/13): netavark-1.7.0-2.module+el8.9.0+21243+ 9.9 MB/s | 3.7 MB 00:00 2024-03-30T00:45:30.004 INFO:teuthology.orchestra.run.smithi130.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 13 MB/s | 2.5 MB 00:00 2024-03-30T00:45:30.128 INFO:teuthology.orchestra.run.smithi130.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+21243+a5 260 kB/s | 49 kB 00:00 2024-03-30T00:45:30.221 INFO:teuthology.orchestra.run.smithi130.stdout:(13/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 19 MB/s | 8.1 MB 00:00 2024-03-30T00:45:30.222 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-30T00:45:30.222 INFO:teuthology.orchestra.run.smithi130.stdout:Total 25 MB/s | 26 MB 00:01 2024-03-30T00:45:30.507 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-30T00:45:30.527 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-30T00:45:30.527 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-30T00:45:30.562 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-30T00:45:30.758 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-30T00:45:30.767 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-30T00:45:30.909 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-30T00:45:31.046 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-03-30T00:45:31.077 INFO:teuthology.orchestra.run.smithi116.stdout:========================================================================================================== 2024-03-30T00:45:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-30T00:45:31.077 INFO:teuthology.orchestra.run.smithi116.stdout:========================================================================================================== 2024-03-30T00:45:31.077 INFO:teuthology.orchestra.run.smithi116.stdout:Installing group/module packages: 2024-03-30T00:45:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-03-30T00:45:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-03-30T00:45:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-03-30T00:45:31.077 INFO:teuthology.orchestra.run.smithi116.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-03-30T00:45:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-30T00:45:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-03-30T00:45:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-03-30T00:45:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-03-30T00:45:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-03-30T00:45:31.078 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-30T00:45:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-03-30T00:45:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-03-30T00:45:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-03-30T00:45:31.078 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-03-30T00:45:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-30T00:45:31.078 INFO:teuthology.orchestra.run.smithi116.stdout:Installing module profiles: 2024-03-30T00:45:31.078 INFO:teuthology.orchestra.run.smithi116.stdout: container-tools/common 2024-03-30T00:45:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:45:31.079 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling module streams: 2024-03-30T00:45:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: container-tools rhel8 2024-03-30T00:45:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:45:31.079 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-30T00:45:31.079 INFO:teuthology.orchestra.run.smithi116.stdout:========================================================================================================== 2024-03-30T00:45:31.079 INFO:teuthology.orchestra.run.smithi116.stdout:Install 13 Packages 2024-03-30T00:45:31.079 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:45:31.080 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 26 M 2024-03-30T00:45:31.080 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 87 M 2024-03-30T00:45:31.080 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-30T00:45:31.196 INFO:teuthology.orchestra.run.smithi116.stdout:(1/13): python3-pytoml-0.1.14-5.git7dea353.el8. 216 kB/s | 25 kB 00:00 2024-03-30T00:45:31.223 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-03-30T00:45:31.262 INFO:teuthology.orchestra.run.smithi116.stdout:(2/13): python3-pyxdg-0.25-16.el8.noarch.rpm 517 kB/s | 94 kB 00:00 2024-03-30T00:45:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:(3/13): criu-libs-3.18-4.module+el8.9.0+20326+3 198 kB/s | 39 kB 00:00 2024-03-30T00:45:31.313 INFO:teuthology.orchestra.run.smithi116.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 350 kB/s | 41 kB 00:00 2024-03-30T00:45:31.395 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-03-30T00:45:31.438 INFO:teuthology.orchestra.run.smithi116.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 5.7 MB/s | 1.0 MB 00:00 2024-03-30T00:45:31.519 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-30T00:45:31.539 INFO:teuthology.orchestra.run.smithi116.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 3.2 MB/s | 739 kB 00:00 2024-03-30T00:45:31.571 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-03-30T00:45:31.605 INFO:teuthology.orchestra.run.smithi116.stdout:(7/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 1.4 MB/s | 239 kB 00:00 2024-03-30T00:45:31.723 INFO:teuthology.orchestra.run.smithi116.stdout:(8/13): buildah-1.31.3-3.module+el8.9.0+21243+a 20 MB/s | 8.8 MB 00:00 2024-03-30T00:45:31.775 INFO:teuthology.orchestra.run.smithi116.stdout:(9/13): python3-podman-4.6.0-2.module+el8.9.0+2 919 kB/s | 154 kB 00:00 2024-03-30T00:45:31.784 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-03-30T00:45:31.915 INFO:teuthology.orchestra.run.smithi116.stdout:(10/13): netavark-1.7.0-2.module+el8.9.0+21243+ 9.9 MB/s | 3.7 MB 00:00 2024-03-30T00:45:32.002 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-03-30T00:45:32.040 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-03-30T00:45:32.090 INFO:teuthology.orchestra.run.smithi116.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 8.0 MB/s | 2.5 MB 00:00 2024-03-30T00:45:32.115 INFO:teuthology.orchestra.run.smithi116.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+21243+a5 247 kB/s | 49 kB 00:00 2024-03-30T00:45:32.232 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-03-30T00:45:32.232 INFO:teuthology.orchestra.run.smithi116.stdout:(13/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 16 MB/s | 8.1 MB 00:00 2024-03-30T00:45:32.233 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-30T00:45:32.233 INFO:teuthology.orchestra.run.smithi116.stdout:Total 22 MB/s | 26 MB 00:01 2024-03-30T00:45:32.237 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-03-30T00:45:32.448 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-03-30T00:45:32.652 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-03-30T00:45:32.701 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-30T00:45:32.725 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-30T00:45:32.726 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-30T00:45:32.862 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-03-30T00:45:33.108 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-03-30T00:45:33.124 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-30T00:45:33.202 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-30T00:45:33.211 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-30T00:45:33.378 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-03-30T00:45:33.381 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-03-30T00:45:33.753 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-03-30T00:45:33.986 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-30T00:45:34.311 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-30T00:45:34.500 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-03-30T00:45:34.642 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-03-30T00:45:34.701 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-03-30T00:45:34.787 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-03-30T00:45:34.864 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-03-30T00:45:34.882 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-03-30T00:45:35.068 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-03-30T00:45:35.165 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-30T00:45:35.207 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-03-30T00:45:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-03-30T00:45:35.486 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-03-30T00:45:35.587 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-30T00:45:35.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-03-30T00:45:35.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-03-30T00:45:35.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-03-30T00:45:35.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-03-30T00:45:35.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-03-30T00:45:35.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-03-30T00:45:35.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-03-30T00:45:35.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-30T00:45:35.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-03-30T00:45:35.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-03-30T00:45:35.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-03-30T00:45:35.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-03-30T00:45:35.681 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-03-30T00:45:36.130 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-03-30T00:45:36.383 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-03-30T00:45:36.606 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-30T00:45:36.627 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-30T00:45:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-03-30T00:45:37.041 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-30T00:45:37.041 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-03-30T00:45:37.041 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-03-30T00:45:37.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-03-30T00:45:37.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-03-30T00:45:37.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-03-30T00:45:37.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-03-30T00:45:37.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-03-30T00:45:37.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-30T00:45:37.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-03-30T00:45:37.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-03-30T00:45:37.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-03-30T00:45:37.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-03-30T00:45:37.268 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-03-30T00:45:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-03-30T00:45:38.461 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-03-30T00:45:38.608 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-30T00:45:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-30T00:45:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-03-30T00:45:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-03-30T00:45:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-03-30T00:45:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-03-30T00:45:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-03-30T00:45:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-03-30T00:45:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-03-30T00:45:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-30T00:45:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-03-30T00:45:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-03-30T00:45:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-03-30T00:45:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-03-30T00:45:45.929 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-03-30T00:45:45.930 INFO:teuthology.orchestra.run.smithi130.stdout:Installed products updated. 2024-03-30T00:45:46.491 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-30T00:45:46.491 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-30T00:45:46.491 INFO:teuthology.orchestra.run.smithi130.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:46.491 INFO:teuthology.orchestra.run.smithi130.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:46.491 INFO:teuthology.orchestra.run.smithi130.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-03-30T00:45:46.491 INFO:teuthology.orchestra.run.smithi130.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-03-30T00:45:46.491 INFO:teuthology.orchestra.run.smithi130.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:46.491 INFO:teuthology.orchestra.run.smithi130.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:46.492 INFO:teuthology.orchestra.run.smithi130.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-03-30T00:45:46.492 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-30T00:45:46.492 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-30T00:45:46.492 INFO:teuthology.orchestra.run.smithi130.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:46.492 INFO:teuthology.orchestra.run.smithi130.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:46.492 INFO:teuthology.orchestra.run.smithi130.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-03-30T00:45:46.492 INFO:teuthology.orchestra.run.smithi130.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-30T00:45:46.492 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-30T00:45:46.492 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-30T00:45:46.640 DEBUG:teuthology.parallel:result is None 2024-03-30T00:45:47.904 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-03-30T00:45:47.904 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-03-30T00:45:48.451 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:45:48.451 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-30T00:45:48.451 INFO:teuthology.orchestra.run.smithi181.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:48.451 INFO:teuthology.orchestra.run.smithi181.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:48.452 INFO:teuthology.orchestra.run.smithi181.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-03-30T00:45:48.452 INFO:teuthology.orchestra.run.smithi181.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-03-30T00:45:48.452 INFO:teuthology.orchestra.run.smithi181.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:48.452 INFO:teuthology.orchestra.run.smithi181.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:48.452 INFO:teuthology.orchestra.run.smithi181.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-03-30T00:45:48.452 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-30T00:45:48.452 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-30T00:45:48.452 INFO:teuthology.orchestra.run.smithi181.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:48.452 INFO:teuthology.orchestra.run.smithi181.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:48.452 INFO:teuthology.orchestra.run.smithi181.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-03-30T00:45:48.453 INFO:teuthology.orchestra.run.smithi181.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-30T00:45:48.453 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:45:48.453 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-30T00:45:48.601 DEBUG:teuthology.parallel:result is None 2024-03-30T00:45:50.641 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-03-30T00:45:50.642 INFO:teuthology.orchestra.run.smithi116.stdout:Installed products updated. 2024-03-30T00:45:51.269 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:45:51.270 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-30T00:45:51.270 INFO:teuthology.orchestra.run.smithi116.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:51.270 INFO:teuthology.orchestra.run.smithi116.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:51.270 INFO:teuthology.orchestra.run.smithi116.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-03-30T00:45:51.270 INFO:teuthology.orchestra.run.smithi116.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-03-30T00:45:51.270 INFO:teuthology.orchestra.run.smithi116.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:51.270 INFO:teuthology.orchestra.run.smithi116.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:51.270 INFO:teuthology.orchestra.run.smithi116.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-03-30T00:45:51.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-30T00:45:51.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-30T00:45:51.271 INFO:teuthology.orchestra.run.smithi116.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:51.271 INFO:teuthology.orchestra.run.smithi116.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T00:45:51.271 INFO:teuthology.orchestra.run.smithi116.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-03-30T00:45:51.271 INFO:teuthology.orchestra.run.smithi116.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-30T00:45:51.271 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:45:51.271 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-30T00:45:51.499 DEBUG:teuthology.parallel:result is None 2024-03-30T00:45:51.499 INFO:teuthology.run_tasks:Running task install... 2024-03-30T00:45:51.512 DEBUG:teuthology.task.install:project ceph 2024-03-30T00:45:51.512 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f'}} 2024-03-30T00:45:51.513 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f'} 2024-03-30T00:45:51.513 INFO:teuthology.task.install:Using flavor: default 2024-03-30T00:45:51.519 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-30T00:45:51.519 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-03-30T00:45:51.519 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f', 'tag': None, 'wait_for_package': False} 2024-03-30T00:45:51.520 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T00:45:51.522 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f', 'tag': None, 'wait_for_package': False} 2024-03-30T00:45:51.523 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T00:45:51.524 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f', 'tag': None, 'wait_for_package': False} 2024-03-30T00:45:51.524 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T00:45:51.666 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/ 2024-03-30T00:45:51.666 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1018.g470100a4 2024-03-30T00:45:51.679 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/ 2024-03-30T00:45:51.680 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1018.g470100a4 2024-03-30T00:45:51.683 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/ 2024-03-30T00:45:51.683 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1018.g470100a4 2024-03-30T00:45:51.840 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T00:45:51.841 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:45:51.841 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T00:45:51.850 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T00:45:51.850 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-30T00:45:51.850 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T00:45:51.856 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T00:45:51.856 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-30T00:45:51.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T00:45:51.884 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-30T00:45:51.884 DEBUG:teuthology.orchestra.run.smithi130:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/470100a4ccbcf7769052bf09a5413881ccf1b97f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T00:45:51.902 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-30T00:45:51.902 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/470100a4ccbcf7769052bf09a5413881ccf1b97f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T00:45:51.908 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-30T00:45:51.908 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/470100a4ccbcf7769052bf09a5413881ccf1b97f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T00:45:51.955 DEBUG:teuthology.orchestra.run.smithi130:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T00:45:51.972 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T00:45:52.015 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T00:45:52.037 DEBUG:teuthology.orchestra.run.smithi130:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T00:45:52.052 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T00:45:52.104 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T00:45:52.105 INFO:teuthology.orchestra.run.smithi130.stdout:check_obsoletes = 1 2024-03-30T00:45:52.106 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum clean all 2024-03-30T00:45:52.119 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-03-30T00:45:52.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-03-30T00:45:52.167 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-03-30T00:45:52.171 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-03-30T00:45:52.487 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2024-03-30T00:45:52.499 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-03-30T00:45:52.611 INFO:teuthology.orchestra.run.smithi116.stdout:Updating Subscription Management repositories. 2024-03-30T00:45:53.791 INFO:teuthology.orchestra.run.smithi130.stdout:61 files removed 2024-03-30T00:45:53.845 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-30T00:45:53.879 INFO:teuthology.orchestra.run.smithi181.stdout:61 files removed 2024-03-30T00:45:53.930 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-30T00:45:54.002 INFO:teuthology.orchestra.run.smithi116.stdout:61 files removed 2024-03-30T00:45:54.058 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-30T00:45:54.182 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2024-03-30T00:45:54.267 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2024-03-30T00:45:54.497 INFO:teuthology.orchestra.run.smithi116.stdout:Updating Subscription Management repositories. 2024-03-30T00:45:55.991 INFO:teuthology.orchestra.run.smithi130.stdout:ceph packages for x86_64 214 kB/s | 80 kB 00:00 2024-03-30T00:45:56.017 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 264 kB/s | 80 kB 00:00 2024-03-30T00:45:56.316 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 38 kB/s | 9.9 kB 00:00 2024-03-30T00:45:56.355 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 241 kB/s | 80 kB 00:00 2024-03-30T00:45:56.411 INFO:teuthology.orchestra.run.smithi130.stdout:ceph noarch packages 26 kB/s | 9.9 kB 00:00 2024-03-30T00:45:56.610 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 5.6 kB/s | 1.5 kB 00:00 2024-03-30T00:45:56.714 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 31 kB/s | 9.9 kB 00:00 2024-03-30T00:45:56.770 INFO:teuthology.orchestra.run.smithi181.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-03-30T00:45:56.775 INFO:teuthology.orchestra.run.smithi130.stdout:ceph source packages 4.4 kB/s | 1.5 kB 00:00 2024-03-30T00:45:56.943 INFO:teuthology.orchestra.run.smithi130.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-03-30T00:45:57.006 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 5.6 kB/s | 1.5 kB 00:00 2024-03-30T00:45:57.216 INFO:teuthology.orchestra.run.smithi116.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-03-30T00:45:57.964 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-03-30T00:45:58.296 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-03-30T00:46:01.571 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 522 kB/s | 24 kB 00:00 2024-03-30T00:46:01.674 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 3.7 MB/s | 16 MB 00:04 2024-03-30T00:46:01.994 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 271 kB/s | 24 kB 00:00 2024-03-30T00:46:02.997 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 MB/s | 66 MB 00:01 2024-03-30T00:46:03.530 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 45 MB/s | 66 MB 00:01 2024-03-30T00:46:05.882 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 547 kB/s | 24 kB 00:00 2024-03-30T00:46:08.043 INFO:teuthology.orchestra.run.smithi116.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 MB/s | 66 MB 00:02 2024-03-30T00:46:14.090 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 MB/s | 60 MB 00:01 2024-03-30T00:46:14.496 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 48 MB/s | 60 MB 00:01 2024-03-30T00:46:20.386 INFO:teuthology.orchestra.run.smithi116.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 MB/s | 60 MB 00:02 2024-03-30T00:46:25.273 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 9.0 MB/s | 8.8 MB 00:00 2024-03-30T00:46:25.530 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 13 MB/s | 8.8 MB 00:00 2024-03-30T00:46:31.970 INFO:teuthology.orchestra.run.smithi116.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 13 MB/s | 8.8 MB 00:00 2024-03-30T00:46:32.192 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-30T00:46:32.193 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-30T00:46:32.211 INFO:teuthology.orchestra.run.smithi181.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-03-30T00:46:32.212 INFO:teuthology.orchestra.run.smithi181.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-03-30T00:46:32.214 INFO:teuthology.orchestra.run.smithi181.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-03-30T00:46:32.215 INFO:teuthology.orchestra.run.smithi181.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-30T00:46:32.218 INFO:teuthology.orchestra.run.smithi181.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-30T00:46:32.222 INFO:teuthology.orchestra.run.smithi181.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-30T00:46:32.223 INFO:teuthology.orchestra.run.smithi181.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-30T00:46:32.440 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-30T00:46:32.454 INFO:teuthology.orchestra.run.smithi181.stdout:=================================================================================================================================== 2024-03-30T00:46:32.454 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-03-30T00:46:32.454 INFO:teuthology.orchestra.run.smithi181.stdout:=================================================================================================================================== 2024-03-30T00:46:32.454 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-30T00:46:32.454 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-03-30T00:46:32.454 INFO:teuthology.orchestra.run.smithi181.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-03-30T00:46:32.454 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:18.2.2-1018.g470100a4.el8 ceph 6.4 k 2024-03-30T00:46:32.454 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:18.2.2-1018.g470100a4.el8 ceph 5.5 M 2024-03-30T00:46:32.454 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 854 k 2024-03-30T00:46:32.455 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1018.g470100a4.el8 ceph 166 k 2024-03-30T00:46:32.455 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:18.2.2-1018.g470100a4.el8 ceph 1.6 M 2024-03-30T00:46:32.455 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 138 k 2024-03-30T00:46:32.455 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 1.8 M 2024-03-30T00:46:32.455 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 7.4 M 2024-03-30T00:46:32.455 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 49 k 2024-03-30T00:46:32.455 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:18.2.2-1018.g470100a4.el8 ceph 13 M 2024-03-30T00:46:32.455 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:18.2.2-1018.g470100a4.el8 ceph 47 M 2024-03-30T00:46:32.455 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 263 k 2024-03-30T00:46:32.455 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 173 k 2024-03-30T00:46:32.455 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-mirror x86_64 2:18.2.2-1018.g470100a4.el8 ceph 261 k 2024-03-30T00:46:32.455 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-top noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 18 k 2024-03-30T00:46:32.455 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-03-30T00:46:32.456 INFO:teuthology.orchestra.run.smithi181.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-03-30T00:46:32.456 INFO:teuthology.orchestra.run.smithi181.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-03-30T00:46:32.456 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:18.2.2-1018.g470100a4.el8 ceph 31 k 2024-03-30T00:46:32.456 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 775 k 2024-03-30T00:46:32.456 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:18.2.2-1018.g470100a4.el8 ceph 139 k 2024-03-30T00:46:32.456 INFO:teuthology.orchestra.run.smithi181.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-03-30T00:46:32.456 INFO:teuthology.orchestra.run.smithi181.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-03-30T00:46:32.456 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-03-30T00:46:32.456 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-03-30T00:46:32.456 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-03-30T00:46:32.456 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-03-30T00:46:32.456 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:18.2.2-1018.g470100a4.el8 ceph 196 k 2024-03-30T00:46:32.456 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:18.2.2-1018.g470100a4.el8 ceph 371 k 2024-03-30T00:46:32.456 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 366 k 2024-03-30T00:46:32.457 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:18.2.2-1018.g470100a4.el8 ceph 107 k 2024-03-30T00:46:32.457 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 95 k 2024-03-30T00:46:32.457 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.5 M 2024-03-30T00:46:32.457 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 181 k 2024-03-30T00:46:32.457 INFO:teuthology.orchestra.run.smithi181.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-03-30T00:46:32.457 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-03-30T00:46:32.457 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-03-30T00:46:32.457 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-03-30T00:46:32.457 INFO:teuthology.orchestra.run.smithi181.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-03-30T00:46:32.457 INFO:teuthology.orchestra.run.smithi181.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-30T00:46:32.457 INFO:teuthology.orchestra.run.smithi181.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-03-30T00:46:32.457 INFO:teuthology.orchestra.run.smithi181.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-03-30T00:46:32.457 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.7 M 2024-03-30T00:46:32.457 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.6 M 2024-03-30T00:46:32.458 INFO:teuthology.orchestra.run.smithi181.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-03-30T00:46:32.458 INFO:teuthology.orchestra.run.smithi181.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-03-30T00:46:32.458 INFO:teuthology.orchestra.run.smithi181.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-03-30T00:46:32.458 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-03-30T00:46:32.458 INFO:teuthology.orchestra.run.smithi181.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-03-30T00:46:32.458 INFO:teuthology.orchestra.run.smithi181.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-03-30T00:46:32.458 INFO:teuthology.orchestra.run.smithi181.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-03-30T00:46:32.458 INFO:teuthology.orchestra.run.smithi181.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-03-30T00:46:32.458 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-03-30T00:46:32.458 INFO:teuthology.orchestra.run.smithi181.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-03-30T00:46:32.458 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:18.2.2-1018.g470100a4.el8 ceph 22 M 2024-03-30T00:46:32.458 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 23 k 2024-03-30T00:46:32.458 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:18.2.2-1018.g470100a4.el8 ceph 2.3 M 2024-03-30T00:46:32.458 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 245 k 2024-03-30T00:46:32.459 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:18.2.2-1018.g470100a4.el8 ceph 4.8 M 2024-03-30T00:46:32.459 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 18 M 2024-03-30T00:46:32.459 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 15 k 2024-03-30T00:46:32.459 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:18.2.2-1018.g470100a4.el8 ceph 24 k 2024-03-30T00:46:32.459 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-03-30T00:46:32.459 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-30T00:46:32.459 INFO:teuthology.orchestra.run.smithi181.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-03-30T00:46:32.459 INFO:teuthology.orchestra.run.smithi181.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-03-30T00:46:32.459 INFO:teuthology.orchestra.run.smithi181.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-03-30T00:46:32.459 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:18.2.2-1018.g470100a4.el8 ceph 142 k 2024-03-30T00:46:32.459 INFO:teuthology.orchestra.run.smithi181.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-03-30T00:46:32.459 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-03-30T00:46:32.459 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 475 k 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 6.5 M 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-03-30T00:46:32.460 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-03-30T00:46:32.461 INFO:teuthology.orchestra.run.smithi181.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-03-30T00:46:32.461 INFO:teuthology.orchestra.run.smithi181.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-03-30T00:46:32.461 INFO:teuthology.orchestra.run.smithi181.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-30T00:46:32.461 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-30T00:46:32.461 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-03-30T00:46:32.461 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-30T00:46:32.461 INFO:teuthology.orchestra.run.smithi181.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-30T00:46:32.461 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-30T00:46:32.461 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 45 k 2024-03-30T00:46:32.461 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:18.2.2-1018.g470100a4.el8 ceph 129 k 2024-03-30T00:46:32.461 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-30T00:46:32.461 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-03-30T00:46:32.461 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-30T00:46:32.461 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-30T00:46:32.462 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-03-30T00:46:32.462 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-30T00:46:32.462 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-30T00:46:32.462 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-30T00:46:32.462 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-03-30T00:46:32.462 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-03-30T00:46:32.462 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-03-30T00:46:32.462 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-30T00:46:32.462 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-30T00:46:32.462 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-30T00:46:32.462 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-03-30T00:46:32.462 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-30T00:46:32.462 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-30T00:46:32.462 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-30T00:46:32.463 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-30T00:46:32.463 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-03-30T00:46:32.463 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-03-30T00:46:32.463 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-03-30T00:46:32.463 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-30T00:46:32.463 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-30T00:46:32.463 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-03-30T00:46:32.463 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-03-30T00:46:32.463 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-03-30T00:46:32.463 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-03-30T00:46:32.463 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-03-30T00:46:32.463 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-03-30T00:46:32.463 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-30T00:46:32.464 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-03-30T00:46:32.465 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-30T00:46:32.465 INFO:teuthology.orchestra.run.smithi181.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-03-30T00:46:32.465 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-03-30T00:46:32.465 INFO:teuthology.orchestra.run.smithi181.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-03-30T00:46:32.465 INFO:teuthology.orchestra.run.smithi181.stdout:Enabling module streams: 2024-03-30T00:46:32.465 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql 10 2024-03-30T00:46:32.465 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:46:32.465 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-30T00:46:32.465 INFO:teuthology.orchestra.run.smithi181.stdout:=================================================================================================================================== 2024-03-30T00:46:32.465 INFO:teuthology.orchestra.run.smithi181.stdout:Install 130 Packages 2024-03-30T00:46:32.465 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 16 Packages 2024-03-30T00:46:32.465 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:46:32.466 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 213 M 2024-03-30T00:46:32.466 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-30T00:46:32.647 INFO:teuthology.orchestra.run.smithi130.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-30T00:46:32.648 INFO:teuthology.orchestra.run.smithi130.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-30T00:46:32.666 INFO:teuthology.orchestra.run.smithi130.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-03-30T00:46:32.667 INFO:teuthology.orchestra.run.smithi130.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-03-30T00:46:32.669 INFO:teuthology.orchestra.run.smithi130.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-03-30T00:46:32.670 INFO:teuthology.orchestra.run.smithi130.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-30T00:46:32.673 INFO:teuthology.orchestra.run.smithi130.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-30T00:46:32.677 INFO:teuthology.orchestra.run.smithi130.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-30T00:46:32.678 INFO:teuthology.orchestra.run.smithi130.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-30T00:46:32.894 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-30T00:46:32.908 INFO:teuthology.orchestra.run.smithi130.stdout:=================================================================================================================================== 2024-03-30T00:46:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-03-30T00:46:32.908 INFO:teuthology.orchestra.run.smithi130.stdout:=================================================================================================================================== 2024-03-30T00:46:32.908 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-30T00:46:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-03-30T00:46:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-03-30T00:46:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: ceph x86_64 2:18.2.2-1018.g470100a4.el8 ceph 6.4 k 2024-03-30T00:46:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base x86_64 2:18.2.2-1018.g470100a4.el8 ceph 5.5 M 2024-03-30T00:46:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 854 k 2024-03-30T00:46:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1018.g470100a4.el8 ceph 166 k 2024-03-30T00:46:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr x86_64 2:18.2.2-1018.g470100a4.el8 ceph 1.6 M 2024-03-30T00:46:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 138 k 2024-03-30T00:46:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 1.8 M 2024-03-30T00:46:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 7.4 M 2024-03-30T00:46:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 49 k 2024-03-30T00:46:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw x86_64 2:18.2.2-1018.g470100a4.el8 ceph 13 M 2024-03-30T00:46:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test x86_64 2:18.2.2-1018.g470100a4.el8 ceph 47 M 2024-03-30T00:46:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 263 k 2024-03-30T00:46:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 173 k 2024-03-30T00:46:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-mirror x86_64 2:18.2.2-1018.g470100a4.el8 ceph 261 k 2024-03-30T00:46:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-top noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 18 k 2024-03-30T00:46:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-03-30T00:46:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-03-30T00:46:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-03-30T00:46:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel x86_64 2:18.2.2-1018.g470100a4.el8 ceph 31 k 2024-03-30T00:46:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 775 k 2024-03-30T00:46:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel x86_64 2:18.2.2-1018.g470100a4.el8 ceph 139 k 2024-03-30T00:46:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-03-30T00:46:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-03-30T00:46:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-03-30T00:46:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-03-30T00:46:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-03-30T00:46:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-03-30T00:46:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs x86_64 2:18.2.2-1018.g470100a4.el8 ceph 196 k 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados x86_64 2:18.2.2-1018.g470100a4.el8 ceph 371 k 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 366 k 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw x86_64 2:18.2.2-1018.g470100a4.el8 ceph 107 k 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 95 k 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.5 M 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 181 k 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrading: 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-03-30T00:46:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: librados2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.7 M 2024-03-30T00:46:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.6 M 2024-03-30T00:46:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-03-30T00:46:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-03-30T00:46:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-03-30T00:46:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-03-30T00:46:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-03-30T00:46:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-03-30T00:46:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-03-30T00:46:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-03-30T00:46:32.912 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-30T00:46:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-03-30T00:46:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common x86_64 2:18.2.2-1018.g470100a4.el8 ceph 22 M 2024-03-30T00:46:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 23 k 2024-03-30T00:46:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds x86_64 2:18.2.2-1018.g470100a4.el8 ceph 2.3 M 2024-03-30T00:46:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 245 k 2024-03-30T00:46:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon x86_64 2:18.2.2-1018.g470100a4.el8 ceph 4.8 M 2024-03-30T00:46:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 18 M 2024-03-30T00:46:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 15 k 2024-03-30T00:46:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux x86_64 2:18.2.2-1018.g470100a4.el8 ceph 24 k 2024-03-30T00:46:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-03-30T00:46:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-30T00:46:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-03-30T00:46:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-03-30T00:46:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-03-30T00:46:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite x86_64 2:18.2.2-1018.g470100a4.el8 ceph 142 k 2024-03-30T00:46:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-03-30T00:46:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-03-30T00:46:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-30T00:46:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-03-30T00:46:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-03-30T00:46:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-03-30T00:46:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 475 k 2024-03-30T00:46:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-03-30T00:46:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 6.5 M 2024-03-30T00:46:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-03-30T00:46:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-03-30T00:46:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-30T00:46:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-03-30T00:46:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-03-30T00:46:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-03-30T00:46:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-03-30T00:46:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-03-30T00:46:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-03-30T00:46:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-03-30T00:46:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-03-30T00:46:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-30T00:46:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-30T00:46:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-03-30T00:46:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-30T00:46:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-30T00:46:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-30T00:46:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 45 k 2024-03-30T00:46:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common x86_64 2:18.2.2-1018.g470100a4.el8 ceph 129 k 2024-03-30T00:46:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-30T00:46:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-03-30T00:46:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-30T00:46:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-30T00:46:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-03-30T00:46:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-30T00:46:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-30T00:46:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-30T00:46:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-03-30T00:46:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-03-30T00:46:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-03-30T00:46:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-30T00:46:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-30T00:46:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-30T00:46:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-03-30T00:46:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-30T00:46:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-30T00:46:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-30T00:46:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-30T00:46:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-03-30T00:46:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-03-30T00:46:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-03-30T00:46:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-30T00:46:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-30T00:46:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-03-30T00:46:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-03-30T00:46:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-03-30T00:46:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-03-30T00:46:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-03-30T00:46:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-03-30T00:46:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-30T00:46:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-30T00:46:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-30T00:46:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-30T00:46:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-03-30T00:46:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-30T00:46:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-30T00:46:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-30T00:46:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-30T00:46:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-30T00:46:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-30T00:46:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-30T00:46:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-30T00:46:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-03-30T00:46:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-30T00:46:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-03-30T00:46:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-30T00:46:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-03-30T00:46:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-03-30T00:46:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-03-30T00:46:32.919 INFO:teuthology.orchestra.run.smithi130.stdout:Enabling module streams: 2024-03-30T00:46:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql 10 2024-03-30T00:46:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-30T00:46:32.919 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-30T00:46:32.919 INFO:teuthology.orchestra.run.smithi130.stdout:=================================================================================================================================== 2024-03-30T00:46:32.919 INFO:teuthology.orchestra.run.smithi130.stdout:Install 130 Packages 2024-03-30T00:46:32.919 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrade 16 Packages 2024-03-30T00:46:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-30T00:46:32.920 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 213 M 2024-03-30T00:46:32.920 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-30T00:46:33.074 INFO:teuthology.orchestra.run.smithi181.stdout:(1/146): ceph-18.2.2-1018.g470100a4.el8.x86_64. 20 kB/s | 6.4 kB 00:00 2024-03-30T00:46:33.340 INFO:teuthology.orchestra.run.smithi130.stdout:(1/146): ceph-18.2.2-1018.g470100a4.el8.x86_64. 17 kB/s | 6.4 kB 00:00 2024-03-30T00:46:34.049 INFO:teuthology.orchestra.run.smithi130.stdout:(2/146): ceph-fuse-18.2.2-1018.g470100a4.el8.x8 1.2 MB/s | 854 kB 00:00 2024-03-30T00:46:34.191 INFO:teuthology.orchestra.run.smithi130.stdout:(3/146): ceph-immutable-object-cache-18.2.2-101 1.1 MB/s | 166 kB 00:00 2024-03-30T00:46:34.299 INFO:teuthology.orchestra.run.smithi181.stdout:(2/146): ceph-fuse-18.2.2-1018.g470100a4.el8.x8 697 kB/s | 854 kB 00:01 2024-03-30T00:46:34.417 INFO:teuthology.orchestra.run.smithi130.stdout:(4/146): ceph-base-18.2.2-1018.g470100a4.el8.x8 3.8 MB/s | 5.5 MB 00:01 2024-03-30T00:46:34.458 INFO:teuthology.orchestra.run.smithi181.stdout:(3/146): ceph-immutable-object-cache-18.2.2-101 1.0 MB/s | 166 kB 00:00 2024-03-30T00:46:34.641 INFO:teuthology.orchestra.run.smithi130.stdout:(5/146): ceph-mgr-18.2.2-1018.g470100a4.el8.x86 6.9 MB/s | 1.6 MB 00:00 2024-03-30T00:46:34.997 INFO:teuthology.orchestra.run.smithi130.stdout:(6/146): ceph-mds-18.2.2-1018.g470100a4.el8.x86 2.8 MB/s | 2.3 MB 00:00 2024-03-30T00:46:35.375 INFO:teuthology.orchestra.run.smithi130.stdout:(7/146): ceph-mon-18.2.2-1018.g470100a4.el8.x86 6.6 MB/s | 4.8 MB 00:00 2024-03-30T00:46:35.525 INFO:teuthology.orchestra.run.smithi181.stdout:(4/146): ceph-base-18.2.2-1018.g470100a4.el8.x8 2.0 MB/s | 5.5 MB 00:02 2024-03-30T00:46:36.066 INFO:teuthology.orchestra.run.smithi181.stdout:(5/146): ceph-mgr-18.2.2-1018.g470100a4.el8.x86 2.9 MB/s | 1.6 MB 00:00 2024-03-30T00:46:37.058 INFO:teuthology.orchestra.run.smithi130.stdout:(8/146): ceph-common-18.2.2-1018.g470100a4.el8. 5.3 MB/s | 22 MB 00:04 2024-03-30T00:46:37.140 INFO:teuthology.orchestra.run.smithi181.stdout:(6/146): ceph-common-18.2.2-1018.g470100a4.el8. 4.9 MB/s | 22 MB 00:04 2024-03-30T00:46:37.175 INFO:teuthology.orchestra.run.smithi130.stdout:(9/146): ceph-selinux-18.2.2-1018.g470100a4.el8 209 kB/s | 24 kB 00:00 2024-03-30T00:46:37.634 INFO:teuthology.orchestra.run.smithi181.stdout:(7/146): ceph-mds-18.2.2-1018.g470100a4.el8.x86 739 kB/s | 2.3 MB 00:03 2024-03-30T00:46:37.684 INFO:teuthology.orchestra.run.smithi130.stdout:(10/146): ceph-osd-18.2.2-1018.g470100a4.el8.x8 6.8 MB/s | 18 MB 00:02 2024-03-30T00:46:37.700 INFO:teuthology.orchestra.run.smithi181.stdout:(8/146): ceph-mon-18.2.2-1018.g470100a4.el8.x86 3.0 MB/s | 4.8 MB 00:01 2024-03-30T00:46:37.759 INFO:teuthology.orchestra.run.smithi181.stdout:(9/146): ceph-selinux-18.2.2-1018.g470100a4.el8 418 kB/s | 24 kB 00:00 2024-03-30T00:46:37.809 INFO:teuthology.orchestra.run.smithi130.stdout:(11/146): ceph-radosgw-18.2.2-1018.g470100a4.el 5.4 MB/s | 13 MB 00:02 2024-03-30T00:46:37.838 INFO:teuthology.orchestra.run.smithi130.stdout:(12/146): cephfs-mirror-18.2.2-1018.g470100a4.e 1.7 MB/s | 261 kB 00:00 2024-03-30T00:46:37.951 INFO:teuthology.orchestra.run.smithi130.stdout:(13/146): libcephfs-devel-18.2.2-1018.g470100a4 218 kB/s | 31 kB 00:00 2024-03-30T00:46:38.018 INFO:teuthology.orchestra.run.smithi130.stdout:(14/146): libcephfs2-18.2.2-1018.g470100a4.el8. 4.1 MB/s | 775 kB 00:00 2024-03-30T00:46:38.044 INFO:teuthology.orchestra.run.smithi130.stdout:(15/146): libcephsqlite-18.2.2-1018.g470100a4.e 1.5 MB/s | 142 kB 00:00 2024-03-30T00:46:38.118 INFO:teuthology.orchestra.run.smithi130.stdout:(16/146): librados-devel-18.2.2-1018.g470100a4. 1.4 MB/s | 139 kB 00:00 2024-03-30T00:46:38.202 INFO:teuthology.orchestra.run.smithi130.stdout:(17/146): libradosstriper1-18.2.2-1018.g470100a 2.9 MB/s | 475 kB 00:00 2024-03-30T00:46:38.303 INFO:teuthology.orchestra.run.smithi130.stdout:(18/146): python3-ceph-argparse-18.2.2-1018.g47 447 kB/s | 45 kB 00:00 2024-03-30T00:46:38.419 INFO:teuthology.orchestra.run.smithi130.stdout:(19/146): python3-ceph-common-18.2.2-1018.g4701 1.1 MB/s | 129 kB 00:00 2024-03-30T00:46:38.528 INFO:teuthology.orchestra.run.smithi130.stdout:(20/146): python3-cephfs-18.2.2-1018.g470100a4. 1.8 MB/s | 196 kB 00:00 2024-03-30T00:46:38.761 INFO:teuthology.orchestra.run.smithi130.stdout:(21/146): librgw2-18.2.2-1018.g470100a4.el8.x86 10 MB/s | 6.5 MB 00:00 2024-03-30T00:46:38.787 INFO:teuthology.orchestra.run.smithi130.stdout:(22/146): python3-rados-18.2.2-1018.g470100a4.e 1.4 MB/s | 371 kB 00:00 2024-03-30T00:46:38.845 INFO:teuthology.orchestra.run.smithi130.stdout:(23/146): python3-rbd-18.2.2-1018.g470100a4.el8 4.3 MB/s | 366 kB 00:00 2024-03-30T00:46:38.904 INFO:teuthology.orchestra.run.smithi130.stdout:(24/146): python3-rgw-18.2.2-1018.g470100a4.el8 916 kB/s | 107 kB 00:00 2024-03-30T00:46:38.921 INFO:teuthology.orchestra.run.smithi130.stdout:(25/146): rbd-fuse-18.2.2-1018.g470100a4.el8.x8 1.2 MB/s | 95 kB 00:00 2024-03-30T00:46:39.005 INFO:teuthology.orchestra.run.smithi130.stdout:(26/146): rbd-nbd-18.2.2-1018.g470100a4.el8.x86 2.1 MB/s | 181 kB 00:00 2024-03-30T00:46:39.080 INFO:teuthology.orchestra.run.smithi130.stdout:(27/146): ceph-grafana-dashboards-18.2.2-1018.g 310 kB/s | 23 kB 00:00 2024-03-30T00:46:39.172 INFO:teuthology.orchestra.run.smithi130.stdout:(28/146): ceph-mgr-cephadm-18.2.2-1018.g470100a 1.5 MB/s | 138 kB 00:00 2024-03-30T00:46:39.272 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-30T00:46:39.274 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-30T00:46:39.293 INFO:teuthology.orchestra.run.smithi116.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-03-30T00:46:39.294 INFO:teuthology.orchestra.run.smithi116.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-03-30T00:46:39.296 INFO:teuthology.orchestra.run.smithi116.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-03-30T00:46:39.298 INFO:teuthology.orchestra.run.smithi116.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-30T00:46:39.301 INFO:teuthology.orchestra.run.smithi116.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-30T00:46:39.305 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-30T00:46:39.307 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-30T00:46:39.464 INFO:teuthology.orchestra.run.smithi130.stdout:(29/146): ceph-mgr-dashboard-18.2.2-1018.g47010 6.1 MB/s | 1.8 MB 00:00 2024-03-30T00:46:39.530 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-30T00:46:39.545 INFO:teuthology.orchestra.run.smithi116.stdout:=================================================================================================================================== 2024-03-30T00:46:39.545 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-30T00:46:39.545 INFO:teuthology.orchestra.run.smithi116.stdout:=================================================================================================================================== 2024-03-30T00:46:39.545 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-30T00:46:39.545 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-03-30T00:46:39.545 INFO:teuthology.orchestra.run.smithi116.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-03-30T00:46:39.546 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:18.2.2-1018.g470100a4.el8 ceph 6.4 k 2024-03-30T00:46:39.546 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:18.2.2-1018.g470100a4.el8 ceph 5.5 M 2024-03-30T00:46:39.546 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 854 k 2024-03-30T00:46:39.546 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1018.g470100a4.el8 ceph 166 k 2024-03-30T00:46:39.546 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:18.2.2-1018.g470100a4.el8 ceph 1.6 M 2024-03-30T00:46:39.546 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 138 k 2024-03-30T00:46:39.546 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 1.8 M 2024-03-30T00:46:39.546 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 7.4 M 2024-03-30T00:46:39.546 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 49 k 2024-03-30T00:46:39.546 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:18.2.2-1018.g470100a4.el8 ceph 13 M 2024-03-30T00:46:39.546 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:18.2.2-1018.g470100a4.el8 ceph 47 M 2024-03-30T00:46:39.546 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 263 k 2024-03-30T00:46:39.546 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 173 k 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror x86_64 2:18.2.2-1018.g470100a4.el8 ceph 261 k 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 18 k 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:18.2.2-1018.g470100a4.el8 ceph 31 k 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 775 k 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:18.2.2-1018.g470100a4.el8 ceph 139 k 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-03-30T00:46:39.547 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:18.2.2-1018.g470100a4.el8 ceph 196 k 2024-03-30T00:46:39.548 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:18.2.2-1018.g470100a4.el8 ceph 371 k 2024-03-30T00:46:39.548 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 366 k 2024-03-30T00:46:39.548 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:18.2.2-1018.g470100a4.el8 ceph 107 k 2024-03-30T00:46:39.548 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 95 k 2024-03-30T00:46:39.548 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.5 M 2024-03-30T00:46:39.548 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 181 k 2024-03-30T00:46:39.548 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-03-30T00:46:39.548 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-03-30T00:46:39.548 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-03-30T00:46:39.548 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-03-30T00:46:39.548 INFO:teuthology.orchestra.run.smithi116.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-03-30T00:46:39.548 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-30T00:46:39.548 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-03-30T00:46:39.548 INFO:teuthology.orchestra.run.smithi116.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.7 M 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.6 M 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:18.2.2-1018.g470100a4.el8 ceph 22 M 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 23 k 2024-03-30T00:46:39.549 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:18.2.2-1018.g470100a4.el8 ceph 2.3 M 2024-03-30T00:46:39.550 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 245 k 2024-03-30T00:46:39.550 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:18.2.2-1018.g470100a4.el8 ceph 4.8 M 2024-03-30T00:46:39.550 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 18 M 2024-03-30T00:46:39.550 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 15 k 2024-03-30T00:46:39.550 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:18.2.2-1018.g470100a4.el8 ceph 24 k 2024-03-30T00:46:39.550 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-03-30T00:46:39.550 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-30T00:46:39.550 INFO:teuthology.orchestra.run.smithi116.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-03-30T00:46:39.550 INFO:teuthology.orchestra.run.smithi116.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-03-30T00:46:39.550 INFO:teuthology.orchestra.run.smithi116.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-03-30T00:46:39.550 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:18.2.2-1018.g470100a4.el8 ceph 142 k 2024-03-30T00:46:39.550 INFO:teuthology.orchestra.run.smithi116.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-03-30T00:46:39.550 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-03-30T00:46:39.551 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-30T00:46:39.551 INFO:teuthology.orchestra.run.smithi116.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-03-30T00:46:39.551 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-03-30T00:46:39.551 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-03-30T00:46:39.551 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 475 k 2024-03-30T00:46:39.551 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-03-30T00:46:39.551 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 6.5 M 2024-03-30T00:46:39.551 INFO:teuthology.orchestra.run.smithi116.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-03-30T00:46:39.551 INFO:teuthology.orchestra.run.smithi116.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-03-30T00:46:39.551 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-30T00:46:39.551 INFO:teuthology.orchestra.run.smithi116.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-03-30T00:46:39.551 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-03-30T00:46:39.551 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-03-30T00:46:39.551 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 45 k 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:18.2.2-1018.g470100a4.el8 ceph 129 k 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-03-30T00:46:39.552 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-30T00:46:39.553 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-30T00:46:39.553 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-03-30T00:46:39.553 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-30T00:46:39.553 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-30T00:46:39.553 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-30T00:46:39.553 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-03-30T00:46:39.553 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-03-30T00:46:39.553 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-03-30T00:46:39.553 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-30T00:46:39.553 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-30T00:46:39.553 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-30T00:46:39.553 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-03-30T00:46:39.553 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-30T00:46:39.554 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-30T00:46:39.554 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-30T00:46:39.554 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-30T00:46:39.554 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-03-30T00:46:39.554 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-03-30T00:46:39.554 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-03-30T00:46:39.554 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-30T00:46:39.554 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-30T00:46:39.554 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-03-30T00:46:39.554 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-03-30T00:46:39.554 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-03-30T00:46:39.554 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-03-30T00:46:39.554 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-03-30T00:46:39.555 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-03-30T00:46:39.555 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-30T00:46:39.555 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-30T00:46:39.555 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-30T00:46:39.555 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-30T00:46:39.555 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-03-30T00:46:39.555 INFO:teuthology.orchestra.run.smithi116.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-30T00:46:39.555 INFO:teuthology.orchestra.run.smithi116.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-30T00:46:39.555 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-30T00:46:39.555 INFO:teuthology.orchestra.run.smithi116.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-30T00:46:39.555 INFO:teuthology.orchestra.run.smithi116.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-30T00:46:39.555 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-30T00:46:39.555 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-30T00:46:39.555 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-30T00:46:39.556 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-03-30T00:46:39.556 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-30T00:46:39.556 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-03-30T00:46:39.556 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-30T00:46:39.556 INFO:teuthology.orchestra.run.smithi116.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-03-30T00:46:39.556 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-03-30T00:46:39.556 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-03-30T00:46:39.556 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling module streams: 2024-03-30T00:46:39.556 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql 10 2024-03-30T00:46:39.556 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:46:39.556 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-30T00:46:39.556 INFO:teuthology.orchestra.run.smithi116.stdout:=================================================================================================================================== 2024-03-30T00:46:39.556 INFO:teuthology.orchestra.run.smithi116.stdout:Install 130 Packages 2024-03-30T00:46:39.556 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 16 Packages 2024-03-30T00:46:39.557 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:46:39.557 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 213 M 2024-03-30T00:46:39.557 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-30T00:46:39.647 INFO:teuthology.orchestra.run.smithi130.stdout:(30/146): rbd-mirror-18.2.2-1018.g470100a4.el8. 4.6 MB/s | 3.5 MB 00:00 2024-03-30T00:46:39.756 INFO:teuthology.orchestra.run.smithi130.stdout:(31/146): ceph-mgr-modules-core-18.2.2-1018.g47 2.2 MB/s | 245 kB 00:00 2024-03-30T00:46:39.873 INFO:teuthology.orchestra.run.smithi130.stdout:(32/146): ceph-mgr-rook-18.2.2-1018.g470100a4.e 422 kB/s | 49 kB 00:00 2024-03-30T00:46:39.965 INFO:teuthology.orchestra.run.smithi130.stdout:(33/146): ceph-prometheus-alerts-18.2.2-1018.g4 161 kB/s | 15 kB 00:00 2024-03-30T00:46:40.070 INFO:teuthology.orchestra.run.smithi116.stdout:(1/146): ceph-18.2.2-1018.g470100a4.el8.x86_64. 19 kB/s | 6.4 kB 00:00 2024-03-30T00:46:40.115 INFO:teuthology.orchestra.run.smithi130.stdout:(34/146): ceph-volume-18.2.2-1018.g470100a4.el8 1.7 MB/s | 263 kB 00:00 2024-03-30T00:46:40.240 INFO:teuthology.orchestra.run.smithi130.stdout:(35/146): ceph-mgr-diskprediction-local-18.2.2- 9.5 MB/s | 7.4 MB 00:00 2024-03-30T00:46:40.266 INFO:teuthology.orchestra.run.smithi130.stdout:(36/146): cephadm-18.2.2-1018.g470100a4.el8.noa 1.1 MB/s | 173 kB 00:00 2024-03-30T00:46:40.349 INFO:teuthology.orchestra.run.smithi130.stdout:(37/146): cephfs-top-18.2.2-1018.g470100a4.el8. 167 kB/s | 18 kB 00:00 2024-03-30T00:46:40.516 INFO:teuthology.orchestra.run.smithi130.stdout:(38/146): liboath-2.6.2-3.el8.x86_64.rpm 354 kB/s | 59 kB 00:00 2024-03-30T00:46:40.592 INFO:teuthology.orchestra.run.smithi130.stdout:(39/146): gperftools-libs-2.7-9.el8.x86_64.rpm 941 kB/s | 306 kB 00:00 2024-03-30T00:46:40.592 INFO:teuthology.orchestra.run.smithi181.stdout:(10/146): ceph-osd-18.2.2-1018.g470100a4.el8.x8 5.3 MB/s | 18 MB 00:03 2024-03-30T00:46:40.617 INFO:teuthology.orchestra.run.smithi130.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 748 kB/s | 75 kB 00:00 2024-03-30T00:46:40.684 INFO:teuthology.orchestra.run.smithi130.stdout:(41/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 655 kB/s | 44 kB 00:00 2024-03-30T00:46:40.709 INFO:teuthology.orchestra.run.smithi130.stdout:(42/146): python3-asyncssh-2.7.0-2.el8.noarch.r 4.0 MB/s | 476 kB 00:00 2024-03-30T00:46:40.717 INFO:teuthology.orchestra.run.smithi181.stdout:(11/146): cephfs-mirror-18.2.2-1018.g470100a4.e 2.0 MB/s | 261 kB 00:00 2024-03-30T00:46:40.720 INFO:teuthology.orchestra.run.smithi116.stdout:(2/146): ceph-fuse-18.2.2-1018.g470100a4.el8.x8 1.3 MB/s | 854 kB 00:00 2024-03-30T00:46:40.759 INFO:teuthology.orchestra.run.smithi181.stdout:(12/146): libcephfs-devel-18.2.2-1018.g470100a4 747 kB/s | 31 kB 00:00 2024-03-30T00:46:40.767 INFO:teuthology.orchestra.run.smithi130.stdout:(43/146): python3-cachetools-3.1.1-4.el8.noarch 566 kB/s | 33 kB 00:00 2024-03-30T00:46:40.834 INFO:teuthology.orchestra.run.smithi130.stdout:(44/146): python3-beautifulsoup4-4.6.3-2.el8.1. 1.2 MB/s | 185 kB 00:00 2024-03-30T00:46:40.860 INFO:teuthology.orchestra.run.smithi130.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 178 kB/s | 16 kB 00:00 2024-03-30T00:46:40.878 INFO:teuthology.orchestra.run.smithi116.stdout:(3/146): ceph-immutable-object-cache-18.2.2-101 1.0 MB/s | 166 kB 00:00 2024-03-30T00:46:40.902 INFO:teuthology.orchestra.run.smithi130.stdout:(46/146): python3-cheroot-8.5.2-1.el8.noarch.rp 2.5 MB/s | 173 kB 00:00 2024-03-30T00:46:40.935 INFO:teuthology.orchestra.run.smithi130.stdout:(47/146): python3-cherrypy-18.4.0-1.el8.noarch. 5.0 MB/s | 384 kB 00:00 2024-03-30T00:46:40.961 INFO:teuthology.orchestra.run.smithi130.stdout:(48/146): python3-google-auth-1.1.1-10.el8.noar 1.4 MB/s | 81 kB 00:00 2024-03-30T00:46:41.001 INFO:teuthology.orchestra.run.smithi181.stdout:(13/146): libcephfs2-18.2.2-1018.g470100a4.el8. 3.1 MB/s | 775 kB 00:00 2024-03-30T00:46:41.002 INFO:teuthology.orchestra.run.smithi130.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2024-03-30T00:46:41.058 INFO:teuthology.orchestra.run.smithi130.stdout:(50/146): python3-jaraco-functools-2.0-4.el8.no 181 kB/s | 18 kB 00:00 2024-03-30T00:46:41.077 INFO:teuthology.orchestra.run.smithi181.stdout:(14/146): libcephsqlite-18.2.2-1018.g470100a4.e 1.9 MB/s | 142 kB 00:00 2024-03-30T00:46:41.120 INFO:teuthology.orchestra.run.smithi130.stdout:(51/146): python3-kubernetes-11.0.0-6.el8.noarc 10 MB/s | 1.2 MB 00:00 2024-03-30T00:46:41.144 INFO:teuthology.orchestra.run.smithi181.stdout:(15/146): librados-devel-18.2.2-1018.g470100a4. 2.0 MB/s | 139 kB 00:00 2024-03-30T00:46:41.145 INFO:teuthology.orchestra.run.smithi130.stdout:(52/146): python3-logutils-0.3.5-11.el8.noarch. 561 kB/s | 49 kB 00:00 2024-03-30T00:46:41.178 INFO:teuthology.orchestra.run.smithi130.stdout:(53/146): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2024-03-30T00:46:41.204 INFO:teuthology.orchestra.run.smithi130.stdout:(54/146): python3-natsort-7.1.1-2.el8.noarch.rp 1.0 MB/s | 60 kB 00:00 2024-03-30T00:46:41.246 INFO:teuthology.orchestra.run.smithi130.stdout:(55/146): python3-pecan-1.3.2-9.el8.noarch.rpm 4.1 MB/s | 283 kB 00:00 2024-03-30T00:46:41.305 INFO:teuthology.orchestra.run.smithi130.stdout:(56/146): python3-portend-2.6-1.el8.noarch.rpm 154 kB/s | 16 kB 00:00 2024-03-30T00:46:41.327 INFO:teuthology.orchestra.run.smithi181.stdout:(16/146): libradosstriper1-18.2.2-1018.g470100a 2.5 MB/s | 475 kB 00:00 2024-03-30T00:46:41.338 INFO:teuthology.orchestra.run.smithi130.stdout:(57/146): python3-repoze-lru-0.7-6.el8.noarch.r 365 kB/s | 34 kB 00:00 2024-03-30T00:46:41.380 INFO:teuthology.orchestra.run.smithi130.stdout:(58/146): python3-routes-2.4.1-12.el8.noarch.rp 2.6 MB/s | 196 kB 00:00 2024-03-30T00:46:41.406 INFO:teuthology.orchestra.run.smithi130.stdout:(59/146): python3-rsa-4.9-2.el8.noarch.rpm 960 kB/s | 64 kB 00:00 2024-03-30T00:46:41.439 INFO:teuthology.orchestra.run.smithi130.stdout:(60/146): python3-simplegeneric-0.8.1-17.el8.no 320 kB/s | 19 kB 00:00 2024-03-30T00:46:41.464 INFO:teuthology.orchestra.run.smithi130.stdout:(61/146): python3-singledispatch-3.4.0.3-18.el8 410 kB/s | 24 kB 00:00 2024-03-30T00:46:41.498 INFO:teuthology.orchestra.run.smithi130.stdout:(62/146): python3-tempora-1.14.1-5.el8.noarch.r 501 kB/s | 29 kB 00:00 2024-03-30T00:46:41.552 INFO:teuthology.orchestra.run.smithi130.stdout:(63/146): python3-trustme-0.6.0-4.el8.noarch.rp 315 kB/s | 27 kB 00:00 2024-03-30T00:46:41.587 INFO:teuthology.orchestra.run.smithi116.stdout:(4/146): ceph-base-18.2.2-1018.g470100a4.el8.x8 3.0 MB/s | 5.5 MB 00:01 2024-03-30T00:46:41.590 INFO:teuthology.orchestra.run.smithi130.stdout:(64/146): python3-waitress-1.4.3-1.el8.noarch.r 2.7 MB/s | 255 kB 00:00 2024-03-30T00:46:41.615 INFO:teuthology.orchestra.run.smithi130.stdout:(65/146): python3-webob-1.8.5-1.el8.1.noarch.rp 3.9 MB/s | 251 kB 00:00 2024-03-30T00:46:41.649 INFO:teuthology.orchestra.run.smithi130.stdout:(66/146): python3-websocket-client-0.56.0-5.el8 1.0 MB/s | 61 kB 00:00 2024-03-30T00:46:41.674 INFO:teuthology.orchestra.run.smithi130.stdout:(67/146): python3-webtest-2.0.33-1.el8.noarch.r 1.4 MB/s | 85 kB 00:00 2024-03-30T00:46:41.708 INFO:teuthology.orchestra.run.smithi130.stdout:(68/146): python3-zc-lockfile-2.0-2.el8.noarch. 392 kB/s | 23 kB 00:00 2024-03-30T00:46:41.733 INFO:teuthology.orchestra.run.smithi130.stdout:(69/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2024-03-30T00:46:41.807 INFO:teuthology.orchestra.run.smithi130.stdout:(70/146): thrift-0.13.0-2.el8.x86_64.rpm 13 MB/s | 1.7 MB 00:00 2024-03-30T00:46:41.929 INFO:teuthology.orchestra.run.smithi116.stdout:(5/146): ceph-mds-18.2.2-1018.g470100a4.el8.x86 2.2 MB/s | 2.3 MB 00:01 2024-03-30T00:46:42.004 INFO:teuthology.orchestra.run.smithi116.stdout:(6/146): ceph-mgr-18.2.2-1018.g470100a4.el8.x86 3.7 MB/s | 1.6 MB 00:00 2024-03-30T00:46:42.075 INFO:teuthology.orchestra.run.smithi130.stdout:(71/146): python3-jwt-1.6.1-2.el8.noarch.rpm 162 kB/s | 43 kB 00:00 2024-03-30T00:46:42.117 INFO:teuthology.orchestra.run.smithi130.stdout:(72/146): libverto-devel-0.3.0-5.el8.x86_64.rpm 47 kB/s | 18 kB 00:00 2024-03-30T00:46:42.317 INFO:teuthology.orchestra.run.smithi130.stdout:(73/146): python3-pycparser-2.14-14.el8.noarch. 546 kB/s | 109 kB 00:00 2024-03-30T00:46:42.351 INFO:teuthology.orchestra.run.smithi130.stdout:(74/146): python3-oauthlib-2.1.0-1.el8.noarch.r 562 kB/s | 155 kB 00:00 2024-03-30T00:46:42.577 INFO:teuthology.orchestra.run.smithi130.stdout:(75/146): libaio-devel-0.3.112-1.el8.x86_64.rpm 83 kB/s | 19 kB 00:00 2024-03-30T00:46:42.602 INFO:teuthology.orchestra.run.smithi130.stdout:(76/146): python3-requests-oauthlib-1.0.0-1.el8 150 kB/s | 43 kB 00:00 2024-03-30T00:46:43.113 INFO:teuthology.orchestra.run.smithi116.stdout:(7/146): ceph-mon-18.2.2-1018.g470100a4.el8.x86 4.1 MB/s | 4.8 MB 00:01 2024-03-30T00:46:43.285 INFO:teuthology.orchestra.run.smithi130.stdout:(77/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 62 kB/s | 44 kB 00:00 2024-03-30T00:46:43.356 INFO:teuthology.orchestra.run.smithi130.stdout:(78/146): pcre2-devel-10.32-2.el8.x86_64.rpm 802 kB/s | 605 kB 00:00 2024-03-30T00:46:43.394 INFO:teuthology.orchestra.run.smithi181.stdout:(17/146): librgw2-18.2.2-1018.g470100a4.el8.x86 3.2 MB/s | 6.5 MB 00:02 2024-03-30T00:46:43.477 INFO:teuthology.orchestra.run.smithi130.stdout:(79/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.2 MB/s | 229 kB 00:00 2024-03-30T00:46:43.478 INFO:teuthology.orchestra.run.smithi181.stdout:(18/146): python3-ceph-argparse-18.2.2-1018.g47 537 kB/s | 45 kB 00:00 2024-03-30T00:46:43.570 INFO:teuthology.orchestra.run.smithi181.stdout:(19/146): python3-ceph-common-18.2.2-1018.g4701 1.4 MB/s | 129 kB 00:00 2024-03-30T00:46:43.628 INFO:teuthology.orchestra.run.smithi130.stdout:(80/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 814 kB/s | 220 kB 00:00 2024-03-30T00:46:43.678 INFO:teuthology.orchestra.run.smithi181.stdout:(20/146): python3-cephfs-18.2.2-1018.g470100a4. 1.8 MB/s | 196 kB 00:00 2024-03-30T00:46:43.778 INFO:teuthology.orchestra.run.smithi130.stdout:(81/146): libselinux-devel-2.9-5.el8.x86_64.rpm 666 kB/s | 200 kB 00:00 2024-03-30T00:46:43.778 INFO:teuthology.orchestra.run.smithi181.stdout:(21/146): python3-rados-18.2.2-1018.g470100a4.e 3.6 MB/s | 371 kB 00:00 2024-03-30T00:46:43.813 INFO:teuthology.orchestra.run.smithi116.stdout:(8/146): ceph-common-18.2.2-1018.g470100a4.el8. 5.3 MB/s | 22 MB 00:04 2024-03-30T00:46:43.878 INFO:teuthology.orchestra.run.smithi130.stdout:(82/146): keyutils-libs-devel-1.5.10-9.el8.x86_ 192 kB/s | 48 kB 00:00 2024-03-30T00:46:43.912 INFO:teuthology.orchestra.run.smithi181.stdout:(22/146): python3-rbd-18.2.2-1018.g470100a4.el8 2.7 MB/s | 366 kB 00:00 2024-03-30T00:46:43.913 INFO:teuthology.orchestra.run.smithi116.stdout:(9/146): ceph-selinux-18.2.2-1018.g470100a4.el8 244 kB/s | 24 kB 00:00 2024-03-30T00:46:43.987 INFO:teuthology.orchestra.run.smithi181.stdout:(23/146): python3-rgw-18.2.2-1018.g470100a4.el8 1.4 MB/s | 107 kB 00:00 2024-03-30T00:46:44.080 INFO:teuthology.orchestra.run.smithi181.stdout:(24/146): rbd-fuse-18.2.2-1018.g470100a4.el8.x8 1.0 MB/s | 95 kB 00:00 2024-03-30T00:46:44.095 INFO:teuthology.orchestra.run.smithi130.stdout:(83/146): libsepol-devel-2.9-3.el8.x86_64.rpm 275 kB/s | 87 kB 00:00 2024-03-30T00:46:44.187 INFO:teuthology.orchestra.run.smithi130.stdout:(84/146): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 187 kB/s | 58 kB 00:00 2024-03-30T00:46:44.371 INFO:teuthology.orchestra.run.smithi130.stdout:(85/146): libcom_err-devel-1.45.6-4.el8.x86_64. 140 kB/s | 39 kB 00:00 2024-03-30T00:46:44.429 INFO:teuthology.orchestra.run.smithi130.stdout:(86/146): fuse-2.9.7-17.el8.x86_64.rpm 343 kB/s | 83 kB 00:00 2024-03-30T00:46:44.705 INFO:teuthology.orchestra.run.smithi130.stdout:(87/146): libgfortran-8.5.0-20.el8.x86_64.rpm 1.9 MB/s | 645 kB 00:00 2024-03-30T00:46:44.721 INFO:teuthology.orchestra.run.smithi130.stdout:(88/146): libquadmath-8.5.0-20.el8.x86_64.rpm 590 kB/s | 172 kB 00:00 2024-03-30T00:46:44.963 INFO:teuthology.orchestra.run.smithi130.stdout:(89/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 984 kB/s | 238 kB 00:00 2024-03-30T00:46:44.997 INFO:teuthology.orchestra.run.smithi130.stdout:(90/146): libuuid-devel-2.32.1-43.el8.x86_64.rp 342 kB/s | 100 kB 00:00 2024-03-30T00:46:45.080 INFO:teuthology.orchestra.run.smithi116.stdout:(10/146): ceph-radosgw-18.2.2-1018.g470100a4.el 6.7 MB/s | 13 MB 00:01 2024-03-30T00:46:45.096 INFO:teuthology.orchestra.run.smithi181.stdout:(25/146): rbd-mirror-18.2.2-1018.g470100a4.el8. 3.4 MB/s | 3.5 MB 00:01 2024-03-30T00:46:45.180 INFO:teuthology.orchestra.run.smithi181.stdout:(26/146): rbd-nbd-18.2.2-1018.g470100a4.el8.x86 2.1 MB/s | 181 kB 00:00 2024-03-30T00:46:45.180 INFO:teuthology.orchestra.run.smithi116.stdout:(11/146): cephfs-mirror-18.2.2-1018.g470100a4.e 2.6 MB/s | 261 kB 00:00 2024-03-30T00:46:45.238 INFO:teuthology.orchestra.run.smithi181.stdout:(27/146): ceph-grafana-dashboards-18.2.2-1018.g 400 kB/s | 23 kB 00:00 2024-03-30T00:46:45.297 INFO:teuthology.orchestra.run.smithi181.stdout:(28/146): ceph-mgr-cephadm-18.2.2-1018.g470100a 2.3 MB/s | 138 kB 00:00 2024-03-30T00:46:45.314 INFO:teuthology.orchestra.run.smithi116.stdout:(12/146): libcephfs-devel-18.2.2-1018.g470100a4 232 kB/s | 31 kB 00:00 2024-03-30T00:46:45.514 INFO:teuthology.orchestra.run.smithi116.stdout:(13/146): libcephfs2-18.2.2-1018.g470100a4.el8. 3.8 MB/s | 775 kB 00:00 2024-03-30T00:46:45.556 INFO:teuthology.orchestra.run.smithi130.stdout:(91/146): ceph-test-18.2.2-1018.g470100a4.el8.x 5.6 MB/s | 47 MB 00:08 2024-03-30T00:46:45.581 INFO:teuthology.orchestra.run.smithi130.stdout:(92/146): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 118 kB/s | 73 kB 00:00 2024-03-30T00:46:45.590 INFO:teuthology.orchestra.run.smithi116.stdout:(14/146): libcephsqlite-18.2.2-1018.g470100a4.e 1.8 MB/s | 142 kB 00:00 2024-03-30T00:46:45.606 INFO:teuthology.orchestra.run.smithi130.stdout:(93/146): NetworkManager-initscripts-updown-1.4 237 kB/s | 144 kB 00:00 2024-03-30T00:46:45.656 INFO:teuthology.orchestra.run.smithi181.stdout:(29/146): ceph-mgr-dashboard-18.2.2-1018.g47010 5.0 MB/s | 1.8 MB 00:00 2024-03-30T00:46:45.681 INFO:teuthology.orchestra.run.smithi116.stdout:(15/146): librados-devel-18.2.2-1018.g470100a4. 1.5 MB/s | 139 kB 00:00 2024-03-30T00:46:45.798 INFO:teuthology.orchestra.run.smithi116.stdout:(16/146): libradosstriper1-18.2.2-1018.g470100a 4.0 MB/s | 475 kB 00:00 2024-03-30T00:46:45.815 INFO:teuthology.orchestra.run.smithi130.stdout:(94/146): elfutils-libelf-devel-0.189-3.el8.x86 239 kB/s | 62 kB 00:00 2024-03-30T00:46:45.865 INFO:teuthology.orchestra.run.smithi130.stdout:(95/146): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 2.0 MB/s | 562 kB 00:00 2024-03-30T00:46:45.949 INFO:teuthology.orchestra.run.smithi130.stdout:(96/146): openssl-devel-1.1.1k-12.el8_9.x86_64. 6.8 MB/s | 2.3 MB 00:00 2024-03-30T00:46:45.999 INFO:teuthology.orchestra.run.smithi130.stdout:(97/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 256 kB/s | 47 kB 00:00 2024-03-30T00:46:46.023 INFO:teuthology.orchestra.run.smithi181.stdout:(30/146): ceph-radosgw-18.2.2-1018.g470100a4.el 1.6 MB/s | 13 MB 00:08 2024-03-30T00:46:46.049 INFO:teuthology.orchestra.run.smithi130.stdout:(98/146): python3-cryptography-3.2.1-7.el8_9.x8 3.0 MB/s | 559 kB 00:00 2024-03-30T00:46:46.191 INFO:teuthology.orchestra.run.smithi130.stdout:(99/146): python3-jsonpointer-1.10-11.el8.noarc 83 kB/s | 20 kB 00:00 2024-03-30T00:46:46.206 INFO:teuthology.orchestra.run.smithi181.stdout:(31/146): ceph-mgr-modules-core-18.2.2-1018.g47 1.3 MB/s | 245 kB 00:00 2024-03-30T00:46:46.216 INFO:teuthology.orchestra.run.smithi130.stdout:(100/146): python3-pyasn1-modules-0.3.7-6.el8.n 660 kB/s | 110 kB 00:00 2024-03-30T00:46:46.241 INFO:teuthology.orchestra.run.smithi130.stdout:(101/146): python3-prettytable-0.7.2-14.el8.noa 181 kB/s | 44 kB 00:00 2024-03-30T00:46:46.267 INFO:teuthology.orchestra.run.smithi130.stdout:(102/146): python3-jsonpatch-1.21-2.el8.noarch. 351 kB/s | 27 kB 00:00 2024-03-30T00:46:46.290 INFO:teuthology.orchestra.run.smithi181.stdout:(32/146): ceph-mgr-rook-18.2.2-1018.g470100a4.e 592 kB/s | 49 kB 00:00 2024-03-30T00:46:46.373 INFO:teuthology.orchestra.run.smithi181.stdout:(33/146): ceph-prometheus-alerts-18.2.2-1018.g4 177 kB/s | 15 kB 00:00 2024-03-30T00:46:46.382 INFO:teuthology.orchestra.run.smithi116.stdout:(17/146): ceph-osd-18.2.2-1018.g470100a4.el8.x8 4.2 MB/s | 18 MB 00:04 2024-03-30T00:46:46.442 INFO:teuthology.orchestra.run.smithi130.stdout:(103/146): python3-pyasn1-0.3.7-6.el8.noarch.rp 561 kB/s | 126 kB 00:00 2024-03-30T00:46:46.482 INFO:teuthology.orchestra.run.smithi116.stdout:(18/146): python3-ceph-argparse-18.2.2-1018.g47 448 kB/s | 45 kB 00:00 2024-03-30T00:46:46.492 INFO:teuthology.orchestra.run.smithi130.stdout:(104/146): python3-werkzeug-0.12.2-4.el8.noarch 1.8 MB/s | 457 kB 00:00 2024-03-30T00:46:46.524 INFO:teuthology.orchestra.run.smithi181.stdout:(34/146): ceph-volume-18.2.2-1018.g470100a4.el8 1.7 MB/s | 263 kB 00:00 2024-03-30T00:46:46.526 INFO:teuthology.orchestra.run.smithi130.stdout:(105/146): bison-3.0.4-10.el8.x86_64.rpm 2.6 MB/s | 688 kB 00:00 2024-03-30T00:46:46.591 INFO:teuthology.orchestra.run.smithi116.stdout:(19/146): librgw2-18.2.2-1018.g470100a4.el8.x86 8.3 MB/s | 6.5 MB 00:00 2024-03-30T00:46:46.599 INFO:teuthology.orchestra.run.smithi181.stdout:(35/146): cephadm-18.2.2-1018.g470100a4.el8.noa 2.3 MB/s | 173 kB 00:00 2024-03-30T00:46:46.616 INFO:teuthology.orchestra.run.smithi116.stdout:(20/146): python3-ceph-common-18.2.2-1018.g4701 965 kB/s | 129 kB 00:00 2024-03-30T00:46:46.658 INFO:teuthology.orchestra.run.smithi116.stdout:(21/146): python3-cephfs-18.2.2-1018.g470100a4. 2.9 MB/s | 196 kB 00:00 2024-03-30T00:46:46.666 INFO:teuthology.orchestra.run.smithi181.stdout:(36/146): cephfs-top-18.2.2-1018.g470100a4.el8. 272 kB/s | 18 kB 00:00 2024-03-30T00:46:46.767 INFO:teuthology.orchestra.run.smithi116.stdout:(22/146): python3-rbd-18.2.2-1018.g470100a4.el8 3.3 MB/s | 366 kB 00:00 2024-03-30T00:46:46.784 INFO:teuthology.orchestra.run.smithi130.stdout:(106/146): flex-2.6.1-9.el8.x86_64.rpm 934 kB/s | 320 kB 00:00 2024-03-30T00:46:46.792 INFO:teuthology.orchestra.run.smithi116.stdout:(23/146): python3-rados-18.2.2-1018.g470100a4.e 2.1 MB/s | 371 kB 00:00 2024-03-30T00:46:46.834 INFO:teuthology.orchestra.run.smithi116.stdout:(24/146): python3-rgw-18.2.2-1018.g470100a4.el8 1.6 MB/s | 107 kB 00:00 2024-03-30T00:46:46.843 INFO:teuthology.orchestra.run.smithi130.stdout:(107/146): python3-markupsafe-0.23-19.el8.x86_6 122 kB/s | 39 kB 00:00 2024-03-30T00:46:46.859 INFO:teuthology.orchestra.run.smithi116.stdout:(25/146): rbd-fuse-18.2.2-1018.g470100a4.el8.x8 1.4 MB/s | 95 kB 00:00 2024-03-30T00:46:46.910 INFO:teuthology.orchestra.run.smithi130.stdout:(108/146): libtool-2.4.6-25.el8.x86_64.rpm 1.7 MB/s | 709 kB 00:00 2024-03-30T00:46:46.935 INFO:teuthology.orchestra.run.smithi116.stdout:(26/146): rbd-nbd-18.2.2-1018.g470100a4.el8.x86 2.4 MB/s | 181 kB 00:00 2024-03-30T00:46:46.993 INFO:teuthology.orchestra.run.smithi116.stdout:(27/146): ceph-grafana-dashboards-18.2.2-1018.g 401 kB/s | 23 kB 00:00 2024-03-30T00:46:47.052 INFO:teuthology.orchestra.run.smithi130.stdout:(109/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 970 kB/s | 259 kB 00:00 2024-03-30T00:46:47.077 INFO:teuthology.orchestra.run.smithi116.stdout:(28/146): ceph-mgr-cephadm-18.2.2-1018.g470100a 1.6 MB/s | 138 kB 00:00 2024-03-30T00:46:47.152 INFO:teuthology.orchestra.run.smithi116.stdout:(29/146): rbd-mirror-18.2.2-1018.g470100a4.el8. 11 MB/s | 3.5 MB 00:00 2024-03-30T00:46:47.194 INFO:teuthology.orchestra.run.smithi130.stdout:(110/146): uuid-1.6.2-43.el8.x86_64.rpm 183 kB/s | 64 kB 00:00 2024-03-30T00:46:47.234 INFO:teuthology.orchestra.run.smithi181.stdout:(37/146): ceph-test-18.2.2-1018.g470100a4.el8.x 5.0 MB/s | 47 MB 00:09 2024-03-30T00:46:47.277 INFO:teuthology.orchestra.run.smithi130.stdout:(111/146): python3-pyOpenSSL-19.0.0-1.el8.noarc 282 kB/s | 103 kB 00:00 2024-03-30T00:46:47.292 INFO:teuthology.orchestra.run.smithi181.stdout:(38/146): gperftools-libs-2.7-9.el8.x86_64.rpm 490 kB/s | 306 kB 00:00 2024-03-30T00:46:47.358 INFO:teuthology.orchestra.run.smithi181.stdout:(39/146): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2024-03-30T00:46:47.411 INFO:teuthology.orchestra.run.smithi130.stdout:(112/146): python3-babel-2.5.1-7.el8.noarch.rpm 13 MB/s | 4.8 MB 00:00 2024-03-30T00:46:47.436 INFO:teuthology.orchestra.run.smithi116.stdout:(30/146): ceph-mgr-dashboard-18.2.2-1018.g47010 5.0 MB/s | 1.8 MB 00:00 2024-03-30T00:46:47.451 INFO:teuthology.orchestra.run.smithi181.stdout:(40/146): ceph-mgr-diskprediction-local-18.2.2- 4.1 MB/s | 7.4 MB 00:01 2024-03-30T00:46:47.461 INFO:teuthology.orchestra.run.smithi130.stdout:(113/146): python3-jinja2-2.10.1-3.el8.noarch.r 2.0 MB/s | 538 kB 00:00 2024-03-30T00:46:47.467 INFO:teuthology.orchestra.run.smithi181.stdout:(41/146): liboath-2.6.2-3.el8.x86_64.rpm 252 kB/s | 59 kB 00:00 2024-03-30T00:46:47.485 INFO:teuthology.orchestra.run.smithi181.stdout:(42/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.3 MB/s | 44 kB 00:00 2024-03-30T00:46:47.510 INFO:teuthology.orchestra.run.smithi181.stdout:(43/146): python3-asyncssh-2.7.0-2.el8.noarch.r 3.1 MB/s | 476 kB 00:00 2024-03-30T00:46:47.528 INFO:teuthology.orchestra.run.smithi116.stdout:(31/146): ceph-mgr-modules-core-18.2.2-1018.g47 2.6 MB/s | 245 kB 00:00 2024-03-30T00:46:47.535 INFO:teuthology.orchestra.run.smithi181.stdout:(44/146): python3-cachetools-3.1.1-4.el8.noarch 659 kB/s | 33 kB 00:00 2024-03-30T00:46:47.569 INFO:teuthology.orchestra.run.smithi181.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 301 kB/s | 16 kB 00:00 2024-03-30T00:46:47.595 INFO:teuthology.orchestra.run.smithi116.stdout:(32/146): ceph-mgr-rook-18.2.2-1018.g470100a4.e 740 kB/s | 49 kB 00:00 2024-03-30T00:46:47.619 INFO:teuthology.orchestra.run.smithi181.stdout:(46/146): python3-cheroot-8.5.2-1.el8.noarch.rp 2.0 MB/s | 173 kB 00:00 2024-03-30T00:46:47.620 INFO:teuthology.orchestra.run.smithi130.stdout:(114/146): python3-numpy-f2py-1.14.3-10.el8.x86 1.4 MB/s | 225 kB 00:00 2024-03-30T00:46:47.661 INFO:teuthology.orchestra.run.smithi181.stdout:(47/146): python3-beautifulsoup4-4.6.3-2.el8.1. 960 kB/s | 185 kB 00:00 2024-03-30T00:46:47.686 INFO:teuthology.orchestra.run.smithi181.stdout:(48/146): python3-google-auth-1.1.1-10.el8.noar 1.2 MB/s | 81 kB 00:00 2024-03-30T00:46:47.689 INFO:teuthology.orchestra.run.smithi116.stdout:(33/146): ceph-prometheus-alerts-18.2.2-1018.g4 157 kB/s | 15 kB 00:00 2024-03-30T00:46:47.703 INFO:teuthology.orchestra.run.smithi130.stdout:(115/146): python3-numpy-1.14.3-10.el8.x86_64.r 8.8 MB/s | 3.7 MB 00:00 2024-03-30T00:46:47.711 INFO:teuthology.orchestra.run.smithi181.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 220 kB/s | 11 kB 00:00 2024-03-30T00:46:47.728 INFO:teuthology.orchestra.run.smithi130.stdout:(116/146): socat-1.7.4.1-1.el8.x86_64.rpm 1.0 MB/s | 323 kB 00:00 2024-03-30T00:46:47.737 INFO:teuthology.orchestra.run.smithi181.stdout:(50/146): python3-cherrypy-18.4.0-1.el8.noarch. 2.2 MB/s | 384 kB 00:00 2024-03-30T00:46:47.770 INFO:teuthology.orchestra.run.smithi181.stdout:(51/146): python3-jaraco-functools-2.0-4.el8.no 210 kB/s | 18 kB 00:00 2024-03-30T00:46:47.796 INFO:teuthology.orchestra.run.smithi181.stdout:(52/146): python3-logutils-0.3.5-11.el8.noarch. 834 kB/s | 49 kB 00:00 2024-03-30T00:46:47.829 INFO:teuthology.orchestra.run.smithi181.stdout:(53/146): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2024-03-30T00:46:47.831 INFO:teuthology.orchestra.run.smithi116.stdout:(34/146): ceph-mgr-diskprediction-local-18.2.2- 11 MB/s | 7.4 MB 00:00 2024-03-30T00:46:47.856 INFO:teuthology.orchestra.run.smithi116.stdout:(35/146): ceph-volume-18.2.2-1018.g470100a4.el8 1.5 MB/s | 263 kB 00:00 2024-03-30T00:46:47.907 INFO:teuthology.orchestra.run.smithi116.stdout:(36/146): cephadm-18.2.2-1018.g470100a4.el8.noa 2.2 MB/s | 173 kB 00:00 2024-03-30T00:46:47.913 INFO:teuthology.orchestra.run.smithi181.stdout:(54/146): python3-natsort-7.1.1-2.el8.noarch.rp 515 kB/s | 60 kB 00:00 2024-03-30T00:46:47.932 INFO:teuthology.orchestra.run.smithi116.stdout:(37/146): cephfs-top-18.2.2-1018.g470100a4.el8. 241 kB/s | 18 kB 00:00 2024-03-30T00:46:47.937 INFO:teuthology.orchestra.run.smithi130.stdout:(117/146): librdkafka-0.11.4-3.el8.x86_64.rpm 1.1 MB/s | 354 kB 00:00 2024-03-30T00:46:47.955 INFO:teuthology.orchestra.run.smithi181.stdout:(55/146): python3-portend-2.6-1.el8.noarch.rpm 376 kB/s | 16 kB 00:00 2024-03-30T00:46:47.962 INFO:teuthology.orchestra.run.smithi130.stdout:(118/146): python3-mako-1.0.6-14.el8.noarch.rpm 671 kB/s | 157 kB 00:00 2024-03-30T00:46:47.980 INFO:teuthology.orchestra.run.smithi181.stdout:(56/146): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2024-03-30T00:46:48.022 INFO:teuthology.orchestra.run.smithi181.stdout:(57/146): python3-repoze-lru-0.7-6.el8.noarch.r 503 kB/s | 34 kB 00:00 2024-03-30T00:46:48.047 INFO:teuthology.orchestra.run.smithi181.stdout:(58/146): python3-routes-2.4.1-12.el8.noarch.rp 2.9 MB/s | 196 kB 00:00 2024-03-30T00:46:48.081 INFO:teuthology.orchestra.run.smithi181.stdout:(59/146): python3-kubernetes-11.0.0-6.el8.noarc 3.2 MB/s | 1.2 MB 00:00 2024-03-30T00:46:48.106 INFO:teuthology.orchestra.run.smithi181.stdout:(60/146): python3-simplegeneric-0.8.1-17.el8.no 319 kB/s | 19 kB 00:00 2024-03-30T00:46:48.121 INFO:teuthology.orchestra.run.smithi130.stdout:(119/146): openblas-threads-0.3.15-6.el8.x86_64 12 MB/s | 5.2 MB 00:00 2024-03-30T00:46:48.131 INFO:teuthology.orchestra.run.smithi181.stdout:(61/146): python3-singledispatch-3.4.0.3-18.el8 476 kB/s | 24 kB 00:00 2024-03-30T00:46:48.138 INFO:teuthology.orchestra.run.smithi116.stdout:(38/146): liboath-2.6.2-3.el8.x86_64.rpm 285 kB/s | 59 kB 00:00 2024-03-30T00:46:48.157 INFO:teuthology.orchestra.run.smithi181.stdout:(62/146): python3-rsa-4.9-2.el8.noarch.rpm 478 kB/s | 64 kB 00:00 2024-03-30T00:46:48.182 INFO:teuthology.orchestra.run.smithi181.stdout:(63/146): python3-tempora-1.14.1-5.el8.noarch.r 388 kB/s | 29 kB 00:00 2024-03-30T00:46:48.207 INFO:teuthology.orchestra.run.smithi116.stdout:(39/146): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2024-03-30T00:46:48.207 INFO:teuthology.orchestra.run.smithi181.stdout:(64/146): python3-trustme-0.6.0-4.el8.noarch.rp 365 kB/s | 27 kB 00:00 2024-03-30T00:46:48.233 INFO:teuthology.orchestra.run.smithi116.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 799 kB/s | 75 kB 00:00 2024-03-30T00:46:48.341 INFO:teuthology.orchestra.run.smithi181.stdout:(65/146): python3-waitress-1.4.3-1.el8.noarch.r 1.4 MB/s | 255 kB 00:00 2024-03-30T00:46:48.358 INFO:teuthology.orchestra.run.smithi181.stdout:(66/146): python3-websocket-client-0.56.0-5.el8 408 kB/s | 61 kB 00:00 2024-03-30T00:46:48.383 INFO:teuthology.orchestra.run.smithi181.stdout:(67/146): python3-webob-1.8.5-1.el8.1.noarch.rp 1.4 MB/s | 251 kB 00:00 2024-03-30T00:46:48.400 INFO:teuthology.orchestra.run.smithi181.stdout:(68/146): python3-zc-lockfile-2.0-2.el8.noarch. 547 kB/s | 23 kB 00:00 2024-03-30T00:46:48.425 INFO:teuthology.orchestra.run.smithi181.stdout:(69/146): python3-webtest-2.0.33-1.el8.noarch.r 1.0 MB/s | 85 kB 00:00 2024-03-30T00:46:48.450 INFO:teuthology.orchestra.run.smithi181.stdout:(70/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2024-03-30T00:46:48.466 INFO:teuthology.orchestra.run.smithi130.stdout:(120/146): openblas-0.3.15-6.el8.x86_64.rpm 9.6 MB/s | 5.0 MB 00:00 2024-03-30T00:46:48.497 INFO:teuthology.orchestra.run.smithi130.stdout:(121/146): automake-1.16.1-8.el8.noarch.rpm 1.3 MB/s | 714 kB 00:00 2024-03-30T00:46:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:(41/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 158 kB/s | 44 kB 00:00 2024-03-30T00:46:48.533 INFO:teuthology.orchestra.run.smithi116.stdout:(42/146): python3-asyncssh-2.7.0-2.el8.noarch.r 1.4 MB/s | 476 kB 00:00 2024-03-30T00:46:48.567 INFO:teuthology.orchestra.run.smithi181.stdout:(71/146): libverto-devel-0.3.0-5.el8.x86_64.rpm 126 kB/s | 18 kB 00:00 2024-03-30T00:46:48.609 INFO:teuthology.orchestra.run.smithi181.stdout:(72/146): python3-jwt-1.6.1-2.el8.noarch.rpm 278 kB/s | 43 kB 00:00 2024-03-30T00:46:48.624 INFO:teuthology.orchestra.run.smithi116.stdout:(43/146): python3-beautifulsoup4-4.6.3-2.el8.1. 1.6 MB/s | 185 kB 00:00 2024-03-30T00:46:48.643 INFO:teuthology.orchestra.run.smithi181.stdout:(73/146): python3-oauthlib-2.1.0-1.el8.noarch.r 2.0 MB/s | 155 kB 00:00 2024-03-30T00:46:48.659 INFO:teuthology.orchestra.run.smithi116.stdout:(44/146): python3-cachetools-3.1.1-4.el8.noarch 263 kB/s | 33 kB 00:00 2024-03-30T00:46:48.693 INFO:teuthology.orchestra.run.smithi116.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 240 kB/s | 16 kB 00:00 2024-03-30T00:46:48.710 INFO:teuthology.orchestra.run.smithi181.stdout:(74/146): python3-pycparser-2.14-14.el8.noarch. 1.1 MB/s | 109 kB 00:00 2024-03-30T00:46:48.735 INFO:teuthology.orchestra.run.smithi181.stdout:(75/146): python3-requests-oauthlib-1.0.0-1.el8 465 kB/s | 43 kB 00:00 2024-03-30T00:46:48.739 INFO:teuthology.orchestra.run.smithi130.stdout:(122/146): libpq-13.11-1.el8.x86_64.rpm 321 kB/s | 198 kB 00:00 2024-03-30T00:46:48.759 INFO:teuthology.orchestra.run.smithi116.stdout:(46/146): python3-cheroot-8.5.2-1.el8.noarch.rp 1.7 MB/s | 173 kB 00:00 2024-03-30T00:46:48.764 INFO:teuthology.orchestra.run.smithi130.stdout:(123/146): python3-pytz-2017.2-11.el8.noarch.rp 203 kB/s | 54 kB 00:00 2024-03-30T00:46:48.785 INFO:teuthology.orchestra.run.smithi116.stdout:(47/146): python3-cherrypy-18.4.0-1.el8.noarch. 4.1 MB/s | 384 kB 00:00 2024-03-30T00:46:48.825 INFO:teuthology.orchestra.run.smithi130.stdout:(124/146): jq-1.6-7.el8.x86_64.rpm 559 kB/s | 202 kB 00:00 2024-03-30T00:46:48.844 INFO:teuthology.orchestra.run.smithi181.stdout:(76/146): thrift-0.13.0-2.el8.x86_64.rpm 3.8 MB/s | 1.7 MB 00:00 2024-03-30T00:46:48.852 INFO:teuthology.orchestra.run.smithi116.stdout:(48/146): python3-google-auth-1.1.1-10.el8.noar 883 kB/s | 81 kB 00:00 2024-03-30T00:46:48.874 INFO:teuthology.orchestra.run.smithi181.stdout:(77/146): libaio-devel-0.3.112-1.el8.x86_64.rpm 118 kB/s | 19 kB 00:00 2024-03-30T00:46:48.877 INFO:teuthology.orchestra.run.smithi116.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2024-03-30T00:46:48.896 INFO:teuthology.orchestra.run.smithi181.stdout:(78/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 275 kB/s | 44 kB 00:00 2024-03-30T00:46:48.919 INFO:teuthology.orchestra.run.smithi116.stdout:(50/146): python3-jaraco-functools-2.0-4.el8.no 264 kB/s | 18 kB 00:00 2024-03-30T00:46:48.969 INFO:teuthology.orchestra.run.smithi116.stdout:(51/146): python3-logutils-0.3.5-11.el8.noarch. 975 kB/s | 49 kB 00:00 2024-03-30T00:46:48.973 INFO:teuthology.orchestra.run.smithi130.stdout:(125/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 903 kB/s | 188 kB 00:00 2024-03-30T00:46:48.995 INFO:teuthology.orchestra.run.smithi181.stdout:(79/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.8 MB/s | 229 kB 00:00 2024-03-30T00:46:49.012 INFO:teuthology.orchestra.run.smithi116.stdout:(52/146): python3-kubernetes-11.0.0-6.el8.noarc 8.7 MB/s | 1.2 MB 00:00 2024-03-30T00:46:49.028 INFO:teuthology.orchestra.run.smithi116.stdout:(53/146): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2024-03-30T00:46:49.045 INFO:teuthology.orchestra.run.smithi181.stdout:(80/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.5 MB/s | 220 kB 00:00 2024-03-30T00:46:49.104 INFO:teuthology.orchestra.run.smithi116.stdout:(54/146): python3-natsort-7.1.1-2.el8.noarch.rp 656 kB/s | 60 kB 00:00 2024-03-30T00:46:49.104 INFO:teuthology.orchestra.run.smithi181.stdout:(81/146): pcre2-devel-10.32-2.el8.x86_64.rpm 2.3 MB/s | 605 kB 00:00 2024-03-30T00:46:49.129 INFO:teuthology.orchestra.run.smithi181.stdout:(82/146): libselinux-devel-2.9-5.el8.x86_64.rpm 1.5 MB/s | 200 kB 00:00 2024-03-30T00:46:49.130 INFO:teuthology.orchestra.run.smithi116.stdout:(55/146): python3-pecan-1.3.2-9.el8.noarch.rpm 2.8 MB/s | 283 kB 00:00 2024-03-30T00:46:49.163 INFO:teuthology.orchestra.run.smithi116.stdout:(56/146): python3-portend-2.6-1.el8.noarch.rpm 267 kB/s | 16 kB 00:00 2024-03-30T00:46:49.188 INFO:teuthology.orchestra.run.smithi116.stdout:(57/146): python3-repoze-lru-0.7-6.el8.noarch.r 575 kB/s | 34 kB 00:00 2024-03-30T00:46:49.198 INFO:teuthology.orchestra.run.smithi130.stdout:(126/146): python3-scipy-1.0.0-21.module+el8.9. 30 MB/s | 14 MB 00:00 2024-03-30T00:46:49.204 INFO:teuthology.orchestra.run.smithi181.stdout:(83/146): keyutils-libs-devel-1.5.10-9.el8.x86_ 303 kB/s | 48 kB 00:00 2024-03-30T00:46:49.222 INFO:teuthology.orchestra.run.smithi116.stdout:(58/146): python3-routes-2.4.1-12.el8.noarch.rp 3.3 MB/s | 196 kB 00:00 2024-03-30T00:46:49.229 INFO:teuthology.orchestra.run.smithi181.stdout:(84/146): libsepol-devel-2.9-3.el8.x86_64.rpm 694 kB/s | 87 kB 00:00 2024-03-30T00:46:49.240 INFO:teuthology.orchestra.run.smithi130.stdout:(127/146): postgresql-contrib-10.23-4.module+el 3.0 MB/s | 812 kB 00:00 2024-03-30T00:46:49.289 INFO:teuthology.orchestra.run.smithi181.stdout:(85/146): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 363 kB/s | 58 kB 00:00 2024-03-30T00:46:49.297 INFO:teuthology.orchestra.run.smithi116.stdout:(59/146): python3-rsa-4.9-2.el8.noarch.rpm 591 kB/s | 64 kB 00:00 2024-03-30T00:46:49.310 INFO:teuthology.orchestra.run.smithi130.stdout:(128/146): postgresql-10.23-4.module+el8.9.0+21 3.1 MB/s | 1.5 MB 00:00 2024-03-30T00:46:49.323 INFO:teuthology.orchestra.run.smithi116.stdout:(60/146): python3-simplegeneric-0.8.1-17.el8.no 186 kB/s | 19 kB 00:00 2024-03-30T00:46:49.356 INFO:teuthology.orchestra.run.smithi116.stdout:(61/146): python3-singledispatch-3.4.0.3-18.el8 408 kB/s | 24 kB 00:00 2024-03-30T00:46:49.372 INFO:teuthology.orchestra.run.smithi181.stdout:(86/146): libcom_err-devel-1.45.6-4.el8.x86_64. 231 kB/s | 39 kB 00:00 2024-03-30T00:46:49.389 INFO:teuthology.orchestra.run.smithi116.stdout:(62/146): python3-tempora-1.14.1-5.el8.noarch.r 438 kB/s | 29 kB 00:00 2024-03-30T00:46:49.397 INFO:teuthology.orchestra.run.smithi181.stdout:(87/146): fuse-2.9.7-17.el8.x86_64.rpm 496 kB/s | 83 kB 00:00 2024-03-30T00:46:49.415 INFO:teuthology.orchestra.run.smithi116.stdout:(63/146): python3-trustme-0.6.0-4.el8.noarch.rp 470 kB/s | 27 kB 00:00 2024-03-30T00:46:49.456 INFO:teuthology.orchestra.run.smithi181.stdout:(88/146): libgfortran-8.5.0-20.el8.x86_64.rpm 3.8 MB/s | 645 kB 00:00 2024-03-30T00:46:49.457 INFO:teuthology.orchestra.run.smithi116.stdout:(64/146): python3-waitress-1.4.3-1.el8.noarch.r 3.7 MB/s | 255 kB 00:00 2024-03-30T00:46:49.506 INFO:teuthology.orchestra.run.smithi181.stdout:(89/146): libuuid-devel-2.32.1-43.el8.x86_64.rp 919 kB/s | 100 kB 00:00 2024-03-30T00:46:49.555 INFO:teuthology.orchestra.run.smithi181.stdout:(90/146): libquadmath-8.5.0-20.el8.x86_64.rpm 943 kB/s | 172 kB 00:00 2024-03-30T00:46:49.582 INFO:teuthology.orchestra.run.smithi116.stdout:(65/146): python3-websocket-client-0.56.0-5.el8 490 kB/s | 61 kB 00:00 2024-03-30T00:46:49.582 INFO:teuthology.orchestra.run.smithi130.stdout:(129/146): postgresql-server-10.23-4.module+el8 13 MB/s | 5.1 MB 00:00 2024-03-30T00:46:49.607 INFO:teuthology.orchestra.run.smithi181.stdout:(91/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.5 MB/s | 238 kB 00:00 2024-03-30T00:46:49.607 INFO:teuthology.orchestra.run.smithi130.stdout:(130/146): python3-packaging-16.8-9.el8.noarch. 131 kB/s | 48 kB 00:00 2024-03-30T00:46:49.608 INFO:teuthology.orchestra.run.smithi116.stdout:(66/146): python3-webob-1.8.5-1.el8.1.noarch.rp 1.3 MB/s | 251 kB 00:00 2024-03-30T00:46:49.632 INFO:teuthology.orchestra.run.smithi181.stdout:(92/146): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 579 kB/s | 73 kB 00:00 2024-03-30T00:46:49.641 INFO:teuthology.orchestra.run.smithi116.stdout:(67/146): python3-webtest-2.0.33-1.el8.noarch.r 1.4 MB/s | 85 kB 00:00 2024-03-30T00:46:49.676 INFO:teuthology.orchestra.run.smithi116.stdout:(68/146): python3-zc-lockfile-2.0-2.el8.noarch. 344 kB/s | 23 kB 00:00 2024-03-30T00:46:49.700 INFO:teuthology.orchestra.run.smithi116.stdout:(69/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2024-03-30T00:46:49.707 INFO:teuthology.orchestra.run.smithi181.stdout:(93/146): NetworkManager-initscripts-updown-1.4 948 kB/s | 144 kB 00:00 2024-03-30T00:46:49.757 INFO:teuthology.orchestra.run.smithi181.stdout:(94/146): elfutils-libelf-devel-0.189-3.el8.x86 410 kB/s | 62 kB 00:00 2024-03-30T00:46:49.791 INFO:teuthology.orchestra.run.smithi181.stdout:(95/146): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 3.5 MB/s | 562 kB 00:00 2024-03-30T00:46:49.800 INFO:teuthology.orchestra.run.smithi116.stdout:(70/146): thrift-0.13.0-2.el8.x86_64.rpm 11 MB/s | 1.7 MB 00:00 2024-03-30T00:46:49.859 INFO:teuthology.orchestra.run.smithi116.stdout:(71/146): libverto-devel-0.3.0-5.el8.x86_64.rpm 113 kB/s | 18 kB 00:00 2024-03-30T00:46:49.882 INFO:teuthology.orchestra.run.smithi130.stdout:(131/146): librados2-18.2.2-1018.g470100a4.el8. 6.4 MB/s | 3.7 MB 00:00 2024-03-30T00:46:49.900 INFO:teuthology.orchestra.run.smithi181.stdout:(96/146): openssl-devel-1.1.1k-12.el8_9.x86_64. 12 MB/s | 2.3 MB 00:00 2024-03-30T00:46:49.908 INFO:teuthology.orchestra.run.smithi130.stdout:(132/146): elfutils-libelf-0.189-3.el8.x86_64.r 773 kB/s | 232 kB 00:00 2024-03-30T00:46:49.925 INFO:teuthology.orchestra.run.smithi181.stdout:(97/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 281 kB/s | 47 kB 00:00 2024-03-30T00:46:49.926 INFO:teuthology.orchestra.run.smithi116.stdout:(72/146): python3-jwt-1.6.1-2.el8.noarch.rpm 340 kB/s | 43 kB 00:00 2024-03-30T00:46:49.958 INFO:teuthology.orchestra.run.smithi181.stdout:(98/146): python3-cryptography-3.2.1-7.el8_9.x8 3.3 MB/s | 559 kB 00:00 2024-03-30T00:46:49.959 INFO:teuthology.orchestra.run.smithi116.stdout:(73/146): python3-oauthlib-2.1.0-1.el8.noarch.r 1.5 MB/s | 155 kB 00:00 2024-03-30T00:46:50.050 INFO:teuthology.orchestra.run.smithi181.stdout:(99/146): python3-prettytable-0.7.2-14.el8.noar 351 kB/s | 44 kB 00:00 2024-03-30T00:46:50.084 INFO:teuthology.orchestra.run.smithi181.stdout:(100/146): python3-pyasn1-modules-0.3.7-6.el8.n 879 kB/s | 110 kB 00:00 2024-03-30T00:46:50.101 INFO:teuthology.orchestra.run.smithi181.stdout:(101/146): python3-jsonpointer-1.10-11.el8.noar 99 kB/s | 20 kB 00:00 2024-03-30T00:46:50.150 INFO:teuthology.orchestra.run.smithi130.stdout:(133/146): libmount-2.32.1-43.el8.x86_64.rpm 979 kB/s | 237 kB 00:00 2024-03-30T00:46:50.175 INFO:teuthology.orchestra.run.smithi130.stdout:(134/146): elfutils-libs-0.189-3.el8.x86_64.rpm 1.0 MB/s | 303 kB 00:00 2024-03-30T00:46:50.202 INFO:teuthology.orchestra.run.smithi181.stdout:(102/146): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.1 MB/s | 126 kB 00:00 2024-03-30T00:46:50.218 INFO:teuthology.orchestra.run.smithi181.stdout:(103/146): python3-jsonpatch-1.21-2.el8.noarch. 158 kB/s | 27 kB 00:00 2024-03-30T00:46:50.285 INFO:teuthology.orchestra.run.smithi181.stdout:(104/146): python3-werkzeug-0.12.2-4.el8.noarch 2.4 MB/s | 457 kB 00:00 2024-03-30T00:46:50.317 INFO:teuthology.orchestra.run.smithi130.stdout:(135/146): librbd1-18.2.2-1018.g470100a4.el8.x8 4.9 MB/s | 3.6 MB 00:00 2024-03-30T00:46:50.352 INFO:teuthology.orchestra.run.smithi181.stdout:(105/146): flex-2.6.1-9.el8.x86_64.rpm 2.3 MB/s | 320 kB 00:00 2024-03-30T00:46:50.385 INFO:teuthology.orchestra.run.smithi116.stdout:(74/146): ceph-test-18.2.2-1018.g470100a4.el8.x 7.3 MB/s | 47 MB 00:06 2024-03-30T00:46:50.402 INFO:teuthology.orchestra.run.smithi181.stdout:(106/146): bison-3.0.4-10.el8.x86_64.rpm 3.3 MB/s | 688 kB 00:00 2024-03-30T00:46:50.409 INFO:teuthology.orchestra.run.smithi130.stdout:(136/146): libuuid-2.32.1-43.el8.x86_64.rpm 424 kB/s | 99 kB 00:00 2024-03-30T00:46:50.410 INFO:teuthology.orchestra.run.smithi116.stdout:(75/146): python3-requests-oauthlib-1.0.0-1.el8 95 kB/s | 43 kB 00:00 2024-03-30T00:46:50.436 INFO:teuthology.orchestra.run.smithi181.stdout:(107/146): libtool-2.4.6-25.el8.x86_64.rpm 4.6 MB/s | 709 kB 00:00 2024-03-30T00:46:50.442 INFO:teuthology.orchestra.run.smithi130.stdout:(137/146): libsmartcols-2.32.1-43.el8.x86_64.rp 614 kB/s | 179 kB 00:00 2024-03-30T00:46:50.443 INFO:teuthology.orchestra.run.smithi116.stdout:(76/146): python3-pycparser-2.14-14.el8.noarch. 211 kB/s | 109 kB 00:00 2024-03-30T00:46:50.461 INFO:teuthology.orchestra.run.smithi181.stdout:(108/146): python3-markupsafe-0.23-19.el8.x86_6 355 kB/s | 39 kB 00:00 2024-03-30T00:46:50.513 INFO:teuthology.orchestra.run.smithi181.stdout:(109/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.3 MB/s | 259 kB 00:00 2024-03-30T00:46:50.536 INFO:teuthology.orchestra.run.smithi116.stdout:(77/146): libaio-devel-0.3.112-1.el8.x86_64.rpm 124 kB/s | 19 kB 00:00 2024-03-30T00:46:50.561 INFO:teuthology.orchestra.run.smithi116.stdout:(78/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 303 kB/s | 44 kB 00:00 2024-03-30T00:46:50.594 INFO:teuthology.orchestra.run.smithi116.stdout:(79/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.8 MB/s | 229 kB 00:00 2024-03-30T00:46:50.601 INFO:teuthology.orchestra.run.smithi130.stdout:(138/146): util-linux-2.32.1-43.el8.x86_64.rpm 8.7 MB/s | 2.5 MB 00:00 2024-03-30T00:46:50.603 INFO:teuthology.orchestra.run.smithi181.stdout:(110/146): python3-pyOpenSSL-19.0.0-1.el8.noarc 730 kB/s | 103 kB 00:00 2024-03-30T00:46:50.626 INFO:teuthology.orchestra.run.smithi130.stdout:(139/146): util-linux-user-2.32.1-43.el8.x86_64 473 kB/s | 103 kB 00:00 2024-03-30T00:46:50.628 INFO:teuthology.orchestra.run.smithi116.stdout:(80/146): pcre2-devel-10.32-2.el8.x86_64.rpm 3.2 MB/s | 605 kB 00:00 2024-03-30T00:46:50.637 INFO:teuthology.orchestra.run.smithi181.stdout:(111/146): uuid-1.6.2-43.el8.x86_64.rpm 318 kB/s | 64 kB 00:00 2024-03-30T00:46:50.695 INFO:teuthology.orchestra.run.smithi116.stdout:(81/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.6 MB/s | 220 kB 00:00 2024-03-30T00:46:50.737 INFO:teuthology.orchestra.run.smithi181.stdout:(112/146): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2024-03-30T00:46:50.762 INFO:teuthology.orchestra.run.smithi116.stdout:(82/146): keyutils-libs-devel-1.5.10-9.el8.x86_ 360 kB/s | 48 kB 00:00 2024-03-30T00:46:50.787 INFO:teuthology.orchestra.run.smithi116.stdout:(83/146): libselinux-devel-2.9-5.el8.x86_64.rpm 1.0 MB/s | 200 kB 00:00 2024-03-30T00:46:50.846 INFO:teuthology.orchestra.run.smithi181.stdout:(113/146): python3-numpy-1.14.3-10.el8.x86_64.r 18 MB/s | 3.7 MB 00:00 2024-03-30T00:46:50.868 INFO:teuthology.orchestra.run.smithi130.stdout:(140/146): iproute-6.2.0-5.el8_9.x86_64.rpm 2.0 MB/s | 854 kB 00:00 2024-03-30T00:46:50.871 INFO:teuthology.orchestra.run.smithi181.stdout:(114/146): python3-jinja2-2.10.1-3.el8.noarch.r 2.0 MB/s | 538 kB 00:00 2024-03-30T00:46:50.893 INFO:teuthology.orchestra.run.smithi130.stdout:(141/146): libblkid-2.32.1-43.el8.x86_64.rpm 755 kB/s | 220 kB 00:00 2024-03-30T00:46:50.896 INFO:teuthology.orchestra.run.smithi181.stdout:(115/146): socat-1.7.4.1-1.el8.x86_64.rpm 2.0 MB/s | 323 kB 00:00 2024-03-30T00:46:50.912 INFO:teuthology.orchestra.run.smithi116.stdout:(84/146): libsepol-devel-2.9-3.el8.x86_64.rpm 401 kB/s | 87 kB 00:00 2024-03-30T00:46:50.919 INFO:teuthology.orchestra.run.smithi130.stdout:(142/146): libfdisk-2.32.1-43.el8.x86_64.rpm 869 kB/s | 254 kB 00:00 2024-03-30T00:46:50.930 INFO:teuthology.orchestra.run.smithi181.stdout:(116/146): python3-numpy-f2py-1.14.3-10.el8.x86 2.6 MB/s | 225 kB 00:00 2024-03-30T00:46:50.938 INFO:teuthology.orchestra.run.smithi116.stdout:(85/146): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 329 kB/s | 58 kB 00:00 2024-03-30T00:46:50.972 INFO:teuthology.orchestra.run.smithi181.stdout:(117/146): librdkafka-0.11.4-3.el8.x86_64.rpm 3.4 MB/s | 354 kB 00:00 2024-03-30T00:46:50.996 INFO:teuthology.orchestra.run.smithi116.stdout:(86/146): libcom_err-devel-1.45.6-4.el8.x86_64. 185 kB/s | 39 kB 00:00 2024-03-30T00:46:51.063 INFO:teuthology.orchestra.run.smithi116.stdout:(87/146): fuse-2.9.7-17.el8.x86_64.rpm 552 kB/s | 83 kB 00:00 2024-03-30T00:46:51.086 INFO:teuthology.orchestra.run.smithi130.stdout:(143/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.0 MB/s | 333 kB 00:00 2024-03-30T00:46:51.097 INFO:teuthology.orchestra.run.smithi116.stdout:(88/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.7 MB/s | 172 kB 00:00 2024-03-30T00:46:51.122 INFO:teuthology.orchestra.run.smithi181.stdout:(118/146): openblas-threads-0.3.15-6.el8.x86_64 23 MB/s | 5.2 MB 00:00 2024-03-30T00:46:51.136 INFO:teuthology.orchestra.run.smithi130.stdout:(144/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 6.1 MB/s | 1.5 MB 00:00 2024-03-30T00:46:51.139 INFO:teuthology.orchestra.run.smithi116.stdout:(89/146): libgfortran-8.5.0-20.el8.x86_64.rpm 3.1 MB/s | 645 kB 00:00 2024-03-30T00:46:51.147 INFO:teuthology.orchestra.run.smithi181.stdout:(119/146): python3-mako-1.0.6-14.el8.noarch.rpm 722 kB/s | 157 kB 00:00 2024-03-30T00:46:51.169 INFO:teuthology.orchestra.run.smithi130.stdout:(145/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 2.3 MB/s | 711 kB 00:00 2024-03-30T00:46:51.198 INFO:teuthology.orchestra.run.smithi116.stdout:(90/146): libuuid-devel-2.32.1-43.el8.x86_64.rp 744 kB/s | 100 kB 00:00 2024-03-30T00:46:51.232 INFO:teuthology.orchestra.run.smithi116.stdout:(91/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.7 MB/s | 238 kB 00:00 2024-03-30T00:46:51.265 INFO:teuthology.orchestra.run.smithi181.stdout:(120/146): openblas-0.3.15-6.el8.x86_64.rpm 17 MB/s | 5.0 MB 00:00 2024-03-30T00:46:51.290 INFO:teuthology.orchestra.run.smithi116.stdout:(92/146): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 483 kB/s | 73 kB 00:00 2024-03-30T00:46:51.298 INFO:teuthology.orchestra.run.smithi181.stdout:(121/146): automake-1.16.1-8.el8.noarch.rpm 4.0 MB/s | 714 kB 00:00 2024-03-30T00:46:51.336 INFO:teuthology.orchestra.run.smithi130.stdout:(146/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 4.3 MB/s | 1.1 MB 00:00 2024-03-30T00:46:51.339 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-30T00:46:51.339 INFO:teuthology.orchestra.run.smithi130.stdout:Total 12 MB/s | 213 MB 00:18 2024-03-30T00:46:51.358 INFO:teuthology.orchestra.run.smithi181.stdout:(122/146): libpq-13.11-1.el8.x86_64.rpm 947 kB/s | 198 kB 00:00 2024-03-30T00:46:51.374 INFO:teuthology.orchestra.run.smithi116.stdout:(93/146): elfutils-libelf-devel-0.189-3.el8.x86 435 kB/s | 62 kB 00:00 2024-03-30T00:46:51.399 INFO:teuthology.orchestra.run.smithi181.stdout:(123/146): jq-1.6-7.el8.x86_64.rpm 1.5 MB/s | 202 kB 00:00 2024-03-30T00:46:51.415 INFO:teuthology.orchestra.run.smithi116.stdout:(94/146): NetworkManager-initscripts-updown-1.4 663 kB/s | 144 kB 00:00 2024-03-30T00:46:51.482 INFO:teuthology.orchestra.run.smithi181.stdout:(124/146): python3-pytz-2017.2-11.el8.noarch.rp 295 kB/s | 54 kB 00:00 2024-03-30T00:46:51.483 INFO:teuthology.orchestra.run.smithi116.stdout:(95/146): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 2.9 MB/s | 562 kB 00:00 2024-03-30T00:46:51.566 INFO:teuthology.orchestra.run.smithi181.stdout:(125/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-03-30T00:46:51.591 INFO:teuthology.orchestra.run.smithi116.stdout:(96/146): openssl-devel-1.1.1k-12.el8_9.x86_64. 11 MB/s | 2.3 MB 00:00 2024-03-30T00:46:51.625 INFO:teuthology.orchestra.run.smithi116.stdout:(97/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 226 kB/s | 47 kB 00:00 2024-03-30T00:46:51.658 INFO:teuthology.orchestra.run.smithi181.stdout:(126/146): postgresql-10.23-4.module+el8.9.0+21 8.6 MB/s | 1.5 MB 00:00 2024-03-30T00:46:51.658 INFO:teuthology.orchestra.run.smithi116.stdout:(98/146): python3-cryptography-3.2.1-7.el8_9.x8 3.1 MB/s | 559 kB 00:00 2024-03-30T00:46:51.716 INFO:teuthology.orchestra.run.smithi181.stdout:(127/146): postgresql-contrib-10.23-4.module+el 5.3 MB/s | 812 kB 00:00 2024-03-30T00:46:51.717 INFO:teuthology.orchestra.run.smithi116.stdout:(99/146): python3-jsonpointer-1.10-11.el8.noarc 159 kB/s | 20 kB 00:00 2024-03-30T00:46:51.759 INFO:teuthology.orchestra.run.smithi116.stdout:(100/146): python3-prettytable-0.7.2-14.el8.noa 328 kB/s | 44 kB 00:00 2024-03-30T00:46:51.784 INFO:teuthology.orchestra.run.smithi116.stdout:(101/146): python3-pyasn1-modules-0.3.7-6.el8.n 876 kB/s | 110 kB 00:00 2024-03-30T00:46:51.809 INFO:teuthology.orchestra.run.smithi116.stdout:(102/146): python3-jsonpatch-1.21-2.el8.noarch. 288 kB/s | 27 kB 00:00 2024-03-30T00:46:51.917 INFO:teuthology.orchestra.run.smithi181.stdout:(128/146): python3-scipy-1.0.0-21.module+el8.9. 24 MB/s | 14 MB 00:00 2024-03-30T00:46:51.942 INFO:teuthology.orchestra.run.smithi181.stdout:(129/146): python3-packaging-16.8-9.el8.noarch. 214 kB/s | 48 kB 00:00 2024-03-30T00:46:51.951 INFO:teuthology.orchestra.run.smithi116.stdout:(103/146): python3-werkzeug-0.12.2-4.el8.noarch 2.7 MB/s | 457 kB 00:00 2024-03-30T00:46:51.985 INFO:teuthology.orchestra.run.smithi116.stdout:(104/146): bison-3.0.4-10.el8.x86_64.rpm 3.8 MB/s | 688 kB 00:00 2024-03-30T00:46:52.042 INFO:teuthology.orchestra.run.smithi181.stdout:(130/146): postgresql-server-10.23-4.module+el8 13 MB/s | 5.1 MB 00:00 2024-03-30T00:46:52.111 INFO:teuthology.orchestra.run.smithi116.stdout:(105/146): libtool-2.4.6-25.el8.x86_64.rpm 5.6 MB/s | 709 kB 00:00 2024-03-30T00:46:52.156 INFO:teuthology.orchestra.run.smithi116.stdout:(106/146): flex-2.6.1-9.el8.x86_64.rpm 1.6 MB/s | 320 kB 00:00 2024-03-30T00:46:52.177 INFO:teuthology.orchestra.run.smithi116.stdout:(107/146): python3-pyasn1-0.3.7-6.el8.noarch.rp 302 kB/s | 126 kB 00:00 2024-03-30T00:46:52.209 INFO:teuthology.orchestra.run.smithi181.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.4 MB/s | 232 kB 00:00 2024-03-30T00:46:52.278 INFO:teuthology.orchestra.run.smithi116.stdout:(108/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.0 MB/s | 259 kB 00:00 2024-03-30T00:46:52.369 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-30T00:46:52.395 INFO:teuthology.orchestra.run.smithi116.stdout:(109/146): python3-markupsafe-0.23-19.el8.x86_6 136 kB/s | 39 kB 00:00 2024-03-30T00:46:52.420 INFO:teuthology.orchestra.run.smithi116.stdout:(110/146): uuid-1.6.2-43.el8.x86_64.rpm 264 kB/s | 64 kB 00:00 2024-03-30T00:46:52.437 INFO:teuthology.orchestra.run.smithi116.stdout:(111/146): python3-pyOpenSSL-19.0.0-1.el8.noarc 652 kB/s | 103 kB 00:00 2024-03-30T00:46:52.484 INFO:teuthology.orchestra.run.smithi181.stdout:(132/146): librbd1-18.2.2-1018.g470100a4.el8.x8 6.6 MB/s | 3.6 MB 00:00 2024-03-30T00:46:52.509 INFO:teuthology.orchestra.run.smithi181.stdout:(133/146): elfutils-libs-0.189-3.el8.x86_64.rpm 1.0 MB/s | 303 kB 00:00 2024-03-30T00:46:52.545 INFO:teuthology.orchestra.run.smithi116.stdout:(112/146): python3-jinja2-2.10.1-3.el8.noarch.r 4.2 MB/s | 538 kB 00:00 2024-03-30T00:46:52.601 INFO:teuthology.orchestra.run.smithi181.stdout:(134/146): libmount-2.32.1-43.el8.x86_64.rpm 2.0 MB/s | 237 kB 00:00 2024-03-30T00:46:52.604 INFO:teuthology.orchestra.run.smithi116.stdout:(113/146): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2024-03-30T00:46:52.644 INFO:teuthology.orchestra.run.smithi181.stdout:(135/146): libsmartcols-2.32.1-43.el8.x86_64.rp 1.3 MB/s | 179 kB 00:00 2024-03-30T00:46:52.680 INFO:teuthology.orchestra.run.smithi116.stdout:(114/146): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2024-03-30T00:46:52.735 INFO:teuthology.orchestra.run.smithi181.stdout:(136/146): librados2-18.2.2-1018.g470100a4.el8. 4.5 MB/s | 3.7 MB 00:00 2024-03-30T00:46:52.760 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-30T00:46:52.760 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-30T00:46:52.761 INFO:teuthology.orchestra.run.smithi181.stdout:(137/146): libuuid-2.32.1-43.el8.x86_64.rpm 623 kB/s | 99 kB 00:00 2024-03-30T00:46:52.771 INFO:teuthology.orchestra.run.smithi116.stdout:(115/146): python3-numpy-1.14.3-10.el8.x86_64.r 11 MB/s | 3.7 MB 00:00 2024-03-30T00:46:52.805 INFO:teuthology.orchestra.run.smithi116.stdout:(116/146): python3-numpy-f2py-1.14.3-10.el8.x86 1.1 MB/s | 225 kB 00:00 2024-03-30T00:46:52.866 INFO:teuthology.orchestra.run.smithi116.stdout:(117/146): librdkafka-0.11.4-3.el8.x86_64.rpm 1.9 MB/s | 354 kB 00:00 2024-03-30T00:46:52.869 INFO:teuthology.orchestra.run.smithi181.stdout:(138/146): util-linux-2.32.1-43.el8.x86_64.rpm 11 MB/s | 2.5 MB 00:00 2024-03-30T00:46:52.945 INFO:teuthology.orchestra.run.smithi181.stdout:(139/146): util-linux-user-2.32.1-43.el8.x86_64 491 kB/s | 103 kB 00:00 2024-03-30T00:46:52.998 INFO:teuthology.orchestra.run.smithi116.stdout:(118/146): python3-mako-1.0.6-14.el8.noarch.rpm 816 kB/s | 157 kB 00:00 2024-03-30T00:46:53.028 INFO:teuthology.orchestra.run.smithi181.stdout:(140/146): iproute-6.2.0-5.el8_9.x86_64.rpm 3.1 MB/s | 854 kB 00:00 2024-03-30T00:46:53.123 INFO:teuthology.orchestra.run.smithi116.stdout:(119/146): openblas-0.3.15-6.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2024-03-30T00:46:53.137 INFO:teuthology.orchestra.run.smithi181.stdout:(141/146): libblkid-2.32.1-43.el8.x86_64.rpm 825 kB/s | 220 kB 00:00 2024-03-30T00:46:53.162 INFO:teuthology.orchestra.run.smithi181.stdout:(142/146): libfdisk-2.32.1-43.el8.x86_64.rpm 1.1 MB/s | 254 kB 00:00 2024-03-30T00:46:53.173 INFO:teuthology.orchestra.run.smithi116.stdout:(120/146): automake-1.16.1-8.el8.noarch.rpm 4.0 MB/s | 714 kB 00:00 2024-03-30T00:46:53.196 INFO:teuthology.orchestra.run.smithi181.stdout:(143/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 4.2 MB/s | 711 kB 00:00 2024-03-30T00:46:53.290 INFO:teuthology.orchestra.run.smithi116.stdout:(121/146): openblas-threads-0.3.15-6.el8.x86_64 10 MB/s | 5.2 MB 00:00 2024-03-30T00:46:53.304 INFO:teuthology.orchestra.run.smithi181.stdout:(144/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 8.8 MB/s | 1.5 MB 00:00 2024-03-30T00:46:53.328 INFO:teuthology.orchestra.run.smithi116.stdout:(122/146): libpq-13.11-1.el8.x86_64.rpm 963 kB/s | 198 kB 00:00 2024-03-30T00:46:53.329 INFO:teuthology.orchestra.run.smithi181.stdout:(145/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.0 MB/s | 333 kB 00:00 2024-03-30T00:46:53.382 INFO:teuthology.orchestra.run.smithi116.stdout:(123/146): jq-1.6-7.el8.x86_64.rpm 970 kB/s | 202 kB 00:00 2024-03-30T00:46:53.518 INFO:teuthology.orchestra.run.smithi116.stdout:(124/146): python3-pytz-2017.2-11.el8.noarch.rp 238 kB/s | 54 kB 00:00 2024-03-30T00:46:53.549 INFO:teuthology.orchestra.run.smithi116.stdout:(125/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-03-30T00:46:53.630 INFO:teuthology.orchestra.run.smithi181.stdout:(146/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 2.5 MB/s | 1.1 MB 00:00 2024-03-30T00:46:53.635 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-30T00:46:53.635 INFO:teuthology.orchestra.run.smithi181.stdout:Total 10 MB/s | 213 MB 00:21 2024-03-30T00:46:53.716 INFO:teuthology.orchestra.run.smithi116.stdout:(126/146): postgresql-10.23-4.module+el8.9.0+21 7.6 MB/s | 1.5 MB 00:00 2024-03-30T00:46:53.750 INFO:teuthology.orchestra.run.smithi116.stdout:(127/146): postgresql-contrib-10.23-4.module+el 4.0 MB/s | 812 kB 00:00 2024-03-30T00:46:53.942 INFO:teuthology.orchestra.run.smithi116.stdout:(128/146): python3-scipy-1.0.0-21.module+el8.9. 22 MB/s | 14 MB 00:00 2024-03-30T00:46:53.959 INFO:teuthology.orchestra.run.smithi116.stdout:(129/146): python3-packaging-16.8-9.el8.noarch. 231 kB/s | 48 kB 00:00 2024-03-30T00:46:54.084 INFO:teuthology.orchestra.run.smithi116.stdout:(130/146): postgresql-server-10.23-4.module+el8 14 MB/s | 5.1 MB 00:00 2024-03-30T00:46:54.215 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-30T00:46:54.226 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-30T00:46:54.309 INFO:teuthology.orchestra.run.smithi116.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.0 MB/s | 232 kB 00:00 2024-03-30T00:46:54.493 INFO:teuthology.orchestra.run.smithi116.stdout:(132/146): elfutils-libs-0.189-3.el8.x86_64.rpm 1.6 MB/s | 303 kB 00:00 2024-03-30T00:46:54.593 INFO:teuthology.orchestra.run.smithi116.stdout:(133/146): librados2-18.2.2-1018.g470100a4.el8. 5.6 MB/s | 3.7 MB 00:00 2024-03-30T00:46:54.643 INFO:teuthology.orchestra.run.smithi116.stdout:(134/146): libmount-2.32.1-43.el8.x86_64.rpm 1.5 MB/s | 237 kB 00:00 2024-03-30T00:46:54.705 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-30T00:46:54.744 INFO:teuthology.orchestra.run.smithi116.stdout:(135/146): libsmartcols-2.32.1-43.el8.x86_64.rp 1.2 MB/s | 179 kB 00:00 2024-03-30T00:46:54.844 INFO:teuthology.orchestra.run.smithi116.stdout:(136/146): libuuid-2.32.1-43.el8.x86_64.rpm 494 kB/s | 99 kB 00:00 2024-03-30T00:46:54.895 INFO:teuthology.orchestra.run.smithi116.stdout:(137/146): librbd1-18.2.2-1018.g470100a4.el8.x8 3.8 MB/s | 3.6 MB 00:00 2024-03-30T00:46:54.970 INFO:teuthology.orchestra.run.smithi116.stdout:(138/146): util-linux-2.32.1-43.el8.x86_64.rpm 11 MB/s | 2.5 MB 00:00 2024-03-30T00:46:54.995 INFO:teuthology.orchestra.run.smithi116.stdout:(139/146): util-linux-user-2.32.1-43.el8.x86_64 683 kB/s | 103 kB 00:00 2024-03-30T00:46:55.047 INFO:teuthology.orchestra.run.smithi116.stdout:(140/146): iproute-6.2.0-5.el8_9.x86_64.rpm 5.5 MB/s | 854 kB 00:00 2024-03-30T00:46:55.100 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-30T00:46:55.100 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-30T00:46:55.120 INFO:teuthology.orchestra.run.smithi116.stdout:(141/146): libfdisk-2.32.1-43.el8.x86_64.rpm 2.0 MB/s | 254 kB 00:00 2024-03-30T00:46:55.154 INFO:teuthology.orchestra.run.smithi116.stdout:(142/146): libblkid-2.32.1-43.el8.x86_64.rpm 1.2 MB/s | 220 kB 00:00 2024-03-30T00:46:55.271 INFO:teuthology.orchestra.run.smithi116.stdout:(143/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 3.1 MB/s | 711 kB 00:00 2024-03-30T00:46:55.313 INFO:teuthology.orchestra.run.smithi116.stdout:(144/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.1 MB/s | 333 kB 00:00 2024-03-30T00:46:55.354 INFO:teuthology.orchestra.run.smithi116.stdout:(145/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 6.3 MB/s | 1.5 MB 00:00 2024-03-30T00:46:55.455 INFO:teuthology.orchestra.run.smithi116.stdout:(146/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 5.8 MB/s | 1.1 MB 00:00 2024-03-30T00:46:55.457 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-30T00:46:55.457 INFO:teuthology.orchestra.run.smithi116.stdout:Total 13 MB/s | 213 MB 00:15 2024-03-30T00:46:56.300 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-30T00:46:56.311 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-30T00:46:57.559 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-30T00:46:58.006 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-30T00:46:58.006 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-30T00:46:59.571 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-30T00:46:59.583 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-30T00:47:00.289 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-30T00:47:00.424 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-03-30T00:47:00.752 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-30T00:47:00.969 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-30T00:47:01.131 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-03-30T00:47:01.347 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-30T00:47:01.412 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-30T00:47:01.597 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-30T00:47:01.691 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-30T00:47:01.893 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-30T00:47:01.943 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-03-30T00:47:02.027 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-03-30T00:47:02.130 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 6/162 2024-03-30T00:47:02.277 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-30T00:47:02.356 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-30T00:47:02.490 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-30T00:47:02.495 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-30T00:47:02.672 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-03-30T00:47:02.705 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-30T00:47:02.861 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-30T00:47:02.940 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-30T00:47:03.105 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-30T00:47:03.128 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-30T00:47:03.201 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-30T00:47:03.380 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librados2-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T00:47:03.432 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-03-30T00:47:03.668 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 6/162 2024-03-30T00:47:03.768 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T00:47:03.853 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-30T00:47:03.934 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-30T00:47:03.972 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T00:47:04.097 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T00:47:04.175 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-30T00:47:04.288 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T00:47:04.419 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T00:47:04.759 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-30T00:47:04.766 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 12/162 2024-03-30T00:47:04.953 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T00:47:05.089 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-30T00:47:05.177 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-30T00:47:05.346 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T00:47:05.428 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-03-30T00:47:05.519 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T00:47:05.651 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T00:47:05.700 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-03-30T00:47:05.805 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-30T00:47:05.860 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T00:47:05.917 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-03-30T00:47:05.988 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T00:47:06.012 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-03-30T00:47:06.245 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-30T00:47:06.321 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 12/162 2024-03-30T00:47:06.326 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-03-30T00:47:06.523 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-30T00:47:06.585 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-30T00:47:06.611 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-03-30T00:47:06.654 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-30T00:47:06.705 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-03-30T00:47:06.763 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T00:47:06.887 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-30T00:47:06.892 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-03-30T00:47:06.902 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T00:47:06.971 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-30T00:47:07.088 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 20/162 2024-03-30T00:47:07.115 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-03-30T00:47:07.136 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-30T00:47:07.166 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 20/162 2024-03-30T00:47:07.223 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-30T00:47:07.289 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-03-30T00:47:07.367 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 21/162 2024-03-30T00:47:07.442 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-03-30T00:47:07.647 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/162 2024-03-30T00:47:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-03-30T00:47:07.663 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 6/162 2024-03-30T00:47:07.895 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-30T00:47:07.915 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-03-30T00:47:07.977 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/162 2024-03-30T00:47:08.028 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-30T00:47:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T00:47:08.179 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-30T00:47:08.190 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T00:47:08.203 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-30T00:47:08.354 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-30T00:47:08.368 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 20/162 2024-03-30T00:47:08.428 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 20/162 2024-03-30T00:47:08.562 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-30T00:47:08.580 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 21/162 2024-03-30T00:47:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-30T00:47:08.827 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/162 2024-03-30T00:47:08.908 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T00:47:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/162 2024-03-30T00:47:09.185 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-30T00:47:09.235 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-30T00:47:09.329 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T00:47:09.375 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-30T00:47:09.379 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-30T00:47:09.511 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T00:47:09.559 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-30T00:47:09.666 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T00:47:09.894 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T00:47:09.966 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-30T00:47:10.019 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T00:47:10.146 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-0.3.15-6.el8.x86_64 27/162 2024-03-30T00:47:10.180 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-30T00:47:10.360 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-30T00:47:10.375 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 12/162 2024-03-30T00:47:10.669 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-30T00:47:10.781 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-30T00:47:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 27/162 2024-03-30T00:47:10.971 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-30T00:47:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-03-30T00:47:11.152 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.15-6.el8.x86_64 27/162 2024-03-30T00:47:11.210 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 28/162 2024-03-30T00:47:11.285 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-03-30T00:47:11.410 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 29/162 2024-03-30T00:47:11.483 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-03-30T00:47:11.673 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-03-30T00:47:11.836 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 27/162 2024-03-30T00:47:11.844 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-common-2:18.2.2-1018.g470100a4.el8. 31/162 2024-03-30T00:47:11.947 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-03-30T00:47:12.023 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-argparse-2:18.2.2-1018.g470100a4.el 32/162 2024-03-30T00:47:12.091 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 28/162 2024-03-30T00:47:12.225 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-03-30T00:47:12.317 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_6 33/162 2024-03-30T00:47:12.332 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 29/162 2024-03-30T00:47:12.378 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T00:47:12.504 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T00:47:12.520 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-30T00:47:12.561 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-03-30T00:47:12.686 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 20/162 2024-03-30T00:47:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:18.2.2-1018.g470100a4.el8. 31/162 2024-03-30T00:47:12.784 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 20/162 2024-03-30T00:47:12.862 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:18.2.2-1018.g470100a4.el 32/162 2024-03-30T00:47:12.942 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 21/162 2024-03-30T00:47:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_6 33/162 2024-03-30T00:47:13.219 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/162 2024-03-30T00:47:13.351 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-30T00:47:13.469 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/162 2024-03-30T00:47:13.662 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-30T00:47:13.826 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-30T00:47:13.952 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-30T00:47:14.257 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-30T00:47:14.656 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 35/162 2024-03-30T00:47:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-30T00:47:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-30T00:47:15.095 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-30T00:47:15.429 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 35/162 2024-03-30T00:47:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-30T00:47:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.15-6.el8.x86_64 27/162 2024-03-30T00:47:15.959 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 36/162 2024-03-30T00:47:16.317 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 37/162 2024-03-30T00:47:16.410 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 36/162 2024-03-30T00:47:16.491 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 27/162 2024-03-30T00:47:16.587 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 38/162 2024-03-30T00:47:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 37/162 2024-03-30T00:47:16.768 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 28/162 2024-03-30T00:47:16.832 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-03-30T00:47:16.994 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 29/162 2024-03-30T00:47:17.018 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 38/162 2024-03-30T00:47:17.067 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-03-30T00:47:17.237 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-03-30T00:47:17.246 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-03-30T00:47:17.295 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 41/162 2024-03-30T00:47:17.445 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-03-30T00:47:17.485 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librados-devel-2:18.2.2-1018.g470100a4.el8.x86_6 42/162 2024-03-30T00:47:17.492 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:18.2.2-1018.g470100a4.el8. 31/162 2024-03-30T00:47:17.623 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:18.2.2-1018.g470100a4.el 32/162 2024-03-30T00:47:17.681 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 41/162 2024-03-30T00:47:17.721 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-03-30T00:47:17.782 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-03-30T00:47:17.840 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:18.2.2-1018.g470100a4.el8.x86_6 42/162 2024-03-30T00:47:17.938 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_6 33/162 2024-03-30T00:47:18.045 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-30T00:47:18.053 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-03-30T00:47:18.126 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-30T00:47:18.130 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-03-30T00:47:18.158 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-30T00:47:18.342 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-30T00:47:18.393 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-03-30T00:47:18.425 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-30T00:47:18.634 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-03-30T00:47:18.671 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-03-30T00:47:18.853 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-30T00:47:18.878 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-03-30T00:47:19.075 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-30T00:47:19.319 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-30T00:47:19.512 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-30T00:47:19.695 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-30T00:47:19.721 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-30T00:47:19.817 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-30T00:47:19.843 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-30T00:47:19.956 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-packaging-16.8-9.el8.noarch 49/162 2024-03-30T00:47:20.036 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-16.8-9.el8.noarch 49/162 2024-03-30T00:47:20.145 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 50/162 2024-03-30T00:47:20.151 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-30T00:47:20.200 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 50/162 2024-03-30T00:47:20.229 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 50/162 2024-03-30T00:47:20.285 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 50/162 2024-03-30T00:47:20.490 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 35/162 2024-03-30T00:47:20.517 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : jq-1.6-7.el8.x86_64 51/162 2024-03-30T00:47:20.552 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : jq-1.6-7.el8.x86_64 51/162 2024-03-30T00:47:20.817 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : automake-1.16.1-8.el8.noarch 52/162 2024-03-30T00:47:20.827 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : automake-1.16.1-8.el8.noarch 52/162 2024-03-30T00:47:21.008 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : uuid-1.6.2-43.el8.x86_64 53/162 2024-03-30T00:47:21.011 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : uuid-1.6.2-43.el8.x86_64 53/162 2024-03-30T00:47:21.080 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 53/162 2024-03-30T00:47:21.114 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 53/162 2024-03-30T00:47:21.248 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 54/162 2024-03-30T00:47:21.321 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 54/162 2024-03-30T00:47:21.432 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 55/162 2024-03-30T00:47:21.511 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 55/162 2024-03-30T00:47:21.576 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 36/162 2024-03-30T00:47:21.627 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 56/162 2024-03-30T00:47:21.730 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 56/162 2024-03-30T00:47:21.821 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.7-17.el8.x86_64 57/162 2024-03-30T00:47:21.960 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 37/162 2024-03-30T00:47:21.966 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : fuse-2.9.7-17.el8.x86_64 57/162 2024-03-30T00:47:22.024 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 58/162 2024-03-30T00:47:22.205 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 59/162 2024-03-30T00:47:22.210 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 58/162 2024-03-30T00:47:22.224 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 38/162 2024-03-30T00:47:22.416 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 59/162 2024-03-30T00:47:22.420 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 60/162 2024-03-30T00:47:22.461 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-03-30T00:47:22.625 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 60/162 2024-03-30T00:47:22.628 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 61/162 2024-03-30T00:47:22.698 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-03-30T00:47:22.865 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 61/162 2024-03-30T00:47:22.900 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 41/162 2024-03-30T00:47:22.927 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 62/162 2024-03-30T00:47:23.093 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:18.2.2-1018.g470100a4.el8.x86_6 42/162 2024-03-30T00:47:23.200 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 62/162 2024-03-30T00:47:23.204 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 63/162 2024-03-30T00:47:23.280 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-03-30T00:47:23.378 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-03-30T00:47:23.396 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 64/162 2024-03-30T00:47:23.535 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 63/162 2024-03-30T00:47:23.562 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 65/162 2024-03-30T00:47:23.617 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-30T00:47:23.702 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-30T00:47:23.768 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 66/162 2024-03-30T00:47:23.791 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 64/162 2024-03-30T00:47:23.893 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-03-30T00:47:23.950 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 65/162 2024-03-30T00:47:24.122 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-03-30T00:47:24.139 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 67/162 2024-03-30T00:47:24.217 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 66/162 2024-03-30T00:47:24.317 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-30T00:47:24.436 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 68/162 2024-03-30T00:47:24.599 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 67/162 2024-03-30T00:47:24.749 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 69/162 2024-03-30T00:47:24.909 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-30T00:47:24.931 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 68/162 2024-03-30T00:47:24.949 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 70/162 2024-03-30T00:47:25.117 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 71/162 2024-03-30T00:47:25.218 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-30T00:47:25.252 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 69/162 2024-03-30T00:47:25.262 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-30T00:47:25.344 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 72/162 2024-03-30T00:47:25.481 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-packaging-16.8-9.el8.noarch 49/162 2024-03-30T00:47:25.544 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 70/162 2024-03-30T00:47:25.628 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 73/162 2024-03-30T00:47:25.642 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 50/162 2024-03-30T00:47:25.724 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 50/162 2024-03-30T00:47:25.754 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 71/162 2024-03-30T00:47:25.833 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/162 2024-03-30T00:47:26.027 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 75/162 2024-03-30T00:47:26.028 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 72/162 2024-03-30T00:47:26.042 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jq-1.6-7.el8.x86_64 51/162 2024-03-30T00:47:26.238 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 76/162 2024-03-30T00:47:26.299 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : automake-1.16.1-8.el8.noarch 52/162 2024-03-30T00:47:26.316 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 73/162 2024-03-30T00:47:26.468 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 77/162 2024-03-30T00:47:26.500 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : uuid-1.6.2-43.el8.x86_64 53/162 2024-03-30T00:47:26.553 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/162 2024-03-30T00:47:26.593 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 53/162 2024-03-30T00:47:26.640 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-03-30T00:47:26.774 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 75/162 2024-03-30T00:47:26.776 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 54/162 2024-03-30T00:47:26.802 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-03-30T00:47:26.976 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 55/162 2024-03-30T00:47:27.020 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 76/162 2024-03-30T00:47:27.117 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-03-30T00:47:27.154 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 56/162 2024-03-30T00:47:27.307 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 77/162 2024-03-30T00:47:27.330 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-03-30T00:47:27.356 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.7-17.el8.x86_64 57/162 2024-03-30T00:47:27.503 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-03-30T00:47:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-03-30T00:47:27.547 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 58/162 2024-03-30T00:47:27.718 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-03-30T00:47:27.718 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-03-30T00:47:27.768 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 59/162 2024-03-30T00:47:27.916 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-03-30T00:47:27.956 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 60/162 2024-03-30T00:47:28.059 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-03-30T00:47:28.127 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-03-30T00:47:28.148 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 61/162 2024-03-30T00:47:28.326 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-03-30T00:47:28.377 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-03-30T00:47:28.503 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 62/162 2024-03-30T00:47:28.531 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-03-30T00:47:28.594 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-03-30T00:47:28.722 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-03-30T00:47:28.771 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-03-30T00:47:28.775 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 63/162 2024-03-30T00:47:28.951 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-03-30T00:47:28.997 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 64/162 2024-03-30T00:47:29.010 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-03-30T00:47:29.159 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 65/162 2024-03-30T00:47:29.206 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-03-30T00:47:29.210 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-03-30T00:47:29.396 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-03-30T00:47:29.409 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 66/162 2024-03-30T00:47:29.497 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-03-30T00:47:29.627 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-03-30T00:47:29.707 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-03-30T00:47:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 67/162 2024-03-30T00:47:29.881 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-03-30T00:47:29.907 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-03-30T00:47:30.106 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-03-30T00:47:30.181 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-03-30T00:47:30.347 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-03-30T00:47:30.431 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-03-30T00:47:30.458 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 68/162 2024-03-30T00:47:30.641 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-03-30T00:47:30.790 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 69/162 2024-03-30T00:47:30.916 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-03-30T00:47:31.009 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 70/162 2024-03-30T00:47:31.209 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-03-30T00:47:31.238 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 71/162 2024-03-30T00:47:31.427 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-03-30T00:47:31.430 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-03-30T00:47:31.518 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 72/162 2024-03-30T00:47:31.686 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-03-30T00:47:31.740 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-03-30T00:47:31.785 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 73/162 2024-03-30T00:47:31.969 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/162 2024-03-30T00:47:31.972 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-03-30T00:47:32.194 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 75/162 2024-03-30T00:47:32.256 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-03-30T00:47:32.391 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 76/162 2024-03-30T00:47:32.532 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-03-30T00:47:32.669 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 77/162 2024-03-30T00:47:32.731 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-03-30T00:47:32.812 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-03-30T00:47:32.851 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-03-30T00:47:33.044 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-03-30T00:47:33.155 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-03-30T00:47:33.404 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-03-30T00:47:33.450 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-03-30T00:47:33.537 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-03-30T00:47:33.666 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-03-30T00:47:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 103/162 2024-03-30T00:47:33.777 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-03-30T00:47:33.796 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 103/162 2024-03-30T00:47:33.838 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-03-30T00:47:34.037 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-03-30T00:47:34.040 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 104/162 2024-03-30T00:47:34.053 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-03-30T00:47:34.234 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-03-30T00:47:34.270 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-03-30T00:47:34.464 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-03-30T00:47:34.679 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-03-30T00:47:34.836 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-03-30T00:47:35.005 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-03-30T00:47:35.046 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-03-30T00:47:35.251 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-03-30T00:47:35.267 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 103/162 2024-03-30T00:47:35.342 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 103/162 2024-03-30T00:47:35.478 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-03-30T00:47:35.545 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 104/162 2024-03-30T00:47:35.647 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-03-30T00:47:35.905 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-03-30T00:47:36.149 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-03-30T00:47:36.341 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-03-30T00:47:36.587 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-03-30T00:47:37.882 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-03-30T00:47:38.176 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-03-30T00:47:38.256 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T00:47:38.445 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-03-30T00:47:38.497 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T00:47:38.752 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-03-30T00:47:39.012 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-03-30T00:47:39.091 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T00:47:39.254 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-03-30T00:47:39.371 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 106/162 2024-03-30T00:47:39.456 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 106/162 2024-03-30T00:47:39.589 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 107/162 2024-03-30T00:47:39.605 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T00:47:39.884 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T00:47:40.078 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-03-30T00:47:40.281 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 103/162 2024-03-30T00:47:40.374 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 103/162 2024-03-30T00:47:40.473 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T00:47:40.797 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 106/162 2024-03-30T00:47:40.876 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 106/162 2024-03-30T00:47:40.924 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 104/162 2024-03-30T00:47:41.083 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 107/162 2024-03-30T00:47:45.767 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T00:47:46.523 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T00:47:47.219 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T00:47:47.926 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 106/162 2024-03-30T00:47:47.926 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 106/162 2024-03-30T00:47:47.926 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 107/162 2024-03-30T00:47:59.268 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 107/162 2024-03-30T00:47:59.268 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-03-30T00:47:59.268 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-03-30T00:47:59.269 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /dev 2024-03-30T00:47:59.269 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /run 2024-03-30T00:47:59.269 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-03-30T00:47:59.269 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-03-30T00:47:59.269 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-03-30T00:47:59.269 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-03-30T00:47:59.269 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /dev 2024-03-30T00:47:59.269 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:47:59.628 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 108/162 2024-03-30T00:48:00.461 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 107/162 2024-03-30T00:48:00.461 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /sys 2024-03-30T00:48:00.461 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /proc 2024-03-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /dev 2024-03-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /run 2024-03-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /mnt 2024-03-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /var/tmp 2024-03-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /home 2024-03-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /tmp 2024-03-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /dev 2024-03-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-30T00:48:00.922 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 108/162 2024-03-30T00:48:01.259 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 108/162 2024-03-30T00:48:01.468 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 109/162 2024-03-30T00:48:01.688 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 109/162 2024-03-30T00:48:01.844 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 110/162 2024-03-30T00:48:02.320 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 110/162 2024-03-30T00:48:02.486 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 111/162 2024-03-30T00:48:02.503 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 111/162 2024-03-30T00:48:02.551 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 108/162 2024-03-30T00:48:02.777 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 109/162 2024-03-30T00:48:02.997 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 109/162 2024-03-30T00:48:03.197 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 110/162 2024-03-30T00:48:03.671 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 110/162 2024-03-30T00:48:03.866 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 111/162 2024-03-30T00:48:03.883 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 111/162 2024-03-30T00:48:04.185 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:18.2.2-1018.g470100a4.el8.noarch 112/162 2024-03-30T00:48:04.378 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:18.2.2-1018.g470100a4.el8.noarch 112/162 2024-03-30T00:48:04.516 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.e 113/162 2024-03-30T00:48:04.721 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1018.g470100a4. 114/162 2024-03-30T00:48:04.954 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 115/162 2024-03-30T00:48:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 115/162 2024-03-30T00:48:05.484 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 116/162 2024-03-30T00:48:05.748 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephadm-2:18.2.2-1018.g470100a4.el8.noarch 112/162 2024-03-30T00:48:05.912 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephadm-2:18.2.2-1018.g470100a4.el8.noarch 112/162 2024-03-30T00:48:06.094 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.e 113/162 2024-03-30T00:48:06.317 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1018.g470100a4. 114/162 2024-03-30T00:48:06.446 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 116/162 2024-03-30T00:48:06.592 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 115/162 2024-03-30T00:48:06.634 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 117/162 2024-03-30T00:48:06.729 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 117/162 2024-03-30T00:48:06.815 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 115/162 2024-03-30T00:48:07.067 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 116/162 2024-03-30T00:48:07.087 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el 118/162 2024-03-30T00:48:07.259 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 119/162 2024-03-30T00:48:07.297 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 119/162 2024-03-30T00:48:07.468 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 120/162 2024-03-30T00:48:07.479 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 120/162 2024-03-30T00:48:08.032 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 116/162 2024-03-30T00:48:08.245 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 117/162 2024-03-30T00:48:08.338 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 117/162 2024-03-30T00:48:08.481 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 107/162 2024-03-30T00:48:08.481 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-03-30T00:48:08.481 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-03-30T00:48:08.481 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /dev 2024-03-30T00:48:08.481 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /run 2024-03-30T00:48:08.481 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-03-30T00:48:08.481 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-03-30T00:48:08.481 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-03-30T00:48:08.482 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-03-30T00:48:08.482 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /dev 2024-03-30T00:48:08.482 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:48:08.721 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el 118/162 2024-03-30T00:48:08.744 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:18.2.2-1018.g470100a4.el8.x86_64 121/162 2024-03-30T00:48:08.853 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 108/162 2024-03-30T00:48:08.921 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 119/162 2024-03-30T00:48:08.960 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 119/162 2024-03-30T00:48:09.017 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 122/162 2024-03-30T00:48:09.053 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 122/162 2024-03-30T00:48:09.120 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 120/162 2024-03-30T00:48:09.132 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 120/162 2024-03-30T00:48:09.251 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1018.g47010 123/162 2024-03-30T00:48:09.296 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1018.g47010 123/162 2024-03-30T00:48:09.476 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 124/162 2024-03-30T00:48:09.824 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 124/162 2024-03-30T00:48:10.001 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 125/162 2024-03-30T00:48:10.443 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-2:18.2.2-1018.g470100a4.el8.x86_64 121/162 2024-03-30T00:48:10.615 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 108/162 2024-03-30T00:48:10.709 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 122/162 2024-03-30T00:48:10.745 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 122/162 2024-03-30T00:48:10.828 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 109/162 2024-03-30T00:48:10.937 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1018.g47010 123/162 2024-03-30T00:48:10.980 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1018.g47010 123/162 2024-03-30T00:48:11.094 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 109/162 2024-03-30T00:48:11.179 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 124/162 2024-03-30T00:48:11.244 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 110/162 2024-03-30T00:48:11.525 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 124/162 2024-03-30T00:48:11.712 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 125/162 2024-03-30T00:48:11.781 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 110/162 2024-03-30T00:48:11.969 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 111/162 2024-03-30T00:48:12.012 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 111/162 2024-03-30T00:48:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 125/162 2024-03-30T00:48:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:18.2.2-1018.g470100a4.el8.noarch 112/162 2024-03-30T00:48:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:18.2.2-1018.g470100a4.el8.noarch 112/162 2024-03-30T00:48:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.e 113/162 2024-03-30T00:48:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1018.g470100a4. 114/162 2024-03-30T00:48:15.432 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 126/162 2024-03-30T00:48:15.640 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 115/162 2024-03-30T00:48:15.721 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-03-30T00:48:15.910 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 115/162 2024-03-30T00:48:16.041 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 128/162 2024-03-30T00:48:16.151 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 116/162 2024-03-30T00:48:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 125/162 2024-03-30T00:48:16.435 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 129/162 2024-03-30T00:48:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 130/162 2024-03-30T00:48:17.000 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libtool-2.4.6-25.el8.x86_64 131/162 2024-03-30T00:48:17.014 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 131/162 2024-03-30T00:48:17.181 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 126/162 2024-03-30T00:48:17.236 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 116/162 2024-03-30T00:48:17.244 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-03-30T00:48:17.392 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 117/162 2024-03-30T00:48:17.462 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-03-30T00:48:17.501 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-03-30T00:48:17.522 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 117/162 2024-03-30T00:48:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_ 134/162 2024-03-30T00:48:17.812 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 128/162 2024-03-30T00:48:17.908 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el 118/162 2024-03-30T00:48:18.102 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 119/162 2024-03-30T00:48:18.172 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 119/162 2024-03-30T00:48:18.189 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 129/162 2024-03-30T00:48:18.310 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 120/162 2024-03-30T00:48:18.414 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 120/162 2024-03-30T00:48:18.557 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 130/162 2024-03-30T00:48:18.810 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libtool-2.4.6-25.el8.x86_64 131/162 2024-03-30T00:48:18.824 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 131/162 2024-03-30T00:48:19.100 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-03-30T00:48:19.324 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-03-30T00:48:19.545 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_ 134/162 2024-03-30T00:48:19.726 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:18.2.2-1018.g470100a4.el8.x86_64 121/162 2024-03-30T00:48:19.952 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T00:48:19.960 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 122/162 2024-03-30T00:48:20.022 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 122/162 2024-03-30T00:48:20.193 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1018.g47010 123/162 2024-03-30T00:48:20.242 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T00:48:20.264 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1018.g47010 123/162 2024-03-30T00:48:20.281 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T00:48:20.476 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 124/162 2024-03-30T00:48:20.701 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-03-30T00:48:20.881 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 124/162 2024-03-30T00:48:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-03-30T00:48:21.083 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 125/162 2024-03-30T00:48:21.303 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-03-30T00:48:21.499 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 139/162 2024-03-30T00:48:21.724 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 140/162 2024-03-30T00:48:21.904 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T00:48:22.011 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 141/162 2024-03-30T00:48:22.238 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T00:48:22.277 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T00:48:22.343 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-03-30T00:48:22.552 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/162 2024-03-30T00:48:22.629 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/162 2024-03-30T00:48:22.773 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-03-30T00:48:22.899 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/162 2024-03-30T00:48:22.913 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/162 2024-03-30T00:48:23.074 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-30T00:48:23.083 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-30T00:48:23.169 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-03-30T00:48:23.226 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 146/162 2024-03-30T00:48:23.410 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-03-30T00:48:23.431 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-03-30T00:48:23.601 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-03-30T00:48:23.660 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 139/162 2024-03-30T00:48:23.819 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-03-30T00:48:23.820 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-30T00:48:23.901 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 140/162 2024-03-30T00:48:24.110 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-30T00:48:24.111 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-30T00:48:24.200 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 141/162 2024-03-30T00:48:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-30T00:48:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-30T00:48:24.531 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-03-30T00:48:24.622 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-30T00:48:24.799 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/162 2024-03-30T00:48:24.867 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-03-30T00:48:24.881 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/162 2024-03-30T00:48:25.068 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-03-30T00:48:25.129 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/162 2024-03-30T00:48:25.144 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/162 2024-03-30T00:48:25.325 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-03-30T00:48:25.325 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-30T00:48:25.345 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-30T00:48:25.356 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-30T00:48:25.528 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 146/162 2024-03-30T00:48:25.568 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-30T00:48:25.568 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-03-30T00:48:25.728 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-03-30T00:48:25.818 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-03-30T00:48:25.818 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-30T00:48:25.935 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 125/162 2024-03-30T00:48:25.944 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-03-30T00:48:26.084 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-30T00:48:26.084 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-30T00:48:26.163 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-03-30T00:48:26.164 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-30T00:48:26.334 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-30T00:48:26.334 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-30T00:48:26.445 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-30T00:48:26.446 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-30T00:48:26.584 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-30T00:48:26.705 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-30T00:48:26.705 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-30T00:48:26.784 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-03-30T00:48:26.784 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-30T00:48:26.994 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-30T00:48:27.194 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-03-30T00:48:27.247 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 126/162 2024-03-30T00:48:27.403 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-03-30T00:48:27.555 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-03-30T00:48:27.680 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:18.2.2-1018.g470100a4.el8.x86_64 1/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 2/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 3/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 4/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1018.g47010 5/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 6/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 7/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 8/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 12/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 13/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_ 14/162 2024-03-30T00:48:27.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 15/162 2024-03-30T00:48:27.682 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 16/162 2024-03-30T00:48:27.682 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:18.2.2-1018.g470100a4.el8.x86_6 17/162 2024-03-30T00:48:27.682 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 18/162 2024-03-30T00:48:27.682 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T00:48:27.682 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1018.g470100a4.el 20/162 2024-03-30T00:48:27.682 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:18.2.2-1018.g470100a4.el8. 21/162 2024-03-30T00:48:27.682 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_6 22/162 2024-03-30T00:48:27.682 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 23/162 2024-03-30T00:48:27.682 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 24/162 2024-03-30T00:48:27.682 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 25/162 2024-03-30T00:48:27.682 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 26/162 2024-03-30T00:48:27.682 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 27/162 2024-03-30T00:48:27.682 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 28/162 2024-03-30T00:48:27.683 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1018.g470100a4. 29/162 2024-03-30T00:48:27.683 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 30/162 2024-03-30T00:48:27.695 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 31/162 2024-03-30T00:48:27.695 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 32/162 2024-03-30T00:48:27.695 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el 33/162 2024-03-30T00:48:27.695 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 34/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.e 35/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 36/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:18.2.2-1018.g470100a4.el8.noarch 37/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 38/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-03-30T00:48:27.696 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-03-30T00:48:27.697 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-03-30T00:48:27.698 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-03-30T00:48:27.698 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-03-30T00:48:27.698 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-03-30T00:48:27.698 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-03-30T00:48:27.698 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-03-30T00:48:27.698 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-03-30T00:48:27.698 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-03-30T00:48:27.698 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 72/162 2024-03-30T00:48:27.698 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/162 2024-03-30T00:48:27.698 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/162 2024-03-30T00:48:27.698 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 75/162 2024-03-30T00:48:27.698 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/162 2024-03-30T00:48:27.698 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 77/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 78/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 79/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 81/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 82/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 83/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 84/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 86/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 87/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 88/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 89/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 90/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 91/162 2024-03-30T00:48:27.699 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 92/162 2024-03-30T00:48:27.700 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 93/162 2024-03-30T00:48:27.700 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 94/162 2024-03-30T00:48:27.700 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 95/162 2024-03-30T00:48:27.700 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 96/162 2024-03-30T00:48:27.700 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 97/162 2024-03-30T00:48:27.700 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 98/162 2024-03-30T00:48:27.700 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 99/162 2024-03-30T00:48:27.700 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 100/162 2024-03-30T00:48:27.700 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 101/162 2024-03-30T00:48:27.700 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 102/162 2024-03-30T00:48:27.700 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 103/162 2024-03-30T00:48:27.700 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 104/162 2024-03-30T00:48:27.700 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : bison-3.0.4-10.el8.x86_64 105/162 2024-03-30T00:48:27.700 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flex-2.6.1-9.el8.x86_64 106/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 107/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 108/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 109/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 110/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 111/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 112/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 113/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 114/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 115/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 116/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 117/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 118/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 119/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 120/162 2024-03-30T00:48:27.701 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : automake-1.16.1-8.el8.noarch 121/162 2024-03-30T00:48:27.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpq-13.11-1.el8.x86_64 122/162 2024-03-30T00:48:27.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : jq-1.6-7.el8.x86_64 123/162 2024-03-30T00:48:27.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 124/162 2024-03-30T00:48:27.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 125/162 2024-03-30T00:48:27.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 126/162 2024-03-30T00:48:27.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 127/162 2024-03-30T00:48:27.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 128/162 2024-03-30T00:48:27.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 129/162 2024-03-30T00:48:27.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 130/162 2024-03-30T00:48:27.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:18.2.2-1018.g470100a4.el8.x86_64 131/162 2024-03-30T00:48:27.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-03-30T00:48:27.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 133/162 2024-03-30T00:48:27.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-03-30T00:48:27.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-03-30T00:48:27.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-03-30T00:48:27.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-03-30T00:48:27.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-03-30T00:48:27.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-03-30T00:48:27.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-03-30T00:48:27.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-03-30T00:48:27.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-03-30T00:48:27.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-03-30T00:48:27.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-03-30T00:48:27.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-03-30T00:48:27.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-03-30T00:48:27.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-03-30T00:48:27.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-03-30T00:48:27.704 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-03-30T00:48:27.704 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-03-30T00:48:27.704 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-03-30T00:48:27.704 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-03-30T00:48:27.704 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-03-30T00:48:27.704 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-03-30T00:48:27.704 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-03-30T00:48:27.704 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-03-30T00:48:27.704 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-03-30T00:48:27.704 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-03-30T00:48:27.704 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-03-30T00:48:27.704 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-03-30T00:48:27.704 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-03-30T00:48:27.712 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-03-30T00:48:27.712 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-30T00:48:27.811 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 128/162 2024-03-30T00:48:27.987 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-30T00:48:27.987 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-03-30T00:48:28.262 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-03-30T00:48:28.262 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-30T00:48:28.305 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 129/162 2024-03-30T00:48:28.537 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-30T00:48:28.537 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-30T00:48:28.660 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 130/162 2024-03-30T00:48:28.803 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-30T00:48:28.803 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-30T00:48:28.891 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtool-2.4.6-25.el8.x86_64 131/162 2024-03-30T00:48:28.922 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 131/162 2024-03-30T00:48:29.061 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-30T00:48:29.113 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-03-30T00:48:29.245 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-03-30T00:48:29.245 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-30T00:48:29.288 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-03-30T00:48:29.525 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_ 134/162 2024-03-30T00:48:30.219 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-30T00:48:30.219 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-2:18.2.2-1018.g470100a4.el8.x86_64 1/162 2024-03-30T00:48:30.219 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 2/162 2024-03-30T00:48:30.219 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 3/162 2024-03-30T00:48:30.219 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 4/162 2024-03-30T00:48:30.219 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1018.g47010 5/162 2024-03-30T00:48:30.219 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 6/162 2024-03-30T00:48:30.219 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 7/162 2024-03-30T00:48:30.219 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 8/162 2024-03-30T00:48:30.219 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T00:48:30.219 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T00:48:30.220 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T00:48:30.220 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 12/162 2024-03-30T00:48:30.220 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 13/162 2024-03-30T00:48:30.220 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_ 14/162 2024-03-30T00:48:30.220 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 15/162 2024-03-30T00:48:30.220 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 16/162 2024-03-30T00:48:30.220 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados-devel-2:18.2.2-1018.g470100a4.el8.x86_6 17/162 2024-03-30T00:48:30.220 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 18/162 2024-03-30T00:48:30.220 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T00:48:30.220 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1018.g470100a4.el 20/162 2024-03-30T00:48:30.220 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-common-2:18.2.2-1018.g470100a4.el8. 21/162 2024-03-30T00:48:30.220 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_6 22/162 2024-03-30T00:48:30.220 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 23/162 2024-03-30T00:48:30.220 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 24/162 2024-03-30T00:48:30.221 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 25/162 2024-03-30T00:48:30.221 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 26/162 2024-03-30T00:48:30.221 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 27/162 2024-03-30T00:48:30.221 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 28/162 2024-03-30T00:48:30.221 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1018.g470100a4. 29/162 2024-03-30T00:48:30.221 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 30/162 2024-03-30T00:48:30.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 31/162 2024-03-30T00:48:30.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 32/162 2024-03-30T00:48:30.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el 33/162 2024-03-30T00:48:30.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 34/162 2024-03-30T00:48:30.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.e 35/162 2024-03-30T00:48:30.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 36/162 2024-03-30T00:48:30.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephadm-2:18.2.2-1018.g470100a4.el8.noarch 37/162 2024-03-30T00:48:30.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 38/162 2024-03-30T00:48:30.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-03-30T00:48:30.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-03-30T00:48:30.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-03-30T00:48:30.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-03-30T00:48:30.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-03-30T00:48:30.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-03-30T00:48:30.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-03-30T00:48:30.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-03-30T00:48:30.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-03-30T00:48:30.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-03-30T00:48:30.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-03-30T00:48:30.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-03-30T00:48:30.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-03-30T00:48:30.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-03-30T00:48:30.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-03-30T00:48:30.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-03-30T00:48:30.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-03-30T00:48:30.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-03-30T00:48:30.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-03-30T00:48:30.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 72/162 2024-03-30T00:48:30.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/162 2024-03-30T00:48:30.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/162 2024-03-30T00:48:30.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 75/162 2024-03-30T00:48:30.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/162 2024-03-30T00:48:30.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 77/162 2024-03-30T00:48:30.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 78/162 2024-03-30T00:48:30.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 79/162 2024-03-30T00:48:30.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/162 2024-03-30T00:48:30.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 81/162 2024-03-30T00:48:30.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 82/162 2024-03-30T00:48:30.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 83/162 2024-03-30T00:48:30.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 84/162 2024-03-30T00:48:30.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/162 2024-03-30T00:48:30.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 86/162 2024-03-30T00:48:30.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 87/162 2024-03-30T00:48:30.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 88/162 2024-03-30T00:48:30.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 89/162 2024-03-30T00:48:30.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 90/162 2024-03-30T00:48:30.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 91/162 2024-03-30T00:48:30.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 92/162 2024-03-30T00:48:30.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 93/162 2024-03-30T00:48:30.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 94/162 2024-03-30T00:48:30.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 95/162 2024-03-30T00:48:30.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 96/162 2024-03-30T00:48:30.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 97/162 2024-03-30T00:48:30.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 98/162 2024-03-30T00:48:30.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 99/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 100/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 101/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 102/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 103/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 104/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : bison-3.0.4-10.el8.x86_64 105/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flex-2.6.1-9.el8.x86_64 106/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 107/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 108/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 109/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 110/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 111/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 112/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 113/162 2024-03-30T00:48:30.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 114/162 2024-03-30T00:48:30.240 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 115/162 2024-03-30T00:48:30.240 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 116/162 2024-03-30T00:48:30.240 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 117/162 2024-03-30T00:48:30.240 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 118/162 2024-03-30T00:48:30.240 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 119/162 2024-03-30T00:48:30.240 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 120/162 2024-03-30T00:48:30.240 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : automake-1.16.1-8.el8.noarch 121/162 2024-03-30T00:48:30.240 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libpq-13.11-1.el8.x86_64 122/162 2024-03-30T00:48:30.240 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : jq-1.6-7.el8.x86_64 123/162 2024-03-30T00:48:30.240 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 124/162 2024-03-30T00:48:30.240 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 125/162 2024-03-30T00:48:30.240 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 126/162 2024-03-30T00:48:30.240 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 127/162 2024-03-30T00:48:30.240 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 128/162 2024-03-30T00:48:30.241 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 129/162 2024-03-30T00:48:30.241 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 130/162 2024-03-30T00:48:30.241 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:18.2.2-1018.g470100a4.el8.x86_64 131/162 2024-03-30T00:48:30.241 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-03-30T00:48:30.241 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 133/162 2024-03-30T00:48:30.241 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-03-30T00:48:30.241 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-03-30T00:48:30.241 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-03-30T00:48:30.241 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-03-30T00:48:30.241 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-03-30T00:48:30.241 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-03-30T00:48:30.241 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-03-30T00:48:30.241 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-03-30T00:48:30.241 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-03-30T00:48:30.242 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-03-30T00:48:30.242 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-03-30T00:48:30.242 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-03-30T00:48:30.242 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-03-30T00:48:30.242 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-03-30T00:48:30.242 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-03-30T00:48:30.242 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-03-30T00:48:30.242 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-03-30T00:48:30.242 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-03-30T00:48:30.242 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-03-30T00:48:30.242 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-03-30T00:48:30.242 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-03-30T00:48:30.242 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-03-30T00:48:30.242 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-03-30T00:48:30.243 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-03-30T00:48:30.243 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-03-30T00:48:30.243 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-03-30T00:48:30.243 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-03-30T00:48:30.243 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-03-30T00:48:32.893 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T00:48:33.200 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T00:48:33.269 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T00:48:33.710 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-03-30T00:48:34.039 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-03-30T00:48:34.329 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-03-30T00:48:34.558 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 139/162 2024-03-30T00:48:34.784 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 140/162 2024-03-30T00:48:35.095 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 141/162 2024-03-30T00:48:35.434 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-03-30T00:48:35.673 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/162 2024-03-30T00:48:35.789 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/162 2024-03-30T00:48:36.023 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/162 2024-03-30T00:48:36.054 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/162 2024-03-30T00:48:36.207 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-30T00:48:36.232 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-30T00:48:36.368 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 146/162 2024-03-30T00:48:36.569 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-03-30T00:48:36.745 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-03-30T00:48:36.944 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-03-30T00:48:36.944 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-30T00:48:37.219 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-30T00:48:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-30T00:48:37.496 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-30T00:48:37.496 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-30T00:48:37.768 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-30T00:48:38.000 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-03-30T00:48:38.164 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-03-30T00:48:38.408 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-03-30T00:48:38.408 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-30T00:48:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-30T00:48:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-03-30T00:48:38.926 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-03-30T00:48:38.927 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-30T00:48:39.159 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-30T00:48:39.160 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-30T00:48:39.409 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-30T00:48:39.409 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-30T00:48:39.651 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-30T00:48:39.843 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-03-30T00:48:39.843 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-30T00:48:40.437 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-30T00:48:40.437 INFO:teuthology.orchestra.run.smithi116.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-30T00:48:40.438 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:48:41.115 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:18.2.2-1018.g470100a4.el8.x86_64 1/162 2024-03-30T00:48:41.115 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 2/162 2024-03-30T00:48:41.115 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 3/162 2024-03-30T00:48:41.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 4/162 2024-03-30T00:48:41.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1018.g47010 5/162 2024-03-30T00:48:41.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 6/162 2024-03-30T00:48:41.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 7/162 2024-03-30T00:48:41.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 8/162 2024-03-30T00:48:41.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T00:48:41.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T00:48:41.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T00:48:41.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 12/162 2024-03-30T00:48:41.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 13/162 2024-03-30T00:48:41.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_ 14/162 2024-03-30T00:48:41.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 15/162 2024-03-30T00:48:41.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 16/162 2024-03-30T00:48:41.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:18.2.2-1018.g470100a4.el8.x86_6 17/162 2024-03-30T00:48:41.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 18/162 2024-03-30T00:48:41.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T00:48:41.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1018.g470100a4.el 20/162 2024-03-30T00:48:41.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:18.2.2-1018.g470100a4.el8. 21/162 2024-03-30T00:48:41.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_6 22/162 2024-03-30T00:48:41.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 23/162 2024-03-30T00:48:41.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 24/162 2024-03-30T00:48:41.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 25/162 2024-03-30T00:48:41.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 26/162 2024-03-30T00:48:41.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 27/162 2024-03-30T00:48:41.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 28/162 2024-03-30T00:48:41.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1018.g470100a4. 29/162 2024-03-30T00:48:41.130 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 30/162 2024-03-30T00:48:41.130 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 31/162 2024-03-30T00:48:41.130 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 32/162 2024-03-30T00:48:41.131 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el 33/162 2024-03-30T00:48:41.131 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 34/162 2024-03-30T00:48:41.131 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.e 35/162 2024-03-30T00:48:41.131 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 36/162 2024-03-30T00:48:41.131 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:18.2.2-1018.g470100a4.el8.noarch 37/162 2024-03-30T00:48:41.131 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 38/162 2024-03-30T00:48:41.131 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-03-30T00:48:41.131 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-03-30T00:48:41.131 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-03-30T00:48:41.131 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-03-30T00:48:41.131 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-03-30T00:48:41.132 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 72/162 2024-03-30T00:48:41.133 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/162 2024-03-30T00:48:41.134 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/162 2024-03-30T00:48:41.134 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 75/162 2024-03-30T00:48:41.134 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/162 2024-03-30T00:48:41.134 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 77/162 2024-03-30T00:48:41.134 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 78/162 2024-03-30T00:48:41.134 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 79/162 2024-03-30T00:48:41.134 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/162 2024-03-30T00:48:41.134 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 81/162 2024-03-30T00:48:41.134 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 82/162 2024-03-30T00:48:41.134 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 83/162 2024-03-30T00:48:41.134 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 84/162 2024-03-30T00:48:41.134 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/162 2024-03-30T00:48:41.134 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 86/162 2024-03-30T00:48:41.134 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 87/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 88/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 89/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 90/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 91/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 92/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 93/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 94/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 95/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 96/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 97/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 98/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 99/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 100/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 101/162 2024-03-30T00:48:41.135 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 102/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 103/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 104/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : bison-3.0.4-10.el8.x86_64 105/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flex-2.6.1-9.el8.x86_64 106/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 107/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 108/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 109/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 110/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 111/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 112/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 113/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 114/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 115/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 116/162 2024-03-30T00:48:41.136 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 117/162 2024-03-30T00:48:41.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 118/162 2024-03-30T00:48:41.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 119/162 2024-03-30T00:48:41.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 120/162 2024-03-30T00:48:41.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : automake-1.16.1-8.el8.noarch 121/162 2024-03-30T00:48:41.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpq-13.11-1.el8.x86_64 122/162 2024-03-30T00:48:41.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jq-1.6-7.el8.x86_64 123/162 2024-03-30T00:48:41.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 124/162 2024-03-30T00:48:41.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 125/162 2024-03-30T00:48:41.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 126/162 2024-03-30T00:48:41.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 127/162 2024-03-30T00:48:41.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 128/162 2024-03-30T00:48:41.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 129/162 2024-03-30T00:48:41.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 130/162 2024-03-30T00:48:41.137 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:18.2.2-1018.g470100a4.el8.x86_64 131/162 2024-03-30T00:48:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-03-30T00:48:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 133/162 2024-03-30T00:48:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-03-30T00:48:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-03-30T00:48:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-03-30T00:48:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-03-30T00:48:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-03-30T00:48:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-03-30T00:48:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-03-30T00:48:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-03-30T00:48:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-03-30T00:48:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-03-30T00:48:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-03-30T00:48:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-03-30T00:48:41.139 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-03-30T00:48:41.139 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-03-30T00:48:41.139 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-03-30T00:48:41.139 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-03-30T00:48:41.139 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-03-30T00:48:41.139 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-03-30T00:48:41.139 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-03-30T00:48:41.139 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-03-30T00:48:41.139 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-03-30T00:48:41.139 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-03-30T00:48:41.139 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-03-30T00:48:41.139 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-03-30T00:48:41.139 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-03-30T00:48:41.139 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-03-30T00:48:41.140 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-03-30T00:48:41.140 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-03-30T00:48:41.331 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-03-30T00:48:41.331 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2024-03-30T00:48:42.790 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:48:42.790 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-03-30T00:48:42.790 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-03-30T00:48:42.790 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-03-30T00:48:42.790 INFO:teuthology.orchestra.run.smithi181.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-03-30T00:48:42.790 INFO:teuthology.orchestra.run.smithi181.stdout: libblkid-2.32.1-43.el8.x86_64 2024-03-30T00:48:42.790 INFO:teuthology.orchestra.run.smithi181.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-03-30T00:48:42.790 INFO:teuthology.orchestra.run.smithi181.stdout: libmount-2.32.1-43.el8.x86_64 2024-03-30T00:48:42.790 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-03-30T00:48:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: libuuid-2.32.1-43.el8.x86_64 2024-03-30T00:48:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-03-30T00:48:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-03-30T00:48:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: util-linux-2.32.1-43.el8.x86_64 2024-03-30T00:48:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-03-30T00:48:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-03-30T00:48:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-30T00:48:42.791 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-30T00:48:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-03-30T00:48:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: automake-1.16.1-8.el8.noarch 2024-03-30T00:48:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: bison-3.0.4-10.el8.x86_64 2024-03-30T00:48:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.792 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.792 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.792 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.792 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:42.792 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.792 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.792 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.792 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:42.792 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:42.792 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:42.792 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:42.792 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:42.792 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.792 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.793 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:42.793 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.793 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.793 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.793 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:42.793 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:42.793 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.793 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:42.793 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-03-30T00:48:42.793 INFO:teuthology.orchestra.run.smithi181.stdout: flex-2.6.1-9.el8.x86_64 2024-03-30T00:48:42.793 INFO:teuthology.orchestra.run.smithi181.stdout: fuse-2.9.7-17.el8.x86_64 2024-03-30T00:48:42.793 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-30T00:48:42.793 INFO:teuthology.orchestra.run.smithi181.stdout: jq-1.6-7.el8.x86_64 2024-03-30T00:48:42.794 INFO:teuthology.orchestra.run.smithi181.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-30T00:48:42.794 INFO:teuthology.orchestra.run.smithi181.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-03-30T00:48:42.794 INFO:teuthology.orchestra.run.smithi181.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-30T00:48:42.794 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.794 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.794 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.794 INFO:teuthology.orchestra.run.smithi181.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-03-30T00:48:42.794 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-03-30T00:48:42.794 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-30T00:48:42.794 INFO:teuthology.orchestra.run.smithi181.stdout: libpq-13.11-1.el8.x86_64 2024-03-30T00:48:42.794 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-03-30T00:48:42.794 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-03-30T00:48:42.794 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.795 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.795 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-03-30T00:48:42.795 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.795 INFO:teuthology.orchestra.run.smithi181.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-03-30T00:48:42.795 INFO:teuthology.orchestra.run.smithi181.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-30T00:48:42.795 INFO:teuthology.orchestra.run.smithi181.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-30T00:48:42.795 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-30T00:48:42.795 INFO:teuthology.orchestra.run.smithi181.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-03-30T00:48:42.795 INFO:teuthology.orchestra.run.smithi181.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-03-30T00:48:42.795 INFO:teuthology.orchestra.run.smithi181.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-30T00:48:42.795 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-30T00:48:42.795 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-03-30T00:48:42.795 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.15-6.el8.x86_64 2024-03-30T00:48:42.795 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-03-30T00:48:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-03-30T00:48:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-03-30T00:48:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-03-30T00:48:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-03-30T00:48:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T00:48:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T00:48:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T00:48:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-30T00:48:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-30T00:48:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-30T00:48:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-30T00:48:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-30T00:48:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.797 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.797 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-30T00:48:42.797 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-30T00:48:42.797 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-30T00:48:42.797 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-30T00:48:42.797 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-03-30T00:48:42.797 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-30T00:48:42.797 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-30T00:48:42.797 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-30T00:48:42.797 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-03-30T00:48:42.797 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-30T00:48:42.797 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-30T00:48:42.797 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-30T00:48:42.797 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-30T00:48:42.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-30T00:48:42.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-30T00:48:42.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-30T00:48:42.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-30T00:48:42.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-30T00:48:42.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-30T00:48:42.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-30T00:48:42.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-30T00:48:42.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-30T00:48:42.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-30T00:48:42.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-2.6-1.el8.noarch 2024-03-30T00:48:42.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-30T00:48:42.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-30T00:48:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-30T00:48:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-30T00:48:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-30T00:48:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-30T00:48:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-30T00:48:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-30T00:48:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-30T00:48:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-30T00:48:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-03-30T00:48:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-30T00:48:42.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-30T00:48:42.800 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-30T00:48:42.801 INFO:teuthology.orchestra.run.smithi181.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-03-30T00:48:42.801 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:48:42.801 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-30T00:48:43.657 DEBUG:teuthology.parallel:result is None 2024-03-30T00:48:44.968 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-03-30T00:48:44.968 INFO:teuthology.orchestra.run.smithi130.stdout:Installed products updated. 2024-03-30T00:48:45.612 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-30T00:48:45.612 INFO:teuthology.orchestra.run.smithi130.stdout:Upgraded: 2024-03-30T00:48:45.613 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-03-30T00:48:45.613 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-03-30T00:48:45.613 INFO:teuthology.orchestra.run.smithi130.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-03-30T00:48:45.613 INFO:teuthology.orchestra.run.smithi130.stdout: libblkid-2.32.1-43.el8.x86_64 2024-03-30T00:48:45.613 INFO:teuthology.orchestra.run.smithi130.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-03-30T00:48:45.613 INFO:teuthology.orchestra.run.smithi130.stdout: libmount-2.32.1-43.el8.x86_64 2024-03-30T00:48:45.613 INFO:teuthology.orchestra.run.smithi130.stdout: librados2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.613 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.613 INFO:teuthology.orchestra.run.smithi130.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-03-30T00:48:45.613 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-2.32.1-43.el8.x86_64 2024-03-30T00:48:45.613 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-03-30T00:48:45.613 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-03-30T00:48:45.613 INFO:teuthology.orchestra.run.smithi130.stdout: util-linux-2.32.1-43.el8.x86_64 2024-03-30T00:48:45.614 INFO:teuthology.orchestra.run.smithi130.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-03-30T00:48:45.614 INFO:teuthology.orchestra.run.smithi130.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-03-30T00:48:45.614 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-30T00:48:45.614 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-30T00:48:45.614 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-03-30T00:48:45.614 INFO:teuthology.orchestra.run.smithi130.stdout: automake-1.16.1-8.el8.noarch 2024-03-30T00:48:45.614 INFO:teuthology.orchestra.run.smithi130.stdout: bison-3.0.4-10.el8.x86_64 2024-03-30T00:48:45.614 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.614 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.614 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.614 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.614 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:45.614 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.615 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.615 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.615 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:45.615 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:45.615 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:45.615 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:45.615 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:45.615 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.615 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.615 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:45.615 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.615 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.615 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.615 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:45.616 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:45.616 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.616 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:45.616 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-03-30T00:48:45.616 INFO:teuthology.orchestra.run.smithi130.stdout: flex-2.6.1-9.el8.x86_64 2024-03-30T00:48:45.616 INFO:teuthology.orchestra.run.smithi130.stdout: fuse-2.9.7-17.el8.x86_64 2024-03-30T00:48:45.616 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-30T00:48:45.616 INFO:teuthology.orchestra.run.smithi130.stdout: jq-1.6-7.el8.x86_64 2024-03-30T00:48:45.616 INFO:teuthology.orchestra.run.smithi130.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-30T00:48:45.616 INFO:teuthology.orchestra.run.smithi130.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-03-30T00:48:45.616 INFO:teuthology.orchestra.run.smithi130.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-30T00:48:45.616 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.616 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.616 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: libpq-13.11-1.el8.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-30T00:48:45.617 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-03-30T00:48:45.618 INFO:teuthology.orchestra.run.smithi130.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-03-30T00:48:45.618 INFO:teuthology.orchestra.run.smithi130.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-30T00:48:45.618 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-30T00:48:45.618 INFO:teuthology.orchestra.run.smithi130.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-03-30T00:48:45.618 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-0.3.15-6.el8.x86_64 2024-03-30T00:48:45.618 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-03-30T00:48:45.618 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-03-30T00:48:45.618 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-03-30T00:48:45.618 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-03-30T00:48:45.618 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-03-30T00:48:45.618 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T00:48:45.618 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T00:48:45.618 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T00:48:45.618 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-30T00:48:45.619 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-30T00:48:45.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-03-30T00:48:45.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-30T00:48:45.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-30T00:48:45.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-30T00:48:45.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-30T00:48:45.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-30T00:48:45.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-30T00:48:45.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-30T00:48:45.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-30T00:48:45.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-30T00:48:45.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-30T00:48:45.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-30T00:48:45.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-30T00:48:45.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend-2.6-1.el8.noarch 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-30T00:48:45.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-30T00:48:45.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-03-30T00:48:45.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-30T00:48:45.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-30T00:48:45.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-30T00:48:45.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-30T00:48:45.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-30T00:48:45.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-30T00:48:45.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-30T00:48:45.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-30T00:48:45.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-30T00:48:45.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-30T00:48:45.622 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.622 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.622 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:45.623 INFO:teuthology.orchestra.run.smithi130.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-30T00:48:45.623 INFO:teuthology.orchestra.run.smithi130.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-30T00:48:45.623 INFO:teuthology.orchestra.run.smithi130.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-30T00:48:45.623 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-30T00:48:45.623 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-30T00:48:45.623 INFO:teuthology.orchestra.run.smithi130.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-03-30T00:48:45.623 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-30T00:48:45.623 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-30T00:48:45.874 DEBUG:teuthology.parallel:result is None 2024-03-30T00:48:52.016 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-03-30T00:48:52.017 INFO:teuthology.orchestra.run.smithi116.stdout:Installed products updated. 2024-03-30T00:48:52.717 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:48:52.717 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-03-30T00:48:52.717 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-03-30T00:48:52.717 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-03-30T00:48:52.717 INFO:teuthology.orchestra.run.smithi116.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-03-30T00:48:52.717 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid-2.32.1-43.el8.x86_64 2024-03-30T00:48:52.717 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-03-30T00:48:52.718 INFO:teuthology.orchestra.run.smithi116.stdout: libmount-2.32.1-43.el8.x86_64 2024-03-30T00:48:52.718 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.718 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.718 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-03-30T00:48:52.718 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-2.32.1-43.el8.x86_64 2024-03-30T00:48:52.718 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-03-30T00:48:52.718 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-03-30T00:48:52.718 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-2.32.1-43.el8.x86_64 2024-03-30T00:48:52.718 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-03-30T00:48:52.718 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-03-30T00:48:52.718 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-30T00:48:52.718 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-30T00:48:52.718 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-03-30T00:48:52.718 INFO:teuthology.orchestra.run.smithi116.stdout: automake-1.16.1-8.el8.noarch 2024-03-30T00:48:52.719 INFO:teuthology.orchestra.run.smithi116.stdout: bison-3.0.4-10.el8.x86_64 2024-03-30T00:48:52.719 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.719 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.719 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.719 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.719 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:52.719 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.719 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.719 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.719 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:52.719 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:52.719 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:52.719 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:52.720 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:52.720 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.720 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.720 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:52.720 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.720 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.720 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.720 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:52.720 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:52.720 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.720 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T00:48:52.720 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-03-30T00:48:52.721 INFO:teuthology.orchestra.run.smithi116.stdout: flex-2.6.1-9.el8.x86_64 2024-03-30T00:48:52.721 INFO:teuthology.orchestra.run.smithi116.stdout: fuse-2.9.7-17.el8.x86_64 2024-03-30T00:48:52.721 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-30T00:48:52.721 INFO:teuthology.orchestra.run.smithi116.stdout: jq-1.6-7.el8.x86_64 2024-03-30T00:48:52.721 INFO:teuthology.orchestra.run.smithi116.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-30T00:48:52.721 INFO:teuthology.orchestra.run.smithi116.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-03-30T00:48:52.721 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-30T00:48:52.721 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.721 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.721 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.721 INFO:teuthology.orchestra.run.smithi116.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-03-30T00:48:52.721 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-03-30T00:48:52.722 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-30T00:48:52.722 INFO:teuthology.orchestra.run.smithi116.stdout: libpq-13.11-1.el8.x86_64 2024-03-30T00:48:52.722 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-03-30T00:48:52.722 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-03-30T00:48:52.722 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.722 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.722 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-03-30T00:48:52.722 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.722 INFO:teuthology.orchestra.run.smithi116.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-03-30T00:48:52.722 INFO:teuthology.orchestra.run.smithi116.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-30T00:48:52.722 INFO:teuthology.orchestra.run.smithi116.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-30T00:48:52.722 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-30T00:48:52.722 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-03-30T00:48:52.722 INFO:teuthology.orchestra.run.smithi116.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.15-6.el8.x86_64 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-30T00:48:52.723 INFO:teuthology.orchestra.run.smithi116.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-30T00:48:52.724 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-30T00:48:52.725 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-30T00:48:52.725 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-30T00:48:52.725 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-30T00:48:52.725 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-30T00:48:52.725 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-30T00:48:52.725 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-30T00:48:52.725 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-30T00:48:52.725 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-30T00:48:52.725 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-30T00:48:52.725 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-30T00:48:52.725 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-30T00:48:52.725 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-30T00:48:52.725 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-2.6-1.el8.noarch 2024-03-30T00:48:52.725 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-30T00:48:52.726 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-30T00:48:52.726 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-30T00:48:52.726 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-30T00:48:52.726 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-30T00:48:52.726 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-30T00:48:52.726 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.726 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.726 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-30T00:48:52.726 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-30T00:48:52.726 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.726 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-30T00:48:52.726 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-30T00:48:52.726 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-03-30T00:48:52.726 INFO:teuthology.orchestra.run.smithi116.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-30T00:48:52.727 INFO:teuthology.orchestra.run.smithi116.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-30T00:48:52.728 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-30T00:48:52.728 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-30T00:48:52.728 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-03-30T00:48:52.728 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:48:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-30T00:48:53.096 DEBUG:teuthology.parallel:result is None 2024-03-30T00:48:53.097 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T00:48:53.244 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T00:48:53.295 INFO:teuthology.orchestra.run.smithi116.stdout:18.2.2-1018.g470100a4.el8 2024-03-30T00:48:53.296 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1018.g470100a4.el8 2024-03-30T00:48:53.296 INFO:teuthology.task.install:The correct ceph version 18.2.2-1018.g470100a4 is installed. 2024-03-30T00:48:53.298 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T00:48:53.438 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T00:48:53.467 INFO:teuthology.orchestra.run.smithi130.stdout:18.2.2-1018.g470100a4.el8 2024-03-30T00:48:53.467 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1018.g470100a4.el8 2024-03-30T00:48:53.468 INFO:teuthology.task.install:The correct ceph version 18.2.2-1018.g470100a4 is installed. 2024-03-30T00:48:53.469 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T00:48:53.607 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T00:48:53.636 INFO:teuthology.orchestra.run.smithi181.stdout:18.2.2-1018.g470100a4.el8 2024-03-30T00:48:53.636 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1018.g470100a4.el8 2024-03-30T00:48:53.636 INFO:teuthology.task.install:The correct ceph version 18.2.2-1018.g470100a4 is installed. 2024-03-30T00:48:53.638 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-30T00:48:53.638 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:48:53.639 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T00:48:53.727 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-30T00:48:53.727 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T00:48:53.777 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-30T00:48:53.777 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T00:48:53.826 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-30T00:48:53.827 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:48:53.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T00:48:53.892 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T00:48:53.997 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-30T00:48:53.997 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T00:48:54.031 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T00:48:54.100 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-30T00:48:54.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T00:48:54.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T00:48:54.204 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-30T00:48:54.204 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:48:54.204 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T00:48:54.270 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T00:48:54.374 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-30T00:48:54.374 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T00:48:54.406 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T00:48:54.477 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-30T00:48:54.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T00:48:54.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T00:48:54.581 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-30T00:48:54.581 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:48:54.581 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T00:48:54.645 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T00:48:54.713 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-30T00:48:54.714 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T00:48:54.746 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T00:48:54.818 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-30T00:48:54.819 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T00:48:54.852 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T00:48:54.925 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-30T00:48:55.131 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-30T00:48:55.132 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T00:48:55.132 INFO:tasks.cephadm:Cluster fsid is 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:48:55.132 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-30T00:48:55.132 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.116', 'mon.b': '172.21.15.130', 'mon.c': '172.21.15.181'} 2024-03-30T00:48:55.133 INFO:tasks.cephadm:First mon is mon.a on smithi116 2024-03-30T00:48:55.133 INFO:tasks.cephadm:First mgr is x 2024-03-30T00:48:55.133 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-30T00:48:55.133 DEBUG:teuthology.orchestra.run.smithi116:> sudo hostname $(hostname -s) 2024-03-30T00:48:55.199 DEBUG:teuthology.orchestra.run.smithi130:> sudo hostname $(hostname -s) 2024-03-30T00:48:55.232 DEBUG:teuthology.orchestra.run.smithi181:> sudo hostname $(hostname -s) 2024-03-30T00:48:55.265 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-30T00:48:55.265 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T00:48:55.482 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/78078/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.2.2-1018-g470100a4', 'node_name': '172.21.2.5+braggi05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.2.2-1018.g470100a4'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-03-29 13:05:59.448245', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240327.051042-reef', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-30T00:48:55.621 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-rishabh-testing-20240327.051042-reef, sha1 470100a4ccbcf7769052bf09a5413881ccf1b97f from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T00:48:55.622 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/x86_64/flavors/default/cephadm 2024-03-30T00:48:55.622 DEBUG:teuthology.orchestra.run.smithi116:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-30T00:48:56.245 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-rw-r--. 1 ubuntu ubuntu 703022 Mar 30 00:48 /home/ubuntu/cephtest/cephadm 2024-03-30T00:48:56.245 DEBUG:teuthology.orchestra.run.smithi130:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-30T00:48:56.806 INFO:teuthology.orchestra.run.smithi130.stdout:-rw-rw-r--. 1 ubuntu ubuntu 703022 Mar 30 00:48 /home/ubuntu/cephtest/cephadm 2024-03-30T00:48:56.807 DEBUG:teuthology.orchestra.run.smithi181:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-30T00:48:57.222 INFO:teuthology.orchestra.run.smithi181.stdout:-rw-rw-r--. 1 ubuntu ubuntu 703022 Mar 30 00:48 /home/ubuntu/cephtest/cephadm 2024-03-30T00:48:57.223 DEBUG:teuthology.orchestra.run.smithi116:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-30T00:48:57.273 DEBUG:teuthology.orchestra.run.smithi130:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-30T00:48:57.294 DEBUG:teuthology.orchestra.run.smithi181:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-30T00:48:57.331 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f on all hosts... 2024-03-30T00:48:57.331 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f pull 2024-03-30T00:48:57.335 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f pull 2024-03-30T00:48:57.337 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f pull 2024-03-30T00:48:58.268 INFO:teuthology.orchestra.run.smithi181.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f... 2024-03-30T00:48:58.653 INFO:teuthology.orchestra.run.smithi130.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f... 2024-03-30T00:48:59.810 INFO:teuthology.orchestra.run.smithi116.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f... 2024-03-30T00:49:23.948 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-03-30T00:49:23.949 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)", 2024-03-30T00:49:23.949 INFO:teuthology.orchestra.run.smithi130.stdout: "image_id": "e7c444a08cf749270ac19a49d6d49787223f88c26222d2dfc1d77b830f40791d", 2024-03-30T00:49:23.949 INFO:teuthology.orchestra.run.smithi130.stdout: "repo_digests": [ 2024-03-30T00:49:23.949 INFO:teuthology.orchestra.run.smithi130.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35" 2024-03-30T00:49:23.949 INFO:teuthology.orchestra.run.smithi130.stdout: ] 2024-03-30T00:49:23.949 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-03-30T00:49:25.612 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-30T00:49:25.612 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_version": "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)", 2024-03-30T00:49:25.612 INFO:teuthology.orchestra.run.smithi181.stdout: "image_id": "e7c444a08cf749270ac19a49d6d49787223f88c26222d2dfc1d77b830f40791d", 2024-03-30T00:49:25.612 INFO:teuthology.orchestra.run.smithi181.stdout: "repo_digests": [ 2024-03-30T00:49:25.612 INFO:teuthology.orchestra.run.smithi181.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35" 2024-03-30T00:49:25.613 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-30T00:49:25.613 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-30T00:49:30.818 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:49:30.818 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)", 2024-03-30T00:49:30.818 INFO:teuthology.orchestra.run.smithi116.stdout: "image_id": "e7c444a08cf749270ac19a49d6d49787223f88c26222d2dfc1d77b830f40791d", 2024-03-30T00:49:30.818 INFO:teuthology.orchestra.run.smithi116.stdout: "repo_digests": [ 2024-03-30T00:49:30.819 INFO:teuthology.orchestra.run.smithi116.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35" 2024-03-30T00:49:30.819 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-30T00:49:30.819 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:49:30.846 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph 2024-03-30T00:49:30.915 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /etc/ceph 2024-03-30T00:49:30.953 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph 2024-03-30T00:49:30.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /etc/ceph 2024-03-30T00:49:31.055 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 777 /etc/ceph 2024-03-30T00:49:31.088 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /etc/ceph 2024-03-30T00:49:31.120 INFO:tasks.cephadm:Writing seed config... 2024-03-30T00:49:31.121 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-30T00:49:31.121 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-30T00:49:31.121 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-30T00:49:31.121 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-30T00:49:31.121 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-30T00:49:31.121 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-30T00:49:31.121 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-30T00:49:31.121 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-30T00:49:31.121 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-30T00:49:31.121 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-30T00:49:31.122 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-30T00:49:31.122 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-30T00:49:31.122 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-30T00:49:31.122 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-30T00:49:31.122 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-30T00:49:31.122 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-30T00:49:31.122 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-30T00:49:31.122 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-30T00:49:31.122 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-30T00:49:31.122 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-30T00:49:31.122 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-30T00:49:31.122 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-30T00:49:31.122 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-30T00:49:31.122 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-30T00:49:31.123 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-30T00:49:31.123 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-30T00:49:31.123 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-30T00:49:31.123 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-30T00:49:31.123 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-30T00:49:31.123 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-30T00:49:31.123 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-30T00:49:31.123 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-30T00:49:31.123 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-30T00:49:31.123 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-30T00:49:31.123 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-30T00:49:31.123 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-30T00:49:31.124 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-30T00:49:31.124 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-30T00:49:31.124 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-30T00:49:31.124 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-30T00:49:31.124 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-30T00:49:31.124 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-30T00:49:31.124 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-30T00:49:31.124 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-30T00:49:31.125 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:49:31.125 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-30T00:49:31.165 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 4909f004-ee2f-11ee-b647-cb9ed24678a4 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-30T00:49:31.166 DEBUG:teuthology.orchestra.run.smithi116:mon.a> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mon.a.service 2024-03-30T00:49:31.210 DEBUG:teuthology.orchestra.run.smithi116:mgr.x> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.x.service 2024-03-30T00:49:31.254 INFO:tasks.cephadm:Bootstrapping... 2024-03-30T00:49:31.254 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f -v bootstrap --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.116 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-30T00:49:31.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:-- Logs begin at Sat 2024-03-30 00:33:55 UTC. -- 2024-03-30T00:49:31.317 INFO:journalctl@ceph.mgr.x.smithi116.stdout:-- Logs begin at Sat 2024-03-30 00:33:55 UTC. -- 2024-03-30T00:49:31.466 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-30T00:49:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f', '-v', 'bootstrap', '--fsid', '4909f004-ee2f-11ee-b647-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.116', '--skip-admin-label'] 2024-03-30T00:49:31.512 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.6.1 2024-03-30T00:49:31.512 INFO:teuthology.orchestra.run.smithi116.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-30T00:49:31.512 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying podman|docker is present... 2024-03-30T00:49:31.555 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.6.1 2024-03-30T00:49:31.555 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying lvm2 is present... 2024-03-30T00:49:31.555 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying time synchronization is in place... 2024-03-30T00:49:31.568 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-30T00:49:31.569 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-30T00:49:31.580 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-30T00:49:31.580 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-30T00:49:31.593 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout enabled 2024-03-30T00:49:31.605 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout active 2024-03-30T00:49:31.605 INFO:teuthology.orchestra.run.smithi116.stdout:Unit chronyd.service is enabled and running 2024-03-30T00:49:31.605 INFO:teuthology.orchestra.run.smithi116.stdout:Repeating the final host check... 2024-03-30T00:49:31.648 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.6.1 2024-03-30T00:49:31.648 INFO:teuthology.orchestra.run.smithi116.stdout:podman (/bin/podman) version 4.6.1 is present 2024-03-30T00:49:31.648 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl is present 2024-03-30T00:49:31.648 INFO:teuthology.orchestra.run.smithi116.stdout:lvcreate is present 2024-03-30T00:49:31.660 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-30T00:49:31.660 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-30T00:49:31.672 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-30T00:49:31.672 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-30T00:49:31.685 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout enabled 2024-03-30T00:49:31.697 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout active 2024-03-30T00:49:31.697 INFO:teuthology.orchestra.run.smithi116.stdout:Unit chronyd.service is enabled and running 2024-03-30T00:49:31.697 INFO:teuthology.orchestra.run.smithi116.stdout:Host looks OK 2024-03-30T00:49:31.697 INFO:teuthology.orchestra.run.smithi116.stdout:Cluster fsid: 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:49:31.697 INFO:teuthology.orchestra.run.smithi116.stdout:Acquiring lock 140430616144360 on /run/cephadm/4909f004-ee2f-11ee-b647-cb9ed24678a4.lock 2024-03-30T00:49:31.698 INFO:teuthology.orchestra.run.smithi116.stdout:Lock 140430616144360 acquired on /run/cephadm/4909f004-ee2f-11ee-b647-cb9ed24678a4.lock 2024-03-30T00:49:31.698 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying IP 172.21.15.116 port 3300 ... 2024-03-30T00:49:31.699 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying IP 172.21.15.116 port 6789 ... 2024-03-30T00:49:31.699 INFO:teuthology.orchestra.run.smithi116.stdout:Base mon IP(s) is [172.21.15.116:3300, 172.21.15.116:6789], mon addrv is [v2:172.21.15.116:3300,v1:172.21.15.116:6789] 2024-03-30T00:49:31.706 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2024-03-30T00:49:31.707 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.116 metric 100 2024-03-30T00:49:31.713 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-30T00:49:31.713 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-30T00:49:31.714 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1787sec hoplimit 64 pref medium 2024-03-30T00:49:31.720 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-30T00:49:31.720 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-30T00:49:31.720 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-30T00:49:31.720 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-30T00:49:31.720 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-03-30T00:49:31.720 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-30T00:49:31.721 INFO:teuthology.orchestra.run.smithi116.stdout:Mon IP `172.21.15.116` is in CIDR network `172.21.0.0/20` 2024-03-30T00:49:31.721 INFO:teuthology.orchestra.run.smithi116.stdout:Mon IP `172.21.15.116` is in CIDR network `172.21.0.0/20` 2024-03-30T00:49:31.721 INFO:teuthology.orchestra.run.smithi116.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-30T00:49:31.722 INFO:teuthology.orchestra.run.smithi116.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-30T00:49:31.723 INFO:teuthology.orchestra.run.smithi116.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f... 2024-03-30T00:49:32.119 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout e7c444a08cf749270ac19a49d6d49787223f88c26222d2dfc1d77b830f40791d 2024-03-30T00:49:32.120 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f... 2024-03-30T00:49:32.120 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Getting image source signatures 2024-03-30T00:49:32.120 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying blob sha256:86ca209f044bd2e3705adc9da4fc1145309402ec2e51b9b9b051617a85f31417 2024-03-30T00:49:32.120 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying blob sha256:11da15500eca1f826c328eb3265caa30b0a3730510e2a32023084d6fe9cee20c 2024-03-30T00:49:32.120 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying config sha256:e7c444a08cf749270ac19a49d6d49787223f88c26222d2dfc1d77b830f40791d 2024-03-30T00:49:32.120 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-30T00:49:32.844 INFO:teuthology.orchestra.run.smithi116.stdout:ceph: stdout ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable) 2024-03-30T00:49:32.845 INFO:teuthology.orchestra.run.smithi116.stdout:Ceph version: ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable) 2024-03-30T00:49:32.845 INFO:teuthology.orchestra.run.smithi116.stdout:Extracting ceph user uid/gid from container image... 2024-03-30T00:49:33.471 INFO:teuthology.orchestra.run.smithi116.stdout:stat: stdout 167 167 2024-03-30T00:49:33.471 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial keys... 2024-03-30T00:49:34.048 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQCdYQdm9tgVLBAAb9Js/MOTqTBcLbLggXuxaw== 2024-03-30T00:49:34.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQCeYQdmZZI+ExAAsfix8b3JItHvaqJCN5D9aA== 2024-03-30T00:49:35.264 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQCeYQdmFxDxOBAAKV+h0EzN1D0EFusMEZLayQ== 2024-03-30T00:49:35.265 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial monmap... 2024-03-30T00:49:35.872 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-30T00:49:35.872 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-03-30T00:49:35.872 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:49:35.872 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-30T00:49:35.872 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool for a [v2:172.21.15.116:3300,v1:172.21.15.116:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-30T00:49:35.872 INFO:teuthology.orchestra.run.smithi116.stdout:setting min_mon_release = pacific 2024-03-30T00:49:35.872 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: set fsid to 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:49:35.872 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-30T00:49:35.872 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:49:35.872 INFO:teuthology.orchestra.run.smithi116.stdout:Creating mon... 2024-03-30T00:49:36.521 INFO:teuthology.orchestra.run.smithi116.stdout:create mon.a on 2024-03-30T00:49:36.752 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-03-30T00:49:36.939 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-30T00:49:37.167 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4.target. 2024-03-30T00:49:37.167 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4.target. 2024-03-30T00:49:37.487 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mon.a 2024-03-30T00:49:37.487 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mon.a.service: Unit ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mon.a.service not loaded. 2024-03-30T00:49:37.673 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4.target.wants/ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mon.a.service → /etc/systemd/system/ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@.service. 2024-03-30T00:49:37.756 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:37 smithi116 systemd[1]: Starting Ceph mon.a for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:49:38.157 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T00:49:38.157 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-30T00:49:38.175 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T00:49:38.175 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-30T00:49:38.175 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-30T00:49:38.175 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to enable service . firewalld.service is not available 2024-03-30T00:49:38.175 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mon to start... 2024-03-30T00:49:38.175 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mon... 2024-03-30T00:49:38.176 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:38 smithi116 bash[37553]: 39fd1e0845d792dba88b4bea5b81c4e96859f0ede144c5248bd7c11122d26b2d 2024-03-30T00:49:38.176 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:38 smithi116 systemd[1]: Started Ceph mon.a for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:49:38.176 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:38 smithi116 ceph-mon[37590]: mkfs 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:49:38.386 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:38 smithi116 ceph-mon[37590]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T00:49:39.052 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout cluster: 2024-03-30T00:49:39.052 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout id: 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:49:39.052 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-30T00:49:39.052 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-30T00:49:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout services: 2024-03-30T00:49:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.553503s) 2024-03-30T00:49:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-30T00:49:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-30T00:49:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-30T00:49:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout data: 2024-03-30T00:49:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-30T00:49:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-30T00:49:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-30T00:49:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout pgs: 2024-03-30T00:49:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-30T00:49:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.634+0000 7fe02b600700 1 Processor -- start 2024-03-30T00:49:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.634+0000 7fe02b600700 1 -- start start 2024-03-30T00:49:39.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.634+0000 7fe02b600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02407be00 0x7fe02407a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:49:39.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.634+0000 7fe02b600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02407a7f0 con 0x7fe02407be00 2024-03-30T00:49:39.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.635+0000 7fe028c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02407be00 0x7fe02407a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:49:39.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.635+0000 7fe028c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02407be00 0x7fe02407a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33728/0 (socket says 172.21.15.116:33728) 2024-03-30T00:49:39.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.635+0000 7fe028c00700 1 -- 172.21.15.116:0/2029897973 learned_addr learned my addr 172.21.15.116:0/2029897973 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:49:39.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.636+0000 7fe028c00700 1 -- 172.21.15.116:0/2029897973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02407a930 con 0x7fe02407be00 2024-03-30T00:49:39.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.636+0000 7fe028c00700 1 --2- 172.21.15.116:0/2029897973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02407be00 0x7fe02407a2b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe0180098a0 tx=0x7fe018009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87daef5e4dfe3aa server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:49:39.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.636+0000 7fe023400700 1 -- 172.21.15.116:0/2029897973 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe018004160 con 0x7fe02407be00 2024-03-30T00:49:39.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.636+0000 7fe023400700 1 -- 172.21.15.116:0/2029897973 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0180042c0 con 0x7fe02407be00 2024-03-30T00:49:39.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.636+0000 7fe023400700 1 -- 172.21.15.116:0/2029897973 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0180045a0 con 0x7fe02407be00 2024-03-30T00:49:39.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.637+0000 7fe02b600700 1 -- 172.21.15.116:0/2029897973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02407be00 msgr2=0x7fe02407a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:49:39.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.637+0000 7fe02b600700 1 --2- 172.21.15.116:0/2029897973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02407be00 0x7fe02407a2b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe0180098a0 tx=0x7fe018009bb0 comp rx=0 tx=0).stop 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.637+0000 7fe02b600700 1 -- 172.21.15.116:0/2029897973 shutdown_connections 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.637+0000 7fe02b600700 1 --2- 172.21.15.116:0/2029897973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02407be00 0x7fe02407a2b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.637+0000 7fe02b600700 1 -- 172.21.15.116:0/2029897973 >> 172.21.15.116:0/2029897973 conn(0x7fe024101af0 msgr2=0x7fe024103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.637+0000 7fe02b600700 1 -- 172.21.15.116:0/2029897973 shutdown_connections 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.637+0000 7fe02b600700 1 -- 172.21.15.116:0/2029897973 wait complete. 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.638+0000 7fe02b600700 1 Processor -- start 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.638+0000 7fe02b600700 1 -- start start 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.638+0000 7fe02b600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02407be00 0x7fe02419bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.638+0000 7fe02b600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02419c100 con 0x7fe02407be00 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.639+0000 7fe028c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02407be00 0x7fe02419bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.639+0000 7fe028c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02407be00 0x7fe02419bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33730/0 (socket says 172.21.15.116:33730) 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.639+0000 7fe028c00700 1 -- 172.21.15.116:0/3436084040 learned_addr learned my addr 172.21.15.116:0/3436084040 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.639+0000 7fe028c00700 1 -- 172.21.15.116:0/3436084040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe018009550 con 0x7fe02407be00 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.639+0000 7fe028c00700 1 --2- 172.21.15.116:0/3436084040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02407be00 0x7fe02419bbc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe02407b320 tx=0x7fe018004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:49:39.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.640+0000 7fe021600700 1 -- 172.21.15.116:0/3436084040 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe018011630 con 0x7fe02407be00 2024-03-30T00:49:39.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.640+0000 7fe021600700 1 -- 172.21.15.116:0/3436084040 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe018011790 con 0x7fe02407be00 2024-03-30T00:49:39.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.640+0000 7fe02b600700 1 -- 172.21.15.116:0/3436084040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02419c300 con 0x7fe02407be00 2024-03-30T00:49:39.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.640+0000 7fe021600700 1 -- 172.21.15.116:0/3436084040 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe018011a30 con 0x7fe02407be00 2024-03-30T00:49:39.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.640+0000 7fe02b600700 1 -- 172.21.15.116:0/3436084040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02419c7a0 con 0x7fe02407be00 2024-03-30T00:49:39.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.640+0000 7fe021600700 1 -- 172.21.15.116:0/3436084040 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe018021070 con 0x7fe02407be00 2024-03-30T00:49:39.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.640+0000 7fe021600700 1 -- 172.21.15.116:0/3436084040 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe01801b440 con 0x7fe02407be00 2024-03-30T00:49:39.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.641+0000 7fe02b600700 1 -- 172.21.15.116:0/3436084040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0241959d0 con 0x7fe02407be00 2024-03-30T00:49:39.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.644+0000 7fe021600700 1 -- 172.21.15.116:0/3436084040 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fe01801a460 con 0x7fe02407be00 2024-03-30T00:49:39.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.715+0000 7fe02b600700 1 -- 172.21.15.116:0/3436084040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe02407ae80 con 0x7fe02407be00 2024-03-30T00:49:39.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.716+0000 7fe021600700 1 -- 172.21.15.116:0/3436084040 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fe01801a600 con 0x7fe02407be00 2024-03-30T00:49:39.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.717+0000 7fe02b600700 1 -- 172.21.15.116:0/3436084040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02407be00 msgr2=0x7fe02419bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:49:39.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.717+0000 7fe02b600700 1 --2- 172.21.15.116:0/3436084040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02407be00 0x7fe02419bbc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe02407b320 tx=0x7fe018004750 comp rx=0 tx=0).stop 2024-03-30T00:49:39.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.717+0000 7fe02b600700 1 -- 172.21.15.116:0/3436084040 shutdown_connections 2024-03-30T00:49:39.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.717+0000 7fe02b600700 1 --2- 172.21.15.116:0/3436084040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02407be00 0x7fe02419bbc0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:49:39.057 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.717+0000 7fe02b600700 1 -- 172.21.15.116:0/3436084040 >> 172.21.15.116:0/3436084040 conn(0x7fe024101af0 msgr2=0x7fe024103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:49:39.057 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.717+0000 7fe02b600700 1 -- 172.21.15.116:0/3436084040 shutdown_connections 2024-03-30T00:49:39.057 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:38.717+0000 7fe02b600700 1 -- 172.21.15.116:0/3436084040 wait complete. 2024-03-30T00:49:39.057 INFO:teuthology.orchestra.run.smithi116.stdout:mon is available 2024-03-30T00:49:39.057 INFO:teuthology.orchestra.run.smithi116.stdout:Assimilating anything we can from ceph.conf... 2024-03-30T00:49:39.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:39 smithi116 ceph-mon[37590]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T00:49:39.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:39 smithi116 ceph-mon[37590]: monmap e1: 1 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T00:49:39.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:39 smithi116 ceph-mon[37590]: fsmap 2024-03-30T00:49:39.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:39 smithi116 ceph-mon[37590]: osdmap e1: 0 total, 0 up, 0 in 2024-03-30T00:49:39.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:39 smithi116 ceph-mon[37590]: mgrmap e1: no daemons active 2024-03-30T00:49:39.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:39 smithi116 ceph-mon[37590]: from='client.? 172.21.15.116:0/3436084040' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-30T00:49:39.907 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-30T00:49:39.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [global] 2024-03-30T00:49:39.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout fsid = 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:49:39.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.116:3300,v1:172.21.15.116:6789] 2024-03-30T00:49:39.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-30T00:49:39.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-30T00:49:39.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-30T00:49:39.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-30T00:49:39.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-30T00:49:39.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-30T00:49:39.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-30T00:49:39.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-30T00:49:39.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [osd] 2024-03-30T00:49:39.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-30T00:49:39.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-30T00:49:39.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-30T00:49:39.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-30T00:49:39.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-30T00:49:39.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.457+0000 7fedbc800700 1 Processor -- start 2024-03-30T00:49:39.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.457+0000 7fedbc800700 1 -- start start 2024-03-30T00:49:39.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.457+0000 7fedbc800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedb407be20 0x7fedb407a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:49:39.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.457+0000 7fedbc800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb407a810 con 0x7fedb407be20 2024-03-30T00:49:39.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.457+0000 7fedb9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedb407be20 0x7fedb407a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:49:39.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.458+0000 7fedb9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedb407be20 0x7fedb407a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33746/0 (socket says 172.21.15.116:33746) 2024-03-30T00:49:39.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.458+0000 7fedb9e00700 1 -- 172.21.15.116:0/634732281 learned_addr learned my addr 172.21.15.116:0/634732281 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:49:39.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.458+0000 7fedb9e00700 1 -- 172.21.15.116:0/634732281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb407a950 con 0x7fedb407be20 2024-03-30T00:49:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.458+0000 7fedb9e00700 1 --2- 172.21.15.116:0/634732281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedb407be20 0x7fedb407a2d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7feda80098a0 tx=0x7feda8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95dbe058d55ec9fd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:49:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.458+0000 7fedb8a00700 1 -- 172.21.15.116:0/634732281 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feda8004160 con 0x7fedb407be20 2024-03-30T00:49:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.458+0000 7fedb8a00700 1 -- 172.21.15.116:0/634732281 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda80042c0 con 0x7fedb407be20 2024-03-30T00:49:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.458+0000 7fedb8a00700 1 -- 172.21.15.116:0/634732281 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feda80045a0 con 0x7fedb407be20 2024-03-30T00:49:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.459+0000 7fedbc800700 1 -- 172.21.15.116:0/634732281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedb407be20 msgr2=0x7fedb407a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:49:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.459+0000 7fedbc800700 1 --2- 172.21.15.116:0/634732281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedb407be20 0x7fedb407a2d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7feda80098a0 tx=0x7feda8009bb0 comp rx=0 tx=0).stop 2024-03-30T00:49:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.459+0000 7fedbc800700 1 -- 172.21.15.116:0/634732281 shutdown_connections 2024-03-30T00:49:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.459+0000 7fedbc800700 1 --2- 172.21.15.116:0/634732281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedb407be20 0x7fedb407a2d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:49:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.459+0000 7fedbc800700 1 -- 172.21.15.116:0/634732281 >> 172.21.15.116:0/634732281 conn(0x7fedb4101af0 msgr2=0x7fedb4103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:49:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.459+0000 7fedbc800700 1 -- 172.21.15.116:0/634732281 shutdown_connections 2024-03-30T00:49:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.459+0000 7fedbc800700 1 -- 172.21.15.116:0/634732281 wait complete. 2024-03-30T00:49:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.460+0000 7fedbc800700 1 Processor -- start 2024-03-30T00:49:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.460+0000 7fedbc800700 1 -- start start 2024-03-30T00:49:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.460+0000 7fedbc800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedb407be20 0x7fedb419bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.460+0000 7fedbc800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb419c120 con 0x7fedb407be20 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.461+0000 7fedb9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedb407be20 0x7fedb419bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.461+0000 7fedb9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedb407be20 0x7fedb419bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33756/0 (socket says 172.21.15.116:33756) 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.461+0000 7fedb9e00700 1 -- 172.21.15.116:0/4015202102 learned_addr learned my addr 172.21.15.116:0/4015202102 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.461+0000 7fedb9e00700 1 -- 172.21.15.116:0/4015202102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda8009550 con 0x7fedb407be20 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.461+0000 7fedb9e00700 1 --2- 172.21.15.116:0/4015202102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedb407be20 0x7fedb419bbe0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fedb407b340 tx=0x7feda8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.461+0000 7fedaea00700 1 -- 172.21.15.116:0/4015202102 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feda8011630 con 0x7fedb407be20 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.462+0000 7fedaea00700 1 -- 172.21.15.116:0/4015202102 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda8011790 con 0x7fedb407be20 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.462+0000 7fedaea00700 1 -- 172.21.15.116:0/4015202102 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feda8011a30 con 0x7fedb407be20 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.462+0000 7fedbc800700 1 -- 172.21.15.116:0/4015202102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb419c320 con 0x7fedb407be20 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.462+0000 7fedbc800700 1 -- 172.21.15.116:0/4015202102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb419c7c0 con 0x7fedb407be20 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.462+0000 7fedaea00700 1 -- 172.21.15.116:0/4015202102 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7feda8021070 con 0x7fedb407be20 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.462+0000 7fedaea00700 1 -- 172.21.15.116:0/4015202102 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7feda801b440 con 0x7fedb407be20 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.463+0000 7fedbc800700 1 -- 172.21.15.116:0/4015202102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb41959f0 con 0x7fedb407be20 2024-03-30T00:49:39.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.465+0000 7fedaea00700 1 -- 172.21.15.116:0/4015202102 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7feda801a460 con 0x7fedb407be20 2024-03-30T00:49:39.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.535+0000 7fedbc800700 1 -- 172.21.15.116:0/4015202102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fedb407aef0 con 0x7fedb407be20 2024-03-30T00:49:39.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.539+0000 7fedaea00700 1 -- 172.21.15.116:0/4015202102 <== mon.0 v2:172.21.15.116:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feda801a600 con 0x7fedb407be20 2024-03-30T00:49:39.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.539+0000 7fedaea00700 1 -- 172.21.15.116:0/4015202102 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7feda8035590 con 0x7fedb407be20 2024-03-30T00:49:39.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.540+0000 7fedbc800700 1 -- 172.21.15.116:0/4015202102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedb407be20 msgr2=0x7fedb419bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:49:39.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.540+0000 7fedbc800700 1 --2- 172.21.15.116:0/4015202102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedb407be20 0x7fedb419bbe0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fedb407b340 tx=0x7feda8004750 comp rx=0 tx=0).stop 2024-03-30T00:49:39.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.540+0000 7fedbc800700 1 -- 172.21.15.116:0/4015202102 shutdown_connections 2024-03-30T00:49:39.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.540+0000 7fedbc800700 1 --2- 172.21.15.116:0/4015202102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedb407be20 0x7fedb419bbe0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:49:39.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.540+0000 7fedbc800700 1 -- 172.21.15.116:0/4015202102 >> 172.21.15.116:0/4015202102 conn(0x7fedb4101af0 msgr2=0x7fedb4103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:49:39.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.540+0000 7fedbc800700 1 -- 172.21.15.116:0/4015202102 shutdown_connections 2024-03-30T00:49:39.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:39.540+0000 7fedbc800700 1 -- 172.21.15.116:0/4015202102 wait complete. 2024-03-30T00:49:39.917 INFO:teuthology.orchestra.run.smithi116.stdout:Generating new minimal ceph.conf... 2024-03-30T00:49:40.554 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:40 smithi116 ceph-mon[37590]: from='client.? 172.21.15.116:0/4015202102' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-30T00:49:40.554 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:40 smithi116 ceph-mon[37590]: from='client.? 172.21.15.116:0/4015202102' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-30T00:49:40.691 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.289+0000 7f8822400700 1 Processor -- start 2024-03-30T00:49:40.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.289+0000 7f8822400700 1 -- start start 2024-03-30T00:49:40.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.290+0000 7f8822400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881c07be20 0x7f881c07a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:49:40.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.290+0000 7f8822400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c07a810 con 0x7f881c07be20 2024-03-30T00:49:40.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.290+0000 7f881b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881c07be20 0x7f881c07a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:49:40.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.290+0000 7f881b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881c07be20 0x7f881c07a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33758/0 (socket says 172.21.15.116:33758) 2024-03-30T00:49:40.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.290+0000 7f881b400700 1 -- 172.21.15.116:0/1823077880 learned_addr learned my addr 172.21.15.116:0/1823077880 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:49:40.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.290+0000 7f881b400700 1 -- 172.21.15.116:0/1823077880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881c07a950 con 0x7f881c07be20 2024-03-30T00:49:40.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.291+0000 7f881b400700 1 --2- 172.21.15.116:0/1823077880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881c07be20 0x7f881c07a2d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f880c0098a0 tx=0x7f880c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c5541be31561a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:49:40.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.291+0000 7f881aa00700 1 -- 172.21.15.116:0/1823077880 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f880c004160 con 0x7f881c07be20 2024-03-30T00:49:40.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.291+0000 7f881aa00700 1 -- 172.21.15.116:0/1823077880 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f880c0042c0 con 0x7f881c07be20 2024-03-30T00:49:40.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.291+0000 7f881aa00700 1 -- 172.21.15.116:0/1823077880 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f880c011a00 con 0x7f881c07be20 2024-03-30T00:49:40.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.292+0000 7f8822400700 1 -- 172.21.15.116:0/1823077880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881c07be20 msgr2=0x7f881c07a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:49:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.292+0000 7f8822400700 1 --2- 172.21.15.116:0/1823077880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881c07be20 0x7f881c07a2d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f880c0098a0 tx=0x7f880c009bb0 comp rx=0 tx=0).stop 2024-03-30T00:49:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.292+0000 7f8822400700 1 -- 172.21.15.116:0/1823077880 shutdown_connections 2024-03-30T00:49:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.292+0000 7f8822400700 1 --2- 172.21.15.116:0/1823077880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881c07be20 0x7f881c07a2d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:49:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.292+0000 7f8822400700 1 -- 172.21.15.116:0/1823077880 >> 172.21.15.116:0/1823077880 conn(0x7f881c101af0 msgr2=0x7f881c103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:49:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.292+0000 7f8822400700 1 -- 172.21.15.116:0/1823077880 shutdown_connections 2024-03-30T00:49:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.292+0000 7f8822400700 1 -- 172.21.15.116:0/1823077880 wait complete. 2024-03-30T00:49:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.293+0000 7f8822400700 1 Processor -- start 2024-03-30T00:49:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.293+0000 7f8822400700 1 -- start start 2024-03-30T00:49:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.293+0000 7f8822400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881c07be20 0x7f881c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:49:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.293+0000 7f8822400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c19c960 con 0x7f881c07be20 2024-03-30T00:49:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.293+0000 7f881b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881c07be20 0x7f881c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:49:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.293+0000 7f881b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881c07be20 0x7f881c19c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33772/0 (socket says 172.21.15.116:33772) 2024-03-30T00:49:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.293+0000 7f881b400700 1 -- 172.21.15.116:0/2894775534 learned_addr learned my addr 172.21.15.116:0/2894775534 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:49:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.294+0000 7f881b400700 1 -- 172.21.15.116:0/2894775534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f880c009550 con 0x7f881c07be20 2024-03-30T00:49:40.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.294+0000 7f881b400700 1 --2- 172.21.15.116:0/2894775534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881c07be20 0x7f881c19c420 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f880c009130 tx=0x7f880c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:49:40.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.294+0000 7f8818c00700 1 -- 172.21.15.116:0/2894775534 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f880c0038b0 con 0x7f881c07be20 2024-03-30T00:49:40.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.294+0000 7f8818c00700 1 -- 172.21.15.116:0/2894775534 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f880c003a10 con 0x7f881c07be20 2024-03-30T00:49:40.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.294+0000 7f8822400700 1 -- 172.21.15.116:0/2894775534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f881c19cb60 con 0x7f881c07be20 2024-03-30T00:49:40.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.294+0000 7f8818c00700 1 -- 172.21.15.116:0/2894775534 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f880c019550 con 0x7f881c07be20 2024-03-30T00:49:40.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.294+0000 7f8822400700 1 -- 172.21.15.116:0/2894775534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f881c19d000 con 0x7f881c07be20 2024-03-30T00:49:40.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.295+0000 7f8818c00700 1 -- 172.21.15.116:0/2894775534 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f880c019b10 con 0x7f881c07be20 2024-03-30T00:49:40.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.295+0000 7f8818c00700 1 -- 172.21.15.116:0/2894775534 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f880c02de30 con 0x7f881c07be20 2024-03-30T00:49:40.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.296+0000 7f8822400700 1 -- 172.21.15.116:0/2894775534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f881c10c840 con 0x7f881c07be20 2024-03-30T00:49:40.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.298+0000 7f8818c00700 1 -- 172.21.15.116:0/2894775534 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f880c003b80 con 0x7f881c07be20 2024-03-30T00:49:40.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.367+0000 7f8822400700 1 -- 172.21.15.116:0/2894775534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f881c10ca30 con 0x7f881c07be20 2024-03-30T00:49:40.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.367+0000 7f8818c00700 1 -- 172.21.15.116:0/2894775534 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f880c01a960 con 0x7f881c07be20 2024-03-30T00:49:40.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.369+0000 7f8822400700 1 -- 172.21.15.116:0/2894775534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881c07be20 msgr2=0x7f881c19c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:49:40.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.369+0000 7f8822400700 1 --2- 172.21.15.116:0/2894775534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881c07be20 0x7f881c19c420 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f880c009130 tx=0x7f880c011ad0 comp rx=0 tx=0).stop 2024-03-30T00:49:40.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.369+0000 7f8822400700 1 -- 172.21.15.116:0/2894775534 shutdown_connections 2024-03-30T00:49:40.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.369+0000 7f8822400700 1 --2- 172.21.15.116:0/2894775534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f881c07be20 0x7f881c19c420 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:49:40.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.369+0000 7f8822400700 1 -- 172.21.15.116:0/2894775534 >> 172.21.15.116:0/2894775534 conn(0x7f881c101af0 msgr2=0x7f881c1026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:49:40.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.369+0000 7f8822400700 1 -- 172.21.15.116:0/2894775534 shutdown_connections 2024-03-30T00:49:40.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:40.369+0000 7f8822400700 1 -- 172.21.15.116:0/2894775534 wait complete. 2024-03-30T00:49:40.695 INFO:teuthology.orchestra.run.smithi116.stdout:Restarting the monitor... 2024-03-30T00:49:40.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:40 smithi116 systemd[1]: Stopping Ceph mon.a for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:49:41.244 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:40 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-a[37567]: 2024-03-30T00:49:40.972+0000 7f8fba800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T00:49:41.244 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:40 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-a[37567]: 2024-03-30T00:49:40.972+0000 7f8fba800700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-30T00:49:41.244 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:41 smithi116 bash[37923]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-a 2024-03-30T00:49:41.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:41 smithi116 systemd[1]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mon.a.service: Succeeded. 2024-03-30T00:49:41.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:41 smithi116 systemd[1]: Stopped Ceph mon.a for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:49:41.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:41 smithi116 systemd[1]: Starting Ceph mon.a for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:49:42.011 INFO:teuthology.orchestra.run.smithi116.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-30T00:49:42.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:41 smithi116 bash[38048]: d8d2be658684c6f8b476287668e8ba88a980f564f502d1a5d393a4f7e440b2ba 2024-03-30T00:49:42.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 systemd[1]: Started Ceph mon.a for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:49:42.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T00:49:42.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable), process ceph-mon, pid 7 2024-03-30T00:49:42.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: pidfile_write: ignore empty --pid-file 2024-03-30T00:49:42.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: load: jerasure load: lrc 2024-03-30T00:49:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: RocksDB version: 7.9.2 2024-03-30T00:49:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Git sha 0 2024-03-30T00:49:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Compile date 2024-03-29 11:49:16 2024-03-30T00:49:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: DB SUMMARY 2024-03-30T00:49:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: DB Session ID: EW5SV5DY6KT2SBJA397N 2024-03-30T00:49:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: CURRENT file: CURRENT 2024-03-30T00:49:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T00:49:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-30T00:49:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-30T00:49:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 97598 ; 2024-03-30T00:49:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.error_if_exists: 0 2024-03-30T00:49:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.create_if_missing: 0 2024-03-30T00:49:42.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.paranoid_checks: 1 2024-03-30T00:49:42.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-30T00:49:42.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T00:49:42.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-30T00:49:42.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.env: 0x55dacc72e720 2024-03-30T00:49:42.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.fs: PosixFileSystem 2024-03-30T00:49:42.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.info_log: 0x55dacf391520 2024-03-30T00:49:42.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T00:49:42.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.statistics: (nil) 2024-03-30T00:49:42.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.use_fsync: 0 2024-03-30T00:49:42.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_log_file_size: 0 2024-03-30T00:49:42.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T00:49:42.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T00:49:42.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.allow_fallocate: 1 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.use_direct_reads: 0 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.db_log_dir: 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.wal_dir: 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T00:49:42.264 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.write_buffer_manager: 0x55dace628320 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.unordered_write: 0 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T00:49:42.265 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T00:49:42.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.row_cache: None 2024-03-30T00:49:42.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.wal_filter: None 2024-03-30T00:49:42.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T00:49:42.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T00:49:42.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.two_write_queues: 0 2024-03-30T00:49:42.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T00:49:42.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.wal_compression: 0 2024-03-30T00:49:42.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.atomic_flush: 0 2024-03-30T00:49:42.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T00:49:42.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T00:49:42.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T00:49:42.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.log_readahead_size: 0 2024-03-30T00:49:42.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_background_jobs: 2 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_background_compactions: -1 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_subcompactions: 1 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T00:49:42.267 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_open_files: -1 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_background_flushes: -1 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Compression algorithms supported: 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: kZSTD supported: 0 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: kXpressCompression supported: 0 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: kZlibCompression supported: 1 2024-03-30T00:49:42.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: kSnappyCompression supported: 1 2024-03-30T00:49:42.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: kLZ4Compression supported: 1 2024-03-30T00:49:42.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: kBZip2Compression supported: 0 2024-03-30T00:49:42.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T00:49:42.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-30T00:49:42.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-30T00:49:42.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T00:49:42.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T00:49:42.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.merge_operator: 2024-03-30T00:49:42.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compaction_filter: None 2024-03-30T00:49:42.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T00:49:42.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T00:49:42.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T00:49:42.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T00:49:42.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dacf391620) 2024-03-30T00:49:42.270 INFO:journalctl@ceph.mon.a.smithi116.stdout: cache_index_and_filter_blocks: 1 2024-03-30T00:49:42.270 INFO:journalctl@ceph.mon.a.smithi116.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T00:49:42.270 INFO:journalctl@ceph.mon.a.smithi116.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T00:49:42.270 INFO:journalctl@ceph.mon.a.smithi116.stdout: pin_top_level_index_and_filter: 1 2024-03-30T00:49:42.270 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_type: 0 2024-03-30T00:49:42.270 INFO:journalctl@ceph.mon.a.smithi116.stdout: data_block_index_type: 0 2024-03-30T00:49:42.270 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_shortening: 1 2024-03-30T00:49:42.270 INFO:journalctl@ceph.mon.a.smithi116.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T00:49:42.270 INFO:journalctl@ceph.mon.a.smithi116.stdout: checksum: 4 2024-03-30T00:49:42.270 INFO:journalctl@ceph.mon.a.smithi116.stdout: no_block_cache: 0 2024-03-30T00:49:42.270 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache: 0x55dace6a9350 2024-03-30T00:49:42.270 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_name: BinnedLRUCache 2024-03-30T00:49:42.270 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_options: 2024-03-30T00:49:42.271 INFO:journalctl@ceph.mon.a.smithi116.stdout: capacity : 536870912 2024-03-30T00:49:42.271 INFO:journalctl@ceph.mon.a.smithi116.stdout: num_shard_bits : 4 2024-03-30T00:49:42.271 INFO:journalctl@ceph.mon.a.smithi116.stdout: strict_capacity_limit : 0 2024-03-30T00:49:42.271 INFO:journalctl@ceph.mon.a.smithi116.stdout: high_pri_pool_ratio: 0.000 2024-03-30T00:49:42.271 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_compressed: (nil) 2024-03-30T00:49:42.271 INFO:journalctl@ceph.mon.a.smithi116.stdout: persistent_cache: (nil) 2024-03-30T00:49:42.271 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_size: 4096 2024-03-30T00:49:42.271 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_size_deviation: 10 2024-03-30T00:49:42.271 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_restart_interval: 16 2024-03-30T00:49:42.271 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_block_restart_interval: 1 2024-03-30T00:49:42.271 INFO:journalctl@ceph.mon.a.smithi116.stdout: metadata_block_size: 4096 2024-03-30T00:49:42.271 INFO:journalctl@ceph.mon.a.smithi116.stdout: partition_filters: 0 2024-03-30T00:49:42.271 INFO:journalctl@ceph.mon.a.smithi116.stdout: use_delta_encoding: 1 2024-03-30T00:49:42.271 INFO:journalctl@ceph.mon.a.smithi116.stdout: filter_policy: bloomfilter 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout: whole_key_filtering: 1 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout: verify_compression: 0 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout: read_amp_bytes_per_bit: 0 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout: format_version: 5 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout: enable_index_compression: 1 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_align: 0 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout: max_auto_readahead_size: 262144 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout: prepopulate_block_cache: 0 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout: initial_auto_readahead_size: 8192 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compression: NoCompression 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T00:49:42.272 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.num_levels: 7 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T00:49:42.273 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T00:49:42.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T00:49:42.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T00:49:42.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T00:49:42.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T00:49:42.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T00:49:42.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T00:49:42.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T00:49:42.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T00:49:42.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T00:49:42.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T00:49:42.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T00:49:42.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T00:49:42.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T00:49:42.274 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T00:49:42.275 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.table_properties_collectors: 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.inplace_update_support: 0 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T00:49:42.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.bloom_locality: 0 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.max_successive_merges: 0 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.ttl: 2592000 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.enable_blob_files: false 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.min_blob_size: 0 2024-03-30T00:49:42.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-30T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-30T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: bf9def87-9277-4d1b-be17-c2a5386899de 2024-03-30T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711759782023863, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-30T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-30T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711759782026134, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92967, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 307, "table_properties": {"data_size": 90978, "index_size": 271, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13606, "raw_average_key_size": 50, "raw_value_size": 83590, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 270, "num_filter_entries": 270, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711759782, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bf9def87-9277-4d1b-be17-c2a5386899de", "db_session_id": "EW5SV5DY6KT2SBJA397N", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-30T00:49:42.278 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711759782026266, "job": 1, "event": "recovery_finished"} 2024-03-30T00:49:42.279 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-30T00:49:42.279 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T00:49:42.279 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55dace746000 2024-03-30T00:49:42.279 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: DB pointer 0x55dace734000 2024-03-30T00:49:42.279 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T00:49:42.279 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T00:49:42.279 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** DB Stats ** 2024-03-30T00:49:42.279 INFO:journalctl@ceph.mon.a.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T00:49:42.279 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T00:49:42.279 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T00:49:42.279 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T00:49:42.279 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T00:49:42.279 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T00:49:42.279 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: L0 2/0 92.62 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 55.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: Sum 2/0 92.62 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 55.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 55.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 55.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T00:49:42.280 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative compaction: 0.00 GB write, 11.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval compaction: 0.00 GB write, 11.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout: Block cache BinnedLRUCache@0x55dace6a9350#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: starting mon.a rank 0 at public addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] at bind addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:49:42.281 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: mon.a@-1(???) e1 preinit fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: mon.a@-1(???).mds e1 new map 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: mon.a@-1(???).mds e1 print_map 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout: e1 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout: legacy client fscid: -1 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout: No filesystems configured 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T00:49:42.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: monmap e1: 1 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T00:49:42.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: fsmap 2024-03-30T00:49:42.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: osdmap e1: 0 total, 0 up, 0 in 2024-03-30T00:49:42.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:42 smithi116 ceph-mon[38086]: mgrmap e1: no daemons active 2024-03-30T00:49:42.891 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.461+0000 7f43d8a00700 1 Processor -- start 2024-03-30T00:49:42.891 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.461+0000 7f43d8a00700 1 -- start start 2024-03-30T00:49:42.891 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.461+0000 7f43d8a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d407be40 0x7f43d407a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:49:42.891 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.461+0000 7f43d8a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d407a8c0 con 0x7f43d407be40 2024-03-30T00:49:42.891 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.462+0000 7f43d1e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d407be40 0x7f43d407a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:49:42.891 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.462+0000 7f43d1e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d407be40 0x7f43d407a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33784/0 (socket says 172.21.15.116:33784) 2024-03-30T00:49:42.891 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.462+0000 7f43d1e00700 1 -- 172.21.15.116:0/4049366873 learned_addr learned my addr 172.21.15.116:0/4049366873 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:49:42.891 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.462+0000 7f43d1e00700 1 -- 172.21.15.116:0/4049366873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d407aa00 con 0x7f43d407be40 2024-03-30T00:49:42.891 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.462+0000 7f43d1e00700 1 --2- 172.21.15.116:0/4049366873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d407be40 0x7f43d407a2f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f43c40098a0 tx=0x7f43c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4276283cacbfe1c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:49:42.891 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.463+0000 7f43d0a00700 1 -- 172.21.15.116:0/4049366873 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43c4004160 con 0x7f43d407be40 2024-03-30T00:49:42.891 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.463+0000 7f43d0a00700 1 -- 172.21.15.116:0/4049366873 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43c40042c0 con 0x7f43d407be40 2024-03-30T00:49:42.891 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.463+0000 7f43d0a00700 1 -- 172.21.15.116:0/4049366873 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43c4011a00 con 0x7f43d407be40 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.463+0000 7f43d8a00700 1 -- 172.21.15.116:0/4049366873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d407be40 msgr2=0x7f43d407a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.463+0000 7f43d8a00700 1 --2- 172.21.15.116:0/4049366873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d407be40 0x7f43d407a2f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f43c40098a0 tx=0x7f43c4009bb0 comp rx=0 tx=0).stop 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.463+0000 7f43d8a00700 1 -- 172.21.15.116:0/4049366873 shutdown_connections 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.463+0000 7f43d8a00700 1 --2- 172.21.15.116:0/4049366873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d407be40 0x7f43d407a2f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.463+0000 7f43d8a00700 1 -- 172.21.15.116:0/4049366873 >> 172.21.15.116:0/4049366873 conn(0x7f43d4101af0 msgr2=0x7f43d4103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.463+0000 7f43d8a00700 1 -- 172.21.15.116:0/4049366873 shutdown_connections 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.464+0000 7f43d8a00700 1 -- 172.21.15.116:0/4049366873 wait complete. 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.464+0000 7f43d8a00700 1 Processor -- start 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.465+0000 7f43d8a00700 1 -- start start 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.465+0000 7f43d8a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d407be40 0x7f43d419c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.465+0000 7f43d8a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d419c960 con 0x7f43d407be40 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.465+0000 7f43d1e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d407be40 0x7f43d419c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.465+0000 7f43d1e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d407be40 0x7f43d419c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33796/0 (socket says 172.21.15.116:33796) 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.465+0000 7f43d1e00700 1 -- 172.21.15.116:0/1437469820 learned_addr learned my addr 172.21.15.116:0/1437469820 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.466+0000 7f43d1e00700 1 -- 172.21.15.116:0/1437469820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c4009550 con 0x7f43d407be40 2024-03-30T00:49:42.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.466+0000 7f43d1e00700 1 --2- 172.21.15.116:0/1437469820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d407be40 0x7f43d419c420 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f43c4011a20 tx=0x7f43c4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.466+0000 7f43caa00700 1 -- 172.21.15.116:0/1437469820 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43c4003af0 con 0x7f43d407be40 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.466+0000 7f43caa00700 1 -- 172.21.15.116:0/1437469820 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43c4003c50 con 0x7f43d407be40 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.466+0000 7f43d8a00700 1 -- 172.21.15.116:0/1437469820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d419cb60 con 0x7f43d407be40 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.466+0000 7f43caa00700 1 -- 172.21.15.116:0/1437469820 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43c4019760 con 0x7f43d407be40 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.466+0000 7f43d8a00700 1 -- 172.21.15.116:0/1437469820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d419d000 con 0x7f43d407be40 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.466+0000 7f43caa00700 1 -- 172.21.15.116:0/1437469820 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f43c40199a0 con 0x7f43d407be40 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.466+0000 7f43caa00700 1 -- 172.21.15.116:0/1437469820 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f43c400b3e0 con 0x7f43d407be40 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.467+0000 7f43d8a00700 1 -- 172.21.15.116:0/1437469820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d410c880 con 0x7f43d407be40 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.470+0000 7f43caa00700 1 -- 172.21.15.116:0/1437469820 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f43c402a060 con 0x7f43d407be40 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.539+0000 7f43d8a00700 1 -- 172.21.15.116:0/1437469820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f43d419f890 con 0x7f43d407be40 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.544+0000 7f43caa00700 1 -- 172.21.15.116:0/1437469820 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f43c4019de0 con 0x7f43d407be40 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.545+0000 7f43d8a00700 1 -- 172.21.15.116:0/1437469820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d407be40 msgr2=0x7f43d419c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.545+0000 7f43d8a00700 1 --2- 172.21.15.116:0/1437469820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d407be40 0x7f43d419c420 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f43c4011a20 tx=0x7f43c4011b00 comp rx=0 tx=0).stop 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.545+0000 7f43d8a00700 1 -- 172.21.15.116:0/1437469820 shutdown_connections 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.545+0000 7f43d8a00700 1 --2- 172.21.15.116:0/1437469820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d407be40 0x7f43d419c420 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:49:42.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.545+0000 7f43d8a00700 1 -- 172.21.15.116:0/1437469820 >> 172.21.15.116:0/1437469820 conn(0x7f43d4101af0 msgr2=0x7f43d4102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:49:42.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.545+0000 7f43d8a00700 1 -- 172.21.15.116:0/1437469820 shutdown_connections 2024-03-30T00:49:42.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:42.545+0000 7f43d8a00700 1 -- 172.21.15.116:0/1437469820 wait complete. 2024-03-30T00:49:42.894 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-30T00:49:42.924 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-30T00:49:42.924 INFO:teuthology.orchestra.run.smithi116.stdout:Creating mgr... 2024-03-30T00:49:42.925 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-30T00:49:42.925 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-30T00:49:43.261 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.x 2024-03-30T00:49:43.261 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.x.service: Unit ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.x.service not loaded. 2024-03-30T00:49:43.455 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4.target.wants/ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.x.service → /etc/systemd/system/ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@.service. 2024-03-30T00:49:43.549 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:43 smithi116 systemd[1]: Starting Ceph mgr.x for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:49:43.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:43 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1437469820' entity='client.admin' 2024-03-30T00:49:43.981 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T00:49:43.981 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-30T00:49:43.999 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T00:49:43.999 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-30T00:49:44.000 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-30T00:49:44.000 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to enable service . firewalld.service is not available 2024-03-30T00:49:44.017 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T00:49:44.017 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-30T00:49:44.034 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T00:49:44.034 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-30T00:49:44.034 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-30T00:49:44.034 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-30T00:49:44.035 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr to start... 2024-03-30T00:49:44.035 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr... 2024-03-30T00:49:44.216 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:43 smithi116 bash[38322]: c520bb0ab8100a3abbd259c0db96e09900ee043a92cd33ea6913bbf5091f3651 2024-03-30T00:49:44.216 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:43 smithi116 systemd[1]: Started Ceph mgr.x for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:49:44.774 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:44 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3250927242' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T00:49:44.774 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:44 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:44.511+0000 7f089d761200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T00:49:44.774 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:44 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:44.644+0000 7f089d761200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T00:49:44.774 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:44 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:44.768+0000 7f089d761200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T00:49:44.805 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-30T00:49:44.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-30T00:49:44.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "4909f004-ee2f-11ee-b647-cb9ed24678a4", 2024-03-30T00:49:44.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T00:49:44.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T00:49:44.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T00:49:44.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T00:49:44.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:44.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T00:49:44.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T00:49:44.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-30T00:49:44.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-30T00:49:44.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T00:49:44.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T00:49:44.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-30T00:49:44.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-30T00:49:44.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T00:49:44.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:49:44.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-30T00:49:44.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T00:49:44.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:44.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T00:49:44.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T00:49:44.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T00:49:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-30T00:49:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T00:49:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T00:49:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:49:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T00:49:38.164631+0000", 2024-03-30T00:49:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T00:49:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T00:49:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-30T00:49:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.430+0000 7fdecca00700 1 Processor -- start 2024-03-30T00:49:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.430+0000 7fdecca00700 1 -- start start 2024-03-30T00:49:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.431+0000 7fdecca00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec8106510 0x7fdec8106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:49:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.431+0000 7fdecca00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec8106f00 con 0x7fdec8106510 2024-03-30T00:49:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.431+0000 7fdec5e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec8106510 0x7fdec8106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:49:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.431+0000 7fdec5e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec8106510 0x7fdec8106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33832/0 (socket says 172.21.15.116:33832) 2024-03-30T00:49:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.431+0000 7fdec5e00700 1 -- 172.21.15.116:0/2098607170 learned_addr learned my addr 172.21.15.116:0/2098607170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:49:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.432+0000 7fdec5e00700 1 -- 172.21.15.116:0/2098607170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec81076c0 con 0x7fdec8106510 2024-03-30T00:49:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.432+0000 7fdec5e00700 1 --2- 172.21.15.116:0/2098607170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec8106510 0x7fdec8106930 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80098a0 tx=0x7fdeb8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e988588646bd3fe server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:49:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.432+0000 7fdec4a00700 1 -- 172.21.15.116:0/2098607170 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdeb8004160 con 0x7fdec8106510 2024-03-30T00:49:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.432+0000 7fdec4a00700 1 -- 172.21.15.116:0/2098607170 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdeb80042c0 con 0x7fdec8106510 2024-03-30T00:49:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.432+0000 7fdec4a00700 1 -- 172.21.15.116:0/2098607170 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdeb8011a00 con 0x7fdec8106510 2024-03-30T00:49:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.433+0000 7fdecca00700 1 -- 172.21.15.116:0/2098607170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec8106510 msgr2=0x7fdec8106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:49:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.433+0000 7fdecca00700 1 --2- 172.21.15.116:0/2098607170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec8106510 0x7fdec8106930 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80098a0 tx=0x7fdeb8009bb0 comp rx=0 tx=0).stop 2024-03-30T00:49:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.433+0000 7fdecca00700 1 -- 172.21.15.116:0/2098607170 shutdown_connections 2024-03-30T00:49:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.433+0000 7fdecca00700 1 --2- 172.21.15.116:0/2098607170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec8106510 0x7fdec8106930 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:49:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.433+0000 7fdecca00700 1 -- 172.21.15.116:0/2098607170 >> 172.21.15.116:0/2098607170 conn(0x7fdec8101af0 msgr2=0x7fdec8103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:49:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.433+0000 7fdecca00700 1 -- 172.21.15.116:0/2098607170 shutdown_connections 2024-03-30T00:49:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.433+0000 7fdecca00700 1 -- 172.21.15.116:0/2098607170 wait complete. 2024-03-30T00:49:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.434+0000 7fdecca00700 1 Processor -- start 2024-03-30T00:49:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.434+0000 7fdecca00700 1 -- start start 2024-03-30T00:49:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.434+0000 7fdecca00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec8106510 0x7fdec819c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:49:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.434+0000 7fdecca00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec819c9a0 con 0x7fdec8106510 2024-03-30T00:49:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.435+0000 7fdec5e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec8106510 0x7fdec819c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:49:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.435+0000 7fdec5e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec8106510 0x7fdec819c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33844/0 (socket says 172.21.15.116:33844) 2024-03-30T00:49:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.435+0000 7fdec5e00700 1 -- 172.21.15.116:0/3250927242 learned_addr learned my addr 172.21.15.116:0/3250927242 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:49:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.435+0000 7fdec5e00700 1 -- 172.21.15.116:0/3250927242 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb8009550 con 0x7fdec8106510 2024-03-30T00:49:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.435+0000 7fdec5e00700 1 --2- 172.21.15.116:0/3250927242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec8106510 0x7fdec819c460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8011710 tx=0x7fdeb8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:49:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.436+0000 7fdebea00700 1 -- 172.21.15.116:0/3250927242 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdeb8003aa0 con 0x7fdec8106510 2024-03-30T00:49:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.436+0000 7fdebea00700 1 -- 172.21.15.116:0/3250927242 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdeb8003c00 con 0x7fdec8106510 2024-03-30T00:49:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.436+0000 7fdecca00700 1 -- 172.21.15.116:0/3250927242 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec819cba0 con 0x7fdec8106510 2024-03-30T00:49:44.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.436+0000 7fdebea00700 1 -- 172.21.15.116:0/3250927242 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdeb8024800 con 0x7fdec8106510 2024-03-30T00:49:44.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.436+0000 7fdecca00700 1 -- 172.21.15.116:0/3250927242 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec819d040 con 0x7fdec8106510 2024-03-30T00:49:44.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.436+0000 7fdebea00700 1 -- 172.21.15.116:0/3250927242 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fdeb8024d30 con 0x7fdec8106510 2024-03-30T00:49:44.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.436+0000 7fdebea00700 1 -- 172.21.15.116:0/3250927242 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdeb80193c0 con 0x7fdec8106510 2024-03-30T00:49:44.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.437+0000 7fdecca00700 1 -- 172.21.15.116:0/3250927242 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec80623e0 con 0x7fdec8106510 2024-03-30T00:49:44.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.439+0000 7fdebea00700 1 -- 172.21.15.116:0/3250927242 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fdeb801abd0 con 0x7fdec8106510 2024-03-30T00:49:44.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.514+0000 7fdecca00700 1 -- 172.21.15.116:0/3250927242 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdec810a8f0 con 0x7fdec8106510 2024-03-30T00:49:44.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.515+0000 7fdebea00700 1 -- 172.21.15.116:0/3250927242 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fdeb801ad70 con 0x7fdec8106510 2024-03-30T00:49:44.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.516+0000 7fdecca00700 1 -- 172.21.15.116:0/3250927242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec8106510 msgr2=0x7fdec819c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:49:44.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.516+0000 7fdecca00700 1 --2- 172.21.15.116:0/3250927242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec8106510 0x7fdec819c460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8011710 tx=0x7fdeb8011ad0 comp rx=0 tx=0).stop 2024-03-30T00:49:44.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.516+0000 7fdecca00700 1 -- 172.21.15.116:0/3250927242 shutdown_connections 2024-03-30T00:49:44.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.516+0000 7fdecca00700 1 --2- 172.21.15.116:0/3250927242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec8106510 0x7fdec819c460 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:49:44.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.516+0000 7fdecca00700 1 -- 172.21.15.116:0/3250927242 >> 172.21.15.116:0/3250927242 conn(0x7fdec8101af0 msgr2=0x7fdec8102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:49:44.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.516+0000 7fdecca00700 1 -- 172.21.15.116:0/3250927242 shutdown_connections 2024-03-30T00:49:44.823 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:44.516+0000 7fdecca00700 1 -- 172.21.15.116:0/3250927242 wait complete. 2024-03-30T00:49:44.823 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (1/15)... 2024-03-30T00:49:45.174 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:44 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:44.884+0000 7f089d761200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T00:49:46.053 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:45 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:45.729+0000 7f089d761200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T00:49:46.054 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:46 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:46.047+0000 7f089d761200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T00:49:46.424 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:46 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:46.157+0000 7f089d761200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T00:49:46.424 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:46 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:46.296+0000 7f089d761200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T00:49:46.799 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:46 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:46.506+0000 7f089d761200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T00:49:46.799 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:46 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:46.678+0000 7f089d761200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T00:49:47.174 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:46 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:46.793+0000 7f089d761200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T00:49:47.674 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:47 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:47.328+0000 7f089d761200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T00:49:48.096 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:47 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:47.825+0000 7f089d761200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T00:49:48.097 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:48 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:48.090+0000 7f089d761200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T00:49:48.360 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:48 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3774252127' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T00:49:48.361 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:48 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:48.219+0000 7f089d761200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T00:49:48.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-30T00:49:48.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-30T00:49:48.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "4909f004-ee2f-11ee-b647-cb9ed24678a4", 2024-03-30T00:49:48.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T00:49:48.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T00:49:48.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T00:49:48.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T00:49:48.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:48.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T00:49:48.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T00:49:48.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-30T00:49:48.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-30T00:49:48.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T00:49:48.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T00:49:48.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-30T00:49:48.497 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-30T00:49:48.497 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T00:49:48.497 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:49:48.497 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-30T00:49:48.497 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T00:49:48.497 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:48.497 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T00:49:48.497 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:49:48.497 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T00:49:48.497 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T00:49:48.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T00:49:48.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T00:49:48.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T00:49:48.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T00:49:48.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:48.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T00:49:48.505 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T00:49:48.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T00:49:48.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T00:49:48.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T00:49:48.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T00:49:48.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T00:49:48.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T00:49:48.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T00:49:48.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:48.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T00:49:48.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:49:48.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T00:49:48.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T00:49:48.506 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T00:49:38.164631+0000", 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:48.507 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T00:49:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-30T00:49:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.051+0000 7ff017a00700 1 Processor -- start 2024-03-30T00:49:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.051+0000 7ff017a00700 1 -- start start 2024-03-30T00:49:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.051+0000 7ff017a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff01007be20 0x7ff01007a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:49:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.051+0000 7ff017a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01007a810 con 0x7ff01007be20 2024-03-30T00:49:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.052+0000 7ff015000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff01007be20 0x7ff01007a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:49:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.052+0000 7ff015000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff01007be20 0x7ff01007a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42612/0 (socket says 172.21.15.116:42612) 2024-03-30T00:49:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.052+0000 7ff015000700 1 -- 172.21.15.116:0/2935098778 learned_addr learned my addr 172.21.15.116:0/2935098778 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:49:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.052+0000 7ff015000700 1 -- 172.21.15.116:0/2935098778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01007a950 con 0x7ff01007be20 2024-03-30T00:49:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.052+0000 7ff015000700 1 --2- 172.21.15.116:0/2935098778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff01007be20 0x7ff01007a2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff0040098a0 tx=0x7ff004009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2afba98d373a50 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:49:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.053+0000 7ff00f400700 1 -- 172.21.15.116:0/2935098778 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff004004160 con 0x7ff01007be20 2024-03-30T00:49:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.053+0000 7ff00f400700 1 -- 172.21.15.116:0/2935098778 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff0040042c0 con 0x7ff01007be20 2024-03-30T00:49:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.053+0000 7ff00f400700 1 -- 172.21.15.116:0/2935098778 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff004011a00 con 0x7ff01007be20 2024-03-30T00:49:48.508 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.053+0000 7ff017a00700 1 -- 172.21.15.116:0/2935098778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff01007be20 msgr2=0x7ff01007a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.053+0000 7ff017a00700 1 --2- 172.21.15.116:0/2935098778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff01007be20 0x7ff01007a2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff0040098a0 tx=0x7ff004009bb0 comp rx=0 tx=0).stop 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.053+0000 7ff017a00700 1 -- 172.21.15.116:0/2935098778 shutdown_connections 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.053+0000 7ff017a00700 1 --2- 172.21.15.116:0/2935098778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff01007be20 0x7ff01007a2d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.053+0000 7ff017a00700 1 -- 172.21.15.116:0/2935098778 >> 172.21.15.116:0/2935098778 conn(0x7ff010101af0 msgr2=0x7ff010103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.053+0000 7ff017a00700 1 -- 172.21.15.116:0/2935098778 shutdown_connections 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.053+0000 7ff017a00700 1 -- 172.21.15.116:0/2935098778 wait complete. 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.054+0000 7ff017a00700 1 Processor -- start 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.055+0000 7ff017a00700 1 -- start start 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.055+0000 7ff017a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff01007be20 0x7ff01019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.055+0000 7ff017a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01019c960 con 0x7ff01007be20 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.055+0000 7ff015000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff01007be20 0x7ff01019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.055+0000 7ff015000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff01007be20 0x7ff01019c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42624/0 (socket says 172.21.15.116:42624) 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.055+0000 7ff015000700 1 -- 172.21.15.116:0/3774252127 learned_addr learned my addr 172.21.15.116:0/3774252127 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.056+0000 7ff015000700 1 -- 172.21.15.116:0/3774252127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff004009550 con 0x7ff01007be20 2024-03-30T00:49:48.509 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.056+0000 7ff015000700 1 --2- 172.21.15.116:0/3774252127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff01007be20 0x7ff01019c420 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff004011a20 tx=0x7ff004011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:49:48.510 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.056+0000 7ff00d600700 1 -- 172.21.15.116:0/3774252127 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff004003af0 con 0x7ff01007be20 2024-03-30T00:49:48.510 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.056+0000 7ff00d600700 1 -- 172.21.15.116:0/3774252127 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff004003c50 con 0x7ff01007be20 2024-03-30T00:49:48.510 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.056+0000 7ff017a00700 1 -- 172.21.15.116:0/3774252127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff01019cb60 con 0x7ff01007be20 2024-03-30T00:49:48.510 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.056+0000 7ff017a00700 1 -- 172.21.15.116:0/3774252127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01019d000 con 0x7ff01007be20 2024-03-30T00:49:48.510 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.056+0000 7ff00d600700 1 -- 172.21.15.116:0/3774252127 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff004019770 con 0x7ff01007be20 2024-03-30T00:49:48.510 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.057+0000 7ff00d600700 1 -- 172.21.15.116:0/3774252127 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff004019910 con 0x7ff01007be20 2024-03-30T00:49:48.510 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.057+0000 7ff00d600700 1 -- 172.21.15.116:0/3774252127 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff00400b3e0 con 0x7ff01007be20 2024-03-30T00:49:48.510 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.057+0000 7ff017a00700 1 -- 172.21.15.116:0/3774252127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff01010c840 con 0x7ff01007be20 2024-03-30T00:49:48.510 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.060+0000 7ff00d600700 1 -- 172.21.15.116:0/3774252127 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7ff00402a060 con 0x7ff01007be20 2024-03-30T00:49:48.510 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.134+0000 7ff017a00700 1 -- 172.21.15.116:0/3774252127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff01010ca00 con 0x7ff01007be20 2024-03-30T00:49:48.510 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.135+0000 7ff00d600700 1 -- 172.21.15.116:0/3774252127 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7ff004019da0 con 0x7ff01007be20 2024-03-30T00:49:48.510 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.136+0000 7ff017a00700 1 -- 172.21.15.116:0/3774252127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff01007be20 msgr2=0x7ff01019c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:49:48.510 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.136+0000 7ff017a00700 1 --2- 172.21.15.116:0/3774252127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff01007be20 0x7ff01019c420 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff004011a20 tx=0x7ff004011b00 comp rx=0 tx=0).stop 2024-03-30T00:49:48.511 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.136+0000 7ff017a00700 1 -- 172.21.15.116:0/3774252127 shutdown_connections 2024-03-30T00:49:48.511 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.136+0000 7ff017a00700 1 --2- 172.21.15.116:0/3774252127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff01007be20 0x7ff01019c420 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:49:48.511 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.136+0000 7ff017a00700 1 -- 172.21.15.116:0/3774252127 >> 172.21.15.116:0/3774252127 conn(0x7ff010101af0 msgr2=0x7ff0101026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:49:48.511 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.136+0000 7ff017a00700 1 -- 172.21.15.116:0/3774252127 shutdown_connections 2024-03-30T00:49:48.511 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:48.136+0000 7ff017a00700 1 -- 172.21.15.116:0/3774252127 wait complete. 2024-03-30T00:49:48.511 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (2/15)... 2024-03-30T00:49:49.510 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:49 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:49.192+0000 7f089d761200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T00:49:49.510 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:49 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:49.315+0000 7f089d761200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T00:49:49.510 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:49 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:49.504+0000 7f089d761200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T00:49:50.647 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:50 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:50.298+0000 7f089d761200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T00:49:50.648 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:50 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:50.540+0000 7f089d761200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T00:49:50.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:50 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:50.816+0000 7f089d761200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T00:49:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-30T00:49:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-30T00:49:51.335 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:51 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/4089762409' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T00:49:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "4909f004-ee2f-11ee-b647-cb9ed24678a4", 2024-03-30T00:49:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T00:49:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T00:49:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T00:49:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T00:49:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T00:49:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T00:49:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-30T00:49:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-30T00:49:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T00:49:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T00:49:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-30T00:49:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-30T00:49:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T00:49:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:49:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-30T00:49:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T00:49:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T00:49:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:49:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T00:49:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T00:49:51.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T00:49:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T00:49:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T00:49:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T00:49:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T00:49:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T00:49:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T00:49:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T00:49:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T00:49:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T00:49:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T00:49:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T00:49:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-30T00:49:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T00:49:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T00:49:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:49:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T00:49:38.164631+0000", 2024-03-30T00:49:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T00:49:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:49:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T00:49:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-30T00:49:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.898+0000 7fb0c0800700 1 Processor -- start 2024-03-30T00:49:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.898+0000 7fb0c0800700 1 -- start start 2024-03-30T00:49:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.898+0000 7fb0c0800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b8106510 0x7fb0b8106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:49:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.898+0000 7fb0c0800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b8106f00 con 0x7fb0b8106510 2024-03-30T00:49:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.899+0000 7fb0bde00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b8106510 0x7fb0b8106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:49:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.899+0000 7fb0bde00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b8106510 0x7fb0b8106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42626/0 (socket says 172.21.15.116:42626) 2024-03-30T00:49:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.899+0000 7fb0bde00700 1 -- 172.21.15.116:0/3247084482 learned_addr learned my addr 172.21.15.116:0/3247084482 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:49:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.899+0000 7fb0bde00700 1 -- 172.21.15.116:0/3247084482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b81076c0 con 0x7fb0b8106510 2024-03-30T00:49:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.899+0000 7fb0bde00700 1 --2- 172.21.15.116:0/3247084482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b8106510 0x7fb0b8106930 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac0098a0 tx=0x7fb0ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1570748dbe26b2b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:49:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.899+0000 7fb0bca00700 1 -- 172.21.15.116:0/3247084482 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0ac004160 con 0x7fb0b8106510 2024-03-30T00:49:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.900+0000 7fb0bca00700 1 -- 172.21.15.116:0/3247084482 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0ac0042c0 con 0x7fb0b8106510 2024-03-30T00:49:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.900+0000 7fb0bca00700 1 -- 172.21.15.116:0/3247084482 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0ac011a00 con 0x7fb0b8106510 2024-03-30T00:49:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.900+0000 7fb0c0800700 1 -- 172.21.15.116:0/3247084482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b8106510 msgr2=0x7fb0b8106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:49:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.900+0000 7fb0c0800700 1 --2- 172.21.15.116:0/3247084482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b8106510 0x7fb0b8106930 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac0098a0 tx=0x7fb0ac009bb0 comp rx=0 tx=0).stop 2024-03-30T00:49:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.900+0000 7fb0c0800700 1 -- 172.21.15.116:0/3247084482 shutdown_connections 2024-03-30T00:49:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.900+0000 7fb0c0800700 1 --2- 172.21.15.116:0/3247084482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b8106510 0x7fb0b8106930 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:49:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.900+0000 7fb0c0800700 1 -- 172.21.15.116:0/3247084482 >> 172.21.15.116:0/3247084482 conn(0x7fb0b8101af0 msgr2=0x7fb0b8103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:49:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.900+0000 7fb0c0800700 1 -- 172.21.15.116:0/3247084482 shutdown_connections 2024-03-30T00:49:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.900+0000 7fb0c0800700 1 -- 172.21.15.116:0/3247084482 wait complete. 2024-03-30T00:49:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.901+0000 7fb0c0800700 1 Processor -- start 2024-03-30T00:49:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.902+0000 7fb0c0800700 1 -- start start 2024-03-30T00:49:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.902+0000 7fb0c0800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b8106510 0x7fb0b819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:49:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.902+0000 7fb0c0800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b819c9c0 con 0x7fb0b8106510 2024-03-30T00:49:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.902+0000 7fb0bde00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b8106510 0x7fb0b819c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:49:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.902+0000 7fb0bde00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b8106510 0x7fb0b819c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42638/0 (socket says 172.21.15.116:42638) 2024-03-30T00:49:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.902+0000 7fb0bde00700 1 -- 172.21.15.116:0/4089762409 learned_addr learned my addr 172.21.15.116:0/4089762409 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:49:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.903+0000 7fb0bde00700 1 -- 172.21.15.116:0/4089762409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac009550 con 0x7fb0b8106510 2024-03-30T00:49:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.903+0000 7fb0bde00700 1 --2- 172.21.15.116:0/4089762409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b8106510 0x7fb0b819c480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac011710 tx=0x7fb0ac011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:49:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.903+0000 7fb0b2a00700 1 -- 172.21.15.116:0/4089762409 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0ac003aa0 con 0x7fb0b8106510 2024-03-30T00:49:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.903+0000 7fb0c0800700 1 -- 172.21.15.116:0/4089762409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b819cbc0 con 0x7fb0b8106510 2024-03-30T00:49:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.903+0000 7fb0b2a00700 1 -- 172.21.15.116:0/4089762409 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0ac003c00 con 0x7fb0b8106510 2024-03-30T00:49:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.903+0000 7fb0b2a00700 1 -- 172.21.15.116:0/4089762409 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0ac024800 con 0x7fb0b8106510 2024-03-30T00:49:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.903+0000 7fb0c0800700 1 -- 172.21.15.116:0/4089762409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b819d060 con 0x7fb0b8106510 2024-03-30T00:49:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.903+0000 7fb0b2a00700 1 -- 172.21.15.116:0/4089762409 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb0ac024c60 con 0x7fb0b8106510 2024-03-30T00:49:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.903+0000 7fb0b2a00700 1 -- 172.21.15.116:0/4089762409 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb0ac0193c0 con 0x7fb0b8106510 2024-03-30T00:49:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.904+0000 7fb0c0800700 1 -- 172.21.15.116:0/4089762409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b80623e0 con 0x7fb0b8106510 2024-03-30T00:49:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.907+0000 7fb0b2a00700 1 -- 172.21.15.116:0/4089762409 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fb0ac02de40 con 0x7fb0b8106510 2024-03-30T00:49:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.996+0000 7fb0c0800700 1 -- 172.21.15.116:0/4089762409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb0b810a8f0 con 0x7fb0b8106510 2024-03-30T00:49:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.997+0000 7fb0b2a00700 1 -- 172.21.15.116:0/4089762409 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fb0ac01ac80 con 0x7fb0b8106510 2024-03-30T00:49:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.998+0000 7fb0c0800700 1 -- 172.21.15.116:0/4089762409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b8106510 msgr2=0x7fb0b819c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:49:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.998+0000 7fb0c0800700 1 --2- 172.21.15.116:0/4089762409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b8106510 0x7fb0b819c480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac011710 tx=0x7fb0ac011ad0 comp rx=0 tx=0).stop 2024-03-30T00:49:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.998+0000 7fb0c0800700 1 -- 172.21.15.116:0/4089762409 shutdown_connections 2024-03-30T00:49:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.998+0000 7fb0c0800700 1 --2- 172.21.15.116:0/4089762409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b8106510 0x7fb0b819c480 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:49:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.998+0000 7fb0c0800700 1 -- 172.21.15.116:0/4089762409 >> 172.21.15.116:0/4089762409 conn(0x7fb0b8101af0 msgr2=0x7fb0b8102630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:49:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.998+0000 7fb0c0800700 1 -- 172.21.15.116:0/4089762409 shutdown_connections 2024-03-30T00:49:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:50.998+0000 7fb0c0800700 1 -- 172.21.15.116:0/4089762409 wait complete. 2024-03-30T00:49:51.356 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (3/15)... 2024-03-30T00:49:52.357 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:51 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:51.983+0000 7f089d761200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T00:49:52.673 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:52 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:52.351+0000 7f089d761200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T00:49:52.674 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:52 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:52.466+0000 7f089d761200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T00:49:53.043 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:49:52 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:49:52.716+0000 7f089d761200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T00:49:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:53 smithi116 ceph-mon[38086]: Activating manager daemon x 2024-03-30T00:49:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:53 smithi116 ceph-mon[38086]: mgrmap e2: x(active, starting, since 0.00482839s) 2024-03-30T00:49:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:53 smithi116 ceph-mon[38086]: from='mgr.14100 172.21.15.116:0/1134815118' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T00:49:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:53 smithi116 ceph-mon[38086]: from='mgr.14100 172.21.15.116:0/1134815118' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T00:49:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:53 smithi116 ceph-mon[38086]: from='mgr.14100 172.21.15.116:0/1134815118' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T00:49:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:53 smithi116 ceph-mon[38086]: from='mgr.14100 172.21.15.116:0/1134815118' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:49:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:53 smithi116 ceph-mon[38086]: from='mgr.14100 172.21.15.116:0/1134815118' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T00:49:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:53 smithi116 ceph-mon[38086]: Manager daemon x is now available 2024-03-30T00:49:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:53 smithi116 ceph-mon[38086]: from='mgr.14100 172.21.15.116:0/1134815118' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-30T00:49:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:53 smithi116 ceph-mon[38086]: from='mgr.14100 172.21.15.116:0/1134815118' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-30T00:49:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:53 smithi116 ceph-mon[38086]: from='mgr.14100 172.21.15.116:0/1134815118' entity='mgr.x' 2024-03-30T00:49:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:53 smithi116 ceph-mon[38086]: from='mgr.14100 172.21.15.116:0/1134815118' entity='mgr.x' 2024-03-30T00:49:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:53 smithi116 ceph-mon[38086]: from='mgr.14100 172.21.15.116:0/1134815118' entity='mgr.x' 2024-03-30T00:49:55.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:54 smithi116 ceph-mon[38086]: mgrmap e3: x(active, since 1.00964s) 2024-03-30T00:49:56.115 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:55 smithi116 ceph-mon[38086]: mgrmap e4: x(active, since 2s) 2024-03-30T00:49:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:49:58 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2551014633' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T00:50:00.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-30T00:50:00.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-30T00:50:00.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "4909f004-ee2f-11ee-b647-cb9ed24678a4", 2024-03-30T00:50:00.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T00:50:00.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T00:50:00.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T00:50:00.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T00:50:00.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:50:00.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T00:50:00.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T00:50:00.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-30T00:50:00.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-30T00:50:00.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T00:50:00.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T00:50:00.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-30T00:50:00.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-03-30T00:50:00.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T00:50:00.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:50:00.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-30T00:50:00.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T00:50:00.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:50:00.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T00:50:00.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:50:00.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T00:50:00.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T00:50:00.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T00:50:00.806 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T00:50:00.806 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T00:50:00.806 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T00:50:00.806 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:50:00.806 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T00:50:00.806 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T00:50:00.806 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T00:50:00.806 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T00:50:00.807 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T00:50:00.808 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T00:50:00.808 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T00:50:00.808 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T00:50:00.808 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-30T00:50:00.808 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T00:50:00.808 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:50:00.808 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T00:50:00.808 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T00:50:00.808 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T00:49:38.164631+0000", 2024-03-30T00:50:00.808 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T00:50:00.808 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-30T00:50:00.808 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T00:50:00.808 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-30T00:50:00.808 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.766+0000 7fd91d000700 1 Processor -- start 2024-03-30T00:50:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.766+0000 7fd91d000700 1 -- start start 2024-03-30T00:50:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.767+0000 7fd91d000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91807be00 0x7fd91807a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.767+0000 7fd91d000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91807a7f0 con 0x7fd91807be00 2024-03-30T00:50:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.767+0000 7fd915e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91807be00 0x7fd91807a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.767+0000 7fd915e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91807be00 0x7fd91807a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46158/0 (socket says 172.21.15.116:46158) 2024-03-30T00:50:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.767+0000 7fd915e00700 1 -- 172.21.15.116:0/1365391896 learned_addr learned my addr 172.21.15.116:0/1365391896 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.767+0000 7fd915e00700 1 -- 172.21.15.116:0/1365391896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91807a930 con 0x7fd91807be00 2024-03-30T00:50:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.768+0000 7fd915e00700 1 --2- 172.21.15.116:0/1365391896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91807be00 0x7fd91807a2b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd9080098a0 tx=0x7fd908009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1eb8e6e584f40 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.768+0000 7fd914a00700 1 -- 172.21.15.116:0/1365391896 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd908004160 con 0x7fd91807be00 2024-03-30T00:50:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.768+0000 7fd914a00700 1 -- 172.21.15.116:0/1365391896 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9080042c0 con 0x7fd91807be00 2024-03-30T00:50:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.768+0000 7fd914a00700 1 -- 172.21.15.116:0/1365391896 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd908011a00 con 0x7fd91807be00 2024-03-30T00:50:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.768+0000 7fd91d000700 1 -- 172.21.15.116:0/1365391896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91807be00 msgr2=0x7fd91807a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.768+0000 7fd91d000700 1 --2- 172.21.15.116:0/1365391896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91807be00 0x7fd91807a2b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd9080098a0 tx=0x7fd908009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.769+0000 7fd91d000700 1 -- 172.21.15.116:0/1365391896 shutdown_connections 2024-03-30T00:50:00.810 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.769+0000 7fd91d000700 1 --2- 172.21.15.116:0/1365391896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91807be00 0x7fd91807a2b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:00.810 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.769+0000 7fd91d000700 1 -- 172.21.15.116:0/1365391896 >> 172.21.15.116:0/1365391896 conn(0x7fd918101af0 msgr2=0x7fd918103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:00.810 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.769+0000 7fd91d000700 1 -- 172.21.15.116:0/1365391896 shutdown_connections 2024-03-30T00:50:00.810 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.769+0000 7fd91d000700 1 -- 172.21.15.116:0/1365391896 wait complete. 2024-03-30T00:50:00.810 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.770+0000 7fd91d000700 1 Processor -- start 2024-03-30T00:50:00.810 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.770+0000 7fd91d000700 1 -- start start 2024-03-30T00:50:00.810 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.770+0000 7fd91d000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91807be00 0x7fd91819c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:00.810 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.770+0000 7fd91d000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91819c970 con 0x7fd91807be00 2024-03-30T00:50:00.810 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.770+0000 7fd915e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91807be00 0x7fd91819c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:00.810 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.770+0000 7fd915e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91807be00 0x7fd91819c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46166/0 (socket says 172.21.15.116:46166) 2024-03-30T00:50:00.810 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.770+0000 7fd915e00700 1 -- 172.21.15.116:0/2551014633 learned_addr learned my addr 172.21.15.116:0/2551014633 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:00.810 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.770+0000 7fd915e00700 1 -- 172.21.15.116:0/2551014633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd908009550 con 0x7fd91807be00 2024-03-30T00:50:00.810 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.771+0000 7fd915e00700 1 --2- 172.21.15.116:0/2551014633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91807be00 0x7fd91819c430 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd908011a20 tx=0x7fd908011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.771+0000 7fd90ea00700 1 -- 172.21.15.116:0/2551014633 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd908003af0 con 0x7fd91807be00 2024-03-30T00:50:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.771+0000 7fd90ea00700 1 -- 172.21.15.116:0/2551014633 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd908003c50 con 0x7fd91807be00 2024-03-30T00:50:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.771+0000 7fd91d000700 1 -- 172.21.15.116:0/2551014633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91819cb70 con 0x7fd91807be00 2024-03-30T00:50:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.771+0000 7fd90ea00700 1 -- 172.21.15.116:0/2551014633 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd908019760 con 0x7fd91807be00 2024-03-30T00:50:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.771+0000 7fd91d000700 1 -- 172.21.15.116:0/2551014633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91819d010 con 0x7fd91807be00 2024-03-30T00:50:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.772+0000 7fd90ea00700 1 -- 172.21.15.116:0/2551014633 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 45522+0+0 (secure 0 0 0) 0x7fd90801ab20 con 0x7fd91807be00 2024-03-30T00:50:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.772+0000 7fd91d000700 1 -- 172.21.15.116:0/2551014633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91810c860 con 0x7fd91807be00 2024-03-30T00:50:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.772+0000 7fd90ea00700 1 --2- 172.21.15.116:0/2551014633 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7fd8fc038700 0x7fd8fc03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.773+0000 7fd90ea00700 1 -- 172.21.15.116:0/2551014633 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd90804c970 con 0x7fd91807be00 2024-03-30T00:50:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.773+0000 7fd915400700 1 --2- 172.21.15.116:0/2551014633 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7fd8fc038700 0x7fd8fc03abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.773+0000 7fd915400700 1 --2- 172.21.15.116:0/2551014633 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7fd8fc038700 0x7fd8fc03abc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd904006fd0 tx=0x7fd904006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:57.778+0000 7fd90ea00700 1 -- 172.21.15.116:0/2551014633 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd90801a7b0 con 0x7fd91807be00 2024-03-30T00:50:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:58.063+0000 7fd91d000700 1 -- 172.21.15.116:0/2551014633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd91810cad0 con 0x7fd91807be00 2024-03-30T00:50:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:58.063+0000 7fd90ea00700 1 -- 172.21.15.116:0/2551014633 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fd908023370 con 0x7fd91807be00 2024-03-30T00:50:00.812 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:58.065+0000 7fd91d000700 1 -- 172.21.15.116:0/2551014633 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7fd8fc038700 msgr2=0x7fd8fc03abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:00.812 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:58.065+0000 7fd91d000700 1 --2- 172.21.15.116:0/2551014633 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7fd8fc038700 0x7fd8fc03abc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd904006fd0 tx=0x7fd904006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:00.812 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:58.066+0000 7fd91d000700 1 -- 172.21.15.116:0/2551014633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91807be00 msgr2=0x7fd91819c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:00.812 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:58.066+0000 7fd91d000700 1 --2- 172.21.15.116:0/2551014633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91807be00 0x7fd91819c430 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd908011a20 tx=0x7fd908011b00 comp rx=0 tx=0).stop 2024-03-30T00:50:00.812 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:58.066+0000 7fd91d000700 1 -- 172.21.15.116:0/2551014633 shutdown_connections 2024-03-30T00:50:00.812 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:58.066+0000 7fd91d000700 1 --2- 172.21.15.116:0/2551014633 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7fd8fc038700 0x7fd8fc03abc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:00.812 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:58.066+0000 7fd91d000700 1 --2- 172.21.15.116:0/2551014633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd91807be00 0x7fd91819c430 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:00.812 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:58.066+0000 7fd91d000700 1 -- 172.21.15.116:0/2551014633 >> 172.21.15.116:0/2551014633 conn(0x7fd918101af0 msgr2=0x7fd918102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:00.812 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:58.066+0000 7fd91d000700 1 -- 172.21.15.116:0/2551014633 shutdown_connections 2024-03-30T00:50:00.812 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:49:58.066+0000 7fd91d000700 1 -- 172.21.15.116:0/2551014633 wait complete. 2024-03-30T00:50:00.812 INFO:teuthology.orchestra.run.smithi116.stdout:mgr is available 2024-03-30T00:50:01.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:01 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1482855715' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-30T00:50:01.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-30T00:50:01.723 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [global] 2024-03-30T00:50:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout fsid = 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:50:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-30T00:50:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-30T00:50:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-30T00:50:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-30T00:50:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-30T00:50:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-30T00:50:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-30T00:50:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-30T00:50:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [osd] 2024-03-30T00:50:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-30T00:50:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-30T00:50:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-30T00:50:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-30T00:50:01.725 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-30T00:50:01.725 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.171+0000 7fd816400700 1 Processor -- start 2024-03-30T00:50:01.725 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.172+0000 7fd816400700 1 -- start start 2024-03-30T00:50:01.725 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.172+0000 7fd816400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81007be20 0x7fd81007a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:01.725 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.172+0000 7fd816400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81007a810 con 0x7fd81007be20 2024-03-30T00:50:01.725 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.172+0000 7fd80f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81007be20 0x7fd81007a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:01.725 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.172+0000 7fd80f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81007be20 0x7fd81007a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46180/0 (socket says 172.21.15.116:46180) 2024-03-30T00:50:01.725 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.172+0000 7fd80f400700 1 -- 172.21.15.116:0/3534389561 learned_addr learned my addr 172.21.15.116:0/3534389561 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:01.725 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.173+0000 7fd80f400700 1 -- 172.21.15.116:0/3534389561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81007a950 con 0x7fd81007be20 2024-03-30T00:50:01.725 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.173+0000 7fd80f400700 1 --2- 172.21.15.116:0/3534389561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81007be20 0x7fd81007a2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd8000098a0 tx=0x7fd800009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66aa6ddee381e68 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:01.725 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.173+0000 7fd80ea00700 1 -- 172.21.15.116:0/3534389561 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd800004160 con 0x7fd81007be20 2024-03-30T00:50:01.725 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.173+0000 7fd80ea00700 1 -- 172.21.15.116:0/3534389561 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd8000042c0 con 0x7fd81007be20 2024-03-30T00:50:01.726 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.173+0000 7fd80ea00700 1 -- 172.21.15.116:0/3534389561 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd800011a00 con 0x7fd81007be20 2024-03-30T00:50:01.726 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.174+0000 7fd816400700 1 -- 172.21.15.116:0/3534389561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81007be20 msgr2=0x7fd81007a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:01.726 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.174+0000 7fd816400700 1 --2- 172.21.15.116:0/3534389561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81007be20 0x7fd81007a2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd8000098a0 tx=0x7fd800009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:01.726 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.174+0000 7fd816400700 1 -- 172.21.15.116:0/3534389561 shutdown_connections 2024-03-30T00:50:01.726 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.174+0000 7fd816400700 1 --2- 172.21.15.116:0/3534389561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81007be20 0x7fd81007a2d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:01.726 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.174+0000 7fd816400700 1 -- 172.21.15.116:0/3534389561 >> 172.21.15.116:0/3534389561 conn(0x7fd810101af0 msgr2=0x7fd810103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:01.726 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.174+0000 7fd816400700 1 -- 172.21.15.116:0/3534389561 shutdown_connections 2024-03-30T00:50:01.726 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.174+0000 7fd816400700 1 -- 172.21.15.116:0/3534389561 wait complete. 2024-03-30T00:50:01.726 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.175+0000 7fd816400700 1 Processor -- start 2024-03-30T00:50:01.726 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.175+0000 7fd816400700 1 -- start start 2024-03-30T00:50:01.726 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.175+0000 7fd816400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81007be20 0x7fd81019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:01.726 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.175+0000 7fd816400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81019c960 con 0x7fd81007be20 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.176+0000 7fd80f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81007be20 0x7fd81019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.176+0000 7fd80f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81007be20 0x7fd81019c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46190/0 (socket says 172.21.15.116:46190) 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.176+0000 7fd80f400700 1 -- 172.21.15.116:0/1482855715 learned_addr learned my addr 172.21.15.116:0/1482855715 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.176+0000 7fd80f400700 1 -- 172.21.15.116:0/1482855715 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd800009550 con 0x7fd81007be20 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.176+0000 7fd80f400700 1 --2- 172.21.15.116:0/1482855715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81007be20 0x7fd81019c420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd800011a20 tx=0x7fd800011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.176+0000 7fd80cc00700 1 -- 172.21.15.116:0/1482855715 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd800003af0 con 0x7fd81007be20 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.176+0000 7fd80cc00700 1 -- 172.21.15.116:0/1482855715 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd800003c50 con 0x7fd81007be20 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.176+0000 7fd80cc00700 1 -- 172.21.15.116:0/1482855715 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd800019760 con 0x7fd81007be20 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.176+0000 7fd816400700 1 -- 172.21.15.116:0/1482855715 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81019cb60 con 0x7fd81007be20 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.177+0000 7fd816400700 1 -- 172.21.15.116:0/1482855715 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81019d000 con 0x7fd81007be20 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.178+0000 7fd80cc00700 1 -- 172.21.15.116:0/1482855715 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 45522+0+0 (secure 0 0 0) 0x7fd80001ab20 con 0x7fd81007be20 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.178+0000 7fd816400700 1 -- 172.21.15.116:0/1482855715 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81010c840 con 0x7fd81007be20 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.178+0000 7fd80cc00700 1 --2- 172.21.15.116:0/1482855715 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7fd7f4038700 0x7fd7f403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.178+0000 7fd80cc00700 1 -- 172.21.15.116:0/1482855715 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd80004c860 con 0x7fd81007be20 2024-03-30T00:50:01.727 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.178+0000 7fd807e00700 1 --2- 172.21.15.116:0/1482855715 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7fd7f4038700 0x7fd7f403abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:01.728 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.179+0000 7fd807e00700 1 --2- 172.21.15.116:0/1482855715 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7fd7f4038700 0x7fd7f403abc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8006fd0 tx=0x7fd7f8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:01.728 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.183+0000 7fd80cc00700 1 -- 172.21.15.116:0/1482855715 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd80002dbd0 con 0x7fd81007be20 2024-03-30T00:50:01.728 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.381+0000 7fd816400700 1 -- 172.21.15.116:0/1482855715 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fd81010ca80 con 0x7fd81007be20 2024-03-30T00:50:01.728 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.382+0000 7fd80cc00700 1 -- 172.21.15.116:0/1482855715 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fd80002d7c0 con 0x7fd81007be20 2024-03-30T00:50:01.728 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.384+0000 7fd816400700 1 -- 172.21.15.116:0/1482855715 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7fd7f4038700 msgr2=0x7fd7f403abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:01.728 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.384+0000 7fd816400700 1 --2- 172.21.15.116:0/1482855715 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7fd7f4038700 0x7fd7f403abc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8006fd0 tx=0x7fd7f8006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:01.728 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.384+0000 7fd816400700 1 -- 172.21.15.116:0/1482855715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81007be20 msgr2=0x7fd81019c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:01.728 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.384+0000 7fd816400700 1 --2- 172.21.15.116:0/1482855715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81007be20 0x7fd81019c420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd800011a20 tx=0x7fd800011b00 comp rx=0 tx=0).stop 2024-03-30T00:50:01.728 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.384+0000 7fd816400700 1 -- 172.21.15.116:0/1482855715 shutdown_connections 2024-03-30T00:50:01.728 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.384+0000 7fd816400700 1 --2- 172.21.15.116:0/1482855715 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7fd7f4038700 0x7fd7f403abc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:01.728 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.384+0000 7fd816400700 1 --2- 172.21.15.116:0/1482855715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81007be20 0x7fd81019c420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:01.728 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.384+0000 7fd816400700 1 -- 172.21.15.116:0/1482855715 >> 172.21.15.116:0/1482855715 conn(0x7fd810101af0 msgr2=0x7fd8101026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:01.728 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.385+0000 7fd816400700 1 -- 172.21.15.116:0/1482855715 shutdown_connections 2024-03-30T00:50:01.728 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:01.385+0000 7fd816400700 1 -- 172.21.15.116:0/1482855715 wait complete. 2024-03-30T00:50:01.729 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling cephadm module... 2024-03-30T00:50:03.367 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:03 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2004735339' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-30T00:50:03.367 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:03 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: ignoring --setuser ceph since I am not root 2024-03-30T00:50:03.367 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:03 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: ignoring --setgroup ceph since I am not root 2024-03-30T00:50:03.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.122+0000 7f69e8a00700 1 Processor -- start 2024-03-30T00:50:03.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.122+0000 7f69e8a00700 1 -- start start 2024-03-30T00:50:03.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.122+0000 7f69e8a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69e407beb0 0x7f69e407a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:03.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.122+0000 7f69e8a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e407a8e0 con 0x7f69e407beb0 2024-03-30T00:50:03.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.123+0000 7f69e2800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69e407beb0 0x7f69e407a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:03.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.123+0000 7f69e2800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69e407beb0 0x7f69e407a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46206/0 (socket says 172.21.15.116:46206) 2024-03-30T00:50:03.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.123+0000 7f69e2800700 1 -- 172.21.15.116:0/2368070077 learned_addr learned my addr 172.21.15.116:0/2368070077 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:03.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.123+0000 7f69e2800700 1 -- 172.21.15.116:0/2368070077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e407aa20 con 0x7f69e407beb0 2024-03-30T00:50:03.374 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.123+0000 7f69e2800700 1 --2- 172.21.15.116:0/2368070077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69e407beb0 0x7f69e407a310 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f69d40098a0 tx=0x7f69d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6bccf3850a04668 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:03.374 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.123+0000 7f69e1400700 1 -- 172.21.15.116:0/2368070077 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69d4004160 con 0x7f69e407beb0 2024-03-30T00:50:03.374 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.124+0000 7f69e1400700 1 -- 172.21.15.116:0/2368070077 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69d40042c0 con 0x7f69e407beb0 2024-03-30T00:50:03.374 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.124+0000 7f69e1400700 1 -- 172.21.15.116:0/2368070077 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69d4011a00 con 0x7f69e407beb0 2024-03-30T00:50:03.374 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.124+0000 7f69e8a00700 1 -- 172.21.15.116:0/2368070077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69e407beb0 msgr2=0x7f69e407a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:03.374 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.124+0000 7f69e8a00700 1 --2- 172.21.15.116:0/2368070077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69e407beb0 0x7f69e407a310 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f69d40098a0 tx=0x7f69d4009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:03.374 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.124+0000 7f69e8a00700 1 -- 172.21.15.116:0/2368070077 shutdown_connections 2024-03-30T00:50:03.374 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.124+0000 7f69e8a00700 1 --2- 172.21.15.116:0/2368070077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69e407beb0 0x7f69e407a310 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:03.374 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.124+0000 7f69e8a00700 1 -- 172.21.15.116:0/2368070077 >> 172.21.15.116:0/2368070077 conn(0x7f69e4101870 msgr2=0x7f69e4103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:03.374 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.124+0000 7f69e8a00700 1 -- 172.21.15.116:0/2368070077 shutdown_connections 2024-03-30T00:50:03.374 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.124+0000 7f69e8a00700 1 -- 172.21.15.116:0/2368070077 wait complete. 2024-03-30T00:50:03.374 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.125+0000 7f69e8a00700 1 Processor -- start 2024-03-30T00:50:03.375 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.125+0000 7f69e8a00700 1 -- start start 2024-03-30T00:50:03.375 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.126+0000 7f69e8a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69e407beb0 0x7f69e419c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:03.375 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.126+0000 7f69e8a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e419c7b0 con 0x7f69e407beb0 2024-03-30T00:50:03.375 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.126+0000 7f69e2800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69e407beb0 0x7f69e419c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:03.375 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.126+0000 7f69e2800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69e407beb0 0x7f69e419c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46210/0 (socket says 172.21.15.116:46210) 2024-03-30T00:50:03.375 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.126+0000 7f69e2800700 1 -- 172.21.15.116:0/2004735339 learned_addr learned my addr 172.21.15.116:0/2004735339 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:03.375 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.126+0000 7f69e2800700 1 -- 172.21.15.116:0/2004735339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d4009550 con 0x7f69e407beb0 2024-03-30T00:50:03.375 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.127+0000 7f69e2800700 1 --2- 172.21.15.116:0/2004735339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69e407beb0 0x7f69e419c270 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f69d4011710 tx=0x7f69d4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:03.375 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.127+0000 7f69db400700 1 -- 172.21.15.116:0/2004735339 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69d4003aa0 con 0x7f69e407beb0 2024-03-30T00:50:03.375 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.127+0000 7f69db400700 1 -- 172.21.15.116:0/2004735339 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69d4003c00 con 0x7f69e407beb0 2024-03-30T00:50:03.375 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.127+0000 7f69db400700 1 -- 172.21.15.116:0/2004735339 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69d4024800 con 0x7f69e407beb0 2024-03-30T00:50:03.375 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.127+0000 7f69e8a00700 1 -- 172.21.15.116:0/2004735339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e419c9b0 con 0x7f69e407beb0 2024-03-30T00:50:03.375 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.127+0000 7f69e8a00700 1 -- 172.21.15.116:0/2004735339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e419ce50 con 0x7f69e407beb0 2024-03-30T00:50:03.375 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.128+0000 7f69e8a00700 1 -- 172.21.15.116:0/2004735339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e410c680 con 0x7f69e407beb0 2024-03-30T00:50:03.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.128+0000 7f69db400700 1 -- 172.21.15.116:0/2004735339 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 45522+0+0 (secure 0 0 0) 0x7f69d401abd0 con 0x7f69e407beb0 2024-03-30T00:50:03.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.128+0000 7f69db400700 1 --2- 172.21.15.116:0/2004735339 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f69c8038700 0x7f69c803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:03.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.128+0000 7f69db400700 1 -- 172.21.15.116:0/2004735339 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f69d404c280 con 0x7f69e407beb0 2024-03-30T00:50:03.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.129+0000 7f69e1e00700 1 --2- 172.21.15.116:0/2004735339 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f69c8038700 0x7f69c803abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:03.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.129+0000 7f69e1e00700 1 --2- 172.21.15.116:0/2004735339 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f69c8038700 0x7f69c803abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f69d0006fd0 tx=0x7f69d0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:03.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.133+0000 7f69db400700 1 -- 172.21.15.116:0/2004735339 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f69d402cb40 con 0x7f69e407beb0 2024-03-30T00:50:03.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:02.385+0000 7f69e8a00700 1 -- 172.21.15.116:0/2004735339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f69e410c8c0 con 0x7f69e407beb0 2024-03-30T00:50:03.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.043+0000 7f69db400700 1 -- 172.21.15.116:0/2004735339 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 5) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f69d402c350 con 0x7f69e407beb0 2024-03-30T00:50:03.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.045+0000 7f69db400700 1 -- 172.21.15.116:0/2004735339 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f69d404bfe0 con 0x7f69e407beb0 2024-03-30T00:50:03.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.048+0000 7f69e8a00700 1 -- 172.21.15.116:0/2004735339 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f69c8038700 msgr2=0x7f69c803abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:03.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.048+0000 7f69e8a00700 1 --2- 172.21.15.116:0/2004735339 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f69c8038700 0x7f69c803abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f69d0006fd0 tx=0x7f69d0006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:03.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.048+0000 7f69e8a00700 1 -- 172.21.15.116:0/2004735339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69e407beb0 msgr2=0x7f69e419c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:03.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.048+0000 7f69e8a00700 1 --2- 172.21.15.116:0/2004735339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69e407beb0 0x7f69e419c270 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f69d4011710 tx=0x7f69d4011ad0 comp rx=0 tx=0).stop 2024-03-30T00:50:03.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.048+0000 7f69e8a00700 1 -- 172.21.15.116:0/2004735339 shutdown_connections 2024-03-30T00:50:03.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.048+0000 7f69e8a00700 1 --2- 172.21.15.116:0/2004735339 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f69c8038700 0x7f69c803abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:03.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.048+0000 7f69e8a00700 1 --2- 172.21.15.116:0/2004735339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69e407beb0 0x7f69e419c270 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:03.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.048+0000 7f69e8a00700 1 -- 172.21.15.116:0/2004735339 >> 172.21.15.116:0/2004735339 conn(0x7f69e4101870 msgr2=0x7f69e4102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:03.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.048+0000 7f69e8a00700 1 -- 172.21.15.116:0/2004735339 shutdown_connections 2024-03-30T00:50:03.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.048+0000 7f69e8a00700 1 -- 172.21.15.116:0/2004735339 wait complete. 2024-03-30T00:50:03.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:03 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:03.645+0000 7f30d98cb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T00:50:03.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:03 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:03.770+0000 7f30d98cb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T00:50:03.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:03 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:03.885+0000 7f30d98cb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T00:50:04.324 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-30T00:50:04.324 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:03 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:03.997+0000 7f30d98cb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T00:50:04.325 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:04 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2004735339' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-30T00:50:04.325 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:04 smithi116 ceph-mon[38086]: mgrmap e5: x(active, since 10s) 2024-03-30T00:50:04.325 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:04 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1928477956' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-30T00:50:04.331 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-30T00:50:04.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T00:50:04.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-30T00:50:04.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-30T00:50:04.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-30T00:50:04.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.740+0000 7f7d12600700 1 Processor -- start 2024-03-30T00:50:04.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.741+0000 7f7d12600700 1 -- start start 2024-03-30T00:50:04.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.741+0000 7f7d12600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0c07be20 0x7f7d0c07a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:04.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.741+0000 7f7d12600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c07a810 con 0x7f7d0c07be20 2024-03-30T00:50:04.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.741+0000 7f7d0b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0c07be20 0x7f7d0c07a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:04.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.741+0000 7f7d0b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0c07be20 0x7f7d0c07a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46234/0 (socket says 172.21.15.116:46234) 2024-03-30T00:50:04.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.741+0000 7f7d0b400700 1 -- 172.21.15.116:0/3772213914 learned_addr learned my addr 172.21.15.116:0/3772213914 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:04.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.742+0000 7f7d0b400700 1 -- 172.21.15.116:0/3772213914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0c07a950 con 0x7f7d0c07be20 2024-03-30T00:50:04.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.742+0000 7f7d0b400700 1 --2- 172.21.15.116:0/3772213914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0c07be20 0x7f7d0c07a2d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc0098a0 tx=0x7f7cfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e45d6883d24386 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:04.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.742+0000 7f7d0a000700 1 -- 172.21.15.116:0/3772213914 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7cfc004160 con 0x7f7d0c07be20 2024-03-30T00:50:04.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.742+0000 7f7d0a000700 1 -- 172.21.15.116:0/3772213914 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cfc0042c0 con 0x7f7d0c07be20 2024-03-30T00:50:04.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.742+0000 7f7d0a000700 1 -- 172.21.15.116:0/3772213914 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7cfc011a00 con 0x7f7d0c07be20 2024-03-30T00:50:04.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.743+0000 7f7d12600700 1 -- 172.21.15.116:0/3772213914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0c07be20 msgr2=0x7f7d0c07a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:04.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.743+0000 7f7d12600700 1 --2- 172.21.15.116:0/3772213914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0c07be20 0x7f7d0c07a2d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc0098a0 tx=0x7f7cfc009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:04.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.743+0000 7f7d12600700 1 -- 172.21.15.116:0/3772213914 shutdown_connections 2024-03-30T00:50:04.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.743+0000 7f7d12600700 1 --2- 172.21.15.116:0/3772213914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0c07be20 0x7f7d0c07a2d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:04.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.743+0000 7f7d12600700 1 -- 172.21.15.116:0/3772213914 >> 172.21.15.116:0/3772213914 conn(0x7f7d0c101af0 msgr2=0x7f7d0c103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:04.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.743+0000 7f7d12600700 1 -- 172.21.15.116:0/3772213914 shutdown_connections 2024-03-30T00:50:04.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.743+0000 7f7d12600700 1 -- 172.21.15.116:0/3772213914 wait complete. 2024-03-30T00:50:04.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.744+0000 7f7d12600700 1 Processor -- start 2024-03-30T00:50:04.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.744+0000 7f7d12600700 1 -- start start 2024-03-30T00:50:04.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.744+0000 7f7d12600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0c07be20 0x7f7d0c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:04.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.744+0000 7f7d12600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c19c960 con 0x7f7d0c07be20 2024-03-30T00:50:04.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.745+0000 7f7d0b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0c07be20 0x7f7d0c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:04.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.745+0000 7f7d0b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0c07be20 0x7f7d0c19c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46242/0 (socket says 172.21.15.116:46242) 2024-03-30T00:50:04.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.745+0000 7f7d0b400700 1 -- 172.21.15.116:0/1928477956 learned_addr learned my addr 172.21.15.116:0/1928477956 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:04.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.745+0000 7f7d0b400700 1 -- 172.21.15.116:0/1928477956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cfc009550 con 0x7f7d0c07be20 2024-03-30T00:50:04.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.745+0000 7f7d0b400700 1 --2- 172.21.15.116:0/1928477956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0c07be20 0x7f7d0c19c420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc011a20 tx=0x7f7cfc011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:04.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.745+0000 7f7d03e00700 1 -- 172.21.15.116:0/1928477956 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7cfc003af0 con 0x7f7d0c07be20 2024-03-30T00:50:04.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.745+0000 7f7d03e00700 1 -- 172.21.15.116:0/1928477956 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cfc003c50 con 0x7f7d0c07be20 2024-03-30T00:50:04.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.745+0000 7f7d12600700 1 -- 172.21.15.116:0/1928477956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0c19cb60 con 0x7f7d0c07be20 2024-03-30T00:50:04.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.745+0000 7f7d03e00700 1 -- 172.21.15.116:0/1928477956 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7cfc019760 con 0x7f7d0c07be20 2024-03-30T00:50:04.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.746+0000 7f7d12600700 1 -- 172.21.15.116:0/1928477956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d0c19d000 con 0x7f7d0c07be20 2024-03-30T00:50:04.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.747+0000 7f7d12600700 1 -- 172.21.15.116:0/1928477956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0c10c840 con 0x7f7d0c07be20 2024-03-30T00:50:04.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.747+0000 7f7d03e00700 1 -- 172.21.15.116:0/1928477956 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f7cfc01ab20 con 0x7f7d0c07be20 2024-03-30T00:50:04.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.747+0000 7f7d03e00700 1 --2- 172.21.15.116:0/1928477956 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f7cf00387a0 0x7f7cf003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:04.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.747+0000 7f7d03e00700 1 -- 172.21.15.116:0/1928477956 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7cfc04c860 con 0x7f7d0c07be20 2024-03-30T00:50:04.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.747+0000 7f7d0aa00700 1 -- 172.21.15.116:0/1928477956 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f7cf00387a0 msgr2=0x7f7cf003ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3183509903 2024-03-30T00:50:04.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.748+0000 7f7d0aa00700 1 --2- 172.21.15.116:0/1928477956 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f7cf00387a0 0x7f7cf003ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T00:50:04.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.752+0000 7f7d03e00700 1 -- 172.21.15.116:0/1928477956 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7cfc02db90 con 0x7f7d0c07be20 2024-03-30T00:50:04.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.949+0000 7f7d0aa00700 1 -- 172.21.15.116:0/1928477956 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f7cf00387a0 msgr2=0x7f7cf003ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3183509903 2024-03-30T00:50:04.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:03.949+0000 7f7d0aa00700 1 --2- 172.21.15.116:0/1928477956 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f7cf00387a0 0x7f7cf003ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T00:50:04.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.014+0000 7f7d12600700 1 -- 172.21.15.116:0/1928477956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7d0c19f890 con 0x7f7d0c07be20 2024-03-30T00:50:04.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.015+0000 7f7d03e00700 1 -- 172.21.15.116:0/1928477956 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f7cfc02d960 con 0x7f7d0c07be20 2024-03-30T00:50:04.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.017+0000 7f7d12600700 1 -- 172.21.15.116:0/1928477956 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f7cf00387a0 msgr2=0x7f7cf003ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T00:50:04.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.017+0000 7f7d12600700 1 --2- 172.21.15.116:0/1928477956 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f7cf00387a0 0x7f7cf003ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:04.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.017+0000 7f7d12600700 1 -- 172.21.15.116:0/1928477956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0c07be20 msgr2=0x7f7d0c19c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:04.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.017+0000 7f7d12600700 1 --2- 172.21.15.116:0/1928477956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0c07be20 0x7f7d0c19c420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc011a20 tx=0x7f7cfc011b00 comp rx=0 tx=0).stop 2024-03-30T00:50:04.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.017+0000 7f7d12600700 1 -- 172.21.15.116:0/1928477956 shutdown_connections 2024-03-30T00:50:04.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.017+0000 7f7d12600700 1 --2- 172.21.15.116:0/1928477956 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f7cf00387a0 0x7f7cf003ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:04.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.017+0000 7f7d12600700 1 --2- 172.21.15.116:0/1928477956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d0c07be20 0x7f7d0c19c420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:04.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.017+0000 7f7d12600700 1 -- 172.21.15.116:0/1928477956 >> 172.21.15.116:0/1928477956 conn(0x7f7d0c101af0 msgr2=0x7f7d0c1026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:04.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.018+0000 7f7d12600700 1 -- 172.21.15.116:0/1928477956 shutdown_connections 2024-03-30T00:50:04.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.018+0000 7f7d12600700 1 -- 172.21.15.116:0/1928477956 wait complete. 2024-03-30T00:50:04.337 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for the mgr to restart... 2024-03-30T00:50:04.337 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr epoch 5... 2024-03-30T00:50:05.174 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:04 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:04.864+0000 7f30d98cb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T00:50:05.629 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:05 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:05.181+0000 7f30d98cb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T00:50:05.629 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:05 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:05.290+0000 7f30d98cb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T00:50:05.629 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:05 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:05.429+0000 7f30d98cb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T00:50:05.904 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:05 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:05.623+0000 7f30d98cb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T00:50:05.904 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:05 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:05.789+0000 7f30d98cb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T00:50:06.174 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:05 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:05.898+0000 7f30d98cb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T00:50:06.674 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:06 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:06.360+0000 7f30d98cb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T00:50:07.174 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:06 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:06.841+0000 7f30d98cb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T00:50:07.174 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:07 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:07.089+0000 7f30d98cb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T00:50:07.674 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:07 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:07.210+0000 7f30d98cb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T00:50:08.423 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:08 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:08.150+0000 7f30d98cb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T00:50:08.424 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:08 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:08.274+0000 7f30d98cb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T00:50:08.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:08 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:08.463+0000 7f30d98cb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T00:50:09.674 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:09 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:09.259+0000 7f30d98cb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T00:50:09.674 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:09 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:09.493+0000 7f30d98cb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T00:50:10.174 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:09 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:09.745+0000 7f30d98cb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T00:50:11.174 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:10 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:10.872+0000 7f30d98cb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T00:50:11.613 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:11.241+0000 7f30d98cb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T00:50:11.614 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:11.356+0000 7f30d98cb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T00:50:11.614 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:11.607+0000 7f30d98cb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T00:50:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-mon[38086]: Active manager daemon x restarted 2024-03-30T00:50:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-mon[38086]: Activating manager daemon x 2024-03-30T00:50:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-mon[38086]: osdmap e2: 0 total, 0 up, 0 in 2024-03-30T00:50:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-mon[38086]: mgrmap e6: x(active, starting, since 0.00370308s) 2024-03-30T00:50:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:50:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T00:50:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T00:50:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T00:50:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T00:50:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-mon[38086]: Manager daemon x is now available 2024-03-30T00:50:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' 2024-03-30T00:50:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' 2024-03-30T00:50:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:50:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:11 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:50:12.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:12 smithi116 ceph-mon[38086]: Found migration_current of "None". Setting to last migration. 2024-03-30T00:50:12.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:12 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-30T00:50:12.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:12 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-30T00:50:12.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:12 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' 2024-03-30T00:50:12.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:12 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' 2024-03-30T00:50:12.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:12 smithi116 ceph-mon[38086]: mgrmap e7: x(active, since 1.00715s) 2024-03-30T00:50:12.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-30T00:50:12.957 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-30T00:50:12.957 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-30T00:50:12.957 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-30T00:50:12.957 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.749+0000 7f9078c00700 1 Processor -- start 2024-03-30T00:50:12.957 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.749+0000 7f9078c00700 1 -- start start 2024-03-30T00:50:12.957 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.750+0000 7f9078c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907407be40 0x7f907407a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:12.957 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.750+0000 7f9078c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907407a8c0 con 0x7f907407be40 2024-03-30T00:50:12.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.750+0000 7f9071e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907407be40 0x7f907407a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:12.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.750+0000 7f9071e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907407be40 0x7f907407a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46248/0 (socket says 172.21.15.116:46248) 2024-03-30T00:50:12.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.750+0000 7f9071e00700 1 -- 172.21.15.116:0/3594273953 learned_addr learned my addr 172.21.15.116:0/3594273953 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:12.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.750+0000 7f9071e00700 1 -- 172.21.15.116:0/3594273953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907407aa00 con 0x7f907407be40 2024-03-30T00:50:12.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.751+0000 7f9071e00700 1 --2- 172.21.15.116:0/3594273953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907407be40 0x7f907407a2f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f90640098a0 tx=0x7f9064009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9276ffe4bb5a88fa server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:12.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.751+0000 7f9070a00700 1 -- 172.21.15.116:0/3594273953 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9064004160 con 0x7f907407be40 2024-03-30T00:50:12.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.751+0000 7f9070a00700 1 -- 172.21.15.116:0/3594273953 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90640042c0 con 0x7f907407be40 2024-03-30T00:50:12.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.751+0000 7f9070a00700 1 -- 172.21.15.116:0/3594273953 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9064011a00 con 0x7f907407be40 2024-03-30T00:50:12.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.752+0000 7f9078c00700 1 -- 172.21.15.116:0/3594273953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907407be40 msgr2=0x7f907407a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:12.958 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.752+0000 7f9078c00700 1 --2- 172.21.15.116:0/3594273953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907407be40 0x7f907407a2f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f90640098a0 tx=0x7f9064009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:12.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.752+0000 7f9078c00700 1 -- 172.21.15.116:0/3594273953 shutdown_connections 2024-03-30T00:50:12.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.752+0000 7f9078c00700 1 --2- 172.21.15.116:0/3594273953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907407be40 0x7f907407a2f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:12.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.752+0000 7f9078c00700 1 -- 172.21.15.116:0/3594273953 >> 172.21.15.116:0/3594273953 conn(0x7f9074101af0 msgr2=0x7f9074103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:12.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.752+0000 7f9078c00700 1 -- 172.21.15.116:0/3594273953 shutdown_connections 2024-03-30T00:50:12.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.752+0000 7f9078c00700 1 -- 172.21.15.116:0/3594273953 wait complete. 2024-03-30T00:50:12.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.753+0000 7f9078c00700 1 Processor -- start 2024-03-30T00:50:12.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.753+0000 7f9078c00700 1 -- start start 2024-03-30T00:50:12.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.753+0000 7f9078c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907407be40 0x7f907419c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:12.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.753+0000 7f9078c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907419c940 con 0x7f907407be40 2024-03-30T00:50:12.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.753+0000 7f9071e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907407be40 0x7f907419c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:12.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.753+0000 7f9071e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907407be40 0x7f907419c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46260/0 (socket says 172.21.15.116:46260) 2024-03-30T00:50:12.959 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.753+0000 7f9071e00700 1 -- 172.21.15.116:0/2676874557 learned_addr learned my addr 172.21.15.116:0/2676874557 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:12.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.754+0000 7f9071e00700 1 -- 172.21.15.116:0/2676874557 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9064009550 con 0x7f907407be40 2024-03-30T00:50:12.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.754+0000 7f9071e00700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907407be40 0x7f907419c400 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9064011a20 tx=0x7f9064011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:12.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.754+0000 7f906aa00700 1 -- 172.21.15.116:0/2676874557 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9064003af0 con 0x7f907407be40 2024-03-30T00:50:12.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.754+0000 7f906aa00700 1 -- 172.21.15.116:0/2676874557 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9064003c50 con 0x7f907407be40 2024-03-30T00:50:12.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.754+0000 7f906aa00700 1 -- 172.21.15.116:0/2676874557 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9064019760 con 0x7f907407be40 2024-03-30T00:50:12.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.754+0000 7f9078c00700 1 -- 172.21.15.116:0/2676874557 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907419cb40 con 0x7f907407be40 2024-03-30T00:50:12.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.755+0000 7f9078c00700 1 -- 172.21.15.116:0/2676874557 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907419cfe0 con 0x7f907407be40 2024-03-30T00:50:12.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.755+0000 7f906aa00700 1 -- 172.21.15.116:0/2676874557 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f906401a430 con 0x7f907407be40 2024-03-30T00:50:12.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.756+0000 7f906aa00700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 0x7f905803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:12.960 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.756+0000 7f9078c00700 1 -- 172.21.15.116:0/2676874557 --> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f907410c880 con 0x7f9058038750 2024-03-30T00:50:12.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.756+0000 7f906aa00700 1 -- 172.21.15.116:0/2676874557 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f906404c8f0 con 0x7f907407be40 2024-03-30T00:50:12.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.756+0000 7f9071400700 1 -- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 msgr2=0x7f905803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3183509903 2024-03-30T00:50:12.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.756+0000 7f9071400700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 0x7f905803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T00:50:12.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.957+0000 7f9071400700 1 -- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 msgr2=0x7f905803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3183509903 2024-03-30T00:50:12.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:04.957+0000 7f9071400700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 0x7f905803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T00:50:12.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:05.358+0000 7f9071400700 1 -- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 msgr2=0x7f905803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3183509903 2024-03-30T00:50:12.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:05.358+0000 7f9071400700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 0x7f905803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T00:50:12.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:06.158+0000 7f9071400700 1 -- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 msgr2=0x7f905803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3183509903 2024-03-30T00:50:12.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:06.158+0000 7f9071400700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 0x7f905803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T00:50:12.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:07.759+0000 7f9071400700 1 -- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 msgr2=0x7f905803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3183509903 2024-03-30T00:50:12.961 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:07.760+0000 7f9071400700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 0x7f905803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-30T00:50:12.962 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:10.963+0000 7f9071400700 1 -- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 msgr2=0x7f905803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3183509903 2024-03-30T00:50:12.962 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:10.963+0000 7f9071400700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 0x7f905803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-30T00:50:12.962 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:11.611+0000 7f906aa00700 1 -- 172.21.15.116:0/2676874557 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mgrmap(e 6) v1 ==== 45300+0+0 (secure 0 0 0) 0x7f9064022d90 con 0x7f907407be40 2024-03-30T00:50:12.962 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:11.611+0000 7f906aa00700 1 -- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 msgr2=0x7f905803ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T00:50:12.962 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:11.611+0000 7f906aa00700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 0x7f905803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:12.962 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.613+0000 7f906aa00700 1 -- 172.21.15.116:0/2676874557 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 7) v1 ==== 45427+0+0 (secure 0 0 0) 0x7f906404e220 con 0x7f907407be40 2024-03-30T00:50:12.962 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.613+0000 7f906aa00700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f905803be80 0x7f905803e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:12.962 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.613+0000 7f906aa00700 1 -- 172.21.15.116:0/2676874557 --> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f907410c880 con 0x7f905803be80 2024-03-30T00:50:12.962 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.614+0000 7f9071400700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f905803be80 0x7f905803e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:12.962 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.615+0000 7f9071400700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f905803be80 0x7f905803e270 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9060003a10 tx=0x7f90600092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:12.962 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.616+0000 7f906aa00700 1 -- 172.21.15.116:0/2676874557 <== mgr.14120 v2:172.21.15.116:6800/131150228 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f907410c880 con 0x7f905803be80 2024-03-30T00:50:12.962 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.622+0000 7f9078c00700 1 -- 172.21.15.116:0/2676874557 --> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f907410c9f0 con 0x7f905803be80 2024-03-30T00:50:12.962 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.623+0000 7f906aa00700 1 -- 172.21.15.116:0/2676874557 <== mgr.14120 v2:172.21.15.116:6800/131150228 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f907410c9f0 con 0x7f905803be80 2024-03-30T00:50:12.962 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.623+0000 7f9078c00700 1 -- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f905803be80 msgr2=0x7f905803e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:12.963 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.623+0000 7f9078c00700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f905803be80 0x7f905803e270 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9060003a10 tx=0x7f90600092b0 comp rx=0 tx=0).stop 2024-03-30T00:50:12.963 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.623+0000 7f9078c00700 1 -- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907407be40 msgr2=0x7f907419c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:12.963 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.623+0000 7f9078c00700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907407be40 0x7f907419c400 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9064011a20 tx=0x7f9064011b00 comp rx=0 tx=0).stop 2024-03-30T00:50:12.963 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.623+0000 7f9078c00700 1 -- 172.21.15.116:0/2676874557 shutdown_connections 2024-03-30T00:50:12.963 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.623+0000 7f9078c00700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f905803be80 0x7f905803e270 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:12.963 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.623+0000 7f9078c00700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907407be40 0x7f907419c400 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:12.963 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.623+0000 7f9078c00700 1 --2- 172.21.15.116:0/2676874557 >> [v2:172.21.15.116:6800/3183509903,v1:172.21.15.116:6801/3183509903] conn(0x7f9058038750 0x7f905803ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:12.963 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.623+0000 7f9078c00700 1 -- 172.21.15.116:0/2676874557 >> 172.21.15.116:0/2676874557 conn(0x7f9074101af0 msgr2=0x7f90741026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:12.963 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.623+0000 7f9078c00700 1 -- 172.21.15.116:0/2676874557 shutdown_connections 2024-03-30T00:50:12.963 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:12.623+0000 7f9078c00700 1 -- 172.21.15.116:0/2676874557 wait complete. 2024-03-30T00:50:12.964 INFO:teuthology.orchestra.run.smithi116.stdout:mgr epoch 5 is available 2024-03-30T00:50:12.964 INFO:teuthology.orchestra.run.smithi116.stdout:Setting orchestrator backend to cephadm... 2024-03-30T00:50:13.890 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:13 smithi116 ceph-mon[38086]: [30/Mar/2024:00:50:12] ENGINE Bus STARTING 2024-03-30T00:50:13.890 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:13 smithi116 ceph-mon[38086]: [30/Mar/2024:00:50:12] ENGINE Serving on http://172.21.15.116:8765 2024-03-30T00:50:13.890 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:13 smithi116 ceph-mon[38086]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-30T00:50:13.891 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:13 smithi116 ceph-mon[38086]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-30T00:50:13.891 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:13 smithi116 ceph-mon[38086]: [30/Mar/2024:00:50:12] ENGINE Serving on https://172.21.15.116:7150 2024-03-30T00:50:13.891 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:13 smithi116 ceph-mon[38086]: [30/Mar/2024:00:50:12] ENGINE Bus STARTED 2024-03-30T00:50:13.891 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:13 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:50:13.891 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:13 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' 2024-03-30T00:50:13.891 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:13 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:50:13.896 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.327+0000 7f6f0e200700 1 Processor -- start 2024-03-30T00:50:13.896 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.327+0000 7f6f0e200700 1 -- start start 2024-03-30T00:50:13.896 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.327+0000 7f6f0e200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0807be00 0x7f6f0807a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:13.896 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.327+0000 7f6f0e200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0807a7f0 con 0x7f6f0807be00 2024-03-30T00:50:13.896 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.327+0000 7f6f07400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0807be00 0x7f6f0807a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:13.896 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.327+0000 7f6f07400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0807be00 0x7f6f0807a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59636/0 (socket says 172.21.15.116:59636) 2024-03-30T00:50:13.897 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.327+0000 7f6f07400700 1 -- 172.21.15.116:0/1085334370 learned_addr learned my addr 172.21.15.116:0/1085334370 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:13.897 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.328+0000 7f6f07400700 1 -- 172.21.15.116:0/1085334370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0807a930 con 0x7f6f0807be00 2024-03-30T00:50:13.897 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.328+0000 7f6f07400700 1 --2- 172.21.15.116:0/1085334370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0807be00 0x7f6f0807a2b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80098a0 tx=0x7f6ef8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18b0b36d8f0d425 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:13.897 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.328+0000 7f6f06000700 1 -- 172.21.15.116:0/1085334370 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ef8004160 con 0x7f6f0807be00 2024-03-30T00:50:13.897 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.328+0000 7f6f06000700 1 -- 172.21.15.116:0/1085334370 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ef80042c0 con 0x7f6f0807be00 2024-03-30T00:50:13.897 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.328+0000 7f6f06000700 1 -- 172.21.15.116:0/1085334370 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ef8011a00 con 0x7f6f0807be00 2024-03-30T00:50:13.897 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.329+0000 7f6f0e200700 1 -- 172.21.15.116:0/1085334370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0807be00 msgr2=0x7f6f0807a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:13.897 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.329+0000 7f6f0e200700 1 --2- 172.21.15.116:0/1085334370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0807be00 0x7f6f0807a2b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80098a0 tx=0x7f6ef8009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:13.897 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.329+0000 7f6f0e200700 1 -- 172.21.15.116:0/1085334370 shutdown_connections 2024-03-30T00:50:13.897 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.329+0000 7f6f0e200700 1 --2- 172.21.15.116:0/1085334370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0807be00 0x7f6f0807a2b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:13.897 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.329+0000 7f6f0e200700 1 -- 172.21.15.116:0/1085334370 >> 172.21.15.116:0/1085334370 conn(0x7f6f08101af0 msgr2=0x7f6f08103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:13.897 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.329+0000 7f6f0e200700 1 -- 172.21.15.116:0/1085334370 shutdown_connections 2024-03-30T00:50:13.897 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.329+0000 7f6f0e200700 1 -- 172.21.15.116:0/1085334370 wait complete. 2024-03-30T00:50:13.897 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.330+0000 7f6f0e200700 1 Processor -- start 2024-03-30T00:50:13.898 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.330+0000 7f6f0e200700 1 -- start start 2024-03-30T00:50:13.898 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.330+0000 7f6f0e200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0807be00 0x7f6f08109790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:13.898 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.330+0000 7f6f0e200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f08109cd0 con 0x7f6f0807be00 2024-03-30T00:50:13.898 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.330+0000 7f6f07400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0807be00 0x7f6f08109790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:13.898 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.331+0000 7f6f07400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0807be00 0x7f6f08109790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59644/0 (socket says 172.21.15.116:59644) 2024-03-30T00:50:13.898 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.331+0000 7f6f07400700 1 -- 172.21.15.116:0/4293063716 learned_addr learned my addr 172.21.15.116:0/4293063716 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:13.898 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.331+0000 7f6f07400700 1 -- 172.21.15.116:0/4293063716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef8009550 con 0x7f6f0807be00 2024-03-30T00:50:13.898 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.331+0000 7f6f07400700 1 --2- 172.21.15.116:0/4293063716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0807be00 0x7f6f08109790 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8011a20 tx=0x7f6ef8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:13.898 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.331+0000 7f6effe00700 1 -- 172.21.15.116:0/4293063716 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ef8003af0 con 0x7f6f0807be00 2024-03-30T00:50:13.898 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.331+0000 7f6effe00700 1 -- 172.21.15.116:0/4293063716 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ef8003c50 con 0x7f6f0807be00 2024-03-30T00:50:13.898 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.331+0000 7f6f0e200700 1 -- 172.21.15.116:0/4293063716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f08109ed0 con 0x7f6f0807be00 2024-03-30T00:50:13.898 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.331+0000 7f6effe00700 1 -- 172.21.15.116:0/4293063716 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ef8019760 con 0x7f6f0807be00 2024-03-30T00:50:13.898 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.331+0000 7f6f0e200700 1 -- 172.21.15.116:0/4293063716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f08106320 con 0x7f6f0807be00 2024-03-30T00:50:13.898 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.333+0000 7f6f0e200700 1 -- 172.21.15.116:0/4293063716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0810c860 con 0x7f6f0807be00 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.333+0000 7f6effe00700 1 -- 172.21.15.116:0/4293063716 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 7) v1 ==== 45427+0+0 (secure 0 0 0) 0x7f6ef801ab20 con 0x7f6f0807be00 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.333+0000 7f6effe00700 1 --2- 172.21.15.116:0/4293063716 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f6eec038620 0x7f6eec03aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.333+0000 7f6effe00700 1 -- 172.21.15.116:0/4293063716 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6ef804c7a0 con 0x7f6f0807be00 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.333+0000 7f6f06a00700 1 --2- 172.21.15.116:0/4293063716 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f6eec038620 0x7f6eec03aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.334+0000 7f6f06a00700 1 --2- 172.21.15.116:0/4293063716 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f6eec038620 0x7f6eec03aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4006fd0 tx=0x7f6ef4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.338+0000 7f6effe00700 1 -- 172.21.15.116:0/4293063716 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6ef801a7b0 con 0x7f6f0807be00 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.554+0000 7f6f0e200700 1 -- 172.21.15.116:0/4293063716 --> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f08106a80 con 0x7f6eec038620 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.564+0000 7f6effe00700 1 -- 172.21.15.116:0/4293063716 <== mgr.14120 v2:172.21.15.116:6800/131150228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6f08106a80 con 0x7f6eec038620 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.566+0000 7f6f0e200700 1 -- 172.21.15.116:0/4293063716 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f6eec038620 msgr2=0x7f6eec03aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.566+0000 7f6f0e200700 1 --2- 172.21.15.116:0/4293063716 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f6eec038620 0x7f6eec03aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4006fd0 tx=0x7f6ef4006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.567+0000 7f6f0e200700 1 -- 172.21.15.116:0/4293063716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0807be00 msgr2=0x7f6f08109790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.567+0000 7f6f0e200700 1 --2- 172.21.15.116:0/4293063716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0807be00 0x7f6f08109790 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8011a20 tx=0x7f6ef8011b00 comp rx=0 tx=0).stop 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.567+0000 7f6f0e200700 1 -- 172.21.15.116:0/4293063716 shutdown_connections 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.567+0000 7f6f0e200700 1 --2- 172.21.15.116:0/4293063716 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f6eec038620 0x7f6eec03aae0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:13.899 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.567+0000 7f6f0e200700 1 --2- 172.21.15.116:0/4293063716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0807be00 0x7f6f08109790 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:13.900 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.567+0000 7f6f0e200700 1 -- 172.21.15.116:0/4293063716 >> 172.21.15.116:0/4293063716 conn(0x7f6f08101af0 msgr2=0x7f6f08102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:13.900 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.567+0000 7f6f0e200700 1 -- 172.21.15.116:0/4293063716 shutdown_connections 2024-03-30T00:50:13.900 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:13.567+0000 7f6f0e200700 1 -- 172.21.15.116:0/4293063716 wait complete. 2024-03-30T00:50:14.831 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:14 smithi116 ceph-mon[38086]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:50:14.831 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:14 smithi116 ceph-mon[38086]: mgrmap e8: x(active, since 2s) 2024-03-30T00:50:14.836 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-30T00:50:14.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.285+0000 7f4d8cc00700 1 Processor -- start 2024-03-30T00:50:14.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.285+0000 7f4d8cc00700 1 -- start start 2024-03-30T00:50:14.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.286+0000 7f4d8cc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8807be40 0x7f4d8807a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:14.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.286+0000 7f4d8cc00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8807a8c0 con 0x7f4d8807be40 2024-03-30T00:50:14.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.286+0000 7f4d85e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8807be40 0x7f4d8807a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:14.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.286+0000 7f4d85e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8807be40 0x7f4d8807a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59646/0 (socket says 172.21.15.116:59646) 2024-03-30T00:50:14.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.286+0000 7f4d85e00700 1 -- 172.21.15.116:0/4100806413 learned_addr learned my addr 172.21.15.116:0/4100806413 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:14.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.286+0000 7f4d85e00700 1 -- 172.21.15.116:0/4100806413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8807aa00 con 0x7f4d8807be40 2024-03-30T00:50:14.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.287+0000 7f4d85e00700 1 --2- 172.21.15.116:0/4100806413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8807be40 0x7f4d8807a2f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d780098a0 tx=0x7f4d78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75806dd8af578c80 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:14.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.287+0000 7f4d85400700 1 -- 172.21.15.116:0/4100806413 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d78004160 con 0x7f4d8807be40 2024-03-30T00:50:14.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.287+0000 7f4d85400700 1 -- 172.21.15.116:0/4100806413 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d780042c0 con 0x7f4d8807be40 2024-03-30T00:50:14.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.287+0000 7f4d85400700 1 -- 172.21.15.116:0/4100806413 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d78011a00 con 0x7f4d8807be40 2024-03-30T00:50:14.837 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.287+0000 7f4d8cc00700 1 -- 172.21.15.116:0/4100806413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8807be40 msgr2=0x7f4d8807a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:14.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.287+0000 7f4d8cc00700 1 --2- 172.21.15.116:0/4100806413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8807be40 0x7f4d8807a2f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d780098a0 tx=0x7f4d78009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:14.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.287+0000 7f4d8cc00700 1 -- 172.21.15.116:0/4100806413 shutdown_connections 2024-03-30T00:50:14.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.287+0000 7f4d8cc00700 1 --2- 172.21.15.116:0/4100806413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8807be40 0x7f4d8807a2f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:14.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.287+0000 7f4d8cc00700 1 -- 172.21.15.116:0/4100806413 >> 172.21.15.116:0/4100806413 conn(0x7f4d88101af0 msgr2=0x7f4d88103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:14.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.288+0000 7f4d8cc00700 1 -- 172.21.15.116:0/4100806413 shutdown_connections 2024-03-30T00:50:14.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.288+0000 7f4d8cc00700 1 -- 172.21.15.116:0/4100806413 wait complete. 2024-03-30T00:50:14.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.289+0000 7f4d8cc00700 1 Processor -- start 2024-03-30T00:50:14.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.289+0000 7f4d8cc00700 1 -- start start 2024-03-30T00:50:14.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.289+0000 7f4d8cc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8807be40 0x7f4d8819c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:14.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.289+0000 7f4d8cc00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8819c940 con 0x7f4d8807be40 2024-03-30T00:50:14.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.289+0000 7f4d85e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8807be40 0x7f4d8819c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:14.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.289+0000 7f4d85e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8807be40 0x7f4d8819c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59662/0 (socket says 172.21.15.116:59662) 2024-03-30T00:50:14.838 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.289+0000 7f4d85e00700 1 -- 172.21.15.116:0/1331963135 learned_addr learned my addr 172.21.15.116:0/1331963135 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:14.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.290+0000 7f4d85e00700 1 -- 172.21.15.116:0/1331963135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d78009550 con 0x7f4d8807be40 2024-03-30T00:50:14.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.290+0000 7f4d85e00700 1 --2- 172.21.15.116:0/1331963135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8807be40 0x7f4d8819c400 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4d78011a20 tx=0x7f4d78011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:14.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.290+0000 7f4d7ea00700 1 -- 172.21.15.116:0/1331963135 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d78003af0 con 0x7f4d8807be40 2024-03-30T00:50:14.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.290+0000 7f4d7ea00700 1 -- 172.21.15.116:0/1331963135 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d78003c50 con 0x7f4d8807be40 2024-03-30T00:50:14.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.290+0000 7f4d7ea00700 1 -- 172.21.15.116:0/1331963135 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d78019760 con 0x7f4d8807be40 2024-03-30T00:50:14.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.290+0000 7f4d8cc00700 1 -- 172.21.15.116:0/1331963135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8819cb40 con 0x7f4d8807be40 2024-03-30T00:50:14.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.290+0000 7f4d8cc00700 1 -- 172.21.15.116:0/1331963135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8819cfe0 con 0x7f4d8807be40 2024-03-30T00:50:14.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.291+0000 7f4d8cc00700 1 -- 172.21.15.116:0/1331963135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d8810c880 con 0x7f4d8807be40 2024-03-30T00:50:14.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.291+0000 7f4d7ea00700 1 -- 172.21.15.116:0/1331963135 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 7) v1 ==== 45427+0+0 (secure 0 0 0) 0x7f4d7801ab20 con 0x7f4d8807be40 2024-03-30T00:50:14.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.292+0000 7f4d7ea00700 1 --2- 172.21.15.116:0/1331963135 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f4d6c038620 0x7f4d6c03aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:14.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.292+0000 7f4d7ea00700 1 -- 172.21.15.116:0/1331963135 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4d7804c730 con 0x7f4d8807be40 2024-03-30T00:50:14.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.292+0000 7f4d7fe00700 1 --2- 172.21.15.116:0/1331963135 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f4d6c038620 0x7f4d6c03aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:14.839 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.293+0000 7f4d7fe00700 1 --2- 172.21.15.116:0/1331963135 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f4d6c038620 0x7f4d6c03aae0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4d70006fd0 tx=0x7f4d70006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.297+0000 7f4d7ea00700 1 -- 172.21.15.116:0/1331963135 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4d7802db90 con 0x7f4d8807be40 2024-03-30T00:50:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.497+0000 7f4d8cc00700 1 -- 172.21.15.116:0/1331963135 --> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d88102dc0 con 0x7f4d6c038620 2024-03-30T00:50:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.498+0000 7f4d7ea00700 1 -- 172.21.15.116:0/1331963135 <== mgr.14120 v2:172.21.15.116:6800/131150228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f4d88102dc0 con 0x7f4d6c038620 2024-03-30T00:50:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.500+0000 7f4d8cc00700 1 -- 172.21.15.116:0/1331963135 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f4d6c038620 msgr2=0x7f4d6c03aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.500+0000 7f4d8cc00700 1 --2- 172.21.15.116:0/1331963135 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f4d6c038620 0x7f4d6c03aae0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4d70006fd0 tx=0x7f4d70006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.500+0000 7f4d8cc00700 1 -- 172.21.15.116:0/1331963135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8807be40 msgr2=0x7f4d8819c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.500+0000 7f4d8cc00700 1 --2- 172.21.15.116:0/1331963135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8807be40 0x7f4d8819c400 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4d78011a20 tx=0x7f4d78011b00 comp rx=0 tx=0).stop 2024-03-30T00:50:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.500+0000 7f4d8cc00700 1 -- 172.21.15.116:0/1331963135 shutdown_connections 2024-03-30T00:50:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.500+0000 7f4d8cc00700 1 --2- 172.21.15.116:0/1331963135 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f4d6c038620 0x7f4d6c03aae0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.500+0000 7f4d8cc00700 1 --2- 172.21.15.116:0/1331963135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8807be40 0x7f4d8819c400 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.500+0000 7f4d8cc00700 1 -- 172.21.15.116:0/1331963135 >> 172.21.15.116:0/1331963135 conn(0x7f4d88101af0 msgr2=0x7f4d881026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.500+0000 7f4d8cc00700 1 -- 172.21.15.116:0/1331963135 shutdown_connections 2024-03-30T00:50:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:14.500+0000 7f4d8cc00700 1 -- 172.21.15.116:0/1331963135 wait complete. 2024-03-30T00:50:14.840 INFO:teuthology.orchestra.run.smithi116.stdout:Generating ssh key... 2024-03-30T00:50:15.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-mon[38086]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:50:15.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: Generating public/private rsa key pair. 2024-03-30T00:50:15.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: Your identification has been saved in /tmp/tmpouvz1mo_/key. 2024-03-30T00:50:15.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: Your public key has been saved in /tmp/tmpouvz1mo_/key.pub. 2024-03-30T00:50:15.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: The key fingerprint is: 2024-03-30T00:50:15.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: SHA256:Iv6D6Rlv+j02AwchKRfw2L5qLzGGFbu1hJEJ2O/7S0U ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:50:15.925 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: The key's randomart image is: 2024-03-30T00:50:15.925 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: +---[RSA 3072]----+ 2024-03-30T00:50:15.925 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: |oo.+.o | 2024-03-30T00:50:15.925 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: |. B++ . | 2024-03-30T00:50:15.925 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: | .Oo. .E | 2024-03-30T00:50:15.925 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: | +.+ .. | 2024-03-30T00:50:15.925 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: | o =o...S | 2024-03-30T00:50:15.925 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: |. =.oo.o. | 2024-03-30T00:50:15.925 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: | . o++.o | 2024-03-30T00:50:15.925 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: | o.+*o.= | 2024-03-30T00:50:15.925 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: | ..+==*+.+ | 2024-03-30T00:50:15.925 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: +----[SHA256]-----+ 2024-03-30T00:50:16.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.199+0000 7fbe4a000700 1 Processor -- start 2024-03-30T00:50:16.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.199+0000 7fbe4a000700 1 -- start start 2024-03-30T00:50:16.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.200+0000 7fbe4a000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4407be20 0x7fbe4407a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:16.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.200+0000 7fbe4a000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4407a810 con 0x7fbe4407be20 2024-03-30T00:50:16.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.200+0000 7fbe43400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4407be20 0x7fbe4407a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:16.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.200+0000 7fbe43400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4407be20 0x7fbe4407a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35896/0 (socket says 172.21.15.116:35896) 2024-03-30T00:50:16.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.200+0000 7fbe43400700 1 -- 172.21.15.116:0/4229287776 learned_addr learned my addr 172.21.15.116:0/4229287776 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:16.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.200+0000 7fbe43400700 1 -- 172.21.15.116:0/4229287776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4407a950 con 0x7fbe4407be20 2024-03-30T00:50:16.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.201+0000 7fbe43400700 1 --2- 172.21.15.116:0/4229287776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4407be20 0x7fbe4407a2d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbe340098a0 tx=0x7fbe34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8713361ce0408fd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:16.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.201+0000 7fbe42a00700 1 -- 172.21.15.116:0/4229287776 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe34004160 con 0x7fbe4407be20 2024-03-30T00:50:16.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.201+0000 7fbe42a00700 1 -- 172.21.15.116:0/4229287776 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe340042c0 con 0x7fbe4407be20 2024-03-30T00:50:16.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.201+0000 7fbe42a00700 1 -- 172.21.15.116:0/4229287776 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe34011a00 con 0x7fbe4407be20 2024-03-30T00:50:16.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.201+0000 7fbe4a000700 1 -- 172.21.15.116:0/4229287776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4407be20 msgr2=0x7fbe4407a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:16.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.201+0000 7fbe4a000700 1 --2- 172.21.15.116:0/4229287776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4407be20 0x7fbe4407a2d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbe340098a0 tx=0x7fbe34009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:16.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.201+0000 7fbe4a000700 1 -- 172.21.15.116:0/4229287776 shutdown_connections 2024-03-30T00:50:16.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.201+0000 7fbe4a000700 1 --2- 172.21.15.116:0/4229287776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4407be20 0x7fbe4407a2d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:16.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.201+0000 7fbe4a000700 1 -- 172.21.15.116:0/4229287776 >> 172.21.15.116:0/4229287776 conn(0x7fbe44101af0 msgr2=0x7fbe44103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:16.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.202+0000 7fbe4a000700 1 -- 172.21.15.116:0/4229287776 shutdown_connections 2024-03-30T00:50:16.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.202+0000 7fbe4a000700 1 -- 172.21.15.116:0/4229287776 wait complete. 2024-03-30T00:50:16.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.203+0000 7fbe4a000700 1 Processor -- start 2024-03-30T00:50:16.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.203+0000 7fbe4a000700 1 -- start start 2024-03-30T00:50:16.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.204+0000 7fbe4a000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4407be20 0x7fbe4419c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:16.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.204+0000 7fbe4a000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4419c960 con 0x7fbe4407be20 2024-03-30T00:50:16.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.204+0000 7fbe43400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4407be20 0x7fbe4419c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.204+0000 7fbe43400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4407be20 0x7fbe4419c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35902/0 (socket says 172.21.15.116:35902) 2024-03-30T00:50:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.204+0000 7fbe43400700 1 -- 172.21.15.116:0/2238269483 learned_addr learned my addr 172.21.15.116:0/2238269483 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.204+0000 7fbe43400700 1 -- 172.21.15.116:0/2238269483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe34009550 con 0x7fbe4407be20 2024-03-30T00:50:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.204+0000 7fbe43400700 1 --2- 172.21.15.116:0/2238269483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4407be20 0x7fbe4419c420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbe34011a20 tx=0x7fbe34011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.204+0000 7fbe40c00700 1 -- 172.21.15.116:0/2238269483 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe34003af0 con 0x7fbe4407be20 2024-03-30T00:50:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.205+0000 7fbe40c00700 1 -- 172.21.15.116:0/2238269483 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe34003c50 con 0x7fbe4407be20 2024-03-30T00:50:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.205+0000 7fbe4a000700 1 -- 172.21.15.116:0/2238269483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4419cb60 con 0x7fbe4407be20 2024-03-30T00:50:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.205+0000 7fbe40c00700 1 -- 172.21.15.116:0/2238269483 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe34019760 con 0x7fbe4407be20 2024-03-30T00:50:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.205+0000 7fbe4a000700 1 -- 172.21.15.116:0/2238269483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4419d000 con 0x7fbe4407be20 2024-03-30T00:50:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.206+0000 7fbe40c00700 1 -- 172.21.15.116:0/2238269483 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7fbe340199a0 con 0x7fbe4407be20 2024-03-30T00:50:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.206+0000 7fbe4a000700 1 -- 172.21.15.116:0/2238269483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe4410c840 con 0x7fbe4407be20 2024-03-30T00:50:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.206+0000 7fbe40c00700 1 --2- 172.21.15.116:0/2238269483 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fbe28038700 0x7fbe2803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.206+0000 7fbe40c00700 1 -- 172.21.15.116:0/2238269483 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbe3404cb10 con 0x7fbe4407be20 2024-03-30T00:50:16.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.206+0000 7fbe3be00700 1 --2- 172.21.15.116:0/2238269483 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fbe28038700 0x7fbe2803abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:16.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.207+0000 7fbe3be00700 1 --2- 172.21.15.116:0/2238269483 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fbe28038700 0x7fbe2803abc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c006fd0 tx=0x7fbe2c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:16.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.211+0000 7fbe40c00700 1 -- 172.21.15.116:0/2238269483 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbe34018db0 con 0x7fbe4407be20 2024-03-30T00:50:16.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.410+0000 7fbe4a000700 1 -- 172.21.15.116:0/2238269483 --> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe44102dc0 con 0x7fbe28038700 2024-03-30T00:50:16.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.695+0000 7fbe40c00700 1 -- 172.21.15.116:0/2238269483 <== mgr.14120 v2:172.21.15.116:6800/131150228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbe44102dc0 con 0x7fbe28038700 2024-03-30T00:50:16.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.698+0000 7fbe4a000700 1 -- 172.21.15.116:0/2238269483 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fbe28038700 msgr2=0x7fbe2803abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:16.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.698+0000 7fbe4a000700 1 --2- 172.21.15.116:0/2238269483 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fbe28038700 0x7fbe2803abc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c006fd0 tx=0x7fbe2c006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:16.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.698+0000 7fbe4a000700 1 -- 172.21.15.116:0/2238269483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4407be20 msgr2=0x7fbe4419c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:16.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.698+0000 7fbe4a000700 1 --2- 172.21.15.116:0/2238269483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4407be20 0x7fbe4419c420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbe34011a20 tx=0x7fbe34011b00 comp rx=0 tx=0).stop 2024-03-30T00:50:16.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.698+0000 7fbe4a000700 1 -- 172.21.15.116:0/2238269483 shutdown_connections 2024-03-30T00:50:16.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.698+0000 7fbe4a000700 1 --2- 172.21.15.116:0/2238269483 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fbe28038700 0x7fbe2803abc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:16.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.698+0000 7fbe4a000700 1 --2- 172.21.15.116:0/2238269483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4407be20 0x7fbe4419c420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:16.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.698+0000 7fbe4a000700 1 -- 172.21.15.116:0/2238269483 >> 172.21.15.116:0/2238269483 conn(0x7fbe44101af0 msgr2=0x7fbe441026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:16.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.698+0000 7fbe4a000700 1 -- 172.21.15.116:0/2238269483 shutdown_connections 2024-03-30T00:50:16.242 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:15.698+0000 7fbe4a000700 1 -- 172.21.15.116:0/2238269483 wait complete. 2024-03-30T00:50:17.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:16 smithi116 ceph-mon[38086]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:50:17.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:16 smithi116 ceph-mon[38086]: Generating ssh key... 2024-03-30T00:50:17.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:16 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' 2024-03-30T00:50:17.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:16 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' 2024-03-30T00:50:17.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWdt11SKUN0YqjB3gX13kMmn3vKU8S7g8aBJwP57XmZ7TbaSuqA7qbYz3zAXfwrGquzHZs+X6BAJ4Ps3NbBspBUvFcOj/qWdjtUUvf/EqxE+EwNPHiNzLy4HdTwrkoA6nMpVWBDL5B39F11cBfDGWUg7wZWsyySUw4d0T1kyxKip4eCPKLY3EzLrYcqCb038WKF4dNIfFjlSK622Cz2mXsqXIhcv6FO9uujmyqvQ6qrq/2RAcvm2/h67ZtK1scnPxb5PPe5zYU7EkoE2tXXH1V50RJ9EnbrSvo2rNLnc9JAdXKUx6p9b4HowSwS/xidyHgOhFqqfBCXL5s2gWoSpWq+3mmKqJr+kakY/t2YTBYzPOOTJb90Jntbh/Z0OsTiwl1Iyq/8w0BNGs9g8cyvppvL9oZdfk5p72xRHjePqZdRfLOtQxjg6/Jz1DRQfDvkLMq7VWTET2GFHys30FKtDoEhxA+IZX48i0b9Tb/0bw5av2Rb/eK/TrJ0CuQIsiQPyU= ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:50:17.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.608+0000 7fbb24000700 1 Processor -- start 2024-03-30T00:50:17.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.608+0000 7fbb24000700 1 -- start start 2024-03-30T00:50:17.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.608+0000 7fbb24000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c106500 0x7fbb1c106920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:17.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.608+0000 7fbb24000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c106ef0 con 0x7fbb1c106500 2024-03-30T00:50:17.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.608+0000 7fbb21600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c106500 0x7fbb1c106920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:17.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.609+0000 7fbb21600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c106500 0x7fbb1c106920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35904/0 (socket says 172.21.15.116:35904) 2024-03-30T00:50:17.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.609+0000 7fbb21600700 1 -- 172.21.15.116:0/3203246770 learned_addr learned my addr 172.21.15.116:0/3203246770 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:17.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.609+0000 7fbb21600700 1 -- 172.21.15.116:0/3203246770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1c1076b0 con 0x7fbb1c106500 2024-03-30T00:50:17.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.609+0000 7fbb21600700 1 --2- 172.21.15.116:0/3203246770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c106500 0x7fbb1c106920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbb100098a0 tx=0x7fbb10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a777605e7edde98 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:17.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.609+0000 7fbb1be00700 1 -- 172.21.15.116:0/3203246770 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb10004160 con 0x7fbb1c106500 2024-03-30T00:50:17.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.609+0000 7fbb1be00700 1 -- 172.21.15.116:0/3203246770 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb100042c0 con 0x7fbb1c106500 2024-03-30T00:50:17.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.609+0000 7fbb1be00700 1 -- 172.21.15.116:0/3203246770 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb10011a00 con 0x7fbb1c106500 2024-03-30T00:50:17.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.610+0000 7fbb24000700 1 -- 172.21.15.116:0/3203246770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c106500 msgr2=0x7fbb1c106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:17.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.610+0000 7fbb24000700 1 --2- 172.21.15.116:0/3203246770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c106500 0x7fbb1c106920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbb100098a0 tx=0x7fbb10009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:17.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.610+0000 7fbb24000700 1 -- 172.21.15.116:0/3203246770 shutdown_connections 2024-03-30T00:50:17.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.610+0000 7fbb24000700 1 --2- 172.21.15.116:0/3203246770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c106500 0x7fbb1c106920 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:17.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.610+0000 7fbb24000700 1 -- 172.21.15.116:0/3203246770 >> 172.21.15.116:0/3203246770 conn(0x7fbb1c101ac0 msgr2=0x7fbb1c103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:17.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.610+0000 7fbb24000700 1 -- 172.21.15.116:0/3203246770 shutdown_connections 2024-03-30T00:50:17.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.610+0000 7fbb24000700 1 -- 172.21.15.116:0/3203246770 wait complete. 2024-03-30T00:50:17.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.611+0000 7fbb24000700 1 Processor -- start 2024-03-30T00:50:17.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.611+0000 7fbb24000700 1 -- start start 2024-03-30T00:50:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.612+0000 7fbb24000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c106500 0x7fbb1c19c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.612+0000 7fbb24000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c19c930 con 0x7fbb1c106500 2024-03-30T00:50:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.612+0000 7fbb21600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c106500 0x7fbb1c19c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.612+0000 7fbb21600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c106500 0x7fbb1c19c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35910/0 (socket says 172.21.15.116:35910) 2024-03-30T00:50:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.612+0000 7fbb21600700 1 -- 172.21.15.116:0/3064055955 learned_addr learned my addr 172.21.15.116:0/3064055955 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.612+0000 7fbb21600700 1 -- 172.21.15.116:0/3064055955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb10009550 con 0x7fbb1c106500 2024-03-30T00:50:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.612+0000 7fbb21600700 1 --2- 172.21.15.116:0/3064055955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c106500 0x7fbb1c19c3f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009130 tx=0x7fbb10011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.613+0000 7fbb1a000700 1 -- 172.21.15.116:0/3064055955 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb10003ab0 con 0x7fbb1c106500 2024-03-30T00:50:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.613+0000 7fbb24000700 1 -- 172.21.15.116:0/3064055955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1c19cb30 con 0x7fbb1c106500 2024-03-30T00:50:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.613+0000 7fbb1a000700 1 -- 172.21.15.116:0/3064055955 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb10003c10 con 0x7fbb1c106500 2024-03-30T00:50:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.613+0000 7fbb1a000700 1 -- 172.21.15.116:0/3064055955 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb100247d0 con 0x7fbb1c106500 2024-03-30T00:50:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.613+0000 7fbb24000700 1 -- 172.21.15.116:0/3064055955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1c19cfd0 con 0x7fbb1c106500 2024-03-30T00:50:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.614+0000 7fbb1a000700 1 -- 172.21.15.116:0/3064055955 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7fbb1001a430 con 0x7fbb1c106500 2024-03-30T00:50:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.614+0000 7fbb24000700 1 -- 172.21.15.116:0/3064055955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb1c10a8e0 con 0x7fbb1c106500 2024-03-30T00:50:17.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.614+0000 7fbb1a000700 1 --2- 172.21.15.116:0/3064055955 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fbb04038700 0x7fbb0403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:17.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.614+0000 7fbb1a000700 1 -- 172.21.15.116:0/3064055955 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbb1004c8a0 con 0x7fbb1c106500 2024-03-30T00:50:17.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.614+0000 7fbb20c00700 1 --2- 172.21.15.116:0/3064055955 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fbb04038700 0x7fbb0403abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:17.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.615+0000 7fbb20c00700 1 --2- 172.21.15.116:0/3064055955 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fbb04038700 0x7fbb0403abc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbb08006fd0 tx=0x7fbb08006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:17.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.619+0000 7fbb1a000700 1 -- 172.21.15.116:0/3064055955 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbb10018b40 con 0x7fbb1c106500 2024-03-30T00:50:17.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.825+0000 7fbb24000700 1 -- 172.21.15.116:0/3064055955 --> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb1c19f920 con 0x7fbb04038700 2024-03-30T00:50:17.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.826+0000 7fbb1a000700 1 -- 172.21.15.116:0/3064055955 <== mgr.14120 v2:172.21.15.116:6800/131150228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fbb1c19f920 con 0x7fbb04038700 2024-03-30T00:50:17.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.828+0000 7fbb24000700 1 -- 172.21.15.116:0/3064055955 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fbb04038700 msgr2=0x7fbb0403abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:17.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.828+0000 7fbb24000700 1 --2- 172.21.15.116:0/3064055955 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fbb04038700 0x7fbb0403abc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbb08006fd0 tx=0x7fbb08006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:17.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.828+0000 7fbb24000700 1 -- 172.21.15.116:0/3064055955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c106500 msgr2=0x7fbb1c19c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:17.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.828+0000 7fbb24000700 1 --2- 172.21.15.116:0/3064055955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c106500 0x7fbb1c19c3f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009130 tx=0x7fbb10011ad0 comp rx=0 tx=0).stop 2024-03-30T00:50:17.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.829+0000 7fbb24000700 1 -- 172.21.15.116:0/3064055955 shutdown_connections 2024-03-30T00:50:17.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.829+0000 7fbb24000700 1 --2- 172.21.15.116:0/3064055955 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fbb04038700 0x7fbb0403abc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:17.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.829+0000 7fbb24000700 1 --2- 172.21.15.116:0/3064055955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1c106500 0x7fbb1c19c3f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:17.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.829+0000 7fbb24000700 1 -- 172.21.15.116:0/3064055955 >> 172.21.15.116:0/3064055955 conn(0x7fbb1c101ac0 msgr2=0x7fbb1c1026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:17.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.829+0000 7fbb24000700 1 -- 172.21.15.116:0/3064055955 shutdown_connections 2024-03-30T00:50:17.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:16.829+0000 7fbb24000700 1 -- 172.21.15.116:0/3064055955 wait complete. 2024-03-30T00:50:17.191 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-30T00:50:17.191 INFO:teuthology.orchestra.run.smithi116.stdout:Adding key to root@localhost authorized_keys... 2024-03-30T00:50:17.192 INFO:teuthology.orchestra.run.smithi116.stdout:Adding host smithi116... 2024-03-30T00:50:18.161 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:18 smithi116 ceph-mon[38086]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:50:19.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:19 smithi116 ceph-mon[38086]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi116", "addr": "172.21.15.116", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:50:20.291 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:20 smithi116 ceph-mon[38086]: Deploying cephadm binary to smithi116 2024-03-30T00:50:20.990 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Added host 'smithi116' with addr '172.21.15.116' 2024-03-30T00:50:20.991 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.561+0000 7fcb81200700 1 Processor -- start 2024-03-30T00:50:20.991 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.562+0000 7fcb81200700 1 -- start start 2024-03-30T00:50:20.991 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.562+0000 7fcb81200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7c07be40 0x7fcb7c07a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:20.991 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.562+0000 7fcb81200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c07a8c0 con 0x7fcb7c07be40 2024-03-30T00:50:20.991 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.563+0000 7fcb79e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7c07be40 0x7fcb7c07a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:20.991 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.563+0000 7fcb79e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7c07be40 0x7fcb7c07a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35912/0 (socket says 172.21.15.116:35912) 2024-03-30T00:50:20.991 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.563+0000 7fcb79e00700 1 -- 172.21.15.116:0/3581843690 learned_addr learned my addr 172.21.15.116:0/3581843690 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:20.991 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.563+0000 7fcb79e00700 1 -- 172.21.15.116:0/3581843690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c07aa00 con 0x7fcb7c07be40 2024-03-30T00:50:20.991 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.563+0000 7fcb79e00700 1 --2- 172.21.15.116:0/3581843690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7c07be40 0x7fcb7c07a2f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c0098a0 tx=0x7fcb6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ebd2448ec3e9bc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:20.991 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.563+0000 7fcb79400700 1 -- 172.21.15.116:0/3581843690 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb6c004160 con 0x7fcb7c07be40 2024-03-30T00:50:20.991 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.563+0000 7fcb79400700 1 -- 172.21.15.116:0/3581843690 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb6c0042c0 con 0x7fcb7c07be40 2024-03-30T00:50:20.991 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.563+0000 7fcb79400700 1 -- 172.21.15.116:0/3581843690 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb6c011a00 con 0x7fcb7c07be40 2024-03-30T00:50:20.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.564+0000 7fcb81200700 1 -- 172.21.15.116:0/3581843690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7c07be40 msgr2=0x7fcb7c07a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:20.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.564+0000 7fcb81200700 1 --2- 172.21.15.116:0/3581843690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7c07be40 0x7fcb7c07a2f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c0098a0 tx=0x7fcb6c009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:20.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.564+0000 7fcb81200700 1 -- 172.21.15.116:0/3581843690 shutdown_connections 2024-03-30T00:50:20.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.564+0000 7fcb81200700 1 --2- 172.21.15.116:0/3581843690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7c07be40 0x7fcb7c07a2f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:20.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.564+0000 7fcb81200700 1 -- 172.21.15.116:0/3581843690 >> 172.21.15.116:0/3581843690 conn(0x7fcb7c101af0 msgr2=0x7fcb7c103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:20.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.564+0000 7fcb81200700 1 -- 172.21.15.116:0/3581843690 shutdown_connections 2024-03-30T00:50:20.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.564+0000 7fcb81200700 1 -- 172.21.15.116:0/3581843690 wait complete. 2024-03-30T00:50:20.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.565+0000 7fcb81200700 1 Processor -- start 2024-03-30T00:50:20.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.565+0000 7fcb81200700 1 -- start start 2024-03-30T00:50:20.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.565+0000 7fcb81200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7c07be40 0x7fcb7c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.565+0000 7fcb81200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c19c960 con 0x7fcb7c07be40 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.566+0000 7fcb79e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7c07be40 0x7fcb7c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.566+0000 7fcb79e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7c07be40 0x7fcb7c19c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35920/0 (socket says 172.21.15.116:35920) 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.566+0000 7fcb79e00700 1 -- 172.21.15.116:0/264222464 learned_addr learned my addr 172.21.15.116:0/264222464 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.566+0000 7fcb79e00700 1 -- 172.21.15.116:0/264222464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c009550 con 0x7fcb7c07be40 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.566+0000 7fcb79e00700 1 --2- 172.21.15.116:0/264222464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7c07be40 0x7fcb7c19c420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c011a20 tx=0x7fcb6c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.566+0000 7fcb72a00700 1 -- 172.21.15.116:0/264222464 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb6c003af0 con 0x7fcb7c07be40 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.566+0000 7fcb72a00700 1 -- 172.21.15.116:0/264222464 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb6c003c50 con 0x7fcb7c07be40 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.566+0000 7fcb81200700 1 -- 172.21.15.116:0/264222464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7c19cb60 con 0x7fcb7c07be40 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.566+0000 7fcb72a00700 1 -- 172.21.15.116:0/264222464 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb6c019760 con 0x7fcb7c07be40 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.567+0000 7fcb81200700 1 -- 172.21.15.116:0/264222464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7c19d000 con 0x7fcb7c07be40 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.567+0000 7fcb72a00700 1 -- 172.21.15.116:0/264222464 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7fcb6c019950 con 0x7fcb7c07be40 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.568+0000 7fcb81200700 1 -- 172.21.15.116:0/264222464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7c10c880 con 0x7fcb7c07be40 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.568+0000 7fcb72a00700 1 --2- 172.21.15.116:0/264222464 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fcb60038750 0x7fcb6003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.568+0000 7fcb72a00700 1 -- 172.21.15.116:0/264222464 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcb6c04c9a0 con 0x7fcb7c07be40 2024-03-30T00:50:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.568+0000 7fcb73e00700 1 --2- 172.21.15.116:0/264222464 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fcb60038750 0x7fcb6003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.569+0000 7fcb73e00700 1 --2- 172.21.15.116:0/264222464 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fcb60038750 0x7fcb6003ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcb64006fd0 tx=0x7fcb64006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.573+0000 7fcb72a00700 1 -- 172.21.15.116:0/264222464 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcb6c018db0 con 0x7fcb7c07be40 2024-03-30T00:50:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:17.780+0000 7fcb81200700 1 -- 172.21.15.116:0/264222464 --> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi116", "addr": "172.21.15.116", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb7c102d90 con 0x7fcb60038750 2024-03-30T00:50:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:20.690+0000 7fcb72a00700 1 -- 172.21.15.116:0/264222464 <== mgr.14120 v2:172.21.15.116:6800/131150228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fcb7c102d90 con 0x7fcb60038750 2024-03-30T00:50:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:20.693+0000 7fcb81200700 1 -- 172.21.15.116:0/264222464 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fcb60038750 msgr2=0x7fcb6003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:20.693+0000 7fcb81200700 1 --2- 172.21.15.116:0/264222464 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fcb60038750 0x7fcb6003ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcb64006fd0 tx=0x7fcb64006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:20.693+0000 7fcb81200700 1 -- 172.21.15.116:0/264222464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7c07be40 msgr2=0x7fcb7c19c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:20.693+0000 7fcb81200700 1 --2- 172.21.15.116:0/264222464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7c07be40 0x7fcb7c19c420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c011a20 tx=0x7fcb6c011b00 comp rx=0 tx=0).stop 2024-03-30T00:50:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:20.693+0000 7fcb81200700 1 -- 172.21.15.116:0/264222464 shutdown_connections 2024-03-30T00:50:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:20.693+0000 7fcb81200700 1 --2- 172.21.15.116:0/264222464 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fcb60038750 0x7fcb6003ac10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:20.693+0000 7fcb81200700 1 --2- 172.21.15.116:0/264222464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb7c07be40 0x7fcb7c19c420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:20.693+0000 7fcb81200700 1 -- 172.21.15.116:0/264222464 >> 172.21.15.116:0/264222464 conn(0x7fcb7c101af0 msgr2=0x7fcb7c102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:20.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:20.693+0000 7fcb81200700 1 -- 172.21.15.116:0/264222464 shutdown_connections 2024-03-30T00:50:20.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:20.693+0000 7fcb81200700 1 -- 172.21.15.116:0/264222464 wait complete. 2024-03-30T00:50:20.999 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying unmanaged mon service... 2024-03-30T00:50:21.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:21 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' 2024-03-30T00:50:21.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:21 smithi116 ceph-mon[38086]: Added host smithi116 2024-03-30T00:50:21.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:21 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:50:21.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:21 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' 2024-03-30T00:50:22.258 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-30T00:50:22.258 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.411+0000 7f3348c00700 1 Processor -- start 2024-03-30T00:50:22.258 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.411+0000 7f3348c00700 1 -- start start 2024-03-30T00:50:22.258 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.411+0000 7f3348c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f334407be20 0x7f334407a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:22.258 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.411+0000 7f3348c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334407a810 con 0x7f334407be20 2024-03-30T00:50:22.258 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.412+0000 7f3341e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f334407be20 0x7f334407a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:22.258 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.412+0000 7f3341e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f334407be20 0x7f334407a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35926/0 (socket says 172.21.15.116:35926) 2024-03-30T00:50:22.259 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.412+0000 7f3341e00700 1 -- 172.21.15.116:0/3784725967 learned_addr learned my addr 172.21.15.116:0/3784725967 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:22.259 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.412+0000 7f3341e00700 1 -- 172.21.15.116:0/3784725967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334407a950 con 0x7f334407be20 2024-03-30T00:50:22.259 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.412+0000 7f3341e00700 1 --2- 172.21.15.116:0/3784725967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f334407be20 0x7f334407a2d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f33340098a0 tx=0x7f3334009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7351bda829d955d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:22.259 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.412+0000 7f3340a00700 1 -- 172.21.15.116:0/3784725967 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3334004160 con 0x7f334407be20 2024-03-30T00:50:22.259 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.413+0000 7f3340a00700 1 -- 172.21.15.116:0/3784725967 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33340042c0 con 0x7f334407be20 2024-03-30T00:50:22.259 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.413+0000 7f3340a00700 1 -- 172.21.15.116:0/3784725967 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3334011a00 con 0x7f334407be20 2024-03-30T00:50:22.259 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.413+0000 7f3348c00700 1 -- 172.21.15.116:0/3784725967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f334407be20 msgr2=0x7f334407a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:22.259 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.413+0000 7f3348c00700 1 --2- 172.21.15.116:0/3784725967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f334407be20 0x7f334407a2d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f33340098a0 tx=0x7f3334009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:22.259 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.413+0000 7f3348c00700 1 -- 172.21.15.116:0/3784725967 shutdown_connections 2024-03-30T00:50:22.259 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.413+0000 7f3348c00700 1 --2- 172.21.15.116:0/3784725967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f334407be20 0x7f334407a2d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:22.259 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.413+0000 7f3348c00700 1 -- 172.21.15.116:0/3784725967 >> 172.21.15.116:0/3784725967 conn(0x7f3344101af0 msgr2=0x7f3344103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:22.259 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.413+0000 7f3348c00700 1 -- 172.21.15.116:0/3784725967 shutdown_connections 2024-03-30T00:50:22.259 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.413+0000 7f3348c00700 1 -- 172.21.15.116:0/3784725967 wait complete. 2024-03-30T00:50:22.259 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.414+0000 7f3348c00700 1 Processor -- start 2024-03-30T00:50:22.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.414+0000 7f3348c00700 1 -- start start 2024-03-30T00:50:22.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.415+0000 7f3348c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f334407be20 0x7f334419c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:22.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.415+0000 7f3348c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334419c960 con 0x7f334407be20 2024-03-30T00:50:22.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.415+0000 7f3341e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f334407be20 0x7f334419c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:22.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.415+0000 7f3341e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f334407be20 0x7f334419c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35932/0 (socket says 172.21.15.116:35932) 2024-03-30T00:50:22.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.415+0000 7f3341e00700 1 -- 172.21.15.116:0/973628359 learned_addr learned my addr 172.21.15.116:0/973628359 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:22.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.415+0000 7f3341e00700 1 -- 172.21.15.116:0/973628359 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3334009550 con 0x7f334407be20 2024-03-30T00:50:22.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.416+0000 7f3341e00700 1 --2- 172.21.15.116:0/973628359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f334407be20 0x7f334419c420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3334011a20 tx=0x7f3334011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:22.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.416+0000 7f333aa00700 1 -- 172.21.15.116:0/973628359 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3334003af0 con 0x7f334407be20 2024-03-30T00:50:22.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.416+0000 7f333aa00700 1 -- 172.21.15.116:0/973628359 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3334003c50 con 0x7f334407be20 2024-03-30T00:50:22.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.416+0000 7f3348c00700 1 -- 172.21.15.116:0/973628359 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334419cb60 con 0x7f334407be20 2024-03-30T00:50:22.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.416+0000 7f333aa00700 1 -- 172.21.15.116:0/973628359 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3334019760 con 0x7f334407be20 2024-03-30T00:50:22.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.416+0000 7f3348c00700 1 -- 172.21.15.116:0/973628359 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334419d000 con 0x7f334407be20 2024-03-30T00:50:22.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.417+0000 7f3348c00700 1 -- 172.21.15.116:0/973628359 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334410c840 con 0x7f334407be20 2024-03-30T00:50:22.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.417+0000 7f333aa00700 1 -- 172.21.15.116:0/973628359 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f333401ab20 con 0x7f334407be20 2024-03-30T00:50:22.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.418+0000 7f333aa00700 1 --2- 172.21.15.116:0/973628359 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f3328038750 0x7f332803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:22.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.418+0000 7f333aa00700 1 -- 172.21.15.116:0/973628359 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f333404c9a0 con 0x7f334407be20 2024-03-30T00:50:22.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.418+0000 7f3341400700 1 --2- 172.21.15.116:0/973628359 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f3328038750 0x7f332803ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:22.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.418+0000 7f3341400700 1 --2- 172.21.15.116:0/973628359 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f3328038750 0x7f332803ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3330006fd0 tx=0x7f3330006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:22.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.422+0000 7f333aa00700 1 -- 172.21.15.116:0/973628359 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f333401a7b0 con 0x7f334407be20 2024-03-30T00:50:22.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.654+0000 7f3348c00700 1 -- 172.21.15.116:0/973628359 --> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3344102dc0 con 0x7f3328038750 2024-03-30T00:50:22.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.659+0000 7f333aa00700 1 -- 172.21.15.116:0/973628359 <== mgr.14120 v2:172.21.15.116:6800/131150228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3344102dc0 con 0x7f3328038750 2024-03-30T00:50:22.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.662+0000 7f3348c00700 1 -- 172.21.15.116:0/973628359 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f3328038750 msgr2=0x7f332803ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:22.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.662+0000 7f3348c00700 1 --2- 172.21.15.116:0/973628359 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f3328038750 0x7f332803ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3330006fd0 tx=0x7f3330006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:22.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.662+0000 7f3348c00700 1 -- 172.21.15.116:0/973628359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f334407be20 msgr2=0x7f334419c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:22.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.662+0000 7f3348c00700 1 --2- 172.21.15.116:0/973628359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f334407be20 0x7f334419c420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3334011a20 tx=0x7f3334011b00 comp rx=0 tx=0).stop 2024-03-30T00:50:22.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.662+0000 7f3348c00700 1 -- 172.21.15.116:0/973628359 shutdown_connections 2024-03-30T00:50:22.262 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.662+0000 7f3348c00700 1 --2- 172.21.15.116:0/973628359 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f3328038750 0x7f332803ac10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:22.262 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.662+0000 7f3348c00700 1 --2- 172.21.15.116:0/973628359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f334407be20 0x7f334419c420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:22.262 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.662+0000 7f3348c00700 1 -- 172.21.15.116:0/973628359 >> 172.21.15.116:0/973628359 conn(0x7f3344101af0 msgr2=0x7f33441026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:22.262 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.662+0000 7f3348c00700 1 -- 172.21.15.116:0/973628359 shutdown_connections 2024-03-30T00:50:22.262 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:21.662+0000 7f3348c00700 1 -- 172.21.15.116:0/973628359 wait complete. 2024-03-30T00:50:22.262 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying unmanaged mgr service... 2024-03-30T00:50:23.324 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:23 smithi116 ceph-mon[38086]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:50:23.324 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:23 smithi116 ceph-mon[38086]: Saving service mon spec with placement count:5 2024-03-30T00:50:23.324 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:23 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' 2024-03-30T00:50:23.324 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:23 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' 2024-03-30T00:50:23.324 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:23 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' 2024-03-30T00:50:23.424 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-30T00:50:23.425 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.675+0000 7f694da00700 1 Processor -- start 2024-03-30T00:50:23.425 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.675+0000 7f694da00700 1 -- start start 2024-03-30T00:50:23.425 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.676+0000 7f694da00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f694807be40 0x7f694807a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:23.425 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.676+0000 7f694da00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694807a8c0 con 0x7f694807be40 2024-03-30T00:50:23.425 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.676+0000 7f6946a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f694807be40 0x7f694807a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:23.425 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.676+0000 7f6946a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f694807be40 0x7f694807a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35938/0 (socket says 172.21.15.116:35938) 2024-03-30T00:50:23.425 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.676+0000 7f6946a00700 1 -- 172.21.15.116:0/4257787084 learned_addr learned my addr 172.21.15.116:0/4257787084 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:23.425 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.677+0000 7f6946a00700 1 -- 172.21.15.116:0/4257787084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694807aa00 con 0x7f694807be40 2024-03-30T00:50:23.425 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.677+0000 7f6946a00700 1 --2- 172.21.15.116:0/4257787084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f694807be40 0x7f694807a2f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f69380098a0 tx=0x7f6938009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d38e9a4b2046f54 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:23.425 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.677+0000 7f6945600700 1 -- 172.21.15.116:0/4257787084 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6938004160 con 0x7f694807be40 2024-03-30T00:50:23.425 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.677+0000 7f6945600700 1 -- 172.21.15.116:0/4257787084 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69380042c0 con 0x7f694807be40 2024-03-30T00:50:23.425 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.677+0000 7f6945600700 1 -- 172.21.15.116:0/4257787084 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69380119b0 con 0x7f694807be40 2024-03-30T00:50:23.425 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.678+0000 7f694da00700 1 -- 172.21.15.116:0/4257787084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f694807be40 msgr2=0x7f694807a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.678+0000 7f694da00700 1 --2- 172.21.15.116:0/4257787084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f694807be40 0x7f694807a2f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f69380098a0 tx=0x7f6938009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.678+0000 7f694da00700 1 -- 172.21.15.116:0/4257787084 shutdown_connections 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.678+0000 7f694da00700 1 --2- 172.21.15.116:0/4257787084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f694807be40 0x7f694807a2f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.678+0000 7f694da00700 1 -- 172.21.15.116:0/4257787084 >> 172.21.15.116:0/4257787084 conn(0x7f6948101af0 msgr2=0x7f6948103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.678+0000 7f694da00700 1 -- 172.21.15.116:0/4257787084 shutdown_connections 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.678+0000 7f694da00700 1 -- 172.21.15.116:0/4257787084 wait complete. 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.679+0000 7f694da00700 1 Processor -- start 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.679+0000 7f694da00700 1 -- start start 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.679+0000 7f694da00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f694807be40 0x7f694819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.679+0000 7f694da00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694819c980 con 0x7f694807be40 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.680+0000 7f6946a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f694807be40 0x7f694819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.680+0000 7f6946a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f694807be40 0x7f694819c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35948/0 (socket says 172.21.15.116:35948) 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.680+0000 7f6946a00700 1 -- 172.21.15.116:0/2527276949 learned_addr learned my addr 172.21.15.116:0/2527276949 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.680+0000 7f6946a00700 1 -- 172.21.15.116:0/2527276949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6938009550 con 0x7f694807be40 2024-03-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.680+0000 7f6946a00700 1 --2- 172.21.15.116:0/2527276949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f694807be40 0x7f694819c440 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6938011980 tx=0x7f6938011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.681+0000 7f693f400700 1 -- 172.21.15.116:0/2527276949 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6938003af0 con 0x7f694807be40 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.681+0000 7f693f400700 1 -- 172.21.15.116:0/2527276949 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6938003c50 con 0x7f694807be40 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.681+0000 7f694da00700 1 -- 172.21.15.116:0/2527276949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694819cb80 con 0x7f694807be40 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.681+0000 7f693f400700 1 -- 172.21.15.116:0/2527276949 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6938019760 con 0x7f694807be40 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.681+0000 7f694da00700 1 -- 172.21.15.116:0/2527276949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f694819d020 con 0x7f694807be40 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.682+0000 7f693f400700 1 -- 172.21.15.116:0/2527276949 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f693801ab10 con 0x7f694807be40 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.682+0000 7f694da00700 1 -- 172.21.15.116:0/2527276949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f694810c8d0 con 0x7f694807be40 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.682+0000 7f693f400700 1 --2- 172.21.15.116:0/2527276949 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f692c038750 0x7f692c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.682+0000 7f693f400700 1 -- 172.21.15.116:0/2527276949 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f693804c9d0 con 0x7f694807be40 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.683+0000 7f6946000700 1 --2- 172.21.15.116:0/2527276949 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f692c038750 0x7f692c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.683+0000 7f6946000700 1 --2- 172.21.15.116:0/2527276949 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f692c038750 0x7f692c03ac10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6934006fd0 tx=0x7f6934006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.688+0000 7f693f400700 1 -- 172.21.15.116:0/2527276949 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f693801a430 con 0x7f694807be40 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.924+0000 7f694da00700 1 -- 172.21.15.116:0/2527276949 --> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f694819f910 con 0x7f692c038750 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.931+0000 7f693f400700 1 -- 172.21.15.116:0/2527276949 <== mgr.14120 v2:172.21.15.116:6800/131150228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f694819f910 con 0x7f692c038750 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.933+0000 7f694da00700 1 -- 172.21.15.116:0/2527276949 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f692c038750 msgr2=0x7f692c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.933+0000 7f694da00700 1 --2- 172.21.15.116:0/2527276949 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f692c038750 0x7f692c03ac10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6934006fd0 tx=0x7f6934006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:23.428 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.933+0000 7f694da00700 1 -- 172.21.15.116:0/2527276949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f694807be40 msgr2=0x7f694819c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:23.428 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.933+0000 7f694da00700 1 --2- 172.21.15.116:0/2527276949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f694807be40 0x7f694819c440 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6938011980 tx=0x7f6938011a60 comp rx=0 tx=0).stop 2024-03-30T00:50:23.428 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.933+0000 7f694da00700 1 -- 172.21.15.116:0/2527276949 shutdown_connections 2024-03-30T00:50:23.428 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.933+0000 7f694da00700 1 --2- 172.21.15.116:0/2527276949 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f692c038750 0x7f692c03ac10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:23.428 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.933+0000 7f694da00700 1 --2- 172.21.15.116:0/2527276949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f694807be40 0x7f694819c440 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:23.428 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.933+0000 7f694da00700 1 -- 172.21.15.116:0/2527276949 >> 172.21.15.116:0/2527276949 conn(0x7f6948101af0 msgr2=0x7f6948102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:23.428 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.934+0000 7f694da00700 1 -- 172.21.15.116:0/2527276949 shutdown_connections 2024-03-30T00:50:23.428 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:22.934+0000 7f694da00700 1 -- 172.21.15.116:0/2527276949 wait complete. 2024-03-30T00:50:24.308 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:24 smithi116 ceph-mon[38086]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:50:24.308 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:24 smithi116 ceph-mon[38086]: Saving service mgr spec with placement count:2 2024-03-30T00:50:24.308 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:24 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' 2024-03-30T00:50:24.419 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.832+0000 7f4725a00700 1 Processor -- start 2024-03-30T00:50:24.419 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.833+0000 7f4725a00700 1 -- start start 2024-03-30T00:50:24.419 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.833+0000 7f4725a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f472007be20 0x7f472007a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:24.419 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.833+0000 7f4725a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472007a810 con 0x7f472007be20 2024-03-30T00:50:24.419 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.833+0000 7f471ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f472007be20 0x7f472007a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:24.419 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.833+0000 7f471ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f472007be20 0x7f472007a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35958/0 (socket says 172.21.15.116:35958) 2024-03-30T00:50:24.419 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.833+0000 7f471ea00700 1 -- 172.21.15.116:0/3742198220 learned_addr learned my addr 172.21.15.116:0/3742198220 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:24.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.834+0000 7f471ea00700 1 -- 172.21.15.116:0/3742198220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472007a950 con 0x7f472007be20 2024-03-30T00:50:24.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.834+0000 7f471ea00700 1 --2- 172.21.15.116:0/3742198220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f472007be20 0x7f472007a2d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f47100098a0 tx=0x7f4710009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8228fec39a3a0011 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:24.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.834+0000 7f471d600700 1 -- 172.21.15.116:0/3742198220 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4710004160 con 0x7f472007be20 2024-03-30T00:50:24.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.834+0000 7f471d600700 1 -- 172.21.15.116:0/3742198220 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47100042c0 con 0x7f472007be20 2024-03-30T00:50:24.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.834+0000 7f471d600700 1 -- 172.21.15.116:0/3742198220 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47100119b0 con 0x7f472007be20 2024-03-30T00:50:24.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.835+0000 7f4725a00700 1 -- 172.21.15.116:0/3742198220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f472007be20 msgr2=0x7f472007a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:24.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.835+0000 7f4725a00700 1 --2- 172.21.15.116:0/3742198220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f472007be20 0x7f472007a2d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f47100098a0 tx=0x7f4710009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:24.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.835+0000 7f4725a00700 1 -- 172.21.15.116:0/3742198220 shutdown_connections 2024-03-30T00:50:24.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.835+0000 7f4725a00700 1 --2- 172.21.15.116:0/3742198220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f472007be20 0x7f472007a2d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:24.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.835+0000 7f4725a00700 1 -- 172.21.15.116:0/3742198220 >> 172.21.15.116:0/3742198220 conn(0x7f4720101af0 msgr2=0x7f4720103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:24.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.835+0000 7f4725a00700 1 -- 172.21.15.116:0/3742198220 shutdown_connections 2024-03-30T00:50:24.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.835+0000 7f4725a00700 1 -- 172.21.15.116:0/3742198220 wait complete. 2024-03-30T00:50:24.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.836+0000 7f4725a00700 1 Processor -- start 2024-03-30T00:50:24.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.836+0000 7f4725a00700 1 -- start start 2024-03-30T00:50:24.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.836+0000 7f4725a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f472007be20 0x7f472019c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:24.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.836+0000 7f4725a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472019c950 con 0x7f472007be20 2024-03-30T00:50:24.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.837+0000 7f471ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f472007be20 0x7f472019c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:24.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.837+0000 7f471ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f472007be20 0x7f472019c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35966/0 (socket says 172.21.15.116:35966) 2024-03-30T00:50:24.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.837+0000 7f471ea00700 1 -- 172.21.15.116:0/3545698840 learned_addr learned my addr 172.21.15.116:0/3545698840 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:24.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.837+0000 7f471ea00700 1 -- 172.21.15.116:0/3545698840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4710009550 con 0x7f472007be20 2024-03-30T00:50:24.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.837+0000 7f471ea00700 1 --2- 172.21.15.116:0/3545698840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f472007be20 0x7f472019c410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4710011980 tx=0x7f4710011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:24.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.838+0000 7f4717400700 1 -- 172.21.15.116:0/3545698840 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4710003b70 con 0x7f472007be20 2024-03-30T00:50:24.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.838+0000 7f4717400700 1 -- 172.21.15.116:0/3545698840 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f471001a8f0 con 0x7f472007be20 2024-03-30T00:50:24.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.838+0000 7f4725a00700 1 -- 172.21.15.116:0/3545698840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472019cb50 con 0x7f472007be20 2024-03-30T00:50:24.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.838+0000 7f4717400700 1 -- 172.21.15.116:0/3545698840 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47100197d0 con 0x7f472007be20 2024-03-30T00:50:24.422 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.838+0000 7f4725a00700 1 -- 172.21.15.116:0/3545698840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472019cff0 con 0x7f472007be20 2024-03-30T00:50:24.422 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.840+0000 7f4717400700 1 -- 172.21.15.116:0/3545698840 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f471001a430 con 0x7f472007be20 2024-03-30T00:50:24.422 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.840+0000 7f4725a00700 1 -- 172.21.15.116:0/3545698840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472010c890 con 0x7f472007be20 2024-03-30T00:50:24.422 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.840+0000 7f4717400700 1 --2- 172.21.15.116:0/3545698840 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f4704038750 0x7f470403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:24.422 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.840+0000 7f4717400700 1 -- 172.21.15.116:0/3545698840 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f471004caf0 con 0x7f472007be20 2024-03-30T00:50:24.422 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.841+0000 7f471e000700 1 --2- 172.21.15.116:0/3545698840 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f4704038750 0x7f470403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:24.422 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.841+0000 7f471e000700 1 --2- 172.21.15.116:0/3545698840 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f4704038750 0x7f470403ac10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f470c006fd0 tx=0x7f470c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:24.422 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:23.849+0000 7f4717400700 1 -- 172.21.15.116:0/3545698840 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4710019a50 con 0x7f472007be20 2024-03-30T00:50:24.422 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.075+0000 7f4725a00700 1 -- 172.21.15.116:0/3545698840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f472010cad0 con 0x7f472007be20 2024-03-30T00:50:24.422 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.082+0000 7f4717400700 1 -- 172.21.15.116:0/3545698840 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f4710018350 con 0x7f472007be20 2024-03-30T00:50:24.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.084+0000 7f4725a00700 1 -- 172.21.15.116:0/3545698840 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f4704038750 msgr2=0x7f470403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:24.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.084+0000 7f4725a00700 1 --2- 172.21.15.116:0/3545698840 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f4704038750 0x7f470403ac10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f470c006fd0 tx=0x7f470c006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:24.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.084+0000 7f4725a00700 1 -- 172.21.15.116:0/3545698840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f472007be20 msgr2=0x7f472019c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:24.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.084+0000 7f4725a00700 1 --2- 172.21.15.116:0/3545698840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f472007be20 0x7f472019c410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4710011980 tx=0x7f4710011a60 comp rx=0 tx=0).stop 2024-03-30T00:50:24.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.084+0000 7f4725a00700 1 -- 172.21.15.116:0/3545698840 shutdown_connections 2024-03-30T00:50:24.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.084+0000 7f4725a00700 1 --2- 172.21.15.116:0/3545698840 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f4704038750 0x7f470403ac10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:24.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.084+0000 7f4725a00700 1 --2- 172.21.15.116:0/3545698840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f472007be20 0x7f472019c410 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:24.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.084+0000 7f4725a00700 1 -- 172.21.15.116:0/3545698840 >> 172.21.15.116:0/3545698840 conn(0x7f4720101af0 msgr2=0x7f4720102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:24.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.085+0000 7f4725a00700 1 -- 172.21.15.116:0/3545698840 shutdown_connections 2024-03-30T00:50:24.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.085+0000 7f4725a00700 1 -- 172.21.15.116:0/3545698840 wait complete. 2024-03-30T00:50:25.371 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:25 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3545698840' entity='client.admin' 2024-03-30T00:50:25.371 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:25 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2178375868' entity='client.admin' 2024-03-30T00:50:25.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.829+0000 7f67bb000700 1 Processor -- start 2024-03-30T00:50:25.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.829+0000 7f67bb000700 1 -- start start 2024-03-30T00:50:25.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.830+0000 7f67bb000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67b407be80 0x7f67b407a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:25.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.830+0000 7f67bb000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b407a900 con 0x7f67b407be80 2024-03-30T00:50:25.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.830+0000 7f67b3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67b407be80 0x7f67b407a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:25.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.830+0000 7f67b3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67b407be80 0x7f67b407a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45508/0 (socket says 172.21.15.116:45508) 2024-03-30T00:50:25.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.830+0000 7f67b3e00700 1 -- 172.21.15.116:0/160510322 learned_addr learned my addr 172.21.15.116:0/160510322 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:25.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.830+0000 7f67b3e00700 1 -- 172.21.15.116:0/160510322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b407aa40 con 0x7f67b407be80 2024-03-30T00:50:25.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.831+0000 7f67b3e00700 1 --2- 172.21.15.116:0/160510322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67b407be80 0x7f67b407a330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f67a40098a0 tx=0x7f67a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81977057f4459043 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:25.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.831+0000 7f67b3400700 1 -- 172.21.15.116:0/160510322 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f67a4004160 con 0x7f67b407be80 2024-03-30T00:50:25.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.831+0000 7f67b3400700 1 -- 172.21.15.116:0/160510322 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67a40042c0 con 0x7f67b407be80 2024-03-30T00:50:25.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.831+0000 7f67b3400700 1 -- 172.21.15.116:0/160510322 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f67a40119b0 con 0x7f67b407be80 2024-03-30T00:50:25.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.831+0000 7f67bb000700 1 -- 172.21.15.116:0/160510322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67b407be80 msgr2=0x7f67b407a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:25.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.831+0000 7f67bb000700 1 --2- 172.21.15.116:0/160510322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67b407be80 0x7f67b407a330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f67a40098a0 tx=0x7f67a4009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:25.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.832+0000 7f67bb000700 1 -- 172.21.15.116:0/160510322 shutdown_connections 2024-03-30T00:50:25.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.832+0000 7f67bb000700 1 --2- 172.21.15.116:0/160510322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67b407be80 0x7f67b407a330 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:25.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.832+0000 7f67bb000700 1 -- 172.21.15.116:0/160510322 >> 172.21.15.116:0/160510322 conn(0x7f67b4101af0 msgr2=0x7f67b4103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:25.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.832+0000 7f67bb000700 1 -- 172.21.15.116:0/160510322 shutdown_connections 2024-03-30T00:50:25.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.832+0000 7f67bb000700 1 -- 172.21.15.116:0/160510322 wait complete. 2024-03-30T00:50:25.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.833+0000 7f67bb000700 1 Processor -- start 2024-03-30T00:50:25.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.833+0000 7f67bb000700 1 -- start start 2024-03-30T00:50:25.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.833+0000 7f67bb000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67b407be80 0x7f67b41a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:25.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.833+0000 7f67bb000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b41a0d80 con 0x7f67b407be80 2024-03-30T00:50:25.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.833+0000 7f67b3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67b407be80 0x7f67b41a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:25.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.833+0000 7f67b3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67b407be80 0x7f67b41a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45520/0 (socket says 172.21.15.116:45520) 2024-03-30T00:50:25.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.833+0000 7f67b3e00700 1 -- 172.21.15.116:0/2178375868 learned_addr learned my addr 172.21.15.116:0/2178375868 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:25.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.834+0000 7f67b3e00700 1 -- 172.21.15.116:0/2178375868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a4009550 con 0x7f67b407be80 2024-03-30T00:50:25.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.834+0000 7f67b3e00700 1 --2- 172.21.15.116:0/2178375868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67b407be80 0x7f67b41a0840 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f67a4011980 tx=0x7f67a4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:25.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.834+0000 7f67b1600700 1 -- 172.21.15.116:0/2178375868 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f67a4003af0 con 0x7f67b407be80 2024-03-30T00:50:25.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.834+0000 7f67b1600700 1 -- 172.21.15.116:0/2178375868 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67a4003c50 con 0x7f67b407be80 2024-03-30T00:50:25.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.834+0000 7f67bb000700 1 -- 172.21.15.116:0/2178375868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b41a0f80 con 0x7f67b407be80 2024-03-30T00:50:25.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.834+0000 7f67b1600700 1 -- 172.21.15.116:0/2178375868 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f67a4019760 con 0x7f67b407be80 2024-03-30T00:50:25.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.834+0000 7f67bb000700 1 -- 172.21.15.116:0/2178375868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b41a1420 con 0x7f67b407be80 2024-03-30T00:50:25.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.835+0000 7f67b1600700 1 -- 172.21.15.116:0/2178375868 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f67a401ab10 con 0x7f67b407be80 2024-03-30T00:50:25.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.835+0000 7f67bb000700 1 -- 172.21.15.116:0/2178375868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b4199fb0 con 0x7f67b407be80 2024-03-30T00:50:25.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.836+0000 7f67b1600700 1 --2- 172.21.15.116:0/2178375868 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f6798038750 0x7f679803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:25.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.836+0000 7f67b1600700 1 -- 172.21.15.116:0/2178375868 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f67a404ca00 con 0x7f67b407be80 2024-03-30T00:50:25.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.836+0000 7f67abe00700 1 --2- 172.21.15.116:0/2178375868 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f6798038750 0x7f679803ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.836+0000 7f67abe00700 1 --2- 172.21.15.116:0/2178375868 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f6798038750 0x7f679803ac10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f679c006fd0 tx=0x7f679c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:24.840+0000 7f67b1600700 1 -- 172.21.15.116:0/2178375868 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f67a401a430 con 0x7f67b407be80 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.051+0000 7f67bb000700 1 -- 172.21.15.116:0/2178375868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f67b41084e0 con 0x7f67b407be80 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.055+0000 7f67b1600700 1 -- 172.21.15.116:0/2178375868 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f67a402db40 con 0x7f67b407be80 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.057+0000 7f67bb000700 1 -- 172.21.15.116:0/2178375868 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f6798038750 msgr2=0x7f679803ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.057+0000 7f67bb000700 1 --2- 172.21.15.116:0/2178375868 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f6798038750 0x7f679803ac10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f679c006fd0 tx=0x7f679c006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.057+0000 7f67bb000700 1 -- 172.21.15.116:0/2178375868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67b407be80 msgr2=0x7f67b41a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.057+0000 7f67bb000700 1 --2- 172.21.15.116:0/2178375868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67b407be80 0x7f67b41a0840 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f67a4011980 tx=0x7f67a4011a60 comp rx=0 tx=0).stop 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.057+0000 7f67bb000700 1 -- 172.21.15.116:0/2178375868 shutdown_connections 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.057+0000 7f67bb000700 1 --2- 172.21.15.116:0/2178375868 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f6798038750 0x7f679803ac10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.057+0000 7f67bb000700 1 --2- 172.21.15.116:0/2178375868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67b407be80 0x7f67b41a0840 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.057+0000 7f67bb000700 1 -- 172.21.15.116:0/2178375868 >> 172.21.15.116:0/2178375868 conn(0x7f67b4101af0 msgr2=0x7f67b4103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.058+0000 7f67bb000700 1 -- 172.21.15.116:0/2178375868 shutdown_connections 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.058+0000 7f67bb000700 1 -- 172.21.15.116:0/2178375868 wait complete. 2024-03-30T00:50:25.380 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling the dashboard module... 2024-03-30T00:50:26.564 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:26 smithi116 ceph-mon[38086]: from='mgr.14120 172.21.15.116:0/3878877564' entity='mgr.x' 2024-03-30T00:50:26.564 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:26 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1488965869' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-30T00:50:26.564 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:26 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: ignoring --setuser ceph since I am not root 2024-03-30T00:50:26.564 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:26 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: ignoring --setgroup ceph since I am not root 2024-03-30T00:50:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.793+0000 7f33f2a00700 1 Processor -- start 2024-03-30T00:50:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.793+0000 7f33f2a00700 1 -- start start 2024-03-30T00:50:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.793+0000 7f33f2a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ec07be40 0x7f33ec07a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.793+0000 7f33f2a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec07a8c0 con 0x7f33ec07be40 2024-03-30T00:50:26.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.793+0000 7f33ebe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ec07be40 0x7f33ec07a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.793+0000 7f33ebe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ec07be40 0x7f33ec07a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45534/0 (socket says 172.21.15.116:45534) 2024-03-30T00:50:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.793+0000 7f33ebe00700 1 -- 172.21.15.116:0/1407351162 learned_addr learned my addr 172.21.15.116:0/1407351162 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.794+0000 7f33ebe00700 1 -- 172.21.15.116:0/1407351162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec07aa00 con 0x7f33ec07be40 2024-03-30T00:50:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.794+0000 7f33ebe00700 1 --2- 172.21.15.116:0/1407351162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ec07be40 0x7f33ec07a2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f33dc0098a0 tx=0x7f33dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f1e6fd8d21da7b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.794+0000 7f33eaa00700 1 -- 172.21.15.116:0/1407351162 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33dc004160 con 0x7f33ec07be40 2024-03-30T00:50:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.794+0000 7f33eaa00700 1 -- 172.21.15.116:0/1407351162 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33dc0042c0 con 0x7f33ec07be40 2024-03-30T00:50:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.794+0000 7f33eaa00700 1 -- 172.21.15.116:0/1407351162 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33dc0119b0 con 0x7f33ec07be40 2024-03-30T00:50:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.795+0000 7f33f2a00700 1 -- 172.21.15.116:0/1407351162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ec07be40 msgr2=0x7f33ec07a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.795+0000 7f33f2a00700 1 --2- 172.21.15.116:0/1407351162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ec07be40 0x7f33ec07a2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f33dc0098a0 tx=0x7f33dc009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.795+0000 7f33f2a00700 1 -- 172.21.15.116:0/1407351162 shutdown_connections 2024-03-30T00:50:26.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.795+0000 7f33f2a00700 1 --2- 172.21.15.116:0/1407351162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ec07be40 0x7f33ec07a2f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.795+0000 7f33f2a00700 1 -- 172.21.15.116:0/1407351162 >> 172.21.15.116:0/1407351162 conn(0x7f33ec101af0 msgr2=0x7f33ec103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.795+0000 7f33f2a00700 1 -- 172.21.15.116:0/1407351162 shutdown_connections 2024-03-30T00:50:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.795+0000 7f33f2a00700 1 -- 172.21.15.116:0/1407351162 wait complete. 2024-03-30T00:50:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.797+0000 7f33f2a00700 1 Processor -- start 2024-03-30T00:50:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.797+0000 7f33f2a00700 1 -- start start 2024-03-30T00:50:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.797+0000 7f33f2a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ec07be40 0x7f33ec109750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.797+0000 7f33f2a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec109c90 con 0x7f33ec07be40 2024-03-30T00:50:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.798+0000 7f33ebe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ec07be40 0x7f33ec109750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.798+0000 7f33ebe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ec07be40 0x7f33ec109750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45540/0 (socket says 172.21.15.116:45540) 2024-03-30T00:50:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.798+0000 7f33ebe00700 1 -- 172.21.15.116:0/1488965869 learned_addr learned my addr 172.21.15.116:0/1488965869 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.798+0000 7f33ebe00700 1 -- 172.21.15.116:0/1488965869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc009550 con 0x7f33ec07be40 2024-03-30T00:50:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.799+0000 7f33ebe00700 1 --2- 172.21.15.116:0/1488965869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ec07be40 0x7f33ec109750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f33dc011980 tx=0x7f33dc011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.799+0000 7f33e8c00700 1 -- 172.21.15.116:0/1488965869 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33dc003af0 con 0x7f33ec07be40 2024-03-30T00:50:26.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.799+0000 7f33e8c00700 1 -- 172.21.15.116:0/1488965869 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33dc003c50 con 0x7f33ec07be40 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.799+0000 7f33f2a00700 1 -- 172.21.15.116:0/1488965869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ec109e90 con 0x7f33ec07be40 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.799+0000 7f33e8c00700 1 -- 172.21.15.116:0/1488965869 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33dc019760 con 0x7f33ec07be40 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.799+0000 7f33f2a00700 1 -- 172.21.15.116:0/1488965869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ec1062e0 con 0x7f33ec07be40 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.800+0000 7f33e8c00700 1 -- 172.21.15.116:0/1488965869 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f33dc0199a0 con 0x7f33ec07be40 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.801+0000 7f33e2000700 1 -- 172.21.15.116:0/1488965869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c40052f0 con 0x7f33ec07be40 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.801+0000 7f33e8c00700 1 --2- 172.21.15.116:0/1488965869 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f33d0038700 0x7f33d003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.801+0000 7f33e8c00700 1 -- 172.21.15.116:0/1488965869 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f33dc04cbf0 con 0x7f33ec07be40 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.801+0000 7f33eb400700 1 --2- 172.21.15.116:0/1488965869 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f33d0038700 0x7f33d003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.802+0000 7f33eb400700 1 --2- 172.21.15.116:0/1488965869 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f33d0038700 0x7f33d003abc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f33d8006fd0 tx=0x7f33d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:25.807+0000 7f33e8c00700 1 -- 172.21.15.116:0/1488965869 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f33dc018db0 con 0x7f33ec07be40 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:26.091+0000 7f33e2000700 1 -- 172.21.15.116:0/1488965869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f33c40055a0 con 0x7f33ec07be40 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:26.335+0000 7f33e8c00700 1 -- 172.21.15.116:0/1488965869 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 9) v1 ==== 45546+0+0 (secure 0 0 0) 0x7f33dc00b900 con 0x7f33ec07be40 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:26.336+0000 7f33e8c00700 1 -- 172.21.15.116:0/1488965869 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f33dc04c590 con 0x7f33ec07be40 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:26.340+0000 7f33e2000700 1 -- 172.21.15.116:0/1488965869 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f33d0038700 msgr2=0x7f33d003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:26.340+0000 7f33e2000700 1 --2- 172.21.15.116:0/1488965869 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f33d0038700 0x7f33d003abc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f33d8006fd0 tx=0x7f33d8006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:26.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:26.340+0000 7f33e2000700 1 -- 172.21.15.116:0/1488965869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ec07be40 msgr2=0x7f33ec109750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:26.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:26.340+0000 7f33e2000700 1 --2- 172.21.15.116:0/1488965869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ec07be40 0x7f33ec109750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f33dc011980 tx=0x7f33dc011a60 comp rx=0 tx=0).stop 2024-03-30T00:50:26.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:26.340+0000 7f33e2000700 1 -- 172.21.15.116:0/1488965869 shutdown_connections 2024-03-30T00:50:26.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:26.340+0000 7f33e2000700 1 --2- 172.21.15.116:0/1488965869 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7f33d0038700 0x7f33d003abc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:26.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:26.340+0000 7f33e2000700 1 --2- 172.21.15.116:0/1488965869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ec07be40 0x7f33ec109750 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:26.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:26.340+0000 7f33e2000700 1 -- 172.21.15.116:0/1488965869 >> 172.21.15.116:0/1488965869 conn(0x7f33ec101af0 msgr2=0x7f33ec102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:26.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:26.340+0000 7f33e2000700 1 -- 172.21.15.116:0/1488965869 shutdown_connections 2024-03-30T00:50:26.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:26.340+0000 7f33e2000700 1 -- 172.21.15.116:0/1488965869 wait complete. 2024-03-30T00:50:27.251 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:26 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:26.996+0000 7fc7e83b6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T00:50:27.251 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:27 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:27.125+0000 7fc7e83b6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T00:50:27.251 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:27 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:27.243+0000 7fc7e83b6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T00:50:27.605 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:27 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:27.356+0000 7fc7e83b6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T00:50:27.605 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:27 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1488965869' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-30T00:50:27.605 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:27 smithi116 ceph-mon[38086]: mgrmap e9: x(active, since 14s) 2024-03-30T00:50:27.748 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-30T00:50:27.755 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-30T00:50:27.755 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T00:50:27.755 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-30T00:50:27.755 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-30T00:50:27.755 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-30T00:50:27.755 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.175+0000 7fc9c0a00700 1 Processor -- start 2024-03-30T00:50:27.755 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.175+0000 7fc9c0a00700 1 -- start start 2024-03-30T00:50:27.755 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.175+0000 7fc9c0a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9bc07be20 0x7fc9bc07a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:27.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.175+0000 7fc9c0a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc07a810 con 0x7fc9bc07be20 2024-03-30T00:50:27.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.176+0000 7fc9b9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9bc07be20 0x7fc9bc07a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:27.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.176+0000 7fc9b9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9bc07be20 0x7fc9bc07a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45564/0 (socket says 172.21.15.116:45564) 2024-03-30T00:50:27.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.176+0000 7fc9b9e00700 1 -- 172.21.15.116:0/3871337569 learned_addr learned my addr 172.21.15.116:0/3871337569 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:27.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.176+0000 7fc9b9e00700 1 -- 172.21.15.116:0/3871337569 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc07a950 con 0x7fc9bc07be20 2024-03-30T00:50:27.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.176+0000 7fc9b9e00700 1 --2- 172.21.15.116:0/3871337569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9bc07be20 0x7fc9bc07a2d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0098a0 tx=0x7fc9ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee94721a6d505205 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:27.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.176+0000 7fc9b8a00700 1 -- 172.21.15.116:0/3871337569 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9ac004160 con 0x7fc9bc07be20 2024-03-30T00:50:27.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.177+0000 7fc9b8a00700 1 -- 172.21.15.116:0/3871337569 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9ac0042c0 con 0x7fc9bc07be20 2024-03-30T00:50:27.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.177+0000 7fc9b8a00700 1 -- 172.21.15.116:0/3871337569 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9ac0119b0 con 0x7fc9bc07be20 2024-03-30T00:50:27.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.177+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3871337569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9bc07be20 msgr2=0x7fc9bc07a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:27.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.177+0000 7fc9c0a00700 1 --2- 172.21.15.116:0/3871337569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9bc07be20 0x7fc9bc07a2d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0098a0 tx=0x7fc9ac009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:27.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.177+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3871337569 shutdown_connections 2024-03-30T00:50:27.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.177+0000 7fc9c0a00700 1 --2- 172.21.15.116:0/3871337569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9bc07be20 0x7fc9bc07a2d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:27.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.177+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3871337569 >> 172.21.15.116:0/3871337569 conn(0x7fc9bc101af0 msgr2=0x7fc9bc103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:27.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.177+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3871337569 shutdown_connections 2024-03-30T00:50:27.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.177+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3871337569 wait complete. 2024-03-30T00:50:27.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.178+0000 7fc9c0a00700 1 Processor -- start 2024-03-30T00:50:27.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.178+0000 7fc9c0a00700 1 -- start start 2024-03-30T00:50:27.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.179+0000 7fc9c0a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9bc07be20 0x7fc9bc19c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:27.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.179+0000 7fc9c0a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc19c970 con 0x7fc9bc07be20 2024-03-30T00:50:27.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.179+0000 7fc9b9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9bc07be20 0x7fc9bc19c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:27.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.179+0000 7fc9b9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9bc07be20 0x7fc9bc19c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45576/0 (socket says 172.21.15.116:45576) 2024-03-30T00:50:27.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.179+0000 7fc9b9e00700 1 -- 172.21.15.116:0/3448410578 learned_addr learned my addr 172.21.15.116:0/3448410578 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:27.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.179+0000 7fc9b9e00700 1 -- 172.21.15.116:0/3448410578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac009550 con 0x7fc9bc07be20 2024-03-30T00:50:27.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.180+0000 7fc9b9e00700 1 --2- 172.21.15.116:0/3448410578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9bc07be20 0x7fc9bc19c430 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac011980 tx=0x7fc9ac011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:27.758 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.180+0000 7fc9b2a00700 1 -- 172.21.15.116:0/3448410578 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9ac003b70 con 0x7fc9bc07be20 2024-03-30T00:50:27.758 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.180+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3448410578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9bc19cb70 con 0x7fc9bc07be20 2024-03-30T00:50:27.758 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.180+0000 7fc9b2a00700 1 -- 172.21.15.116:0/3448410578 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9ac01a8f0 con 0x7fc9bc07be20 2024-03-30T00:50:27.758 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.180+0000 7fc9b2a00700 1 -- 172.21.15.116:0/3448410578 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9ac0197d0 con 0x7fc9bc07be20 2024-03-30T00:50:27.758 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.180+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3448410578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9bc19d010 con 0x7fc9bc07be20 2024-03-30T00:50:27.758 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.181+0000 7fc9b2a00700 1 -- 172.21.15.116:0/3448410578 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 9) v1 ==== 45546+0+0 (secure 0 0 0) 0x7fc9ac01a430 con 0x7fc9bc07be20 2024-03-30T00:50:27.758 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.181+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3448410578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9bc10c890 con 0x7fc9bc07be20 2024-03-30T00:50:27.758 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.181+0000 7fc9b2a00700 1 --2- 172.21.15.116:0/3448410578 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fc9a0038750 0x7fc9a003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:27.758 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.182+0000 7fc9b2a00700 1 -- 172.21.15.116:0/3448410578 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc9ac04cb20 con 0x7fc9bc07be20 2024-03-30T00:50:27.758 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.182+0000 7fc9b9400700 1 -- 172.21.15.116:0/3448410578 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fc9a0038750 msgr2=0x7fc9a003ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/131150228 2024-03-30T00:50:27.758 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.182+0000 7fc9b9400700 1 --2- 172.21.15.116:0/3448410578 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fc9a0038750 0x7fc9a003ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T00:50:27.758 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.187+0000 7fc9b2a00700 1 -- 172.21.15.116:0/3448410578 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc9ac00b3e0 con 0x7fc9bc07be20 2024-03-30T00:50:27.758 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.383+0000 7fc9b9400700 1 -- 172.21.15.116:0/3448410578 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fc9a0038750 msgr2=0x7fc9a003ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/131150228 2024-03-30T00:50:27.758 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.383+0000 7fc9b9400700 1 --2- 172.21.15.116:0/3448410578 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fc9a0038750 0x7fc9a003ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T00:50:27.759 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.455+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3448410578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fc9bc19f900 con 0x7fc9bc07be20 2024-03-30T00:50:27.759 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.455+0000 7fc9b2a00700 1 -- 172.21.15.116:0/3448410578 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fc9ac018350 con 0x7fc9bc07be20 2024-03-30T00:50:27.759 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.457+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3448410578 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fc9a0038750 msgr2=0x7fc9a003ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T00:50:27.759 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.458+0000 7fc9c0a00700 1 --2- 172.21.15.116:0/3448410578 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fc9a0038750 0x7fc9a003ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:27.759 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.458+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3448410578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9bc07be20 msgr2=0x7fc9bc19c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:27.759 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.458+0000 7fc9c0a00700 1 --2- 172.21.15.116:0/3448410578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9bc07be20 0x7fc9bc19c430 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac011980 tx=0x7fc9ac011a60 comp rx=0 tx=0).stop 2024-03-30T00:50:27.759 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.458+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3448410578 shutdown_connections 2024-03-30T00:50:27.759 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.458+0000 7fc9c0a00700 1 --2- 172.21.15.116:0/3448410578 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fc9a0038750 0x7fc9a003ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:27.759 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.458+0000 7fc9c0a00700 1 --2- 172.21.15.116:0/3448410578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9bc07be20 0x7fc9bc19c430 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:27.759 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.458+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3448410578 >> 172.21.15.116:0/3448410578 conn(0x7fc9bc101af0 msgr2=0x7fc9bc1026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:27.759 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.458+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3448410578 shutdown_connections 2024-03-30T00:50:27.759 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:27.458+0000 7fc9c0a00700 1 -- 172.21.15.116:0/3448410578 wait complete. 2024-03-30T00:50:27.759 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for the mgr to restart... 2024-03-30T00:50:27.759 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr epoch 9... 2024-03-30T00:50:28.564 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:28 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3448410578' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-30T00:50:28.564 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:28 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:28.245+0000 7fc7e83b6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T00:50:28.564 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:28 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:28.556+0000 7fc7e83b6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T00:50:28.815 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:28 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:28.669+0000 7fc7e83b6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T00:50:29.173 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:28 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:28.808+0000 7fc7e83b6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T00:50:29.174 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:29 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:29.003+0000 7fc7e83b6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T00:50:29.674 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:29 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:29.167+0000 7fc7e83b6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T00:50:29.674 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:29 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:29.277+0000 7fc7e83b6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T00:50:30.174 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:29 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:29.737+0000 7fc7e83b6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T00:50:30.598 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:30 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:30.222+0000 7fc7e83b6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T00:50:30.598 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:30 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:30.469+0000 7fc7e83b6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T00:50:31.853 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:30 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:30.591+0000 7fc7e83b6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T00:50:31.853 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:31 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:31.534+0000 7fc7e83b6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T00:50:31.853 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:31 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:31.657+0000 7fc7e83b6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T00:50:32.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:31 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:31.846+0000 7fc7e83b6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T00:50:32.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:32 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:32.645+0000 7fc7e83b6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T00:50:32.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:32 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:32.884+0000 7fc7e83b6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T00:50:33.423 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:33 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:33.141+0000 7fc7e83b6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T00:50:34.663 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:34 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:34.279+0000 7fc7e83b6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T00:50:34.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:34 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:34.656+0000 7fc7e83b6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T00:50:34.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:34 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:34.773+0000 7fc7e83b6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T00:50:35.424 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:50:35 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:50:35.026+0000 7fc7e83b6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T00:50:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:35 smithi116 ceph-mon[38086]: Active manager daemon x restarted 2024-03-30T00:50:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:35 smithi116 ceph-mon[38086]: Activating manager daemon x 2024-03-30T00:50:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:35 smithi116 ceph-mon[38086]: osdmap e3: 0 total, 0 up, 0 in 2024-03-30T00:50:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:35 smithi116 ceph-mon[38086]: mgrmap e10: x(active, starting, since 0.00434514s) 2024-03-30T00:50:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:50:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T00:50:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T00:50:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T00:50:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T00:50:35.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:35 smithi116 ceph-mon[38086]: Manager daemon x is now available 2024-03-30T00:50:35.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:50:36.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-30T00:50:36.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-30T00:50:36.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:36 smithi116 ceph-mon[38086]: mgrmap e11: x(active, since 1.01166s) 2024-03-30T00:50:36.464 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-30T00:50:36.464 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-30T00:50:36.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-30T00:50:36.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-30T00:50:36.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.201+0000 7fe6ad600700 1 Processor -- start 2024-03-30T00:50:36.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.201+0000 7fe6ad600700 1 -- start start 2024-03-30T00:50:36.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.202+0000 7fe6ad600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a807be20 0x7fe6a807a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:36.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.202+0000 7fe6ad600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a807a810 con 0x7fe6a807be20 2024-03-30T00:50:36.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.202+0000 7fe6a6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a807be20 0x7fe6a807a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:36.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.202+0000 7fe6a6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a807be20 0x7fe6a807a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45586/0 (socket says 172.21.15.116:45586) 2024-03-30T00:50:36.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.202+0000 7fe6a6a00700 1 -- 172.21.15.116:0/1604105575 learned_addr learned my addr 172.21.15.116:0/1604105575 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:36.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.202+0000 7fe6a6a00700 1 -- 172.21.15.116:0/1604105575 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a807a950 con 0x7fe6a807be20 2024-03-30T00:50:36.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.203+0000 7fe6a6a00700 1 --2- 172.21.15.116:0/1604105575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a807be20 0x7fe6a807a2d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe6980098a0 tx=0x7fe698009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa0824dbd139b58c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:36.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.203+0000 7fe6a5600700 1 -- 172.21.15.116:0/1604105575 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe698004160 con 0x7fe6a807be20 2024-03-30T00:50:36.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.203+0000 7fe6a5600700 1 -- 172.21.15.116:0/1604105575 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6980042c0 con 0x7fe6a807be20 2024-03-30T00:50:36.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.203+0000 7fe6a5600700 1 -- 172.21.15.116:0/1604105575 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6980119b0 con 0x7fe6a807be20 2024-03-30T00:50:36.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.204+0000 7fe6ad600700 1 -- 172.21.15.116:0/1604105575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a807be20 msgr2=0x7fe6a807a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:36.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.204+0000 7fe6ad600700 1 --2- 172.21.15.116:0/1604105575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a807be20 0x7fe6a807a2d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe6980098a0 tx=0x7fe698009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:36.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.204+0000 7fe6ad600700 1 -- 172.21.15.116:0/1604105575 shutdown_connections 2024-03-30T00:50:36.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.204+0000 7fe6ad600700 1 --2- 172.21.15.116:0/1604105575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a807be20 0x7fe6a807a2d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:36.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.204+0000 7fe6ad600700 1 -- 172.21.15.116:0/1604105575 >> 172.21.15.116:0/1604105575 conn(0x7fe6a8101af0 msgr2=0x7fe6a8103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:36.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.204+0000 7fe6ad600700 1 -- 172.21.15.116:0/1604105575 shutdown_connections 2024-03-30T00:50:36.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.204+0000 7fe6ad600700 1 -- 172.21.15.116:0/1604105575 wait complete. 2024-03-30T00:50:36.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.205+0000 7fe6ad600700 1 Processor -- start 2024-03-30T00:50:36.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.205+0000 7fe6ad600700 1 -- start start 2024-03-30T00:50:36.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.205+0000 7fe6ad600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a807be20 0x7fe6a819c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:36.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.205+0000 7fe6ad600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a819c970 con 0x7fe6a807be20 2024-03-30T00:50:36.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.205+0000 7fe6a6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a807be20 0x7fe6a819c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.206+0000 7fe6a6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a807be20 0x7fe6a819c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45590/0 (socket says 172.21.15.116:45590) 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.206+0000 7fe6a6a00700 1 -- 172.21.15.116:0/3314612634 learned_addr learned my addr 172.21.15.116:0/3314612634 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.206+0000 7fe6a6a00700 1 -- 172.21.15.116:0/3314612634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe698009550 con 0x7fe6a807be20 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.206+0000 7fe6a6a00700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a807be20 0x7fe6a819c430 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe698011980 tx=0x7fe698011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.206+0000 7fe69f400700 1 -- 172.21.15.116:0/3314612634 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe698003b70 con 0x7fe6a807be20 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.206+0000 7fe69f400700 1 -- 172.21.15.116:0/3314612634 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe69801a8f0 con 0x7fe6a807be20 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.206+0000 7fe69f400700 1 -- 172.21.15.116:0/3314612634 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6980197d0 con 0x7fe6a807be20 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.206+0000 7fe6ad600700 1 -- 172.21.15.116:0/3314612634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a819cb70 con 0x7fe6a807be20 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.206+0000 7fe6ad600700 1 -- 172.21.15.116:0/3314612634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a819d010 con 0x7fe6a807be20 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.207+0000 7fe69f400700 1 -- 172.21.15.116:0/3314612634 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 9) v1 ==== 45546+0+0 (secure 0 0 0) 0x7fe69801a430 con 0x7fe6a807be20 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.208+0000 7fe69f400700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 0x7fe68c03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.208+0000 7fe69f400700 1 -- 172.21.15.116:0/3314612634 --> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe68c03b370 con 0x7fe68c0387a0 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.208+0000 7fe69f400700 1 -- 172.21.15.116:0/3314612634 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe69804cae0 con 0x7fe6a807be20 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.208+0000 7fe6a6000700 1 -- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 msgr2=0x7fe68c03ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/131150228 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.208+0000 7fe6a6000700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 0x7fe68c03ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.409+0000 7fe6a6000700 1 -- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 msgr2=0x7fe68c03ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/131150228 2024-03-30T00:50:36.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.409+0000 7fe6a6000700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 0x7fe68c03ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.810+0000 7fe6a6000700 1 -- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 msgr2=0x7fe68c03ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/131150228 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:28.810+0000 7fe6a6000700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 0x7fe68c03ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:29.611+0000 7fe6a6000700 1 -- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 msgr2=0x7fe68c03ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/131150228 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:29.611+0000 7fe6a6000700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 0x7fe68c03ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:31.213+0000 7fe6a6000700 1 -- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 msgr2=0x7fe68c03ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/131150228 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:31.213+0000 7fe6a6000700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 0x7fe68c03ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:34.414+0000 7fe6a6000700 1 -- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 msgr2=0x7fe68c03ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/131150228 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:34.414+0000 7fe6a6000700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 0x7fe68c03ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:35.031+0000 7fe69f400700 1 -- 172.21.15.116:0/3314612634 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mgrmap(e 10) v1 ==== 45313+0+0 (secure 0 0 0) 0x7fe698019a00 con 0x7fe6a807be20 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:35.031+0000 7fe69f400700 1 -- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 msgr2=0x7fe68c03ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:35.031+0000 7fe69f400700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 0x7fe68c03ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.036+0000 7fe69f400700 1 -- 172.21.15.116:0/3314612634 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 11) v1 ==== 45440+0+0 (secure 0 0 0) 0x7fe698019d80 con 0x7fe6a807be20 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.037+0000 7fe69f400700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe68c03bff0 0x7fe68c03e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.037+0000 7fe69f400700 1 -- 172.21.15.116:0/3314612634 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe68c03b370 con 0x7fe68c03bff0 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.037+0000 7fe6a6000700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe68c03bff0 0x7fe68c03e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.038+0000 7fe6a6000700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe68c03bff0 0x7fe68c03e3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe694003a10 tx=0x7fe694009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.039+0000 7fe69f400700 1 -- 172.21.15.116:0/3314612634 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fe68c03b370 con 0x7fe68c03bff0 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.051+0000 7fe6ad600700 1 -- 172.21.15.116:0/3314612634 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fe6a810c890 con 0x7fe68c03bff0 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.052+0000 7fe69f400700 1 -- 172.21.15.116:0/3314612634 <== mgr.14152 v2:172.21.15.116:6800/3072243062 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fe6a810c890 con 0x7fe68c03bff0 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.052+0000 7fe6ad600700 1 -- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe68c03bff0 msgr2=0x7fe68c03e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.052+0000 7fe6ad600700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe68c03bff0 0x7fe68c03e3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe694003a10 tx=0x7fe694009270 comp rx=0 tx=0).stop 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.052+0000 7fe6ad600700 1 -- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a807be20 msgr2=0x7fe6a819c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.052+0000 7fe6ad600700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a807be20 0x7fe6a819c430 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe698011980 tx=0x7fe698011a60 comp rx=0 tx=0).stop 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.053+0000 7fe6ad600700 1 -- 172.21.15.116:0/3314612634 shutdown_connections 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.053+0000 7fe6ad600700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe68c03bff0 0x7fe68c03e3e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.053+0000 7fe6ad600700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a807be20 0x7fe6a819c430 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.053+0000 7fe6ad600700 1 --2- 172.21.15.116:0/3314612634 >> [v2:172.21.15.116:6800/131150228,v1:172.21.15.116:6801/131150228] conn(0x7fe68c0387a0 0x7fe68c03ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.053+0000 7fe6ad600700 1 -- 172.21.15.116:0/3314612634 >> 172.21.15.116:0/3314612634 conn(0x7fe6a8101af0 msgr2=0x7fe6a81026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.053+0000 7fe6ad600700 1 -- 172.21.15.116:0/3314612634 shutdown_connections 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.053+0000 7fe6ad600700 1 -- 172.21.15.116:0/3314612634 wait complete. 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:mgr epoch 9 is available 2024-03-30T00:50:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:Generating a dashboard self-signed certificate... 2024-03-30T00:50:37.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:37 smithi116 ceph-mon[38086]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-30T00:50:37.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:37 smithi116 ceph-mon[38086]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-30T00:50:37.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:37 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:37.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:37 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:37.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:37 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:37.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-30T00:50:37.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.905+0000 7f3450a00700 1 Processor -- start 2024-03-30T00:50:37.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.906+0000 7f3450a00700 1 -- start start 2024-03-30T00:50:37.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.906+0000 7f3450a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f344c106510 0x7f344c106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:37.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.906+0000 7f3450a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c106f00 con 0x7f344c106510 2024-03-30T00:50:37.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.907+0000 7f3449e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f344c106510 0x7f344c106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:37.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.907+0000 7f3449e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f344c106510 0x7f344c106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54298/0 (socket says 172.21.15.116:54298) 2024-03-30T00:50:37.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.907+0000 7f3449e00700 1 -- 172.21.15.116:0/3552451922 learned_addr learned my addr 172.21.15.116:0/3552451922 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:37.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.907+0000 7f3449e00700 1 -- 172.21.15.116:0/3552451922 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c1076c0 con 0x7f344c106510 2024-03-30T00:50:37.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.907+0000 7f3449e00700 1 --2- 172.21.15.116:0/3552451922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f344c106510 0x7f344c106930 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f343c0098a0 tx=0x7f343c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73527a9d0a9786eb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:37.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.907+0000 7f3448a00700 1 -- 172.21.15.116:0/3552451922 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f343c004160 con 0x7f344c106510 2024-03-30T00:50:37.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.907+0000 7f3448a00700 1 -- 172.21.15.116:0/3552451922 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f343c0042c0 con 0x7f344c106510 2024-03-30T00:50:37.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.907+0000 7f3448a00700 1 -- 172.21.15.116:0/3552451922 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f343c0119b0 con 0x7f344c106510 2024-03-30T00:50:37.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.908+0000 7f3450a00700 1 -- 172.21.15.116:0/3552451922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f344c106510 msgr2=0x7f344c106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:37.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.908+0000 7f3450a00700 1 --2- 172.21.15.116:0/3552451922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f344c106510 0x7f344c106930 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f343c0098a0 tx=0x7f343c009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:37.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.908+0000 7f3450a00700 1 -- 172.21.15.116:0/3552451922 shutdown_connections 2024-03-30T00:50:37.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.908+0000 7f3450a00700 1 --2- 172.21.15.116:0/3552451922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f344c106510 0x7f344c106930 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:37.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.908+0000 7f3450a00700 1 -- 172.21.15.116:0/3552451922 >> 172.21.15.116:0/3552451922 conn(0x7f344c101af0 msgr2=0x7f344c103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:37.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.908+0000 7f3450a00700 1 -- 172.21.15.116:0/3552451922 shutdown_connections 2024-03-30T00:50:37.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.909+0000 7f3450a00700 1 -- 172.21.15.116:0/3552451922 wait complete. 2024-03-30T00:50:37.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.910+0000 7f3450a00700 1 Processor -- start 2024-03-30T00:50:37.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.910+0000 7f3450a00700 1 -- start start 2024-03-30T00:50:37.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.911+0000 7f3450a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f344c106510 0x7f344c19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:37.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.911+0000 7f3450a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c19a7c0 con 0x7f344c106510 2024-03-30T00:50:37.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.911+0000 7f3449e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f344c106510 0x7f344c19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.911+0000 7f3449e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f344c106510 0x7f344c19a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54308/0 (socket says 172.21.15.116:54308) 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.911+0000 7f3449e00700 1 -- 172.21.15.116:0/1655914372 learned_addr learned my addr 172.21.15.116:0/1655914372 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.911+0000 7f3449e00700 1 -- 172.21.15.116:0/1655914372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c009550 con 0x7f344c106510 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.911+0000 7f3449e00700 1 --2- 172.21.15.116:0/1655914372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f344c106510 0x7f344c19a280 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f343c011670 tx=0x7f343c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.912+0000 7f3442a00700 1 -- 172.21.15.116:0/1655914372 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f343c003af0 con 0x7f344c106510 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.912+0000 7f3442a00700 1 -- 172.21.15.116:0/1655914372 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f343c003c50 con 0x7f344c106510 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.912+0000 7f3450a00700 1 -- 172.21.15.116:0/1655914372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344c19a9c0 con 0x7f344c106510 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.912+0000 7f3442a00700 1 -- 172.21.15.116:0/1655914372 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f343c024820 con 0x7f344c106510 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.912+0000 7f3450a00700 1 -- 172.21.15.116:0/1655914372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f344c19ae60 con 0x7f344c106510 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.913+0000 7f3442a00700 1 -- 172.21.15.116:0/1655914372 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 11) v1 ==== 45440+0+0 (secure 0 0 0) 0x7f343c01a430 con 0x7f344c106510 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.913+0000 7f3450a00700 1 -- 172.21.15.116:0/1655914372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344c10a8f0 con 0x7f344c106510 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.914+0000 7f3442a00700 1 --2- 172.21.15.116:0/1655914372 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3430038670 0x7f343003ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.914+0000 7f3442a00700 1 -- 172.21.15.116:0/1655914372 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f343c04c6b0 con 0x7f344c106510 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.914+0000 7f3449400700 1 --2- 172.21.15.116:0/1655914372 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3430038670 0x7f343003ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:37.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.915+0000 7f3449400700 1 --2- 172.21.15.116:0/1655914372 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3430038670 0x7f343003ab30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3438006fd0 tx=0x7f3438006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:37.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:36.919+0000 7f3442a00700 1 -- 172.21.15.116:0/1655914372 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f343c02c9a0 con 0x7f344c106510 2024-03-30T00:50:37.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:37.161+0000 7f3450a00700 1 -- 172.21.15.116:0/1655914372 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f344c19d6f0 con 0x7f3430038670 2024-03-30T00:50:37.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:37.370+0000 7f3442a00700 1 -- 172.21.15.116:0/1655914372 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f344c19d6f0 con 0x7f3430038670 2024-03-30T00:50:37.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:37.374+0000 7f3450a00700 1 -- 172.21.15.116:0/1655914372 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3430038670 msgr2=0x7f343003ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:37.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:37.374+0000 7f3450a00700 1 --2- 172.21.15.116:0/1655914372 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3430038670 0x7f343003ab30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3438006fd0 tx=0x7f3438006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:37.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:37.374+0000 7f3450a00700 1 -- 172.21.15.116:0/1655914372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f344c106510 msgr2=0x7f344c19a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:37.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:37.374+0000 7f3450a00700 1 --2- 172.21.15.116:0/1655914372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f344c106510 0x7f344c19a280 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f343c011670 tx=0x7f343c011a30 comp rx=0 tx=0).stop 2024-03-30T00:50:37.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:37.374+0000 7f3450a00700 1 -- 172.21.15.116:0/1655914372 shutdown_connections 2024-03-30T00:50:37.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:37.374+0000 7f3450a00700 1 --2- 172.21.15.116:0/1655914372 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3430038670 0x7f343003ab30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:37.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:37.374+0000 7f3450a00700 1 --2- 172.21.15.116:0/1655914372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f344c106510 0x7f344c19a280 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:37.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:37.374+0000 7f3450a00700 1 -- 172.21.15.116:0/1655914372 >> 172.21.15.116:0/1655914372 conn(0x7f344c101af0 msgr2=0x7f344c1026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:37.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:37.375+0000 7f3450a00700 1 -- 172.21.15.116:0/1655914372 shutdown_connections 2024-03-30T00:50:37.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:37.375+0000 7f3450a00700 1 -- 172.21.15.116:0/1655914372 wait complete. 2024-03-30T00:50:37.742 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial admin user... 2024-03-30T00:50:38.603 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:38 smithi116 ceph-mon[38086]: [30/Mar/2024:00:50:36] ENGINE Bus STARTING 2024-03-30T00:50:38.603 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:38 smithi116 ceph-mon[38086]: [30/Mar/2024:00:50:36] ENGINE Serving on https://172.21.15.116:7150 2024-03-30T00:50:38.603 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:38 smithi116 ceph-mon[38086]: [30/Mar/2024:00:50:36] ENGINE Serving on http://172.21.15.116:8765 2024-03-30T00:50:38.603 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:38 smithi116 ceph-mon[38086]: [30/Mar/2024:00:50:36] ENGINE Bus STARTED 2024-03-30T00:50:38.603 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:38 smithi116 ceph-mon[38086]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:50:38.603 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:38 smithi116 ceph-mon[38086]: mgrmap e12: x(active, since 2s) 2024-03-30T00:50:38.603 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:38 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:39.025 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$vDFlZisLysJuxRMIkGGFOeleiv3xni41tF9vD/c/QcxjL/H3cOXI6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711759838, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-30T00:50:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.145+0000 7f97bd000700 1 Processor -- start 2024-03-30T00:50:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.145+0000 7f97bd000700 1 -- start start 2024-03-30T00:50:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.146+0000 7f97bd000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97b807be00 0x7f97b807a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.146+0000 7f97bd000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b807a7f0 con 0x7f97b807be00 2024-03-30T00:50:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.146+0000 7f97b5e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97b807be00 0x7f97b807a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.146+0000 7f97b5e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97b807be00 0x7f97b807a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54324/0 (socket says 172.21.15.116:54324) 2024-03-30T00:50:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.146+0000 7f97b5e00700 1 -- 172.21.15.116:0/511907688 learned_addr learned my addr 172.21.15.116:0/511907688 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.147+0000 7f97b5e00700 1 -- 172.21.15.116:0/511907688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b807a930 con 0x7f97b807be00 2024-03-30T00:50:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.147+0000 7f97b5e00700 1 --2- 172.21.15.116:0/511907688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97b807be00 0x7f97b807a2b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f97a80098a0 tx=0x7f97a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a65d4dc1d95144 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.147+0000 7f97b5400700 1 -- 172.21.15.116:0/511907688 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97a8004160 con 0x7f97b807be00 2024-03-30T00:50:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.147+0000 7f97b5400700 1 -- 172.21.15.116:0/511907688 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97a80042c0 con 0x7f97b807be00 2024-03-30T00:50:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.147+0000 7f97b5400700 1 -- 172.21.15.116:0/511907688 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97a80119b0 con 0x7f97b807be00 2024-03-30T00:50:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.148+0000 7f97bd000700 1 -- 172.21.15.116:0/511907688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97b807be00 msgr2=0x7f97b807a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:39.027 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.148+0000 7f97bd000700 1 --2- 172.21.15.116:0/511907688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97b807be00 0x7f97b807a2b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f97a80098a0 tx=0x7f97a8009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:39.027 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.148+0000 7f97bd000700 1 -- 172.21.15.116:0/511907688 shutdown_connections 2024-03-30T00:50:39.027 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.148+0000 7f97bd000700 1 --2- 172.21.15.116:0/511907688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97b807be00 0x7f97b807a2b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:39.027 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.148+0000 7f97bd000700 1 -- 172.21.15.116:0/511907688 >> 172.21.15.116:0/511907688 conn(0x7f97b8101af0 msgr2=0x7f97b8103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:39.027 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.148+0000 7f97bd000700 1 -- 172.21.15.116:0/511907688 shutdown_connections 2024-03-30T00:50:39.027 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.148+0000 7f97bd000700 1 -- 172.21.15.116:0/511907688 wait complete. 2024-03-30T00:50:39.027 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.149+0000 7f97bd000700 1 Processor -- start 2024-03-30T00:50:39.027 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.149+0000 7f97bd000700 1 -- start start 2024-03-30T00:50:39.027 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.150+0000 7f97bd000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97b807be00 0x7f97b819c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:39.027 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.150+0000 7f97bd000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b819c960 con 0x7f97b807be00 2024-03-30T00:50:39.027 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.150+0000 7f97b5e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97b807be00 0x7f97b819c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:39.027 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.150+0000 7f97b5e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97b807be00 0x7f97b819c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54326/0 (socket says 172.21.15.116:54326) 2024-03-30T00:50:39.027 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.150+0000 7f97b5e00700 1 -- 172.21.15.116:0/1653023873 learned_addr learned my addr 172.21.15.116:0/1653023873 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:39.027 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.151+0000 7f97b5e00700 1 -- 172.21.15.116:0/1653023873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a8009550 con 0x7f97b807be00 2024-03-30T00:50:39.028 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.151+0000 7f97b5e00700 1 --2- 172.21.15.116:0/1653023873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97b807be00 0x7f97b819c420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f97a8011980 tx=0x7f97a8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:39.028 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.151+0000 7f97aea00700 1 -- 172.21.15.116:0/1653023873 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97a8003af0 con 0x7f97b807be00 2024-03-30T00:50:39.028 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.151+0000 7f97aea00700 1 -- 172.21.15.116:0/1653023873 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97a8003c50 con 0x7f97b807be00 2024-03-30T00:50:39.028 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.151+0000 7f97aea00700 1 -- 172.21.15.116:0/1653023873 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97a8019760 con 0x7f97b807be00 2024-03-30T00:50:39.028 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.151+0000 7f97bd000700 1 -- 172.21.15.116:0/1653023873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b819cb60 con 0x7f97b807be00 2024-03-30T00:50:39.028 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.151+0000 7f97bd000700 1 -- 172.21.15.116:0/1653023873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b819d000 con 0x7f97b807be00 2024-03-30T00:50:39.028 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.152+0000 7f97aea00700 1 -- 172.21.15.116:0/1653023873 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 12) v1 ==== 45546+0+0 (secure 0 0 0) 0x7f97a801a430 con 0x7f97b807be00 2024-03-30T00:50:39.028 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.152+0000 7f97bd000700 1 -- 172.21.15.116:0/1653023873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b810c8b0 con 0x7f97b807be00 2024-03-30T00:50:39.028 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.153+0000 7f97aea00700 1 --2- 172.21.15.116:0/1653023873 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f979c0387f0 0x7f979c03acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:39.028 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.153+0000 7f97aea00700 1 -- 172.21.15.116:0/1653023873 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f97a804cad0 con 0x7f97b807be00 2024-03-30T00:50:39.028 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.153+0000 7f97afe00700 1 --2- 172.21.15.116:0/1653023873 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f979c0387f0 0x7f979c03acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:39.028 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.154+0000 7f97afe00700 1 --2- 172.21.15.116:0/1653023873 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f979c0387f0 0x7f979c03acb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f97a0006fd0 tx=0x7f97a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:39.028 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.158+0000 7f97aea00700 1 -- 172.21.15.116:0/1653023873 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f97a802d950 con 0x7f97b807be00 2024-03-30T00:50:39.028 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.403+0000 7f97bd000700 1 -- 172.21.15.116:0/1653023873 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f97b8062200 con 0x7f979c0387f0 2024-03-30T00:50:39.029 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.633+0000 7f97aea00700 1 -- 172.21.15.116:0/1653023873 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f97b8062200 con 0x7f979c0387f0 2024-03-30T00:50:39.029 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.635+0000 7f97bd000700 1 -- 172.21.15.116:0/1653023873 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f979c0387f0 msgr2=0x7f979c03acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:39.029 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.635+0000 7f97bd000700 1 --2- 172.21.15.116:0/1653023873 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f979c0387f0 0x7f979c03acb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f97a0006fd0 tx=0x7f97a0006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:39.029 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.635+0000 7f97bd000700 1 -- 172.21.15.116:0/1653023873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97b807be00 msgr2=0x7f97b819c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:39.029 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.635+0000 7f97bd000700 1 --2- 172.21.15.116:0/1653023873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97b807be00 0x7f97b819c420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f97a8011980 tx=0x7f97a8011a60 comp rx=0 tx=0).stop 2024-03-30T00:50:39.029 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.635+0000 7f97bd000700 1 -- 172.21.15.116:0/1653023873 shutdown_connections 2024-03-30T00:50:39.029 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.635+0000 7f97bd000700 1 --2- 172.21.15.116:0/1653023873 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f979c0387f0 0x7f979c03acb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:39.029 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.635+0000 7f97bd000700 1 --2- 172.21.15.116:0/1653023873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97b807be00 0x7f97b819c420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:39.029 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.635+0000 7f97bd000700 1 -- 172.21.15.116:0/1653023873 >> 172.21.15.116:0/1653023873 conn(0x7f97b8101af0 msgr2=0x7f97b81026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:39.029 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.635+0000 7f97bd000700 1 -- 172.21.15.116:0/1653023873 shutdown_connections 2024-03-30T00:50:39.029 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:38.635+0000 7f97bd000700 1 -- 172.21.15.116:0/1653023873 wait complete. 2024-03-30T00:50:39.029 INFO:teuthology.orchestra.run.smithi116.stdout:Fetching dashboard port number... 2024-03-30T00:50:39.664 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:39 smithi116 ceph-mon[38086]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:50:39.665 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:39 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:39.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 8443 2024-03-30T00:50:39.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.425+0000 7fa5c4a00700 1 Processor -- start 2024-03-30T00:50:39.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.426+0000 7fa5c4a00700 1 -- start start 2024-03-30T00:50:39.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.426+0000 7fa5c4a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5c007be00 0x7fa5c007a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:39.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.426+0000 7fa5c4a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c007a7f0 con 0x7fa5c007be00 2024-03-30T00:50:39.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.426+0000 7fa5bde00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5c007be00 0x7fa5c007a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:39.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.426+0000 7fa5bde00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5c007be00 0x7fa5c007a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54338/0 (socket says 172.21.15.116:54338) 2024-03-30T00:50:39.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.426+0000 7fa5bde00700 1 -- 172.21.15.116:0/975806542 learned_addr learned my addr 172.21.15.116:0/975806542 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:39.996 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.427+0000 7fa5bde00700 1 -- 172.21.15.116:0/975806542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c007a930 con 0x7fa5c007be00 2024-03-30T00:50:39.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.427+0000 7fa5bde00700 1 --2- 172.21.15.116:0/975806542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5c007be00 0x7fa5c007a2b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00098a0 tx=0x7fa5b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6470c0ccbb78f0b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:39.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.427+0000 7fa5bd400700 1 -- 172.21.15.116:0/975806542 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5b0004160 con 0x7fa5c007be00 2024-03-30T00:50:39.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.427+0000 7fa5bd400700 1 -- 172.21.15.116:0/975806542 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5b00042c0 con 0x7fa5c007be00 2024-03-30T00:50:39.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.427+0000 7fa5bd400700 1 -- 172.21.15.116:0/975806542 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5b00119b0 con 0x7fa5c007be00 2024-03-30T00:50:39.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.428+0000 7fa5c4a00700 1 -- 172.21.15.116:0/975806542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5c007be00 msgr2=0x7fa5c007a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:39.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.428+0000 7fa5c4a00700 1 --2- 172.21.15.116:0/975806542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5c007be00 0x7fa5c007a2b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00098a0 tx=0x7fa5b0009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:39.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.428+0000 7fa5c4a00700 1 -- 172.21.15.116:0/975806542 shutdown_connections 2024-03-30T00:50:39.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.428+0000 7fa5c4a00700 1 --2- 172.21.15.116:0/975806542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5c007be00 0x7fa5c007a2b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:39.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.428+0000 7fa5c4a00700 1 -- 172.21.15.116:0/975806542 >> 172.21.15.116:0/975806542 conn(0x7fa5c0101af0 msgr2=0x7fa5c0103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:39.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.428+0000 7fa5c4a00700 1 -- 172.21.15.116:0/975806542 shutdown_connections 2024-03-30T00:50:39.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.428+0000 7fa5c4a00700 1 -- 172.21.15.116:0/975806542 wait complete. 2024-03-30T00:50:39.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.429+0000 7fa5c4a00700 1 Processor -- start 2024-03-30T00:50:39.997 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.429+0000 7fa5c4a00700 1 -- start start 2024-03-30T00:50:39.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.429+0000 7fa5c4a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5c007be00 0x7fa5c019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:39.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.429+0000 7fa5c4a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c019c980 con 0x7fa5c007be00 2024-03-30T00:50:39.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.429+0000 7fa5bde00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5c007be00 0x7fa5c019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:39.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.430+0000 7fa5bde00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5c007be00 0x7fa5c019c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54346/0 (socket says 172.21.15.116:54346) 2024-03-30T00:50:39.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.430+0000 7fa5bde00700 1 -- 172.21.15.116:0/2745974400 learned_addr learned my addr 172.21.15.116:0/2745974400 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:39.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.430+0000 7fa5bde00700 1 -- 172.21.15.116:0/2745974400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b0009550 con 0x7fa5c007be00 2024-03-30T00:50:39.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.430+0000 7fa5bde00700 1 --2- 172.21.15.116:0/2745974400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5c007be00 0x7fa5c019c440 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0011980 tx=0x7fa5b0011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:39.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.430+0000 7fa5b6a00700 1 -- 172.21.15.116:0/2745974400 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5b0003af0 con 0x7fa5c007be00 2024-03-30T00:50:39.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.430+0000 7fa5b6a00700 1 -- 172.21.15.116:0/2745974400 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5b0003c50 con 0x7fa5c007be00 2024-03-30T00:50:39.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.430+0000 7fa5c4a00700 1 -- 172.21.15.116:0/2745974400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c019cb80 con 0x7fa5c007be00 2024-03-30T00:50:39.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.430+0000 7fa5b6a00700 1 -- 172.21.15.116:0/2745974400 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5b0019760 con 0x7fa5c007be00 2024-03-30T00:50:39.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.430+0000 7fa5c4a00700 1 -- 172.21.15.116:0/2745974400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c019d020 con 0x7fa5c007be00 2024-03-30T00:50:39.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.431+0000 7fa5b6a00700 1 -- 172.21.15.116:0/2745974400 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 12) v1 ==== 45546+0+0 (secure 0 0 0) 0x7fa5b001a430 con 0x7fa5c007be00 2024-03-30T00:50:39.998 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.431+0000 7fa5c4a00700 1 -- 172.21.15.116:0/2745974400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c010c8b0 con 0x7fa5c007be00 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.432+0000 7fa5b6a00700 1 --2- 172.21.15.116:0/2745974400 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5a40387a0 0x7fa5a403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.432+0000 7fa5b6a00700 1 -- 172.21.15.116:0/2745974400 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa5b004cad0 con 0x7fa5c007be00 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.432+0000 7fa5b7e00700 1 --2- 172.21.15.116:0/2745974400 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5a40387a0 0x7fa5a403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.433+0000 7fa5b7e00700 1 --2- 172.21.15.116:0/2745974400 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5a40387a0 0x7fa5a403ac60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8006fd0 tx=0x7fa5a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.437+0000 7fa5b6a00700 1 -- 172.21.15.116:0/2745974400 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa5b002d950 con 0x7fa5c007be00 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.635+0000 7fa5c4a00700 1 -- 172.21.15.116:0/2745974400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fa5c010cb20 con 0x7fa5c007be00 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.636+0000 7fa5b6a00700 1 -- 172.21.15.116:0/2745974400 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fa5b002d350 con 0x7fa5c007be00 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.638+0000 7fa5c4a00700 1 -- 172.21.15.116:0/2745974400 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5a40387a0 msgr2=0x7fa5a403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.638+0000 7fa5c4a00700 1 --2- 172.21.15.116:0/2745974400 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5a40387a0 0x7fa5a403ac60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8006fd0 tx=0x7fa5a8006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.638+0000 7fa5c4a00700 1 -- 172.21.15.116:0/2745974400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5c007be00 msgr2=0x7fa5c019c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.638+0000 7fa5c4a00700 1 --2- 172.21.15.116:0/2745974400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5c007be00 0x7fa5c019c440 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0011980 tx=0x7fa5b0011a60 comp rx=0 tx=0).stop 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.639+0000 7fa5c4a00700 1 -- 172.21.15.116:0/2745974400 shutdown_connections 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.639+0000 7fa5c4a00700 1 --2- 172.21.15.116:0/2745974400 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5a40387a0 0x7fa5a403ac60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.639+0000 7fa5c4a00700 1 --2- 172.21.15.116:0/2745974400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5c007be00 0x7fa5c019c440 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.639+0000 7fa5c4a00700 1 -- 172.21.15.116:0/2745974400 >> 172.21.15.116:0/2745974400 conn(0x7fa5c0101af0 msgr2=0x7fa5c0102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:40.000 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.639+0000 7fa5c4a00700 1 -- 172.21.15.116:0/2745974400 shutdown_connections 2024-03-30T00:50:40.000 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:39.639+0000 7fa5c4a00700 1 -- 172.21.15.116:0/2745974400 wait complete. 2024-03-30T00:50:40.012 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T00:50:40.012 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-30T00:50:40.027 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T00:50:40.027 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-30T00:50:40.028 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-30T00:50:40.028 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-30T00:50:40.030 INFO:teuthology.orchestra.run.smithi116.stdout:Ceph Dashboard is now available at: 2024-03-30T00:50:40.030 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:50:40.030 INFO:teuthology.orchestra.run.smithi116.stdout: URL: https://smithi116.front.sepia.ceph.com:8443/ 2024-03-30T00:50:40.030 INFO:teuthology.orchestra.run.smithi116.stdout: User: admin 2024-03-30T00:50:40.030 INFO:teuthology.orchestra.run.smithi116.stdout: Password: htkep5gk2r 2024-03-30T00:50:40.030 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:50:40.031 INFO:teuthology.orchestra.run.smithi116.stdout:Saving cluster configuration to /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/config directory 2024-03-30T00:50:40.032 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling autotune for osd_memory_target 2024-03-30T00:50:40.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:40 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2745974400' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-30T00:50:40.982 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.427+0000 7feb97600700 1 Processor -- start 2024-03-30T00:50:40.982 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.427+0000 7feb97600700 1 -- start start 2024-03-30T00:50:40.982 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.427+0000 7feb97600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb9007be20 0x7feb9007a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:40.983 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.427+0000 7feb97600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9007a810 con 0x7feb9007be20 2024-03-30T00:50:40.983 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.427+0000 7feb94c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb9007be20 0x7feb9007a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:40.983 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.428+0000 7feb94c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb9007be20 0x7feb9007a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54350/0 (socket says 172.21.15.116:54350) 2024-03-30T00:50:40.983 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.428+0000 7feb94c00700 1 -- 172.21.15.116:0/3480635719 learned_addr learned my addr 172.21.15.116:0/3480635719 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:40.983 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.428+0000 7feb94c00700 1 -- 172.21.15.116:0/3480635719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9007a950 con 0x7feb9007be20 2024-03-30T00:50:40.983 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.428+0000 7feb94c00700 1 --2- 172.21.15.116:0/3480635719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb9007be20 0x7feb9007a2d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feb840098a0 tx=0x7feb84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0957d296a6da0e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:40.983 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.428+0000 7feb8f400700 1 -- 172.21.15.116:0/3480635719 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb84004160 con 0x7feb9007be20 2024-03-30T00:50:40.983 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.428+0000 7feb8f400700 1 -- 172.21.15.116:0/3480635719 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb840042c0 con 0x7feb9007be20 2024-03-30T00:50:40.983 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.428+0000 7feb8f400700 1 -- 172.21.15.116:0/3480635719 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb840119b0 con 0x7feb9007be20 2024-03-30T00:50:40.983 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.429+0000 7feb97600700 1 -- 172.21.15.116:0/3480635719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb9007be20 msgr2=0x7feb9007a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:40.983 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.429+0000 7feb97600700 1 --2- 172.21.15.116:0/3480635719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb9007be20 0x7feb9007a2d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feb840098a0 tx=0x7feb84009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:40.983 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.429+0000 7feb97600700 1 -- 172.21.15.116:0/3480635719 shutdown_connections 2024-03-30T00:50:40.983 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.429+0000 7feb97600700 1 --2- 172.21.15.116:0/3480635719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb9007be20 0x7feb9007a2d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:40.983 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.429+0000 7feb97600700 1 -- 172.21.15.116:0/3480635719 >> 172.21.15.116:0/3480635719 conn(0x7feb90101af0 msgr2=0x7feb90103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.429+0000 7feb97600700 1 -- 172.21.15.116:0/3480635719 shutdown_connections 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.429+0000 7feb97600700 1 -- 172.21.15.116:0/3480635719 wait complete. 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.430+0000 7feb97600700 1 Processor -- start 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.430+0000 7feb97600700 1 -- start start 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.431+0000 7feb97600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb9007be20 0x7feb9019c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.431+0000 7feb97600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9019c970 con 0x7feb9007be20 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.431+0000 7feb94c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb9007be20 0x7feb9019c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.431+0000 7feb94c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb9007be20 0x7feb9019c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54360/0 (socket says 172.21.15.116:54360) 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.431+0000 7feb94c00700 1 -- 172.21.15.116:0/3455943037 learned_addr learned my addr 172.21.15.116:0/3455943037 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.431+0000 7feb94c00700 1 -- 172.21.15.116:0/3455943037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb84009550 con 0x7feb9007be20 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.431+0000 7feb94c00700 1 --2- 172.21.15.116:0/3455943037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb9007be20 0x7feb9019c430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feb84011980 tx=0x7feb84011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.432+0000 7feb8d600700 1 -- 172.21.15.116:0/3455943037 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb84003b70 con 0x7feb9007be20 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.432+0000 7feb8d600700 1 -- 172.21.15.116:0/3455943037 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb8401a8f0 con 0x7feb9007be20 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.432+0000 7feb97600700 1 -- 172.21.15.116:0/3455943037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9019cb70 con 0x7feb9007be20 2024-03-30T00:50:40.984 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.432+0000 7feb8d600700 1 -- 172.21.15.116:0/3455943037 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb840197d0 con 0x7feb9007be20 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.432+0000 7feb97600700 1 -- 172.21.15.116:0/3455943037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9019d010 con 0x7feb9007be20 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.433+0000 7feb8d600700 1 -- 172.21.15.116:0/3455943037 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 12) v1 ==== 45546+0+0 (secure 0 0 0) 0x7feb8401a430 con 0x7feb9007be20 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.433+0000 7feb97600700 1 -- 172.21.15.116:0/3455943037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb9010c890 con 0x7feb9007be20 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.433+0000 7feb8d600700 1 --2- 172.21.15.116:0/3455943037 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb780387a0 0x7feb7803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.433+0000 7feb8d600700 1 -- 172.21.15.116:0/3455943037 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7feb8404cb00 con 0x7feb9007be20 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.433+0000 7feb8fe00700 1 --2- 172.21.15.116:0/3455943037 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb780387a0 0x7feb7803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.434+0000 7feb8fe00700 1 --2- 172.21.15.116:0/3455943037 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb780387a0 0x7feb7803ac60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feb7c006fd0 tx=0x7feb7c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.438+0000 7feb8d600700 1 -- 172.21.15.116:0/3455943037 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7feb8400b3e0 con 0x7feb9007be20 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.639+0000 7feb97600700 1 -- 172.21.15.116:0/3455943037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7feb9010cad0 con 0x7feb9007be20 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.640+0000 7feb8d600700 1 -- 172.21.15.116:0/3455943037 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7feb84018350 con 0x7feb9007be20 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.642+0000 7feb97600700 1 -- 172.21.15.116:0/3455943037 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb780387a0 msgr2=0x7feb7803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.642+0000 7feb97600700 1 --2- 172.21.15.116:0/3455943037 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb780387a0 0x7feb7803ac60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feb7c006fd0 tx=0x7feb7c006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.642+0000 7feb97600700 1 -- 172.21.15.116:0/3455943037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb9007be20 msgr2=0x7feb9019c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.642+0000 7feb97600700 1 --2- 172.21.15.116:0/3455943037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb9007be20 0x7feb9019c430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feb84011980 tx=0x7feb84011a60 comp rx=0 tx=0).stop 2024-03-30T00:50:40.985 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.642+0000 7feb97600700 1 -- 172.21.15.116:0/3455943037 shutdown_connections 2024-03-30T00:50:40.986 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.642+0000 7feb97600700 1 --2- 172.21.15.116:0/3455943037 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb780387a0 0x7feb7803ac60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:40.986 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.642+0000 7feb97600700 1 --2- 172.21.15.116:0/3455943037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb9007be20 0x7feb9019c430 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:40.986 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.642+0000 7feb97600700 1 -- 172.21.15.116:0/3455943037 >> 172.21.15.116:0/3455943037 conn(0x7feb90101af0 msgr2=0x7feb901026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:40.986 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.642+0000 7feb97600700 1 -- 172.21.15.116:0/3455943037 shutdown_connections 2024-03-30T00:50:40.986 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:40.642+0000 7feb97600700 1 -- 172.21.15.116:0/3455943037 wait complete. 2024-03-30T00:50:42.030 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.349+0000 7f490cc00700 1 Processor -- start 2024-03-30T00:50:42.030 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.350+0000 7f490cc00700 1 -- start start 2024-03-30T00:50:42.030 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.350+0000 7f490cc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f490807be60 0x7f490807a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:42.030 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.350+0000 7f490cc00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490807a8e0 con 0x7f490807be60 2024-03-30T00:50:42.030 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.350+0000 7f4905e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f490807be60 0x7f490807a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:42.030 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.350+0000 7f4905e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f490807be60 0x7f490807a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54364/0 (socket says 172.21.15.116:54364) 2024-03-30T00:50:42.030 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.350+0000 7f4905e00700 1 -- 172.21.15.116:0/3341056994 learned_addr learned my addr 172.21.15.116:0/3341056994 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:42.030 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.351+0000 7f4905e00700 1 -- 172.21.15.116:0/3341056994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490807aa20 con 0x7f490807be60 2024-03-30T00:50:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.351+0000 7f4905e00700 1 --2- 172.21.15.116:0/3341056994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f490807be60 0x7f490807a310 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f48f80098a0 tx=0x7f48f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0271e0a2fe813b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.351+0000 7f4904a00700 1 -- 172.21.15.116:0/3341056994 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48f8004160 con 0x7f490807be60 2024-03-30T00:50:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.351+0000 7f4904a00700 1 -- 172.21.15.116:0/3341056994 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48f80042c0 con 0x7f490807be60 2024-03-30T00:50:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.351+0000 7f4904a00700 1 -- 172.21.15.116:0/3341056994 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48f80119b0 con 0x7f490807be60 2024-03-30T00:50:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.352+0000 7f490cc00700 1 -- 172.21.15.116:0/3341056994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f490807be60 msgr2=0x7f490807a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.352+0000 7f490cc00700 1 --2- 172.21.15.116:0/3341056994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f490807be60 0x7f490807a310 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f48f80098a0 tx=0x7f48f8009bb0 comp rx=0 tx=0).stop 2024-03-30T00:50:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.352+0000 7f490cc00700 1 -- 172.21.15.116:0/3341056994 shutdown_connections 2024-03-30T00:50:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.352+0000 7f490cc00700 1 --2- 172.21.15.116:0/3341056994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f490807be60 0x7f490807a310 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.352+0000 7f490cc00700 1 -- 172.21.15.116:0/3341056994 >> 172.21.15.116:0/3341056994 conn(0x7f4908101af0 msgr2=0x7f4908103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.352+0000 7f490cc00700 1 -- 172.21.15.116:0/3341056994 shutdown_connections 2024-03-30T00:50:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.352+0000 7f490cc00700 1 -- 172.21.15.116:0/3341056994 wait complete. 2024-03-30T00:50:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.353+0000 7f490cc00700 1 Processor -- start 2024-03-30T00:50:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.353+0000 7f490cc00700 1 -- start start 2024-03-30T00:50:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.353+0000 7f490cc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f490807be60 0x7f490819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:42.032 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.353+0000 7f490cc00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490819c980 con 0x7f490807be60 2024-03-30T00:50:42.032 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.354+0000 7f4905e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f490807be60 0x7f490819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:42.032 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.354+0000 7f4905e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f490807be60 0x7f490819c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54378/0 (socket says 172.21.15.116:54378) 2024-03-30T00:50:42.032 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.354+0000 7f4905e00700 1 -- 172.21.15.116:0/2435465374 learned_addr learned my addr 172.21.15.116:0/2435465374 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:42.032 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.354+0000 7f4905e00700 1 -- 172.21.15.116:0/2435465374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f8009550 con 0x7f490807be60 2024-03-30T00:50:42.032 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.354+0000 7f4905e00700 1 --2- 172.21.15.116:0/2435465374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f490807be60 0x7f490819c440 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f48f8011980 tx=0x7f48f8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:42.032 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.354+0000 7f48fea00700 1 -- 172.21.15.116:0/2435465374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48f8003af0 con 0x7f490807be60 2024-03-30T00:50:42.032 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.354+0000 7f48fea00700 1 -- 172.21.15.116:0/2435465374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48f8003c50 con 0x7f490807be60 2024-03-30T00:50:42.032 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.354+0000 7f48fea00700 1 -- 172.21.15.116:0/2435465374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48f8019760 con 0x7f490807be60 2024-03-30T00:50:42.032 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.354+0000 7f490cc00700 1 -- 172.21.15.116:0/2435465374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490819cb80 con 0x7f490807be60 2024-03-30T00:50:42.032 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.355+0000 7f490cc00700 1 -- 172.21.15.116:0/2435465374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490819d020 con 0x7f490807be60 2024-03-30T00:50:42.032 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.356+0000 7f48fea00700 1 -- 172.21.15.116:0/2435465374 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 12) v1 ==== 45546+0+0 (secure 0 0 0) 0x7f48f801ab10 con 0x7f490807be60 2024-03-30T00:50:42.032 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.356+0000 7f490cc00700 1 -- 172.21.15.116:0/2435465374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490810c8b0 con 0x7f490807be60 2024-03-30T00:50:42.032 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.356+0000 7f48fea00700 1 --2- 172.21.15.116:0/2435465374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f48ec0387a0 0x7f48ec03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.356+0000 7f48fea00700 1 -- 172.21.15.116:0/2435465374 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f48f804cb00 con 0x7f490807be60 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.356+0000 7f4905400700 1 --2- 172.21.15.116:0/2435465374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f48ec0387a0 0x7f48ec03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.357+0000 7f4905400700 1 --2- 172.21.15.116:0/2435465374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f48ec0387a0 0x7f48ec03ac60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f48f4006fd0 tx=0x7f48f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.361+0000 7f48fea00700 1 -- 172.21.15.116:0/2435465374 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f48f801a430 con 0x7f490807be60 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.410+0000 7f48fea00700 1 -- 172.21.15.116:0/2435465374 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7f48f801a7b0 con 0x7f490807be60 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.671+0000 7f490cc00700 1 -- 172.21.15.116:0/2435465374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f490810caf0 con 0x7f490807be60 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.673+0000 7f48fea00700 1 -- 172.21.15.116:0/2435465374 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f48f801a430 con 0x7f490807be60 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.676+0000 7f490cc00700 1 -- 172.21.15.116:0/2435465374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f48ec0387a0 msgr2=0x7f48ec03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.676+0000 7f490cc00700 1 --2- 172.21.15.116:0/2435465374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f48ec0387a0 0x7f48ec03ac60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f48f4006fd0 tx=0x7f48f4006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.676+0000 7f490cc00700 1 -- 172.21.15.116:0/2435465374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f490807be60 msgr2=0x7f490819c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.676+0000 7f490cc00700 1 --2- 172.21.15.116:0/2435465374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f490807be60 0x7f490819c440 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f48f8011980 tx=0x7f48f8011a60 comp rx=0 tx=0).stop 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.676+0000 7f490cc00700 1 -- 172.21.15.116:0/2435465374 shutdown_connections 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.676+0000 7f490cc00700 1 --2- 172.21.15.116:0/2435465374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f48ec0387a0 0x7f48ec03ac60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.676+0000 7f490cc00700 1 --2- 172.21.15.116:0/2435465374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f490807be60 0x7f490819c440 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.676+0000 7f490cc00700 1 -- 172.21.15.116:0/2435465374 >> 172.21.15.116:0/2435465374 conn(0x7f4908101af0 msgr2=0x7f4908102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.676+0000 7f490cc00700 1 -- 172.21.15.116:0/2435465374 shutdown_connections 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-30T00:50:41.676+0000 7f490cc00700 1 -- 172.21.15.116:0/2435465374 wait complete. 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout:Or, if you are only running a single cluster on this host: 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout: ceph telemetry on 2024-03-30T00:50:42.034 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:50:42.035 INFO:teuthology.orchestra.run.smithi116.stdout:For more information see: 2024-03-30T00:50:42.035 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:50:42.035 INFO:teuthology.orchestra.run.smithi116.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-30T00:50:42.035 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:50:42.035 INFO:teuthology.orchestra.run.smithi116.stdout:Bootstrap complete. 2024-03-30T00:50:42.088 INFO:tasks.cephadm:Fetching config... 2024-03-30T00:50:42.089 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:50:42.089 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-30T00:50:42.130 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-30T00:50:42.130 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:50:42.131 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-30T00:50:42.215 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-30T00:50:42.215 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:50:42.215 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/keyring of=/dev/stdout 2024-03-30T00:50:42.330 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-30T00:50:42.330 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:50:42.330 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-30T00:50:42.411 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-30T00:50:42.412 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWdt11SKUN0YqjB3gX13kMmn3vKU8S7g8aBJwP57XmZ7TbaSuqA7qbYz3zAXfwrGquzHZs+X6BAJ4Ps3NbBspBUvFcOj/qWdjtUUvf/EqxE+EwNPHiNzLy4HdTwrkoA6nMpVWBDL5B39F11cBfDGWUg7wZWsyySUw4d0T1kyxKip4eCPKLY3EzLrYcqCb038WKF4dNIfFjlSK622Cz2mXsqXIhcv6FO9uujmyqvQ6qrq/2RAcvm2/h67ZtK1scnPxb5PPe5zYU7EkoE2tXXH1V50RJ9EnbrSvo2rNLnc9JAdXKUx6p9b4HowSwS/xidyHgOhFqqfBCXL5s2gWoSpWq+3mmKqJr+kakY/t2YTBYzPOOTJb90Jntbh/Z0OsTiwl1Iyq/8w0BNGs9g8cyvppvL9oZdfk5p72xRHjePqZdRfLOtQxjg6/Jz1DRQfDvkLMq7VWTET2GFHys30FKtDoEhxA+IZX48i0b9Tb/0bw5av2Rb/eK/TrJ0CuQIsiQPyU= ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T00:50:42.540 INFO:teuthology.orchestra.run.smithi116.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWdt11SKUN0YqjB3gX13kMmn3vKU8S7g8aBJwP57XmZ7TbaSuqA7qbYz3zAXfwrGquzHZs+X6BAJ4Ps3NbBspBUvFcOj/qWdjtUUvf/EqxE+EwNPHiNzLy4HdTwrkoA6nMpVWBDL5B39F11cBfDGWUg7wZWsyySUw4d0T1kyxKip4eCPKLY3EzLrYcqCb038WKF4dNIfFjlSK622Cz2mXsqXIhcv6FO9uujmyqvQ6qrq/2RAcvm2/h67ZtK1scnPxb5PPe5zYU7EkoE2tXXH1V50RJ9EnbrSvo2rNLnc9JAdXKUx6p9b4HowSwS/xidyHgOhFqqfBCXL5s2gWoSpWq+3mmKqJr+kakY/t2YTBYzPOOTJb90Jntbh/Z0OsTiwl1Iyq/8w0BNGs9g8cyvppvL9oZdfk5p72xRHjePqZdRfLOtQxjg6/Jz1DRQfDvkLMq7VWTET2GFHys30FKtDoEhxA+IZX48i0b9Tb/0bw5av2Rb/eK/TrJ0CuQIsiQPyU= ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:50:42.560 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:42 smithi116 ceph-mon[38086]: mgrmap e13: x(active, since 6s) 2024-03-30T00:50:42.560 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:42 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2435465374' entity='client.admin' 2024-03-30T00:50:42.568 DEBUG:teuthology.orchestra.run.smithi130:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWdt11SKUN0YqjB3gX13kMmn3vKU8S7g8aBJwP57XmZ7TbaSuqA7qbYz3zAXfwrGquzHZs+X6BAJ4Ps3NbBspBUvFcOj/qWdjtUUvf/EqxE+EwNPHiNzLy4HdTwrkoA6nMpVWBDL5B39F11cBfDGWUg7wZWsyySUw4d0T1kyxKip4eCPKLY3EzLrYcqCb038WKF4dNIfFjlSK622Cz2mXsqXIhcv6FO9uujmyqvQ6qrq/2RAcvm2/h67ZtK1scnPxb5PPe5zYU7EkoE2tXXH1V50RJ9EnbrSvo2rNLnc9JAdXKUx6p9b4HowSwS/xidyHgOhFqqfBCXL5s2gWoSpWq+3mmKqJr+kakY/t2YTBYzPOOTJb90Jntbh/Z0OsTiwl1Iyq/8w0BNGs9g8cyvppvL9oZdfk5p72xRHjePqZdRfLOtQxjg6/Jz1DRQfDvkLMq7VWTET2GFHys30FKtDoEhxA+IZX48i0b9Tb/0bw5av2Rb/eK/TrJ0CuQIsiQPyU= ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T00:50:42.614 INFO:teuthology.orchestra.run.smithi130.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWdt11SKUN0YqjB3gX13kMmn3vKU8S7g8aBJwP57XmZ7TbaSuqA7qbYz3zAXfwrGquzHZs+X6BAJ4Ps3NbBspBUvFcOj/qWdjtUUvf/EqxE+EwNPHiNzLy4HdTwrkoA6nMpVWBDL5B39F11cBfDGWUg7wZWsyySUw4d0T1kyxKip4eCPKLY3EzLrYcqCb038WKF4dNIfFjlSK622Cz2mXsqXIhcv6FO9uujmyqvQ6qrq/2RAcvm2/h67ZtK1scnPxb5PPe5zYU7EkoE2tXXH1V50RJ9EnbrSvo2rNLnc9JAdXKUx6p9b4HowSwS/xidyHgOhFqqfBCXL5s2gWoSpWq+3mmKqJr+kakY/t2YTBYzPOOTJb90Jntbh/Z0OsTiwl1Iyq/8w0BNGs9g8cyvppvL9oZdfk5p72xRHjePqZdRfLOtQxjg6/Jz1DRQfDvkLMq7VWTET2GFHys30FKtDoEhxA+IZX48i0b9Tb/0bw5av2Rb/eK/TrJ0CuQIsiQPyU= ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:50:42.627 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWdt11SKUN0YqjB3gX13kMmn3vKU8S7g8aBJwP57XmZ7TbaSuqA7qbYz3zAXfwrGquzHZs+X6BAJ4Ps3NbBspBUvFcOj/qWdjtUUvf/EqxE+EwNPHiNzLy4HdTwrkoA6nMpVWBDL5B39F11cBfDGWUg7wZWsyySUw4d0T1kyxKip4eCPKLY3EzLrYcqCb038WKF4dNIfFjlSK622Cz2mXsqXIhcv6FO9uujmyqvQ6qrq/2RAcvm2/h67ZtK1scnPxb5PPe5zYU7EkoE2tXXH1V50RJ9EnbrSvo2rNLnc9JAdXKUx6p9b4HowSwS/xidyHgOhFqqfBCXL5s2gWoSpWq+3mmKqJr+kakY/t2YTBYzPOOTJb90Jntbh/Z0OsTiwl1Iyq/8w0BNGs9g8cyvppvL9oZdfk5p72xRHjePqZdRfLOtQxjg6/Jz1DRQfDvkLMq7VWTET2GFHys30FKtDoEhxA+IZX48i0b9Tb/0bw5av2Rb/eK/TrJ0CuQIsiQPyU= ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T00:50:42.675 INFO:teuthology.orchestra.run.smithi181.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWdt11SKUN0YqjB3gX13kMmn3vKU8S7g8aBJwP57XmZ7TbaSuqA7qbYz3zAXfwrGquzHZs+X6BAJ4Ps3NbBspBUvFcOj/qWdjtUUvf/EqxE+EwNPHiNzLy4HdTwrkoA6nMpVWBDL5B39F11cBfDGWUg7wZWsyySUw4d0T1kyxKip4eCPKLY3EzLrYcqCb038WKF4dNIfFjlSK622Cz2mXsqXIhcv6FO9uujmyqvQ6qrq/2RAcvm2/h67ZtK1scnPxb5PPe5zYU7EkoE2tXXH1V50RJ9EnbrSvo2rNLnc9JAdXKUx6p9b4HowSwS/xidyHgOhFqqfBCXL5s2gWoSpWq+3mmKqJr+kakY/t2YTBYzPOOTJb90Jntbh/Z0OsTiwl1Iyq/8w0BNGs9g8cyvppvL9oZdfk5p72xRHjePqZdRfLOtQxjg6/Jz1DRQfDvkLMq7VWTET2GFHys30FKtDoEhxA+IZX48i0b9Tb/0bw5av2Rb/eK/TrJ0CuQIsiQPyU= ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:50:42.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-30T00:50:43.049 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:50:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.287+0000 7f7abba00700 1 -- 172.21.15.116:0/1444862015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ab40fefa0 msgr2=0x7f7ab41013d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.287+0000 7f7abba00700 1 --2- 172.21.15.116:0/1444862015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ab40fefa0 0x7f7ab41013d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009910 tx=0x7f7aa8009c20 comp rx=0 tx=0).stop 2024-03-30T00:50:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.287+0000 7f7abba00700 1 -- 172.21.15.116:0/1444862015 shutdown_connections 2024-03-30T00:50:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.287+0000 7f7abba00700 1 --2- 172.21.15.116:0/1444862015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ab40fefa0 0x7f7ab41013d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.287+0000 7f7abba00700 1 -- 172.21.15.116:0/1444862015 >> 172.21.15.116:0/1444862015 conn(0x7f7ab40fab80 msgr2=0x7f7ab40fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.288+0000 7f7abba00700 1 -- 172.21.15.116:0/1444862015 shutdown_connections 2024-03-30T00:50:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.288+0000 7f7abba00700 1 -- 172.21.15.116:0/1444862015 wait complete. 2024-03-30T00:50:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.289+0000 7f7abba00700 1 Processor -- start 2024-03-30T00:50:44.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.289+0000 7f7abba00700 1 -- start start 2024-03-30T00:50:44.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.290+0000 7f7abba00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ab40fefa0 0x7f7ab419c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:44.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.290+0000 7f7abba00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab419c670 con 0x7f7ab40fefa0 2024-03-30T00:50:44.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.290+0000 7f7ab9000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ab40fefa0 0x7f7ab419c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:44.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.290+0000 7f7ab9000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ab40fefa0 0x7f7ab419c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54394/0 (socket says 172.21.15.116:54394) 2024-03-30T00:50:44.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.290+0000 7f7ab9000700 1 -- 172.21.15.116:0/1322494766 learned_addr learned my addr 172.21.15.116:0/1322494766 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:44.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.291+0000 7f7ab9000700 1 -- 172.21.15.116:0/1322494766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa80095f0 con 0x7f7ab40fefa0 2024-03-30T00:50:44.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.291+0000 7f7ab9000700 1 --2- 172.21.15.116:0/1322494766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ab40fefa0 0x7f7ab419c130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8005950 tx=0x7f7aa80057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:44.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.291+0000 7f7ab1600700 1 -- 172.21.15.116:0/1322494766 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7aa801d070 con 0x7f7ab40fefa0 2024-03-30T00:50:44.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.291+0000 7f7ab1600700 1 -- 172.21.15.116:0/1322494766 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7aa800bc10 con 0x7f7ab40fefa0 2024-03-30T00:50:44.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.291+0000 7f7abba00700 1 -- 172.21.15.116:0/1322494766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab419c870 con 0x7f7ab40fefa0 2024-03-30T00:50:44.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.291+0000 7f7ab1600700 1 -- 172.21.15.116:0/1322494766 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7aa80039d0 con 0x7f7ab40fefa0 2024-03-30T00:50:44.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.291+0000 7f7abba00700 1 -- 172.21.15.116:0/1322494766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab419cd10 con 0x7f7ab40fefa0 2024-03-30T00:50:44.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.293+0000 7f7abba00700 1 -- 172.21.15.116:0/1322494766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab4196290 con 0x7f7ab40fefa0 2024-03-30T00:50:44.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.293+0000 7f7ab1600700 1 -- 172.21.15.116:0/1322494766 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7f7aa8003b30 con 0x7f7ab40fefa0 2024-03-30T00:50:44.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.294+0000 7f7ab1600700 1 --2- 172.21.15.116:0/1322494766 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7a9c038840 0x7f7a9c03ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:44.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.294+0000 7f7ab1600700 1 -- 172.21.15.116:0/1322494766 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7aa804c620 con 0x7f7ab40fefa0 2024-03-30T00:50:44.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.294+0000 7f7ab3e00700 1 --2- 172.21.15.116:0/1322494766 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7a9c038840 0x7f7a9c03ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:44.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.295+0000 7f7ab3e00700 1 --2- 172.21.15.116:0/1322494766 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7a9c038840 0x7f7a9c03ad00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0006fd0 tx=0x7f7aa0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:44.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.299+0000 7f7ab1600700 1 -- 172.21.15.116:0/1322494766 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7aa8029950 con 0x7f7ab40fefa0 2024-03-30T00:50:44.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.518+0000 7f7abba00700 1 -- 172.21.15.116:0/1322494766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f7ab40623e0 con 0x7f7ab40fefa0 2024-03-30T00:50:44.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.522+0000 7f7ab1600700 1 -- 172.21.15.116:0/1322494766 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7ab40623e0 con 0x7f7ab40fefa0 2024-03-30T00:50:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.524+0000 7f7abba00700 1 -- 172.21.15.116:0/1322494766 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7a9c038840 msgr2=0x7f7a9c03ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.524+0000 7f7abba00700 1 --2- 172.21.15.116:0/1322494766 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7a9c038840 0x7f7a9c03ad00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0006fd0 tx=0x7f7aa0006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.524+0000 7f7abba00700 1 -- 172.21.15.116:0/1322494766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ab40fefa0 msgr2=0x7f7ab419c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.524+0000 7f7abba00700 1 --2- 172.21.15.116:0/1322494766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ab40fefa0 0x7f7ab419c130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8005950 tx=0x7f7aa80057c0 comp rx=0 tx=0).stop 2024-03-30T00:50:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.524+0000 7f7abba00700 1 -- 172.21.15.116:0/1322494766 shutdown_connections 2024-03-30T00:50:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.524+0000 7f7abba00700 1 --2- 172.21.15.116:0/1322494766 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7a9c038840 0x7f7a9c03ad00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.524+0000 7f7abba00700 1 --2- 172.21.15.116:0/1322494766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ab40fefa0 0x7f7ab419c130 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.524+0000 7f7abba00700 1 -- 172.21.15.116:0/1322494766 >> 172.21.15.116:0/1322494766 conn(0x7f7ab40fab80 msgr2=0x7f7ab40fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.525+0000 7f7abba00700 1 -- 172.21.15.116:0/1322494766 shutdown_connections 2024-03-30T00:50:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:44.525+0000 7f7abba00700 1 -- 172.21.15.116:0/1322494766 wait complete. 2024-03-30T00:50:44.921 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-30T00:50:44.921 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-30T00:50:45.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:45.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:45.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:50:45.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:45.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:50:45.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:45.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:45 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1322494766' entity='client.admin' 2024-03-30T00:50:45.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:50:45.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:45.282 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:50:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.466+0000 7fb217200700 1 -- 172.21.15.116:0/2545295818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb210074d80 msgr2=0x7fb2100731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.466+0000 7fb217200700 1 --2- 172.21.15.116:0/2545295818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb210074d80 0x7fb2100731e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb200009910 tx=0x7fb200009c20 comp rx=0 tx=0).stop 2024-03-30T00:50:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.466+0000 7fb217200700 1 -- 172.21.15.116:0/2545295818 shutdown_connections 2024-03-30T00:50:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.466+0000 7fb217200700 1 --2- 172.21.15.116:0/2545295818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb210074d80 0x7fb2100731e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:46.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.466+0000 7fb217200700 1 -- 172.21.15.116:0/2545295818 >> 172.21.15.116:0/2545295818 conn(0x7fb2100fbcd0 msgr2=0x7fb2100fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:46.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.466+0000 7fb217200700 1 -- 172.21.15.116:0/2545295818 shutdown_connections 2024-03-30T00:50:46.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.466+0000 7fb217200700 1 -- 172.21.15.116:0/2545295818 wait complete. 2024-03-30T00:50:46.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.467+0000 7fb217200700 1 Processor -- start 2024-03-30T00:50:46.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.467+0000 7fb217200700 1 -- start start 2024-03-30T00:50:46.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.467+0000 7fb217200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb210074d80 0x7fb210197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:46.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.467+0000 7fb217200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2101982a0 con 0x7fb210074d80 2024-03-30T00:50:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.468+0000 7fb20fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb210074d80 0x7fb210197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.468+0000 7fb20fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb210074d80 0x7fb210197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54046/0 (socket says 172.21.15.116:54046) 2024-03-30T00:50:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.468+0000 7fb20fe00700 1 -- 172.21.15.116:0/135476466 learned_addr learned my addr 172.21.15.116:0/135476466 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.468+0000 7fb20fe00700 1 -- 172.21.15.116:0/135476466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2000095f0 con 0x7fb210074d80 2024-03-30T00:50:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.468+0000 7fb20fe00700 1 --2- 172.21.15.116:0/135476466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb210074d80 0x7fb210197d60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb200005950 tx=0x7fb2000057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.469+0000 7fb20cc00700 1 -- 172.21.15.116:0/135476466 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb20001d070 con 0x7fb210074d80 2024-03-30T00:50:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.469+0000 7fb217200700 1 -- 172.21.15.116:0/135476466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2101984a0 con 0x7fb210074d80 2024-03-30T00:50:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.469+0000 7fb20cc00700 1 -- 172.21.15.116:0/135476466 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb20000bc10 con 0x7fb210074d80 2024-03-30T00:50:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.469+0000 7fb20cc00700 1 -- 172.21.15.116:0/135476466 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2000039d0 con 0x7fb210074d80 2024-03-30T00:50:46.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.469+0000 7fb217200700 1 -- 172.21.15.116:0/135476466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb210198940 con 0x7fb210074d80 2024-03-30T00:50:46.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.470+0000 7fb20cc00700 1 -- 172.21.15.116:0/135476466 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7fb200003c90 con 0x7fb210074d80 2024-03-30T00:50:46.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.470+0000 7fb217200700 1 -- 172.21.15.116:0/135476466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb210191630 con 0x7fb210074d80 2024-03-30T00:50:46.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.470+0000 7fb20cc00700 1 --2- 172.21.15.116:0/135476466 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb1f40387a0 0x7fb1f403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:46.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.470+0000 7fb20cc00700 1 -- 172.21.15.116:0/135476466 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb20004c560 con 0x7fb210074d80 2024-03-30T00:50:46.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.470+0000 7fb20f400700 1 --2- 172.21.15.116:0/135476466 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb1f40387a0 0x7fb1f403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:46.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.471+0000 7fb20f400700 1 --2- 172.21.15.116:0/135476466 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb1f40387a0 0x7fb1f403ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc006fd0 tx=0x7fb1fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:46.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.475+0000 7fb20cc00700 1 -- 172.21.15.116:0/135476466 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb20000ba70 con 0x7fb210074d80 2024-03-30T00:50:46.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.683+0000 7fb217200700 1 -- 172.21.15.116:0/135476466 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2100611f0 con 0x7fb1f40387a0 2024-03-30T00:50:46.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.686+0000 7fb20cc00700 1 -- 172.21.15.116:0/135476466 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb2100611f0 con 0x7fb1f40387a0 2024-03-30T00:50:46.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.689+0000 7fb217200700 1 -- 172.21.15.116:0/135476466 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb1f40387a0 msgr2=0x7fb1f403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:46.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.689+0000 7fb217200700 1 --2- 172.21.15.116:0/135476466 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb1f40387a0 0x7fb1f403ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc006fd0 tx=0x7fb1fc006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:46.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.689+0000 7fb217200700 1 -- 172.21.15.116:0/135476466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb210074d80 msgr2=0x7fb210197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:46.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.689+0000 7fb217200700 1 --2- 172.21.15.116:0/135476466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb210074d80 0x7fb210197d60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb200005950 tx=0x7fb2000057c0 comp rx=0 tx=0).stop 2024-03-30T00:50:46.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.689+0000 7fb217200700 1 -- 172.21.15.116:0/135476466 shutdown_connections 2024-03-30T00:50:46.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.689+0000 7fb217200700 1 --2- 172.21.15.116:0/135476466 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb1f40387a0 0x7fb1f403ac60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:46.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.689+0000 7fb217200700 1 --2- 172.21.15.116:0/135476466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb210074d80 0x7fb210197d60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:46.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.689+0000 7fb217200700 1 -- 172.21.15.116:0/135476466 >> 172.21.15.116:0/135476466 conn(0x7fb2100fbcd0 msgr2=0x7fb2100fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:46.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.689+0000 7fb217200700 1 -- 172.21.15.116:0/135476466 shutdown_connections 2024-03-30T00:50:46.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:46.689+0000 7fb217200700 1 -- 172.21.15.116:0/135476466 wait complete. 2024-03-30T00:50:47.000 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi130 2024-03-30T00:50:47.000 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-30T00:50:47.000 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/etc/ceph/ceph.conf 2024-03-30T00:50:47.021 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-30T00:50:47.021 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T00:50:47.081 INFO:tasks.cephadm:Adding host smithi130 to orchestrator... 2024-03-30T00:50:47.081 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi130 2024-03-30T00:50:47.453 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:50:47.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:47 smithi116 ceph-mon[38086]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:50:47.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:47 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:47.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:47 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:50:47.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:47 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:50:47.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:47 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:50:47.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:47 smithi116 ceph-mon[38086]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-30T00:50:47.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:47 smithi116 ceph-mon[38086]: Updating smithi116:/var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T00:50:48.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.510+0000 7f5c18800700 1 -- 172.21.15.116:0/3338643181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c101066b0 msgr2=0x7f5c10106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:48.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.510+0000 7f5c18800700 1 --2- 172.21.15.116:0/3338643181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c101066b0 0x7f5c10106a90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5c04009910 tx=0x7f5c04009c20 comp rx=0 tx=0).stop 2024-03-30T00:50:48.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.510+0000 7f5c18800700 1 -- 172.21.15.116:0/3338643181 shutdown_connections 2024-03-30T00:50:48.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.510+0000 7f5c18800700 1 --2- 172.21.15.116:0/3338643181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c101066b0 0x7f5c10106a90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:48.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.510+0000 7f5c18800700 1 -- 172.21.15.116:0/3338643181 >> 172.21.15.116:0/3338643181 conn(0x7f5c100fc1e0 msgr2=0x7f5c100fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:48.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.510+0000 7f5c18800700 1 -- 172.21.15.116:0/3338643181 shutdown_connections 2024-03-30T00:50:48.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.510+0000 7f5c18800700 1 -- 172.21.15.116:0/3338643181 wait complete. 2024-03-30T00:50:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.511+0000 7f5c18800700 1 Processor -- start 2024-03-30T00:50:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.511+0000 7f5c18800700 1 -- start start 2024-03-30T00:50:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.511+0000 7f5c18800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c101066b0 0x7f5c10198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.511+0000 7f5c18800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c10198660 con 0x7f5c101066b0 2024-03-30T00:50:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.512+0000 7f5c15e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c101066b0 0x7f5c10198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.512+0000 7f5c15e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c101066b0 0x7f5c10198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54076/0 (socket says 172.21.15.116:54076) 2024-03-30T00:50:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.512+0000 7f5c15e00700 1 -- 172.21.15.116:0/348100863 learned_addr learned my addr 172.21.15.116:0/348100863 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.512+0000 7f5c15e00700 1 -- 172.21.15.116:0/348100863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c040095f0 con 0x7f5c101066b0 2024-03-30T00:50:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.512+0000 7f5c15e00700 1 --2- 172.21.15.116:0/348100863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c101066b0 0x7f5c10198120 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5c04006010 tx=0x7f5c04004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:48.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.513+0000 7f5c0aa00700 1 -- 172.21.15.116:0/348100863 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c0401d070 con 0x7f5c101066b0 2024-03-30T00:50:48.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.513+0000 7f5c0aa00700 1 -- 172.21.15.116:0/348100863 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c0400bbb0 con 0x7f5c101066b0 2024-03-30T00:50:48.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.513+0000 7f5c18800700 1 -- 172.21.15.116:0/348100863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c101988c0 con 0x7f5c101066b0 2024-03-30T00:50:48.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.513+0000 7f5c0aa00700 1 -- 172.21.15.116:0/348100863 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c04003810 con 0x7f5c101066b0 2024-03-30T00:50:48.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.513+0000 7f5c18800700 1 -- 172.21.15.116:0/348100863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c10198d40 con 0x7f5c101066b0 2024-03-30T00:50:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.514+0000 7f5c18800700 1 -- 172.21.15.116:0/348100863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c10101df0 con 0x7f5c101066b0 2024-03-30T00:50:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.514+0000 7f5c0aa00700 1 -- 172.21.15.116:0/348100863 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7f5c0400b6f0 con 0x7f5c101066b0 2024-03-30T00:50:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.515+0000 7f5c0aa00700 1 --2- 172.21.15.116:0/348100863 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5bf80387f0 0x7f5bf803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.515+0000 7f5c0aa00700 1 -- 172.21.15.116:0/348100863 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5c0404c420 con 0x7f5c101066b0 2024-03-30T00:50:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.515+0000 7f5c15400700 1 --2- 172.21.15.116:0/348100863 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5bf80387f0 0x7f5bf803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:48.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.516+0000 7f5c15400700 1 --2- 172.21.15.116:0/348100863 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5bf80387f0 0x7f5bf803acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5c00006fd0 tx=0x7f5c00006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:48.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.520+0000 7f5c0aa00700 1 -- 172.21.15.116:0/348100863 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5c04029540 con 0x7f5c101066b0 2024-03-30T00:50:48.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:48.758+0000 7f5c18800700 1 -- 172.21.15.116:0/348100863 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c1010a9b0 con 0x7f5bf80387f0 2024-03-30T00:50:49.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:48 smithi116 ceph-mon[38086]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-30T00:50:49.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:48 smithi116 ceph-mon[38086]: Updating smithi116:/var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T00:50:50.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:49 smithi116 ceph-mon[38086]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:50:50.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:50.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:50.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:50.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:49 smithi116 ceph-mon[38086]: Deploying cephadm binary to smithi130 2024-03-30T00:50:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:50.968+0000 7f5c0aa00700 1 -- 172.21.15.116:0/348100863 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5c1010a9b0 con 0x7f5bf80387f0 2024-03-30T00:50:50.974 INFO:teuthology.orchestra.run.smithi116.stdout:Added host 'smithi130' with addr '172.21.15.130' 2024-03-30T00:50:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:50.970+0000 7f5c18800700 1 -- 172.21.15.116:0/348100863 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5bf80387f0 msgr2=0x7f5bf803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:50.970+0000 7f5c18800700 1 --2- 172.21.15.116:0/348100863 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5bf80387f0 0x7f5bf803acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5c00006fd0 tx=0x7f5c00006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:50.970+0000 7f5c18800700 1 -- 172.21.15.116:0/348100863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c101066b0 msgr2=0x7f5c10198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:50.970+0000 7f5c18800700 1 --2- 172.21.15.116:0/348100863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c101066b0 0x7f5c10198120 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5c04006010 tx=0x7f5c04004ff0 comp rx=0 tx=0).stop 2024-03-30T00:50:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:50.970+0000 7f5c18800700 1 -- 172.21.15.116:0/348100863 shutdown_connections 2024-03-30T00:50:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:50.970+0000 7f5c18800700 1 --2- 172.21.15.116:0/348100863 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5bf80387f0 0x7f5bf803acb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:50.970+0000 7f5c18800700 1 --2- 172.21.15.116:0/348100863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c101066b0 0x7f5c10198120 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:50.970+0000 7f5c18800700 1 -- 172.21.15.116:0/348100863 >> 172.21.15.116:0/348100863 conn(0x7f5c100fc1e0 msgr2=0x7f5c101082a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:50.971+0000 7f5c18800700 1 -- 172.21.15.116:0/348100863 shutdown_connections 2024-03-30T00:50:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:50.971+0000 7f5c18800700 1 -- 172.21.15.116:0/348100863 wait complete. 2024-03-30T00:50:51.329 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-03-30T00:50:51.675 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:50:52.118 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:51 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:52.118 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:51 smithi116 ceph-mon[38086]: Added host smithi130 2024-03-30T00:50:52.118 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:51 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:50:52.118 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:51 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:52.118 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:51 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:52.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.725+0000 7fdb5d000700 1 -- 172.21.15.116:0/931547363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb581066b0 msgr2=0x7fdb58106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:52.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.725+0000 7fdb5d000700 1 --2- 172.21.15.116:0/931547363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb581066b0 0x7fdb58106a90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009910 tx=0x7fdb48009c20 comp rx=0 tx=0).stop 2024-03-30T00:50:52.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.725+0000 7fdb5d000700 1 -- 172.21.15.116:0/931547363 shutdown_connections 2024-03-30T00:50:52.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.725+0000 7fdb5d000700 1 --2- 172.21.15.116:0/931547363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb581066b0 0x7fdb58106a90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:52.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.725+0000 7fdb5d000700 1 -- 172.21.15.116:0/931547363 >> 172.21.15.116:0/931547363 conn(0x7fdb580fc1e0 msgr2=0x7fdb580fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:52.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.725+0000 7fdb5d000700 1 -- 172.21.15.116:0/931547363 shutdown_connections 2024-03-30T00:50:52.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.725+0000 7fdb5d000700 1 -- 172.21.15.116:0/931547363 wait complete. 2024-03-30T00:50:52.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.726+0000 7fdb5d000700 1 Processor -- start 2024-03-30T00:50:52.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.726+0000 7fdb5d000700 1 -- start start 2024-03-30T00:50:52.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.726+0000 7fdb5d000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb581066b0 0x7fdb581980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:52.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.726+0000 7fdb5d000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb58198630 con 0x7fdb581066b0 2024-03-30T00:50:52.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.727+0000 7fdb55e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb581066b0 0x7fdb581980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:52.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.727+0000 7fdb55e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb581066b0 0x7fdb581980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54088/0 (socket says 172.21.15.116:54088) 2024-03-30T00:50:52.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.727+0000 7fdb55e00700 1 -- 172.21.15.116:0/3921126136 learned_addr learned my addr 172.21.15.116:0/3921126136 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:52.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.727+0000 7fdb55e00700 1 -- 172.21.15.116:0/3921126136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb480095f0 con 0x7fdb581066b0 2024-03-30T00:50:52.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.727+0000 7fdb55e00700 1 --2- 172.21.15.116:0/3921126136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb581066b0 0x7fdb581980f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdb48006010 tx=0x7fdb48004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:52.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.727+0000 7fdb4ea00700 1 -- 172.21.15.116:0/3921126136 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdb4801d070 con 0x7fdb581066b0 2024-03-30T00:50:52.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.727+0000 7fdb5d000700 1 -- 172.21.15.116:0/3921126136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb58198890 con 0x7fdb581066b0 2024-03-30T00:50:52.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.727+0000 7fdb4ea00700 1 -- 172.21.15.116:0/3921126136 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb4800bbb0 con 0x7fdb581066b0 2024-03-30T00:50:52.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.727+0000 7fdb4ea00700 1 -- 172.21.15.116:0/3921126136 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdb48003810 con 0x7fdb581066b0 2024-03-30T00:50:52.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.728+0000 7fdb5d000700 1 -- 172.21.15.116:0/3921126136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb58198d10 con 0x7fdb581066b0 2024-03-30T00:50:52.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.729+0000 7fdb4ea00700 1 -- 172.21.15.116:0/3921126136 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7fdb4800b6f0 con 0x7fdb581066b0 2024-03-30T00:50:52.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.729+0000 7fdb5d000700 1 -- 172.21.15.116:0/3921126136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb58101df0 con 0x7fdb581066b0 2024-03-30T00:50:52.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.729+0000 7fdb4ea00700 1 --2- 172.21.15.116:0/3921126136 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdb3c0387a0 0x7fdb3c03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:52.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.729+0000 7fdb4ea00700 1 -- 172.21.15.116:0/3921126136 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdb4804c460 con 0x7fdb581066b0 2024-03-30T00:50:52.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.729+0000 7fdb55400700 1 --2- 172.21.15.116:0/3921126136 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdb3c0387a0 0x7fdb3c03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:52.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.730+0000 7fdb55400700 1 --2- 172.21.15.116:0/3921126136 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdb3c0387a0 0x7fdb3c03ac60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fdb44006fd0 tx=0x7fdb44006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.734+0000 7fdb4ea00700 1 -- 172.21.15.116:0/3921126136 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdb48029540 con 0x7fdb581066b0 2024-03-30T00:50:52.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.939+0000 7fdb5d000700 1 -- 172.21.15.116:0/3921126136 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdb5810a9b0 con 0x7fdb3c0387a0 2024-03-30T00:50:52.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.941+0000 7fdb4ea00700 1 -- 172.21.15.116:0/3921126136 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fdb5810a9b0 con 0x7fdb3c0387a0 2024-03-30T00:50:52.946 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:50:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:[{"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.130", "hostname": "smithi130", "labels": [], "status": ""}] 2024-03-30T00:50:52.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.943+0000 7fdb5d000700 1 -- 172.21.15.116:0/3921126136 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdb3c0387a0 msgr2=0x7fdb3c03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:52.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.943+0000 7fdb5d000700 1 --2- 172.21.15.116:0/3921126136 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdb3c0387a0 0x7fdb3c03ac60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fdb44006fd0 tx=0x7fdb44006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:52.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.943+0000 7fdb5d000700 1 -- 172.21.15.116:0/3921126136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb581066b0 msgr2=0x7fdb581980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:52.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.943+0000 7fdb5d000700 1 --2- 172.21.15.116:0/3921126136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb581066b0 0x7fdb581980f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdb48006010 tx=0x7fdb48004ff0 comp rx=0 tx=0).stop 2024-03-30T00:50:52.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.943+0000 7fdb5d000700 1 -- 172.21.15.116:0/3921126136 shutdown_connections 2024-03-30T00:50:52.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.943+0000 7fdb5d000700 1 --2- 172.21.15.116:0/3921126136 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdb3c0387a0 0x7fdb3c03ac60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:52.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.943+0000 7fdb5d000700 1 --2- 172.21.15.116:0/3921126136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb581066b0 0x7fdb581980f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:52.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.943+0000 7fdb5d000700 1 -- 172.21.15.116:0/3921126136 >> 172.21.15.116:0/3921126136 conn(0x7fdb580fc1e0 msgr2=0x7fdb581082c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:52.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.944+0000 7fdb5d000700 1 -- 172.21.15.116:0/3921126136 shutdown_connections 2024-03-30T00:50:52.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:52.944+0000 7fdb5d000700 1 -- 172.21.15.116:0/3921126136 wait complete. 2024-03-30T00:50:53.249 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi181 2024-03-30T00:50:53.249 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-30T00:50:53.249 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.conf 2024-03-30T00:50:53.270 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-30T00:50:53.270 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T00:50:53.330 INFO:tasks.cephadm:Adding host smithi181 to orchestrator... 2024-03-30T00:50:53.331 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi181 2024-03-30T00:50:53.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:53.674 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:50:54.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:54 smithi116 ceph-mon[38086]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:50:54.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.771+0000 7fecc6a00700 1 -- 172.21.15.116:0/2803915486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc01066b0 msgr2=0x7fecc0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:54.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.771+0000 7fecc6a00700 1 --2- 172.21.15.116:0/2803915486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc01066b0 0x7fecc0106a90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fecb0009910 tx=0x7fecb0009c20 comp rx=0 tx=0).stop 2024-03-30T00:50:54.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.771+0000 7fecc6a00700 1 -- 172.21.15.116:0/2803915486 shutdown_connections 2024-03-30T00:50:54.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.771+0000 7fecc6a00700 1 --2- 172.21.15.116:0/2803915486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc01066b0 0x7fecc0106a90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:54.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.771+0000 7fecc6a00700 1 -- 172.21.15.116:0/2803915486 >> 172.21.15.116:0/2803915486 conn(0x7fecc00fc1e0 msgr2=0x7fecc00fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:54.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.771+0000 7fecc6a00700 1 -- 172.21.15.116:0/2803915486 shutdown_connections 2024-03-30T00:50:54.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.771+0000 7fecc6a00700 1 -- 172.21.15.116:0/2803915486 wait complete. 2024-03-30T00:50:54.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.773+0000 7fecc6a00700 1 Processor -- start 2024-03-30T00:50:54.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.773+0000 7fecc6a00700 1 -- start start 2024-03-30T00:50:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.773+0000 7fecc6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc01066b0 0x7fecc01a66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.773+0000 7fecc6a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc01a6c20 con 0x7fecc01066b0 2024-03-30T00:50:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.774+0000 7fecbfe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc01066b0 0x7fecc01a66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.774+0000 7fecbfe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc01066b0 0x7fecc01a66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54114/0 (socket says 172.21.15.116:54114) 2024-03-30T00:50:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.774+0000 7fecbfe00700 1 -- 172.21.15.116:0/4165811654 learned_addr learned my addr 172.21.15.116:0/4165811654 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.774+0000 7fecbfe00700 1 -- 172.21.15.116:0/4165811654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb00095f0 con 0x7fecc01066b0 2024-03-30T00:50:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.774+0000 7fecbfe00700 1 --2- 172.21.15.116:0/4165811654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc01066b0 0x7fecc01a66e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fecb0006010 tx=0x7fecb0004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.774+0000 7fecbcc00700 1 -- 172.21.15.116:0/4165811654 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fecb001d070 con 0x7fecc01066b0 2024-03-30T00:50:54.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.775+0000 7fecbcc00700 1 -- 172.21.15.116:0/4165811654 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fecb000bbb0 con 0x7fecc01066b0 2024-03-30T00:50:54.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.775+0000 7fecc6a00700 1 -- 172.21.15.116:0/4165811654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc01a6e20 con 0x7fecc01066b0 2024-03-30T00:50:54.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.775+0000 7fecbcc00700 1 -- 172.21.15.116:0/4165811654 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fecb0003810 con 0x7fecc01066b0 2024-03-30T00:50:54.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.775+0000 7fecc6a00700 1 -- 172.21.15.116:0/4165811654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc01a0820 con 0x7fecc01066b0 2024-03-30T00:50:54.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.776+0000 7fecc6a00700 1 -- 172.21.15.116:0/4165811654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc0101df0 con 0x7fecc01066b0 2024-03-30T00:50:54.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.776+0000 7fecbcc00700 1 -- 172.21.15.116:0/4165811654 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7fecb000b6f0 con 0x7fecc01066b0 2024-03-30T00:50:54.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.776+0000 7fecbcc00700 1 --2- 172.21.15.116:0/4165811654 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feca40387f0 0x7feca403acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:54.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.776+0000 7fecbcc00700 1 -- 172.21.15.116:0/4165811654 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fecb004c390 con 0x7fecc01066b0 2024-03-30T00:50:54.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.776+0000 7fecbf400700 1 --2- 172.21.15.116:0/4165811654 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feca40387f0 0x7feca403acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:54.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.777+0000 7fecbf400700 1 --2- 172.21.15.116:0/4165811654 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feca40387f0 0x7feca403acb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fecac006fd0 tx=0x7fecac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:54.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.781+0000 7fecbcc00700 1 -- 172.21.15.116:0/4165811654 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fecb0029540 con 0x7fecc01066b0 2024-03-30T00:50:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:54.993+0000 7fecc6a00700 1 -- 172.21.15.116:0/4165811654 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}) v1 -- 0x7fecc01a1ce0 con 0x7feca40387f0 2024-03-30T00:50:56.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:56.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:56.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:56.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:56.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:50:56.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:50:56.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:50:56.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: Updating smithi130:/etc/ceph/ceph.conf 2024-03-30T00:50:56.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: Updating smithi130:/var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T00:50:56.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:50:56.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:50:56.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-03-30T00:50:56.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: Updating smithi130:/var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T00:50:56.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:56.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:56.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:57.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:56 smithi116 ceph-mon[38086]: Deploying cephadm binary to smithi181 2024-03-30T00:50:57.217 INFO:teuthology.orchestra.run.smithi116.stdout:Added host 'smithi181' with addr '172.21.15.181' 2024-03-30T00:50:57.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:57.211+0000 7fecbcc00700 1 -- 172.21.15.116:0/4165811654 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fecc01a1ce0 con 0x7feca40387f0 2024-03-30T00:50:57.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:57.214+0000 7fecc6a00700 1 -- 172.21.15.116:0/4165811654 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feca40387f0 msgr2=0x7feca403acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:57.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:57.214+0000 7fecc6a00700 1 --2- 172.21.15.116:0/4165811654 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feca40387f0 0x7feca403acb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fecac006fd0 tx=0x7fecac006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:57.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:57.214+0000 7fecc6a00700 1 -- 172.21.15.116:0/4165811654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc01066b0 msgr2=0x7fecc01a66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:57.214+0000 7fecc6a00700 1 --2- 172.21.15.116:0/4165811654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc01066b0 0x7fecc01a66e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fecb0006010 tx=0x7fecb0004ff0 comp rx=0 tx=0).stop 2024-03-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:57.214+0000 7fecc6a00700 1 -- 172.21.15.116:0/4165811654 shutdown_connections 2024-03-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:57.214+0000 7fecc6a00700 1 --2- 172.21.15.116:0/4165811654 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feca40387f0 0x7feca403acb0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:57.214+0000 7fecc6a00700 1 --2- 172.21.15.116:0/4165811654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc01066b0 0x7fecc01a66e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:57.214+0000 7fecc6a00700 1 -- 172.21.15.116:0/4165811654 >> 172.21.15.116:0/4165811654 conn(0x7fecc00fc1e0 msgr2=0x7fecc0108350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:57.214+0000 7fecc6a00700 1 -- 172.21.15.116:0/4165811654 shutdown_connections 2024-03-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:57.214+0000 7fecc6a00700 1 -- 172.21.15.116:0/4165811654 wait complete. 2024-03-30T00:50:57.595 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-03-30T00:50:57.942 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:50:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:58 smithi116 ceph-mon[38086]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:50:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:58 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:58 smithi116 ceph-mon[38086]: Added host smithi181 2024-03-30T00:50:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:58 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:50:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:58 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:58 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:59.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.048+0000 7f8541400700 1 -- 172.21.15.116:0/1420834096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f853c1066b0 msgr2=0x7f853c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:59.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.048+0000 7f8541400700 1 --2- 172.21.15.116:0/1420834096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f853c1066b0 0x7f853c106a90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f852c009910 tx=0x7f852c009c20 comp rx=0 tx=0).stop 2024-03-30T00:50:59.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.048+0000 7f8541400700 1 -- 172.21.15.116:0/1420834096 shutdown_connections 2024-03-30T00:50:59.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.048+0000 7f8541400700 1 --2- 172.21.15.116:0/1420834096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f853c1066b0 0x7f853c106a90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:59.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.048+0000 7f8541400700 1 -- 172.21.15.116:0/1420834096 >> 172.21.15.116:0/1420834096 conn(0x7f853c0fc1e0 msgr2=0x7f853c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:59.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.048+0000 7f8541400700 1 -- 172.21.15.116:0/1420834096 shutdown_connections 2024-03-30T00:50:59.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.048+0000 7f8541400700 1 -- 172.21.15.116:0/1420834096 wait complete. 2024-03-30T00:50:59.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.049+0000 7f8541400700 1 Processor -- start 2024-03-30T00:50:59.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.049+0000 7f8541400700 1 -- start start 2024-03-30T00:50:59.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.050+0000 7f8541400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f853c1066b0 0x7f853c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:59.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.050+0000 7f8541400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853c074f70 con 0x7f853c1066b0 2024-03-30T00:50:59.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.050+0000 7f853a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f853c1066b0 0x7f853c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:59.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.050+0000 7f853a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f853c1066b0 0x7f853c0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43314/0 (socket says 172.21.15.116:43314) 2024-03-30T00:50:59.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.050+0000 7f853a800700 1 -- 172.21.15.116:0/3397555842 learned_addr learned my addr 172.21.15.116:0/3397555842 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:50:59.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.050+0000 7f853a800700 1 -- 172.21.15.116:0/3397555842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c0095f0 con 0x7f853c1066b0 2024-03-30T00:50:59.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.051+0000 7f853a800700 1 --2- 172.21.15.116:0/3397555842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f853c1066b0 0x7f853c0730b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f852c006010 tx=0x7f852c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:59.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.051+0000 7f8533400700 1 -- 172.21.15.116:0/3397555842 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f852c01d070 con 0x7f853c1066b0 2024-03-30T00:50:59.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.051+0000 7f8533400700 1 -- 172.21.15.116:0/3397555842 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f852c00bbb0 con 0x7f853c1066b0 2024-03-30T00:50:59.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.051+0000 7f8541400700 1 -- 172.21.15.116:0/3397555842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853c0735f0 con 0x7f853c1066b0 2024-03-30T00:50:59.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.051+0000 7f8533400700 1 -- 172.21.15.116:0/3397555842 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f852c003810 con 0x7f853c1066b0 2024-03-30T00:50:59.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.051+0000 7f8541400700 1 -- 172.21.15.116:0/3397555842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853c073a70 con 0x7f853c1066b0 2024-03-30T00:50:59.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.052+0000 7f8533400700 1 -- 172.21.15.116:0/3397555842 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7f852c00bd20 con 0x7f853c1066b0 2024-03-30T00:50:59.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.052+0000 7f8541400700 1 -- 172.21.15.116:0/3397555842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853c101df0 con 0x7f853c1066b0 2024-03-30T00:50:59.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.052+0000 7f8533400700 1 --2- 172.21.15.116:0/3397555842 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f85200387f0 0x7f852003acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:50:59.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.052+0000 7f8533400700 1 -- 172.21.15.116:0/3397555842 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f852c04c410 con 0x7f853c1066b0 2024-03-30T00:50:59.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.053+0000 7f8539e00700 1 --2- 172.21.15.116:0/3397555842 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f85200387f0 0x7f852003acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:50:59.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.053+0000 7f8539e00700 1 --2- 172.21.15.116:0/3397555842 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f85200387f0 0x7f852003acb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8528006fd0 tx=0x7f8528006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:50:59.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.058+0000 7f8533400700 1 -- 172.21.15.116:0/3397555842 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f852c029360 con 0x7f853c1066b0 2024-03-30T00:50:59.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.263+0000 7f8541400700 1 -- 172.21.15.116:0/3397555842 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f853c073cb0 con 0x7f85200387f0 2024-03-30T00:50:59.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.264+0000 7f8533400700 1 -- 172.21.15.116:0/3397555842 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+241 (secure 0 0 0) 0x7f853c073cb0 con 0x7f85200387f0 2024-03-30T00:50:59.270 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:50:59.270 INFO:teuthology.orchestra.run.smithi116.stdout:[{"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.130", "hostname": "smithi130", "labels": [], "status": ""}, {"addr": "172.21.15.181", "hostname": "smithi181", "labels": [], "status": ""}] 2024-03-30T00:50:59.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.266+0000 7f8541400700 1 -- 172.21.15.116:0/3397555842 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f85200387f0 msgr2=0x7f852003acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:59.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.266+0000 7f8541400700 1 --2- 172.21.15.116:0/3397555842 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f85200387f0 0x7f852003acb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8528006fd0 tx=0x7f8528006e40 comp rx=0 tx=0).stop 2024-03-30T00:50:59.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.266+0000 7f8541400700 1 -- 172.21.15.116:0/3397555842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f853c1066b0 msgr2=0x7f853c0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:50:59.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.266+0000 7f8541400700 1 --2- 172.21.15.116:0/3397555842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f853c1066b0 0x7f853c0730b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f852c006010 tx=0x7f852c004ff0 comp rx=0 tx=0).stop 2024-03-30T00:50:59.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.266+0000 7f8541400700 1 -- 172.21.15.116:0/3397555842 shutdown_connections 2024-03-30T00:50:59.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.266+0000 7f8541400700 1 --2- 172.21.15.116:0/3397555842 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f85200387f0 0x7f852003acb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:59.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.266+0000 7f8541400700 1 --2- 172.21.15.116:0/3397555842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f853c1066b0 0x7f853c0730b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:50:59.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.266+0000 7f8541400700 1 -- 172.21.15.116:0/3397555842 >> 172.21.15.116:0/3397555842 conn(0x7f853c0fc1e0 msgr2=0x7f853c108350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:50:59.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.267+0000 7f8541400700 1 -- 172.21.15.116:0/3397555842 shutdown_connections 2024-03-30T00:50:59.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:50:59.267+0000 7f8541400700 1 -- 172.21.15.116:0/3397555842 wait complete. 2024-03-30T00:50:59.633 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-30T00:50:59.634 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd crush tunables default 2024-03-30T00:50:59.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:59 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:50:59.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:59 smithi116 ceph-mon[38086]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:50:59.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:50:59 smithi116 ceph-mon[38086]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:50:59.978 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:51:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.971+0000 7f94a9200700 1 -- 172.21.15.116:0/536797939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a4101880 msgr2=0x7f94a4101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.971+0000 7f94a9200700 1 --2- 172.21.15.116:0/536797939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a4101880 0x7f94a4101c60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9494009910 tx=0x7f9494009c20 comp rx=0 tx=0).stop 2024-03-30T00:51:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.971+0000 7f94a9200700 1 -- 172.21.15.116:0/536797939 shutdown_connections 2024-03-30T00:51:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.971+0000 7f94a9200700 1 --2- 172.21.15.116:0/536797939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a4101880 0x7f94a4101c60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.971+0000 7f94a9200700 1 -- 172.21.15.116:0/536797939 >> 172.21.15.116:0/536797939 conn(0x7f94a4076220 msgr2=0x7f94a40fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.971+0000 7f94a9200700 1 -- 172.21.15.116:0/536797939 shutdown_connections 2024-03-30T00:51:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.972+0000 7f94a9200700 1 -- 172.21.15.116:0/536797939 wait complete. 2024-03-30T00:51:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.973+0000 7f94a9200700 1 Processor -- start 2024-03-30T00:51:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.973+0000 7f94a9200700 1 -- start start 2024-03-30T00:51:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.973+0000 7f94a9200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a4101880 0x7f94a4198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.973+0000 7f94a9200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a4198740 con 0x7f94a4101880 2024-03-30T00:51:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.973+0000 7f94a1e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a4101880 0x7f94a4198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.973+0000 7f94a1e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a4101880 0x7f94a4198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43338/0 (socket says 172.21.15.116:43338) 2024-03-30T00:51:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.973+0000 7f94a1e00700 1 -- 172.21.15.116:0/3975586857 learned_addr learned my addr 172.21.15.116:0/3975586857 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:51:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.974+0000 7f94a1e00700 1 -- 172.21.15.116:0/3975586857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94940095f0 con 0x7f94a4101880 2024-03-30T00:51:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.974+0000 7f94a1e00700 1 --2- 172.21.15.116:0/3975586857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a4101880 0x7f94a4198200 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9494006010 tx=0x7f9494004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.974+0000 7f949aa00700 1 -- 172.21.15.116:0/3975586857 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f949401d070 con 0x7f94a4101880 2024-03-30T00:51:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.974+0000 7f949aa00700 1 -- 172.21.15.116:0/3975586857 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f949400bc10 con 0x7f94a4101880 2024-03-30T00:51:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.974+0000 7f949aa00700 1 -- 172.21.15.116:0/3975586857 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94940038f0 con 0x7f94a4101880 2024-03-30T00:51:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.974+0000 7f94a9200700 1 -- 172.21.15.116:0/3975586857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a4198940 con 0x7f94a4101880 2024-03-30T00:51:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.974+0000 7f94a9200700 1 -- 172.21.15.116:0/3975586857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a4198d60 con 0x7f94a4101880 2024-03-30T00:51:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.975+0000 7f949aa00700 1 -- 172.21.15.116:0/3975586857 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7f949400b6f0 con 0x7f94a4101880 2024-03-30T00:51:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.975+0000 7f94a9200700 1 -- 172.21.15.116:0/3975586857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a4068780 con 0x7f94a4101880 2024-03-30T00:51:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.976+0000 7f949aa00700 1 --2- 172.21.15.116:0/3975586857 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f94880387f0 0x7f948803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.976+0000 7f949aa00700 1 -- 172.21.15.116:0/3975586857 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f949404c4f0 con 0x7f94a4101880 2024-03-30T00:51:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.976+0000 7f94a1400700 1 --2- 172.21.15.116:0/3975586857 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f94880387f0 0x7f948803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.976+0000 7f94a1400700 1 --2- 172.21.15.116:0/3975586857 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f94880387f0 0x7f948803acb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9490006fd0 tx=0x7f9490006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:00.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:00.981+0000 7f949aa00700 1 -- 172.21.15.116:0/3975586857 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9494029540 con 0x7f94a4101880 2024-03-30T00:51:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:01.195+0000 7f94a9200700 1 -- 172.21.15.116:0/3975586857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f94a410aa20 con 0x7f94a4101880 2024-03-30T00:51:01.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:01.832+0000 7f949aa00700 1 -- 172.21.15.116:0/3975586857 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9494026020 con 0x7f94a4101880 2024-03-30T00:51:01.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:01.835+0000 7f94a9200700 1 -- 172.21.15.116:0/3975586857 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f94880387f0 msgr2=0x7f948803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:01.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:01.835+0000 7f94a9200700 1 --2- 172.21.15.116:0/3975586857 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f94880387f0 0x7f948803acb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9490006fd0 tx=0x7f9490006e40 comp rx=0 tx=0).stop 2024-03-30T00:51:01.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:01.835+0000 7f94a9200700 1 -- 172.21.15.116:0/3975586857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a4101880 msgr2=0x7f94a4198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:01.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:01.835+0000 7f94a9200700 1 --2- 172.21.15.116:0/3975586857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a4101880 0x7f94a4198200 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9494006010 tx=0x7f9494004ff0 comp rx=0 tx=0).stop 2024-03-30T00:51:01.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:01.835+0000 7f94a9200700 1 -- 172.21.15.116:0/3975586857 shutdown_connections 2024-03-30T00:51:01.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:01.835+0000 7f94a9200700 1 --2- 172.21.15.116:0/3975586857 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f94880387f0 0x7f948803acb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:01.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:01.835+0000 7f94a9200700 1 --2- 172.21.15.116:0/3975586857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a4101880 0x7f94a4198200 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:01.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:01.835+0000 7f94a9200700 1 -- 172.21.15.116:0/3975586857 >> 172.21.15.116:0/3975586857 conn(0x7f94a4076220 msgr2=0x7f94a41077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:01.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:01.835+0000 7f94a9200700 1 -- 172.21.15.116:0/3975586857 shutdown_connections 2024-03-30T00:51:01.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:01.835+0000 7f94a9200700 1 -- 172.21.15.116:0/3975586857 wait complete. 2024-03-30T00:51:01.842 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2024-03-30T00:51:02.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:02.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:02.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:02.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:02.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:51:02.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:02.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:02.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: Updating smithi181:/etc/ceph/ceph.conf 2024-03-30T00:51:02.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: Updating smithi181:/var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T00:51:02.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:02.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3975586857' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-30T00:51:02.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-03-30T00:51:02.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: Updating smithi181:/var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T00:51:02.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:02.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:02.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:01 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:02.215 INFO:tasks.cephadm:Adding mon.a on smithi116 2024-03-30T00:51:02.215 INFO:tasks.cephadm:Adding mon.b on smithi130 2024-03-30T00:51:02.215 INFO:tasks.cephadm:Adding mon.c on smithi181 2024-03-30T00:51:02.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch apply mon '3;smithi116:172.21.15.116=a;smithi130:172.21.15.130=b;smithi181:172.21.15.181=c' 2024-03-30T00:51:02.465 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T00:51:03.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:02 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3975586857' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-30T00:51:03.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:02 smithi116 ceph-mon[38086]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T00:51:03.385 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.379+0000 7fa0d206c700 1 -- 172.21.15.181:0/913689319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc105650 msgr2=0x7fa0cc105a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:03.385 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.379+0000 7fa0d206c700 1 --2- 172.21.15.181:0/913689319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc105650 0x7fa0cc105a30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009910 tx=0x7fa0c0009c20 comp rx=0 tx=0).stop 2024-03-30T00:51:03.385 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.379+0000 7fa0d206c700 1 -- 172.21.15.181:0/913689319 shutdown_connections 2024-03-30T00:51:03.385 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.379+0000 7fa0d206c700 1 --2- 172.21.15.181:0/913689319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc105650 0x7fa0cc105a30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:03.385 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.379+0000 7fa0d206c700 1 -- 172.21.15.181:0/913689319 >> 172.21.15.181:0/913689319 conn(0x7fa0cc076220 msgr2=0x7fa0cc0fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:03.385 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.379+0000 7fa0d206c700 1 -- 172.21.15.181:0/913689319 shutdown_connections 2024-03-30T00:51:03.385 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.379+0000 7fa0d206c700 1 -- 172.21.15.181:0/913689319 wait complete. 2024-03-30T00:51:03.386 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.379+0000 7fa0d206c700 1 Processor -- start 2024-03-30T00:51:03.387 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.379+0000 7fa0d206c700 1 -- start start 2024-03-30T00:51:03.387 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.380+0000 7fa0d206c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc105650 0x7fa0cc0fc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:03.387 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.380+0000 7fa0d206c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc0fe130 con 0x7fa0cc105650 2024-03-30T00:51:03.387 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.380+0000 7fa0cb7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc105650 0x7fa0cc0fc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:03.387 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.380+0000 7fa0cb7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc105650 0x7fa0cc0fc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.181:48684/0 (socket says 172.21.15.181:48684) 2024-03-30T00:51:03.387 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.380+0000 7fa0cb7fe700 1 -- 172.21.15.181:0/1006541909 learned_addr learned my addr 172.21.15.181:0/1006541909 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-03-30T00:51:03.387 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.380+0000 7fa0cb7fe700 1 -- 172.21.15.181:0/1006541909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c00095f0 con 0x7fa0cc105650 2024-03-30T00:51:03.387 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.381+0000 7fa0cb7fe700 1 --2- 172.21.15.181:0/1006541909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc105650 0x7fa0cc0fc270 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0004f40 tx=0x7fa0c0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:03.388 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.381+0000 7fa0c8ff9700 1 -- 172.21.15.181:0/1006541909 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0c001d070 con 0x7fa0cc105650 2024-03-30T00:51:03.388 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.381+0000 7fa0d206c700 1 -- 172.21.15.181:0/1006541909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0cc0fc7b0 con 0x7fa0cc105650 2024-03-30T00:51:03.388 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.381+0000 7fa0c8ff9700 1 -- 172.21.15.181:0/1006541909 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa0c000b6f0 con 0x7fa0cc105650 2024-03-30T00:51:03.388 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.381+0000 7fa0c8ff9700 1 -- 172.21.15.181:0/1006541909 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0c0003810 con 0x7fa0cc105650 2024-03-30T00:51:03.388 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.381+0000 7fa0d206c700 1 -- 172.21.15.181:0/1006541909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0cc0fcc30 con 0x7fa0cc105650 2024-03-30T00:51:03.389 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.382+0000 7fa0d206c700 1 -- 172.21.15.181:0/1006541909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0cc100d00 con 0x7fa0cc105650 2024-03-30T00:51:03.389 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.382+0000 7fa0c8ff9700 1 -- 172.21.15.181:0/1006541909 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7fa0c0004260 con 0x7fa0cc105650 2024-03-30T00:51:03.389 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.382+0000 7fa0c8ff9700 1 --2- 172.21.15.181:0/1006541909 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa0b40387a0 0x7fa0b403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:03.389 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.382+0000 7fa0c8ff9700 1 -- 172.21.15.181:0/1006541909 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa0c004c410 con 0x7fa0cc105650 2024-03-30T00:51:03.389 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.382+0000 7fa0caffd700 1 --2- 172.21.15.181:0/1006541909 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa0b40387a0 0x7fa0b403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:03.389 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.383+0000 7fa0caffd700 1 --2- 172.21.15.181:0/1006541909 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa0b40387a0 0x7fa0b403ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc006fd0 tx=0x7fa0bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:03.394 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.388+0000 7fa0c8ff9700 1 -- 172.21.15.181:0/1006541909 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa0c000b860 con 0x7fa0cc105650 2024-03-30T00:51:03.597 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.591+0000 7fa0d206c700 1 -- 172.21.15.181:0/1006541909 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi130:172.21.15.130=b;smithi181:172.21.15.181=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0cc0fcf30 con 0x7fa0b40387a0 2024-03-30T00:51:03.604 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.599+0000 7fa0c8ff9700 1 -- 172.21.15.181:0/1006541909 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa0cc0fcf30 con 0x7fa0b40387a0 2024-03-30T00:51:03.605 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mon update... 2024-03-30T00:51:03.607 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.600+0000 7fa0d206c700 1 -- 172.21.15.181:0/1006541909 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa0b40387a0 msgr2=0x7fa0b403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:03.607 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.600+0000 7fa0d206c700 1 --2- 172.21.15.181:0/1006541909 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa0b40387a0 0x7fa0b403ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc006fd0 tx=0x7fa0bc006e40 comp rx=0 tx=0).stop 2024-03-30T00:51:03.607 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.600+0000 7fa0d206c700 1 -- 172.21.15.181:0/1006541909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc105650 msgr2=0x7fa0cc0fc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:03.607 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.600+0000 7fa0d206c700 1 --2- 172.21.15.181:0/1006541909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc105650 0x7fa0cc0fc270 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0004f40 tx=0x7fa0c0005020 comp rx=0 tx=0).stop 2024-03-30T00:51:03.607 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.600+0000 7fa0d206c700 1 -- 172.21.15.181:0/1006541909 shutdown_connections 2024-03-30T00:51:03.607 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.600+0000 7fa0d206c700 1 --2- 172.21.15.181:0/1006541909 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa0b40387a0 0x7fa0b403ac60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:03.607 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.600+0000 7fa0d206c700 1 --2- 172.21.15.181:0/1006541909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc105650 0x7fa0cc0fc270 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:03.607 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.600+0000 7fa0d206c700 1 -- 172.21.15.181:0/1006541909 >> 172.21.15.181:0/1006541909 conn(0x7fa0cc076220 msgr2=0x7fa0cc067e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:03.607 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.600+0000 7fa0d206c700 1 -- 172.21.15.181:0/1006541909 shutdown_connections 2024-03-30T00:51:03.607 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:03.600+0000 7fa0d206c700 1 -- 172.21.15.181:0/1006541909 wait complete. 2024-03-30T00:51:03.979 DEBUG:teuthology.orchestra.run.smithi130:mon.b> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mon.b.service 2024-03-30T00:51:03.981 DEBUG:teuthology.orchestra.run.smithi181:mon.c> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mon.c.service 2024-03-30T00:51:03.983 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T00:51:03.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T00:51:04.014 INFO:journalctl@ceph.mon.c.smithi181.stdout:-- Logs begin at Sat 2024-03-30 00:27:51 UTC. -- 2024-03-30T00:51:04.018 INFO:journalctl@ceph.mon.b.smithi130.stdout:-- Logs begin at Sat 2024-03-30 00:27:58 UTC. -- 2024-03-30T00:51:04.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:03 smithi116 ceph-mon[38086]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:04.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:04.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:04.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:04.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:04.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:04.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:04.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:04.433 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T00:51:05.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:04 smithi116 ceph-mon[38086]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi130:172.21.15.130=b;smithi181:172.21.15.181=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:51:05.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:04 smithi116 ceph-mon[38086]: Saving service mon spec with placement smithi116:172.21.15.116=a;smithi130:172.21.15.130=b;smithi181:172.21.15.181=c;count:3 2024-03-30T00:51:05.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:04 smithi116 ceph-mon[38086]: Deploying daemon mon.c on smithi181 2024-03-30T00:51:05.880 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.873+0000 7f15e8529700 1 -- 172.21.15.181:0/1525292848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15e0105680 msgr2=0x7f15e0105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:05.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.873+0000 7f15e8529700 1 --2- 172.21.15.181:0/1525292848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15e0105680 0x7f15e0105a60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009910 tx=0x7f15dc009c20 comp rx=0 tx=0).stop 2024-03-30T00:51:05.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.873+0000 7f15e8529700 1 -- 172.21.15.181:0/1525292848 shutdown_connections 2024-03-30T00:51:05.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.873+0000 7f15e8529700 1 --2- 172.21.15.181:0/1525292848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15e0105680 0x7f15e0105a60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:05.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.873+0000 7f15e8529700 1 -- 172.21.15.181:0/1525292848 >> 172.21.15.181:0/1525292848 conn(0x7f15e00fb0c0 msgr2=0x7f15e00fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:05.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.873+0000 7f15e8529700 1 -- 172.21.15.181:0/1525292848 shutdown_connections 2024-03-30T00:51:05.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.873+0000 7f15e8529700 1 -- 172.21.15.181:0/1525292848 wait complete. 2024-03-30T00:51:05.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.874+0000 7f15e8529700 1 Processor -- start 2024-03-30T00:51:05.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.874+0000 7f15e8529700 1 -- start start 2024-03-30T00:51:05.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.874+0000 7f15e8529700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15e0105680 0x7f15e0195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:05.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.874+0000 7f15e8529700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e0196420 con 0x7f15e0105680 2024-03-30T00:51:05.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.875+0000 7f15e62c5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15e0105680 0x7f15e0195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:05.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.875+0000 7f15e62c5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15e0105680 0x7f15e0195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.181:48698/0 (socket says 172.21.15.181:48698) 2024-03-30T00:51:05.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.875+0000 7f15e62c5700 1 -- 172.21.15.181:0/3882575852 learned_addr learned my addr 172.21.15.181:0/3882575852 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-03-30T00:51:05.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.875+0000 7f15e62c5700 1 -- 172.21.15.181:0/3882575852 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc0095f0 con 0x7f15e0105680 2024-03-30T00:51:05.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.875+0000 7f15e62c5700 1 --2- 172.21.15.181:0/3882575852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15e0105680 0x7f15e0195ee0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f15dc004f40 tx=0x7f15dc005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:05.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.875+0000 7f15d37fe700 1 -- 172.21.15.181:0/3882575852 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15dc01d070 con 0x7f15e0105680 2024-03-30T00:51:05.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.875+0000 7f15e8529700 1 -- 172.21.15.181:0/3882575852 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e0196680 con 0x7f15e0105680 2024-03-30T00:51:05.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.875+0000 7f15d37fe700 1 -- 172.21.15.181:0/3882575852 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15dc00bc10 con 0x7f15e0105680 2024-03-30T00:51:05.883 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.876+0000 7f15d37fe700 1 -- 172.21.15.181:0/3882575852 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15dc0037d0 con 0x7f15e0105680 2024-03-30T00:51:05.884 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.876+0000 7f15e8529700 1 -- 172.21.15.181:0/3882575852 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e0196b00 con 0x7f15e0105680 2024-03-30T00:51:05.884 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.876+0000 7f15d37fe700 1 -- 172.21.15.181:0/3882575852 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7f15dc003a90 con 0x7f15e0105680 2024-03-30T00:51:05.884 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.877+0000 7f15d37fe700 1 --2- 172.21.15.181:0/3882575852 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f15cc0387a0 0x7f15cc03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:05.884 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.877+0000 7f15e8529700 1 -- 172.21.15.181:0/3882575852 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e0100cf0 con 0x7f15e0105680 2024-03-30T00:51:05.884 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.877+0000 7f15d37fe700 1 -- 172.21.15.181:0/3882575852 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f15dc04c4a0 con 0x7f15e0105680 2024-03-30T00:51:05.884 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.877+0000 7f15e5ac4700 1 --2- 172.21.15.181:0/3882575852 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f15cc0387a0 0x7f15cc03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:05.884 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.877+0000 7f15e5ac4700 1 --2- 172.21.15.181:0/3882575852 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f15cc0387a0 0x7f15cc03ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f15d4006fd0 tx=0x7f15d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:05.888 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:05.883+0000 7f15d37fe700 1 -- 172.21.15.181:0/3882575852 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f15dc026020 con 0x7f15e0105680 2024-03-30T00:51:06.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:05 smithi116 ceph-mon[38086]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:06.193 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:51:06.194 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","modified":"2024-03-30T00:49:35.569110Z","created":"2024-03-30T00:49:35.569110Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T00:51:06.194 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:06.179+0000 7f15e8529700 1 -- 172.21.15.181:0/3882575852 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15e0109850 con 0x7f15e0105680 2024-03-30T00:51:06.194 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:06.181+0000 7f15d37fe700 1 -- 172.21.15.181:0/3882575852 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+740 (secure 0 0 0) 0x7f15dc029360 con 0x7f15e0105680 2024-03-30T00:51:06.194 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:06.186+0000 7f15e8529700 1 -- 172.21.15.181:0/3882575852 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f15cc0387a0 msgr2=0x7f15cc03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:06.194 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:06.186+0000 7f15e8529700 1 --2- 172.21.15.181:0/3882575852 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f15cc0387a0 0x7f15cc03ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f15d4006fd0 tx=0x7f15d4006e40 comp rx=0 tx=0).stop 2024-03-30T00:51:06.194 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:06.186+0000 7f15e8529700 1 -- 172.21.15.181:0/3882575852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15e0105680 msgr2=0x7f15e0195ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:06.194 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:06.186+0000 7f15e8529700 1 --2- 172.21.15.181:0/3882575852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15e0105680 0x7f15e0195ee0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f15dc004f40 tx=0x7f15dc005020 comp rx=0 tx=0).stop 2024-03-30T00:51:06.194 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:06.186+0000 7f15e8529700 1 -- 172.21.15.181:0/3882575852 shutdown_connections 2024-03-30T00:51:06.194 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:06.186+0000 7f15e8529700 1 --2- 172.21.15.181:0/3882575852 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f15cc0387a0 0x7f15cc03ac60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:06.195 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:06.186+0000 7f15e8529700 1 --2- 172.21.15.181:0/3882575852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15e0105680 0x7f15e0195ee0 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f15dc004f40 tx=0x7f15dc005020 comp rx=0 tx=0).stop 2024-03-30T00:51:06.195 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:06.186+0000 7f15e8529700 1 -- 172.21.15.181:0/3882575852 >> 172.21.15.181:0/3882575852 conn(0x7f15e00fb0c0 msgr2=0x7f15e0068710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:06.195 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:06.187+0000 7f15e8529700 1 -- 172.21.15.181:0/3882575852 shutdown_connections 2024-03-30T00:51:06.195 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:06.187+0000 7f15e8529700 1 -- 172.21.15.181:0/3882575852 wait complete. 2024-03-30T00:51:06.198 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-03-30T00:51:07.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:06 smithi116 ceph-mon[38086]: from='client.? 172.21.15.181:0/3882575852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T00:51:07.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 systemd[1]: Started Ceph mon.c for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:51:07.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: mon.c@-1(synchronizing).mds e1 new map 2024-03-30T00:51:07.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-30T00:51:07.502 INFO:journalctl@ceph.mon.c.smithi181.stdout: e1 2024-03-30T00:51:07.503 INFO:journalctl@ceph.mon.c.smithi181.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T00:51:07.503 INFO:journalctl@ceph.mon.c.smithi181.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T00:51:07.503 INFO:journalctl@ceph.mon.c.smithi181.stdout: legacy client fscid: -1 2024-03-30T00:51:07.503 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2024-03-30T00:51:07.503 INFO:journalctl@ceph.mon.c.smithi181.stdout: No filesystems configured 2024-03-30T00:51:07.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-30T00:51:07.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-30T00:51:07.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-30T00:51:07.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-30T00:51:07.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-30T00:51:07.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-30T00:51:07.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T00:51:07.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T00:51:07.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3975586857' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi130:172.21.15.130=b;smithi181:172.21.15.181=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: Saving service mon spec with placement smithi116:172.21.15.116=a;smithi130:172.21.15.130=b;smithi181:172.21.15.181=c;count:3 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: Deploying daemon mon.c on smithi181 2024-03-30T00:51:07.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:07.505 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: from='client.? 172.21.15.181:0/3882575852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T00:51:07.505 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:07 smithi181 ceph-mon[47622]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-30T00:51:07.673 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T00:51:07.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T00:51:07.889 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T00:51:12.483 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.476+0000 7f436c9a0700 1 -- 172.21.15.181:0/2332680725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43540035c0 msgr2=0x7f4354005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:12.483 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.476+0000 7f436c9a0700 1 --2- 172.21.15.181:0/2332680725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43540035c0 0x7f4354005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4368067cc0 tx=0x7f4350014ec0 comp rx=0 tx=0).stop 2024-03-30T00:51:12.483 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.476+0000 7f436c9a0700 1 -- 172.21.15.181:0/2332680725 shutdown_connections 2024-03-30T00:51:12.483 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.476+0000 7f436c9a0700 1 --2- 172.21.15.181:0/2332680725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43540035c0 0x7f4354005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:12.483 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.476+0000 7f436c9a0700 1 -- 172.21.15.181:0/2332680725 >> 172.21.15.181:0/2332680725 conn(0x7f43680fa7a0 msgr2=0x7f43680fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:12.483 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.476+0000 7f436c9a0700 1 -- 172.21.15.181:0/2332680725 shutdown_connections 2024-03-30T00:51:12.483 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.476+0000 7f436c9a0700 1 -- 172.21.15.181:0/2332680725 wait complete. 2024-03-30T00:51:12.484 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.477+0000 7f436c9a0700 1 Processor -- start 2024-03-30T00:51:12.484 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.478+0000 7f436c9a0700 1 -- start start 2024-03-30T00:51:12.485 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.478+0000 7f436c9a0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43540035c0 0x7f4368195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:12.485 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.478+0000 7f436c9a0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43681963c0 0x7f436819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:12.485 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.478+0000 7f436c9a0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4368196950 con 0x7f43681963c0 2024-03-30T00:51:12.485 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.478+0000 7f436c9a0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4368196ac0 con 0x7f43540035c0 2024-03-30T00:51:12.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.479+0000 7f43677fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43540035c0 0x7f4368195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:12.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.479+0000 7f43677fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43540035c0 0x7f4368195e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43600/0 (socket says 172.21.15.181:43600) 2024-03-30T00:51:12.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.479+0000 7f43677fe700 1 -- 172.21.15.181:0/1127327425 learned_addr learned my addr 172.21.15.181:0/1127327425 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-03-30T00:51:12.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.479+0000 7f43677fe700 1 -- 172.21.15.181:0/1127327425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43540035c0 msgr2=0x7f4368195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-30T00:51:12.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.479+0000 7f43677fe700 1 -- 172.21.15.181:0/1127327425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43540035c0 msgr2=0x7f4368195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-30T00:51:12.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.479+0000 7f43677fe700 1 --2- 172.21.15.181:0/1127327425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43540035c0 0x7f4368195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T00:51:12.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.479+0000 7f43677fe700 1 --2- 172.21.15.181:0/1127327425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43540035c0 0x7f4368195e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T00:51:12.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.479+0000 7f4366ffd700 1 --2- 172.21.15.181:0/1127327425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43681963c0 0x7f436819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:12.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.479+0000 7f4366ffd700 1 -- 172.21.15.181:0/1127327425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43540035c0 msgr2=0x7f4368195e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T00:51:12.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.479+0000 7f4366ffd700 1 --2- 172.21.15.181:0/1127327425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43540035c0 0x7f4368195e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:12.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.479+0000 7f4366ffd700 1 -- 172.21.15.181:0/1127327425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4350014910 con 0x7f43681963c0 2024-03-30T00:51:12.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.479+0000 7f4366ffd700 1 --2- 172.21.15.181:0/1127327425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43681963c0 0x7f436819a770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f435800b4d0 tx=0x7f435800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:12.488 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.480+0000 7f4364ff9700 1 -- 172.21.15.181:0/1127327425 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f4358011630 con 0x7f43681963c0 2024-03-30T00:51:12.488 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.480+0000 7f4364ff9700 1 -- 172.21.15.181:0/1127327425 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4358011c50 con 0x7f43681963c0 2024-03-30T00:51:12.488 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.480+0000 7f436c9a0700 1 -- 172.21.15.181:0/1127327425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436819ad10 con 0x7f43681963c0 2024-03-30T00:51:12.489 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.480+0000 7f4364ff9700 1 -- 172.21.15.181:0/1127327425 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f4358010d90 con 0x7f43681963c0 2024-03-30T00:51:12.489 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.480+0000 7f436c9a0700 1 -- 172.21.15.181:0/1127327425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436819b2e0 con 0x7f43681963c0 2024-03-30T00:51:12.489 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.481+0000 7f4364ff9700 1 -- 172.21.15.181:0/1127327425 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7f435801a400 con 0x7f43681963c0 2024-03-30T00:51:12.489 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.481+0000 7f436c9a0700 1 -- 172.21.15.181:0/1127327425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436818fcc0 con 0x7f43681963c0 2024-03-30T00:51:12.489 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.481+0000 7f4364ff9700 1 --2- 172.21.15.181:0/1127327425 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f43480385d0 0x7f434803aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:12.489 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.481+0000 7f4364ff9700 1 -- 172.21.15.181:0/1127327425 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f435804ad80 con 0x7f43681963c0 2024-03-30T00:51:12.489 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.481+0000 7f43677fe700 1 --2- 172.21.15.181:0/1127327425 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f43480385d0 0x7f434803aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:12.489 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.482+0000 7f43677fe700 1 --2- 172.21.15.181:0/1127327425 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f43480385d0 0x7f434803aa90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4350005b40 tx=0x7f4350005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:12.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.487+0000 7f4364ff9700 1 -- 172.21.15.181:0/1127327425 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f43580176d0 con 0x7f43681963c0 2024-03-30T00:51:12.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:12.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:51:12.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:12.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: mon.a calling monitor election 2024-03-30T00:51:12.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:12.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:12.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:12.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: mon.c calling monitor election 2024-03-30T00:51:12.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:12.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:12.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:12.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:12.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-30T00:51:12.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: monmap e2: 2 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T00:51:12.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: fsmap 2024-03-30T00:51:12.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T00:51:12.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: mgrmap e13: x(active, since 37s) 2024-03-30T00:51:12.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: overall HEALTH_OK 2024-03-30T00:51:12.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:12.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:12.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:12.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:12.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:12.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:12.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:51:12.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:12.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: mon.a calling monitor election 2024-03-30T00:51:12.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:12.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:12.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: mon.c calling monitor election 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: monmap e2: 2 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: fsmap 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: mgrmap e13: x(active, since 37s) 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: overall HEALTH_OK 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:12.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:12.767 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.761+0000 7f436c9a0700 1 -- 172.21.15.181:0/1127327425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4368197190 con 0x7f43681963c0 2024-03-30T00:51:12.768 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.762+0000 7f4364ff9700 1 -- 172.21.15.181:0/1127327425 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1010 (secure 0 0 0) 0x7f43580172c0 con 0x7f43681963c0 2024-03-30T00:51:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:51:12.768 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":2,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","modified":"2024-03-30T00:51:07.190147Z","created":"2024-03-30T00:49:35.569110Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-30T00:51:12.770 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.764+0000 7f436c9a0700 1 -- 172.21.15.181:0/1127327425 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f43480385d0 msgr2=0x7f434803aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:12.770 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.764+0000 7f436c9a0700 1 --2- 172.21.15.181:0/1127327425 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f43480385d0 0x7f434803aa90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4350005b40 tx=0x7f4350005a90 comp rx=0 tx=0).stop 2024-03-30T00:51:12.771 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.764+0000 7f436c9a0700 1 -- 172.21.15.181:0/1127327425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43681963c0 msgr2=0x7f436819a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:12.771 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.764+0000 7f436c9a0700 1 --2- 172.21.15.181:0/1127327425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43681963c0 0x7f436819a770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f435800b4d0 tx=0x7f435800b890 comp rx=0 tx=0).stop 2024-03-30T00:51:12.771 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.764+0000 7f436c9a0700 1 -- 172.21.15.181:0/1127327425 shutdown_connections 2024-03-30T00:51:12.771 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.764+0000 7f436c9a0700 1 --2- 172.21.15.181:0/1127327425 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f43480385d0 0x7f434803aa90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:12.771 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.764+0000 7f436c9a0700 1 --2- 172.21.15.181:0/1127327425 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43540035c0 0x7f4368195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:12.771 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.764+0000 7f436c9a0700 1 --2- 172.21.15.181:0/1127327425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43681963c0 0x7f436819a770 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:12.771 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.764+0000 7f436c9a0700 1 -- 172.21.15.181:0/1127327425 >> 172.21.15.181:0/1127327425 conn(0x7f43680fa7a0 msgr2=0x7f43680fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:12.771 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.764+0000 7f436c9a0700 1 -- 172.21.15.181:0/1127327425 shutdown_connections 2024-03-30T00:51:12.771 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:12.764+0000 7f436c9a0700 1 -- 172.21.15.181:0/1127327425 wait complete. 2024-03-30T00:51:12.771 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 2 2024-03-30T00:51:13.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:13 smithi181 ceph-mon[47622]: Deploying daemon mon.b on smithi130 2024-03-30T00:51:13.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:13 smithi181 ceph-mon[47622]: from='client.? 172.21.15.181:0/1127327425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T00:51:13.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:13 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:13.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:13 smithi116 ceph-mon[38086]: Deploying daemon mon.b on smithi130 2024-03-30T00:51:13.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:13 smithi116 ceph-mon[38086]: from='client.? 172.21.15.181:0/1127327425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T00:51:13.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:13 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:14.142 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T00:51:14.142 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T00:51:14.370 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T00:51:14.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:14 smithi181 ceph-mon[47622]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:14.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:14 smithi116 ceph-mon[38086]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:15.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 systemd[1]: Started Ceph mon.b for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:51:15.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: RocksDB version: 7.9.2 2024-03-30T00:51:15.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Git sha 0 2024-03-30T00:51:15.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Compile date 2024-03-29 11:49:16 2024-03-30T00:51:15.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: DB SUMMARY 2024-03-30T00:51:15.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: DB Session ID: 8CS2AO9IQ13SU7CKHIO6 2024-03-30T00:51:15.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: CURRENT file: CURRENT 2024-03-30T00:51:15.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T00:51:15.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-30T00:51:15.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-30T00:51:15.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-30T00:51:15.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.error_if_exists: 0 2024-03-30T00:51:15.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.create_if_missing: 0 2024-03-30T00:51:15.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.paranoid_checks: 1 2024-03-30T00:51:15.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-30T00:51:15.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T00:51:15.349 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-30T00:51:15.349 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.env: 0x561144a9e720 2024-03-30T00:51:15.349 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.fs: PosixFileSystem 2024-03-30T00:51:15.350 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.info_log: 0x561146e134e0 2024-03-30T00:51:15.350 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T00:51:15.350 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.statistics: (nil) 2024-03-30T00:51:15.350 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.use_fsync: 0 2024-03-30T00:51:15.350 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_log_file_size: 0 2024-03-30T00:51:15.350 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T00:51:15.351 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T00:51:15.351 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T00:51:15.351 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T00:51:15.351 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.allow_fallocate: 1 2024-03-30T00:51:15.351 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T00:51:15.351 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T00:51:15.351 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.use_direct_reads: 0 2024-03-30T00:51:15.351 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T00:51:15.351 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T00:51:15.351 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.db_log_dir: 2024-03-30T00:51:15.351 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.wal_dir: 2024-03-30T00:51:15.351 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T00:51:15.351 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T00:51:15.351 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.write_buffer_manager: 0x5611460aa320 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T00:51:15.352 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.unordered_write: 0 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.row_cache: None 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.wal_filter: None 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.two_write_queues: 0 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.wal_compression: 0 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.atomic_flush: 0 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T00:51:15.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.log_readahead_size: 0 2024-03-30T00:51:15.354 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T00:51:15.354 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T00:51:15.354 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T00:51:15.354 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T00:51:15.354 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T00:51:15.354 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T00:51:15.354 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-30T00:51:15.354 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_background_jobs: 2 2024-03-30T00:51:15.354 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_background_compactions: -1 2024-03-30T00:51:15.354 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_subcompactions: 1 2024-03-30T00:51:15.354 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T00:51:15.354 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T00:51:15.354 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T00:51:15.354 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_open_files: -1 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_background_flushes: -1 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Compression algorithms supported: 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: kZSTD supported: 0 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: kXpressCompression supported: 0 2024-03-30T00:51:15.355 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: kZlibCompression supported: 1 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: kSnappyCompression supported: 1 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: kLZ4Compression supported: 1 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: kBZip2Compression supported: 0 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.merge_operator: 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compaction_filter: None 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T00:51:15.356 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T00:51:15.357 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561146e135e0) 2024-03-30T00:51:15.357 INFO:journalctl@ceph.mon.b.smithi130.stdout: cache_index_and_filter_blocks: 1 2024-03-30T00:51:15.357 INFO:journalctl@ceph.mon.b.smithi130.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T00:51:15.357 INFO:journalctl@ceph.mon.b.smithi130.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T00:51:15.357 INFO:journalctl@ceph.mon.b.smithi130.stdout: pin_top_level_index_and_filter: 1 2024-03-30T00:51:15.357 INFO:journalctl@ceph.mon.b.smithi130.stdout: index_type: 0 2024-03-30T00:51:15.357 INFO:journalctl@ceph.mon.b.smithi130.stdout: data_block_index_type: 0 2024-03-30T00:51:15.357 INFO:journalctl@ceph.mon.b.smithi130.stdout: index_shortening: 1 2024-03-30T00:51:15.357 INFO:journalctl@ceph.mon.b.smithi130.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T00:51:15.357 INFO:journalctl@ceph.mon.b.smithi130.stdout: checksum: 4 2024-03-30T00:51:15.357 INFO:journalctl@ceph.mon.b.smithi130.stdout: no_block_cache: 0 2024-03-30T00:51:15.358 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_cache: 0x56114612b1f0 2024-03-30T00:51:15.358 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_cache_name: BinnedLRUCache 2024-03-30T00:51:15.358 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_cache_options: 2024-03-30T00:51:15.358 INFO:journalctl@ceph.mon.b.smithi130.stdout: capacity : 536870912 2024-03-30T00:51:15.358 INFO:journalctl@ceph.mon.b.smithi130.stdout: num_shard_bits : 4 2024-03-30T00:51:15.358 INFO:journalctl@ceph.mon.b.smithi130.stdout: strict_capacity_limit : 0 2024-03-30T00:51:15.358 INFO:journalctl@ceph.mon.b.smithi130.stdout: high_pri_pool_ratio: 0.000 2024-03-30T00:51:15.358 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_cache_compressed: (nil) 2024-03-30T00:51:15.358 INFO:journalctl@ceph.mon.b.smithi130.stdout: persistent_cache: (nil) 2024-03-30T00:51:15.358 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_size: 4096 2024-03-30T00:51:15.358 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_size_deviation: 10 2024-03-30T00:51:15.358 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_restart_interval: 16 2024-03-30T00:51:15.359 INFO:journalctl@ceph.mon.b.smithi130.stdout: index_block_restart_interval: 1 2024-03-30T00:51:15.359 INFO:journalctl@ceph.mon.b.smithi130.stdout: metadata_block_size: 4096 2024-03-30T00:51:15.359 INFO:journalctl@ceph.mon.b.smithi130.stdout: partition_filters: 0 2024-03-30T00:51:15.359 INFO:journalctl@ceph.mon.b.smithi130.stdout: use_delta_encoding: 1 2024-03-30T00:51:15.359 INFO:journalctl@ceph.mon.b.smithi130.stdout: filter_policy: bloomfilter 2024-03-30T00:51:15.359 INFO:journalctl@ceph.mon.b.smithi130.stdout: whole_key_filtering: 1 2024-03-30T00:51:15.359 INFO:journalctl@ceph.mon.b.smithi130.stdout: verify_compression: 0 2024-03-30T00:51:15.359 INFO:journalctl@ceph.mon.b.smithi130.stdout: read_amp_bytes_per_bit: 0 2024-03-30T00:51:15.359 INFO:journalctl@ceph.mon.b.smithi130.stdout: format_version: 5 2024-03-30T00:51:15.359 INFO:journalctl@ceph.mon.b.smithi130.stdout: enable_index_compression: 1 2024-03-30T00:51:15.359 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_align: 0 2024-03-30T00:51:15.359 INFO:journalctl@ceph.mon.b.smithi130.stdout: max_auto_readahead_size: 262144 2024-03-30T00:51:15.359 INFO:journalctl@ceph.mon.b.smithi130.stdout: prepopulate_block_cache: 0 2024-03-30T00:51:15.359 INFO:journalctl@ceph.mon.b.smithi130.stdout: initial_auto_readahead_size: 8192 2024-03-30T00:51:15.360 INFO:journalctl@ceph.mon.b.smithi130.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T00:51:15.360 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T00:51:15.360 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T00:51:15.360 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compression: NoCompression 2024-03-30T00:51:15.360 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T00:51:15.360 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T00:51:15.360 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T00:51:15.360 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.num_levels: 7 2024-03-30T00:51:15.360 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T00:51:15.360 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T00:51:15.360 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T00:51:15.360 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T00:51:15.360 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T00:51:15.360 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T00:51:15.361 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T00:51:15.361 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T00:51:15.361 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T00:51:15.361 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T00:51:15.361 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T00:51:15.361 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T00:51:15.361 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T00:51:15.361 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T00:51:15.361 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T00:51:15.361 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T00:51:15.361 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T00:51:15.361 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T00:51:15.361 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T00:51:15.361 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T00:51:15.362 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T00:51:15.363 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T00:51:15.363 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T00:51:15.363 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T00:51:15.363 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T00:51:15.363 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T00:51:15.363 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T00:51:15.363 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T00:51:15.363 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T00:51:15.363 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T00:51:15.363 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T00:51:15.363 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T00:51:15.363 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T00:51:15.363 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T00:51:15.363 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.table_properties_collectors: 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.inplace_update_support: 0 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.bloom_locality: 0 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.max_successive_merges: 0 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T00:51:15.364 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.ttl: 2592000 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.enable_blob_files: false 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.min_blob_size: 0 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-30T00:51:15.365 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-30T00:51:15.366 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c527597c-dc92-4d08-913b-38e914c927a6 2024-03-30T00:51:15.366 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711759874927017, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-30T00:51:15.366 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-30T00:51:15.366 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711759874927640, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711759874, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c527597c-dc92-4d08-913b-38e914c927a6", "db_session_id": "8CS2AO9IQ13SU7CKHIO6", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-30T00:51:15.366 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711759874927727, "job": 1, "event": "recovery_finished"} 2024-03-30T00:51:15.366 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-30T00:51:15.366 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T00:51:15.366 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5611461c8000 2024-03-30T00:51:15.366 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: DB pointer 0x5611461b6000 2024-03-30T00:51:15.366 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T00:51:15.366 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T00:51:15.366 INFO:journalctl@ceph.mon.b.smithi130.stdout: ** DB Stats ** 2024-03-30T00:51:15.366 INFO:journalctl@ceph.mon.b.smithi130.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T00:51:15.367 INFO:journalctl@ceph.mon.b.smithi130.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T00:51:15.367 INFO:journalctl@ceph.mon.b.smithi130.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T00:51:15.367 INFO:journalctl@ceph.mon.b.smithi130.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T00:51:15.367 INFO:journalctl@ceph.mon.b.smithi130.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T00:51:15.367 INFO:journalctl@ceph.mon.b.smithi130.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T00:51:15.367 INFO:journalctl@ceph.mon.b.smithi130.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T00:51:15.367 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2024-03-30T00:51:15.367 INFO:journalctl@ceph.mon.b.smithi130.stdout: ** Compaction Stats [default] ** 2024-03-30T00:51:15.367 INFO:journalctl@ceph.mon.b.smithi130.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T00:51:15.367 INFO:journalctl@ceph.mon.b.smithi130.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T00:51:15.367 INFO:journalctl@ceph.mon.b.smithi130.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T00:51:15.367 INFO:journalctl@ceph.mon.b.smithi130.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T00:51:15.367 INFO:journalctl@ceph.mon.b.smithi130.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T00:51:15.367 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2024-03-30T00:51:15.368 INFO:journalctl@ceph.mon.b.smithi130.stdout: ** Compaction Stats [default] ** 2024-03-30T00:51:15.368 INFO:journalctl@ceph.mon.b.smithi130.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T00:51:15.368 INFO:journalctl@ceph.mon.b.smithi130.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T00:51:15.368 INFO:journalctl@ceph.mon.b.smithi130.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T00:51:15.368 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2024-03-30T00:51:15.368 INFO:journalctl@ceph.mon.b.smithi130.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T00:51:15.368 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2024-03-30T00:51:15.368 INFO:journalctl@ceph.mon.b.smithi130.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout: Cumulative compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout: Interval compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout: Block cache BinnedLRUCache@0x56114612b1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: using public_addr v2:172.21.15.130:0/0 -> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] 2024-03-30T00:51:15.369 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: starting mon.b rank -1 at public addrs [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] at bind addrs [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:51:15.370 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b@-1(???) e0 preinit fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:51:15.370 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b@-1(synchronizing).mds e1 new map 2024-03-30T00:51:15.370 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-30T00:51:15.370 INFO:journalctl@ceph.mon.b.smithi130.stdout: e1 2024-03-30T00:51:15.370 INFO:journalctl@ceph.mon.b.smithi130.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T00:51:15.370 INFO:journalctl@ceph.mon.b.smithi130.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T00:51:15.370 INFO:journalctl@ceph.mon.b.smithi130.stdout: legacy client fscid: -1 2024-03-30T00:51:15.370 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2024-03-30T00:51:15.370 INFO:journalctl@ceph.mon.b.smithi130.stdout: No filesystems configured 2024-03-30T00:51:15.370 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-30T00:51:15.370 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-30T00:51:15.370 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-30T00:51:15.370 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-30T00:51:15.370 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-30T00:51:15.371 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-30T00:51:15.371 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T00:51:15.371 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T00:51:15.371 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T00:51:15.371 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T00:51:15.371 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3975586857' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-30T00:51:15.371 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T00:51:15.371 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:15.371 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:15.371 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:15.371 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:15.371 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:15.371 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:15.371 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi130:172.21.15.130=b;smithi181:172.21.15.181=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: Saving service mon spec with placement smithi116:172.21.15.116=a;smithi130:172.21.15.130=b;smithi181:172.21.15.181=c;count:3 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: Deploying daemon mon.c on smithi181 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='client.? 172.21.15.181:0/3882575852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.a calling monitor election 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.c calling monitor election 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:15.372 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:15.373 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:15.373 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:15.373 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-30T00:51:15.373 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: monmap e2: 2 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T00:51:15.373 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: fsmap 2024-03-30T00:51:15.373 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T00:51:15.373 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mgrmap e13: x(active, since 37s) 2024-03-30T00:51:15.373 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: overall HEALTH_OK 2024-03-30T00:51:15.373 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:15.373 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:15.373 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:15.373 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:15.373 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:15.374 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: Deploying daemon mon.b on smithi130 2024-03-30T00:51:15.374 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='client.? 172.21.15.181:0/1127327425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T00:51:15.374 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:15.374 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:15.374 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:14 smithi130 ceph-mon[47626]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-30T00:51:20.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:51:20.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: mon.c calling monitor election 2024-03-30T00:51:20.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: mon.a calling monitor election 2024-03-30T00:51:20.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:20.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:20.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:20.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:20.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:20.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:20.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:20.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:20.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:20.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:20.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-30T00:51:20.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T00:51:20.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: fsmap 2024-03-30T00:51:20.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T00:51:20.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: mgrmap e13: x(active, since 44s) 2024-03-30T00:51:20.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-30T00:51:20.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-30T00:51:20.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-30T00:51:20.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: mon.b (rank 2) addr [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] is down (out of quorum) 2024-03-30T00:51:20.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:20.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:20.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:20.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:20.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:51:20.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: mon.c calling monitor election 2024-03-30T00:51:20.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: mon.a calling monitor election 2024-03-30T00:51:20.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:20.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:20.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:20.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: fsmap 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: mgrmap e13: x(active, since 44s) 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: mon.b (rank 2) addr [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] is down (out of quorum) 2024-03-30T00:51:20.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:20.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:20.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:20.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:21.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.298+0000 7f3544f0c700 1 -- 172.21.15.181:0/1136321963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f352c008490 msgr2=0x7f352c005c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:21.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.298+0000 7f3544f0c700 1 --2- 172.21.15.181:0/1136321963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f352c008490 0x7f352c005c70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f35401007b0 tx=0x7f352800fe90 comp rx=0 tx=0).stop 2024-03-30T00:51:21.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.298+0000 7f3544f0c700 1 -- 172.21.15.181:0/1136321963 shutdown_connections 2024-03-30T00:51:21.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.298+0000 7f3544f0c700 1 --2- 172.21.15.181:0/1136321963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f352c008490 0x7f352c005c70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:21.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.298+0000 7f3544f0c700 1 -- 172.21.15.181:0/1136321963 >> 172.21.15.181:0/1136321963 conn(0x7f3540076390 msgr2=0x7f35400fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:21.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.299+0000 7f3544f0c700 1 -- 172.21.15.181:0/1136321963 shutdown_connections 2024-03-30T00:51:21.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.299+0000 7f3544f0c700 1 -- 172.21.15.181:0/1136321963 wait complete. 2024-03-30T00:51:21.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.299+0000 7f3544f0c700 1 Processor -- start 2024-03-30T00:51:21.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.299+0000 7f3544f0c700 1 -- start start 2024-03-30T00:51:21.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f3544f0c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f352c008490 0x7f35401984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:21.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f3544f0c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35401989e0 0x7f354019cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:21.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f3544f0c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f354019d2d0 0x7f354019d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:21.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f3544f0c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35401990d0 con 0x7f35401989e0 2024-03-30T00:51:21.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f3544f0c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35401a5040 con 0x7f352c008490 2024-03-30T00:51:21.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f3544f0c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35401a5180 con 0x7f354019d2d0 2024-03-30T00:51:21.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353ed9d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f354019d2d0 0x7f354019d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:21.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353e59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f352c008490 0x7f35401984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:21.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353ed9d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f354019d2d0 0x7f354019d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:55922/0 (socket says 172.21.15.181:55922) 2024-03-30T00:51:21.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353e59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f352c008490 0x7f35401984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.181:45198/0 (socket says 172.21.15.181:45198) 2024-03-30T00:51:21.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353ed9d700 1 -- 172.21.15.181:0/2247931362 learned_addr learned my addr 172.21.15.181:0/2247931362 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-03-30T00:51:21.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353dd9b700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35401989e0 0x7f354019cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:21.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353e59c700 1 -- 172.21.15.181:0/2247931362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f352c008490 msgr2=0x7f35401984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-30T00:51:21.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353e59c700 1 -- 172.21.15.181:0/2247931362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f352c008490 msgr2=0x7f35401984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-30T00:51:21.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353e59c700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f352c008490 0x7f35401984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T00:51:21.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353e59c700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f352c008490 0x7f35401984a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T00:51:21.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353ed9d700 1 -- 172.21.15.181:0/2247931362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f354019d2d0 msgr2=0x7f354019d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-30T00:51:21.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353ed9d700 1 -- 172.21.15.181:0/2247931362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f354019d2d0 msgr2=0x7f354019d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-30T00:51:21.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353ed9d700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f354019d2d0 0x7f354019d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T00:51:21.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353ed9d700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f354019d2d0 0x7f354019d780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T00:51:21.310 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353dd9b700 1 -- 172.21.15.181:0/2247931362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f354019d2d0 msgr2=0x7f354019d780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T00:51:21.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353dd9b700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f354019d2d0 0x7f354019d780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:21.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353dd9b700 1 -- 172.21.15.181:0/2247931362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f352c008490 msgr2=0x7f35401984a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T00:51:21.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353dd9b700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f352c008490 0x7f35401984a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:21.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.300+0000 7f353dd9b700 1 -- 172.21.15.181:0/2247931362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352800f8e0 con 0x7f35401989e0 2024-03-30T00:51:21.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.301+0000 7f353dd9b700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35401989e0 0x7f354019cd90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f353000b550 tx=0x7f353000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:21.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.301+0000 7f35377fe700 1 -- 172.21.15.181:0/2247931362 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3530004230 con 0x7f35401989e0 2024-03-30T00:51:21.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.301+0000 7f3544f0c700 1 -- 172.21.15.181:0/2247931362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35401a5460 con 0x7f35401989e0 2024-03-30T00:51:21.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.301+0000 7f35377fe700 1 -- 172.21.15.181:0/2247931362 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3530004390 con 0x7f35401989e0 2024-03-30T00:51:21.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.301+0000 7f35377fe700 1 -- 172.21.15.181:0/2247931362 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3530011af0 con 0x7f35401989e0 2024-03-30T00:51:21.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.301+0000 7f3544f0c700 1 -- 172.21.15.181:0/2247931362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35401a59b0 con 0x7f35401989e0 2024-03-30T00:51:21.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.302+0000 7f35377fe700 1 -- 172.21.15.181:0/2247931362 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7f3530011c50 con 0x7f35401989e0 2024-03-30T00:51:21.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.302+0000 7f35377fe700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3520038c20 0x7f352003b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:21.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.302+0000 7f35377fe700 1 -- 172.21.15.181:0/2247931362 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f353004ad80 con 0x7f35401989e0 2024-03-30T00:51:21.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.302+0000 7f3544f0c700 1 -- 172.21.15.181:0/2247931362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3540100e30 con 0x7f35401989e0 2024-03-30T00:51:21.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.302+0000 7f353e59c700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3520038c20 0x7f352003b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:21.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.302+0000 7f353e59c700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3520038c20 0x7f352003b0e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f352801ddd0 tx=0x7f352801dd60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:21.313 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.307+0000 7f35377fe700 1 -- 172.21.15.181:0/2247931362 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f353001a400 con 0x7f35401989e0 2024-03-30T00:51:21.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:21.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:21.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:21.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:21.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:21.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:21.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:21.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:21.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:21.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:21.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:21.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:21.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:21.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:21.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:21.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:21.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:21.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:21.597 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.591+0000 7f3544f0c700 1 -- 172.21.15.181:0/2247931362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35401997b0 con 0x7f35401989e0 2024-03-30T00:51:21.598 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.591+0000 7f35377fe700 1 -- 172.21.15.181:0/2247931362 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1278 (secure 0 0 0) 0x7f353000f2e0 con 0x7f35401989e0 2024-03-30T00:51:21.598 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-30T00:51:21.598 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":3,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","modified":"2024-03-30T00:51:14.962160Z","created":"2024-03-30T00:49:35.569110Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-30T00:51:21.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.593+0000 7f3544f0c700 1 -- 172.21.15.181:0/2247931362 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3520038c20 msgr2=0x7f352003b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:21.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.593+0000 7f3544f0c700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3520038c20 0x7f352003b0e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f352801ddd0 tx=0x7f352801dd60 comp rx=0 tx=0).stop 2024-03-30T00:51:21.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.593+0000 7f3544f0c700 1 -- 172.21.15.181:0/2247931362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35401989e0 msgr2=0x7f354019cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:21.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.593+0000 7f3544f0c700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35401989e0 0x7f354019cd90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f353000b550 tx=0x7f353000b910 comp rx=0 tx=0).stop 2024-03-30T00:51:21.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.594+0000 7f3544f0c700 1 -- 172.21.15.181:0/2247931362 shutdown_connections 2024-03-30T00:51:21.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.594+0000 7f3544f0c700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3520038c20 0x7f352003b0e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:21.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.594+0000 7f3544f0c700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f354019d2d0 0x7f354019d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:21.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.594+0000 7f3544f0c700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f352c008490 0x7f35401984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:21.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.594+0000 7f3544f0c700 1 --2- 172.21.15.181:0/2247931362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35401989e0 0x7f354019cd90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:21.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.594+0000 7f3544f0c700 1 -- 172.21.15.181:0/2247931362 >> 172.21.15.181:0/2247931362 conn(0x7f3540076390 msgr2=0x7f35401011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:21.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.594+0000 7f3544f0c700 1 -- 172.21.15.181:0/2247931362 shutdown_connections 2024-03-30T00:51:21.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:21.594+0000 7f3544f0c700 1 -- 172.21.15.181:0/2247931362 wait complete. 2024-03-30T00:51:21.601 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 3 2024-03-30T00:51:21.984 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-30T00:51:21.984 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-03-30T00:51:22.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:51:22.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: mon.c calling monitor election 2024-03-30T00:51:22.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: mon.a calling monitor election 2024-03-30T00:51:22.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:22.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:51:22.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:22.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:22.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:22.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:22.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:22.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:22.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:22.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:22.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-30T00:51:22.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T00:51:22.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: fsmap 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: mgrmap e13: x(active, since 44s) 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: mon.b (rank 2) addr [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] is down (out of quorum) 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:22.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:22.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:22.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:22.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:22.358 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:51:23.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: mon.b calling monitor election 2024-03-30T00:51:23.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: mon.b calling monitor election 2024-03-30T00:51:23.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: mon.c calling monitor election 2024-03-30T00:51:23.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: mon.a calling monitor election 2024-03-30T00:51:23.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: fsmap 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: mgrmap e13: x(active, since 46s) 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: Cluster is now healthy 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: overall HEALTH_OK 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T00:51:23.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:23.176 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: Reconfiguring daemon mon.a on smithi116 2024-03-30T00:51:23.176 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:22 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:23.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: mon.b calling monitor election 2024-03-30T00:51:23.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: mon.b calling monitor election 2024-03-30T00:51:23.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: mon.c calling monitor election 2024-03-30T00:51:23.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: mon.a calling monitor election 2024-03-30T00:51:23.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-30T00:51:23.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T00:51:23.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: fsmap 2024-03-30T00:51:23.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T00:51:23.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: mgrmap e13: x(active, since 46s) 2024-03-30T00:51:23.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-30T00:51:23.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: Cluster is now healthy 2024-03-30T00:51:23.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: overall HEALTH_OK 2024-03-30T00:51:23.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:23.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:23.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:23.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:23.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T00:51:23.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:23.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T00:51:23.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:23.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: Reconfiguring daemon mon.a on smithi116 2024-03-30T00:51:23.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:22 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:23.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: mon.b calling monitor election 2024-03-30T00:51:23.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: mon.b calling monitor election 2024-03-30T00:51:23.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: mon.c calling monitor election 2024-03-30T00:51:23.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: mon.a calling monitor election 2024-03-30T00:51:23.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-30T00:51:23.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T00:51:23.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: fsmap 2024-03-30T00:51:23.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T00:51:23.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: mgrmap e13: x(active, since 46s) 2024-03-30T00:51:23.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-30T00:51:23.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: Cluster is now healthy 2024-03-30T00:51:23.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: overall HEALTH_OK 2024-03-30T00:51:23.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:23.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:23.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:23.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:23.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T00:51:23.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:23.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T00:51:23.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:23.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: Reconfiguring daemon mon.a on smithi116 2024-03-30T00:51:23.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:22 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:51:23.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.522+0000 7f1705200700 1 -- 172.21.15.116:0/354464720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17001066b0 msgr2=0x7f1700106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:23.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.522+0000 7f1705200700 1 --2- 172.21.15.116:0/354464720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17001066b0 0x7f1700106a90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009910 tx=0x7f16f0009c20 comp rx=0 tx=0).stop 2024-03-30T00:51:23.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.522+0000 7f1705200700 1 -- 172.21.15.116:0/354464720 shutdown_connections 2024-03-30T00:51:23.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.522+0000 7f1705200700 1 --2- 172.21.15.116:0/354464720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17001066b0 0x7f1700106a90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:23.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.522+0000 7f1705200700 1 -- 172.21.15.116:0/354464720 >> 172.21.15.116:0/354464720 conn(0x7f17000fc1e0 msgr2=0x7f17000fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:23.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.522+0000 7f1705200700 1 -- 172.21.15.116:0/354464720 shutdown_connections 2024-03-30T00:51:23.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.522+0000 7f1705200700 1 -- 172.21.15.116:0/354464720 wait complete. 2024-03-30T00:51:23.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.524+0000 7f1705200700 1 Processor -- start 2024-03-30T00:51:23.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.524+0000 7f1705200700 1 -- start start 2024-03-30T00:51:23.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.524+0000 7f1705200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17001066b0 0x7f17000730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:23.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.524+0000 7f1705200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17000735f0 0x7f1700073a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.524+0000 7f1705200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17001a22e0 0x7f17001a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.525+0000 7f1705200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17001a4c10 con 0x7f17000735f0 2024-03-30T00:51:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.525+0000 7f1705200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17001a4d80 con 0x7f17001066b0 2024-03-30T00:51:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.525+0000 7f1705200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17001a4f40 con 0x7f17001a22e0 2024-03-30T00:51:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.525+0000 7f16fde00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17001066b0 0x7f17000730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.525+0000 7f16fde00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17001066b0 0x7f17000730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:44040/0 (socket says 172.21.15.116:44040) 2024-03-30T00:51:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.525+0000 7f16fde00700 1 -- 172.21.15.116:0/2971897742 learned_addr learned my addr 172.21.15.116:0/2971897742 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:51:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.525+0000 7f16fd400700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17000735f0 0x7f1700073a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.525+0000 7f16fe800700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17001a22e0 0x7f17001a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.525+0000 7f16fde00700 1 -- 172.21.15.116:0/2971897742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17001a22e0 msgr2=0x7f17001a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.525+0000 7f16fde00700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17001a22e0 0x7f17001a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.525+0000 7f16fde00700 1 -- 172.21.15.116:0/2971897742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17000735f0 msgr2=0x7f1700073a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.525+0000 7f16fde00700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17000735f0 0x7f1700073a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.525+0000 7f16fde00700 1 -- 172.21.15.116:0/2971897742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f00095f0 con 0x7f17001066b0 2024-03-30T00:51:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.525+0000 7f16fe800700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17001a22e0 0x7f17001a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:51:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.526+0000 7f16fde00700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17001066b0 0x7f17000730b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f16f0004cb0 tx=0x7f16f0004ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.526+0000 7f16f6a00700 1 -- 172.21.15.116:0/2971897742 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f001d070 con 0x7f17001066b0 2024-03-30T00:51:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.526+0000 7f16f6a00700 1 -- 172.21.15.116:0/2971897742 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16f0005000 con 0x7f17001066b0 2024-03-30T00:51:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.526+0000 7f16f6a00700 1 -- 172.21.15.116:0/2971897742 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f0022d70 con 0x7f17001066b0 2024-03-30T00:51:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.527+0000 7f1705200700 1 -- 172.21.15.116:0/2971897742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17001a5220 con 0x7f17001066b0 2024-03-30T00:51:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.527+0000 7f1705200700 1 -- 172.21.15.116:0/2971897742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17001a56d0 con 0x7f17001066b0 2024-03-30T00:51:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.528+0000 7f16f6a00700 1 -- 172.21.15.116:0/2971897742 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7f16f0022470 con 0x7f17001066b0 2024-03-30T00:51:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.528+0000 7f1705200700 1 -- 172.21.15.116:0/2971897742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1700108890 con 0x7f17001066b0 2024-03-30T00:51:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.528+0000 7f16f6a00700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f16e4038c70 0x7f16e403b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.528+0000 7f16f6a00700 1 -- 172.21.15.116:0/2971897742 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f16f004c560 con 0x7f17001066b0 2024-03-30T00:51:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.529+0000 7f16fd400700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f16e4038c70 0x7f16e403b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.529+0000 7f16fd400700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f16e4038c70 0x7f16e403b130 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009ad0 tx=0x7f16ec009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:23.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.534+0000 7f16f6a00700 1 -- 172.21.15.116:0/2971897742 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f16f000f460 con 0x7f17001066b0 2024-03-30T00:51:23.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.745+0000 7f1705200700 1 -- 172.21.15.116:0/2971897742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f1700101df0 con 0x7f17001066b0 2024-03-30T00:51:23.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.745+0000 7f16f6a00700 1 -- 172.21.15.116:0/2971897742 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f16f0027030 con 0x7f17001066b0 2024-03-30T00:51:23.752 INFO:teuthology.orchestra.run.smithi116.stdout:# minimal ceph.conf for 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:51:23.752 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-03-30T00:51:23.753 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:51:23.753 INFO:teuthology.orchestra.run.smithi116.stdout: mon_host = [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2024-03-30T00:51:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.748+0000 7f1705200700 1 -- 172.21.15.116:0/2971897742 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f16e4038c70 msgr2=0x7f16e403b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.748+0000 7f1705200700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f16e4038c70 0x7f16e403b130 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009ad0 tx=0x7f16ec009210 comp rx=0 tx=0).stop 2024-03-30T00:51:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.748+0000 7f1705200700 1 -- 172.21.15.116:0/2971897742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17001066b0 msgr2=0x7f17000730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.748+0000 7f1705200700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17001066b0 0x7f17000730b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f16f0004cb0 tx=0x7f16f0004ce0 comp rx=0 tx=0).stop 2024-03-30T00:51:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.748+0000 7f1705200700 1 -- 172.21.15.116:0/2971897742 shutdown_connections 2024-03-30T00:51:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.748+0000 7f1705200700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f16e4038c70 0x7f16e403b130 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.748+0000 7f1705200700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17001a22e0 0x7f17001a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.748+0000 7f1705200700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17001066b0 0x7f17000730b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.748+0000 7f1705200700 1 --2- 172.21.15.116:0/2971897742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17000735f0 0x7f1700073a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.748+0000 7f1705200700 1 -- 172.21.15.116:0/2971897742 >> 172.21.15.116:0/2971897742 conn(0x7f17000fc1e0 msgr2=0x7f17001021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.748+0000 7f1705200700 1 -- 172.21.15.116:0/2971897742 shutdown_connections 2024-03-30T00:51:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:23.748+0000 7f1705200700 1 -- 172.21.15.116:0/2971897742 wait complete. 2024-03-30T00:51:24.123 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-30T00:51:24.123 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:51:24.123 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T00:51:24.182 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:23 smithi116 ceph-mon[38086]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:24.182 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:24.182 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:24.182 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:24.182 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T00:51:24.183 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:24.183 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:23 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2971897742' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:24.192 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:51:24.192 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T00:51:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:23 smithi181 ceph-mon[47622]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T00:51:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:23 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2971897742' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:24.298 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-30T00:51:24.298 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T00:51:24.329 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:23 smithi130 ceph-mon[47626]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:24.329 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:24.329 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:24.329 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:24.329 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T00:51:24.329 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:24.329 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:23 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2971897742' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:24.332 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-30T00:51:24.332 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T00:51:24.399 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-30T00:51:24.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T00:51:24.436 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-30T00:51:24.436 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T00:51:24.507 INFO:tasks.cephadm:Adding mgr.x on smithi116 2024-03-30T00:51:24.507 INFO:tasks.cephadm:Adding mgr.y on smithi130 2024-03-30T00:51:24.507 INFO:tasks.cephadm:Adding mgr.z on smithi181 2024-03-30T00:51:24.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch apply mgr '3;smithi116=x;smithi130=y;smithi181=z' 2024-03-30T00:51:24.780 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T00:51:25.166 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:24 smithi181 ceph-mon[47622]: Reconfiguring mon.b (monmap changed)... 2024-03-30T00:51:25.166 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:24 smithi181 ceph-mon[47622]: Reconfiguring daemon mon.b on smithi130 2024-03-30T00:51:25.166 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:24 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:25.166 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:24 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:25.166 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:24 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:25.166 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:24 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T00:51:25.166 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:24 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:25.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:24 smithi130 ceph-mon[47626]: Reconfiguring mon.b (monmap changed)... 2024-03-30T00:51:25.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:24 smithi130 ceph-mon[47626]: Reconfiguring daemon mon.b on smithi130 2024-03-30T00:51:25.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:24 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:25.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:24 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:25.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:24 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:25.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:24 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T00:51:25.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:24 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:24 smithi116 ceph-mon[38086]: Reconfiguring mon.b (monmap changed)... 2024-03-30T00:51:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:24 smithi116 ceph-mon[38086]: Reconfiguring daemon mon.b on smithi130 2024-03-30T00:51:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:24 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:24 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:24 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T00:51:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:24 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T00:51:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:24 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:26.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:25 smithi181 ceph-mon[47622]: Reconfiguring mon.c (monmap changed)... 2024-03-30T00:51:26.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:25 smithi181 ceph-mon[47622]: Reconfiguring daemon mon.c on smithi181 2024-03-30T00:51:26.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:25 smithi181 ceph-mon[47622]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:26.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:26.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:26.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.267+0000 7faaf58c8700 1 -- 172.21.15.181:0/3011124417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf0102090 msgr2=0x7faaf01024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:26.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.267+0000 7faaf58c8700 1 --2- 172.21.15.181:0/3011124417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf0102090 0x7faaf01024f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7faae0009990 tx=0x7faae0009ca0 comp rx=0 tx=0).stop 2024-03-30T00:51:26.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.267+0000 7faaf58c8700 1 -- 172.21.15.181:0/3011124417 shutdown_connections 2024-03-30T00:51:26.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.267+0000 7faaf58c8700 1 --2- 172.21.15.181:0/3011124417 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faaf0102a30 0x7faaf01092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:26.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.267+0000 7faaf58c8700 1 --2- 172.21.15.181:0/3011124417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf00ff0b0 0x7faaf00ff4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:26.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.267+0000 7faaf58c8700 1 --2- 172.21.15.181:0/3011124417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf0102090 0x7faaf01024f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:26.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.267+0000 7faaf58c8700 1 -- 172.21.15.181:0/3011124417 >> 172.21.15.181:0/3011124417 conn(0x7faaf00faa10 msgr2=0x7faaf00fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:26.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.267+0000 7faaf58c8700 1 -- 172.21.15.181:0/3011124417 shutdown_connections 2024-03-30T00:51:26.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.267+0000 7faaf58c8700 1 -- 172.21.15.181:0/3011124417 wait complete. 2024-03-30T00:51:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.267+0000 7faaf58c8700 1 Processor -- start 2024-03-30T00:51:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaf58c8700 1 -- start start 2024-03-30T00:51:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaf58c8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf00ff0b0 0x7faaf0196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaf58c8700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faaf0102090 0x7faaf0197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaf58c8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf0102a30 0x7faaf019e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaf58c8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf019e950 con 0x7faaf0102a30 2024-03-30T00:51:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaf58c8700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf019ea90 con 0x7faaf0102090 2024-03-30T00:51:26.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaf58c8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf019ec20 con 0x7faaf00ff0b0 2024-03-30T00:51:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaeeffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf00ff0b0 0x7faaf0196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaeeffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf00ff0b0 0x7faaf0196e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:55938/0 (socket says 172.21.15.181:55938) 2024-03-30T00:51:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaeeffd700 1 -- 172.21.15.181:0/3694881692 learned_addr learned my addr 172.21.15.181:0/3694881692 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-03-30T00:51:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaee7fc700 1 --2- 172.21.15.181:0/3694881692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faaf0102090 0x7faaf0197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaef7fe700 1 --2- 172.21.15.181:0/3694881692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf0102a30 0x7faaf019e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaeeffd700 1 -- 172.21.15.181:0/3694881692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faaf0102090 msgr2=0x7faaf0197350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaeeffd700 1 --2- 172.21.15.181:0/3694881692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faaf0102090 0x7faaf0197350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaeeffd700 1 -- 172.21.15.181:0/3694881692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf0102a30 msgr2=0x7faaf019e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaeeffd700 1 --2- 172.21.15.181:0/3694881692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf0102a30 0x7faaf019e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaeeffd700 1 -- 172.21.15.181:0/3694881692 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae00095f0 con 0x7faaf00ff0b0 2024-03-30T00:51:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaeeffd700 1 --2- 172.21.15.181:0/3694881692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf00ff0b0 0x7faaf0196e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faae400b530 tx=0x7faae400b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaf48c6700 1 -- 172.21.15.181:0/3694881692 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae4011830 con 0x7faaf00ff0b0 2024-03-30T00:51:26.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaf48c6700 1 -- 172.21.15.181:0/3694881692 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faae4011e50 con 0x7faaf00ff0b0 2024-03-30T00:51:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.268+0000 7faaf58c8700 1 -- 172.21.15.181:0/3694881692 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf019ef00 con 0x7faaf00ff0b0 2024-03-30T00:51:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.269+0000 7faaf48c6700 1 -- 172.21.15.181:0/3694881692 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae4010de0 con 0x7faaf00ff0b0 2024-03-30T00:51:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.269+0000 7faaf58c8700 1 -- 172.21.15.181:0/3694881692 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf019f3a0 con 0x7faaf00ff0b0 2024-03-30T00:51:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.269+0000 7faaf48c6700 1 -- 172.21.15.181:0/3694881692 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7faae4010480 con 0x7faaf00ff0b0 2024-03-30T00:51:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.269+0000 7faaf58c8700 1 -- 172.21.15.181:0/3694881692 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf0108340 con 0x7faaf00ff0b0 2024-03-30T00:51:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.270+0000 7faaf48c6700 1 --2- 172.21.15.181:0/3694881692 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7faad80389d0 0x7faad803ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.270+0000 7faaf48c6700 1 -- 172.21.15.181:0/3694881692 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faae404ac10 con 0x7faaf00ff0b0 2024-03-30T00:51:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.270+0000 7faaee7fc700 1 --2- 172.21.15.181:0/3694881692 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7faad80389d0 0x7faad803ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:26.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.270+0000 7faaee7fc700 1 --2- 172.21.15.181:0/3694881692 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7faad80389d0 0x7faad803ae90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faae0006010 tx=0x7faae0003880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:26.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.275+0000 7faaf48c6700 1 -- 172.21.15.181:0/3694881692 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faae4017b20 con 0x7faaf00ff0b0 2024-03-30T00:51:26.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:25 smithi130 ceph-mon[47626]: Reconfiguring mon.c (monmap changed)... 2024-03-30T00:51:26.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:25 smithi130 ceph-mon[47626]: Reconfiguring daemon mon.c on smithi181 2024-03-30T00:51:26.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:25 smithi130 ceph-mon[47626]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:26.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:26.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:26.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:25 smithi116 ceph-mon[38086]: Reconfiguring mon.c (monmap changed)... 2024-03-30T00:51:26.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:25 smithi116 ceph-mon[38086]: Reconfiguring daemon mon.c on smithi181 2024-03-30T00:51:26.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:25 smithi116 ceph-mon[38086]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:26.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:26.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:26.482 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.475+0000 7faaf58c8700 1 -- 172.21.15.181:0/3694881692 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi130=y;smithi181=z", "target": ["mon-mgr", ""]}) v1 -- 0x7faaf002cdc0 con 0x7faad80389d0 2024-03-30T00:51:26.488 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.482+0000 7faaf48c6700 1 -- 172.21.15.181:0/3694881692 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7faaf002cdc0 con 0x7faad80389d0 2024-03-30T00:51:26.488 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mgr update... 2024-03-30T00:51:26.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.484+0000 7faaf58c8700 1 -- 172.21.15.181:0/3694881692 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7faad80389d0 msgr2=0x7faad803ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:26.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.484+0000 7faaf58c8700 1 --2- 172.21.15.181:0/3694881692 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7faad80389d0 0x7faad803ae90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faae0006010 tx=0x7faae0003880 comp rx=0 tx=0).stop 2024-03-30T00:51:26.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.484+0000 7faaf58c8700 1 -- 172.21.15.181:0/3694881692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf00ff0b0 msgr2=0x7faaf0196e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:26.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.484+0000 7faaf58c8700 1 --2- 172.21.15.181:0/3694881692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf00ff0b0 0x7faaf0196e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faae400b530 tx=0x7faae400b840 comp rx=0 tx=0).stop 2024-03-30T00:51:26.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.484+0000 7faaf58c8700 1 -- 172.21.15.181:0/3694881692 shutdown_connections 2024-03-30T00:51:26.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.484+0000 7faaf58c8700 1 --2- 172.21.15.181:0/3694881692 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7faad80389d0 0x7faad803ae90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:26.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.484+0000 7faaf58c8700 1 --2- 172.21.15.181:0/3694881692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faaf00ff0b0 0x7faaf0196e10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:26.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.484+0000 7faaf58c8700 1 --2- 172.21.15.181:0/3694881692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf0102a30 0x7faaf019e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:26.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.484+0000 7faaf58c8700 1 --2- 172.21.15.181:0/3694881692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faaf0102090 0x7faaf0197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:26.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.484+0000 7faaf58c8700 1 -- 172.21.15.181:0/3694881692 >> 172.21.15.181:0/3694881692 conn(0x7faaf00faa10 msgr2=0x7faaf0105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:26.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.484+0000 7faaf58c8700 1 -- 172.21.15.181:0/3694881692 shutdown_connections 2024-03-30T00:51:26.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:51:26.484+0000 7faaf58c8700 1 -- 172.21.15.181:0/3694881692 wait complete. 2024-03-30T00:51:27.075 DEBUG:teuthology.orchestra.run.smithi130:mgr.y> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.y.service 2024-03-30T00:51:27.077 DEBUG:teuthology.orchestra.run.smithi181:mgr.z> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.z.service 2024-03-30T00:51:27.080 INFO:tasks.cephadm:Deploying OSDs... 2024-03-30T00:51:27.080 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:51:27.080 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T00:51:27.109 INFO:journalctl@ceph.mgr.y.smithi130.stdout:-- Logs begin at Sat 2024-03-30 00:27:58 UTC. -- 2024-03-30T00:51:27.111 INFO:journalctl@ceph.mgr.z.smithi181.stdout:-- Logs begin at Sat 2024-03-30 00:27:51 UTC. -- 2024-03-30T00:51:27.125 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T00:51:27.126 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-03-30T00:51:27.211 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T00:51:27.211 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T00:51:27.211 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 448 Links: 1 2024-03-30T00:51:27.211 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T00:51:27.211 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T00:51:27.211 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-30 00:50:45.304364584 +0000 2024-03-30T00:51:27.211 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-30 00:44:12.095310065 +0000 2024-03-30T00:51:27.211 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-30 00:44:12.095310065 +0000 2024-03-30T00:51:27.211 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-30 00:44:12.066310533 +0000 2024-03-30T00:51:27.212 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T00:51:27.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:27.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:27.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:27.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:27.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi130=y;smithi181=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:51:27.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: Saving service mgr spec with placement smithi116=x;smithi130=y;smithi181=z;count:3 2024-03-30T00:51:27.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:27.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:27.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:27.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:27.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:27.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T00:51:27.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T00:51:27.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T00:51:27.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:26 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:27.309 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:27.309 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:27.309 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:27.309 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:27.309 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi130=y;smithi181=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:51:27.309 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: Saving service mgr spec with placement smithi116=x;smithi130=y;smithi181=z;count:3 2024-03-30T00:51:27.309 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:27.309 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:27.309 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:27.310 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:27.310 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:27.310 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T00:51:27.310 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T00:51:27.310 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T00:51:27.310 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:26 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-30T00:51:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-30T00:51:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000348031 s, 1.5 MB/s 2024-03-30T00:51:27.318 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T00:51:27.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:27.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:27.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:27.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:27.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi130=y;smithi181=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:51:27.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: Saving service mgr spec with placement smithi116=x;smithi130=y;smithi181=z;count:3 2024-03-30T00:51:27.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:27.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:27.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:27.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:27.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:27.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T00:51:27.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T00:51:27.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T00:51:27.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:26 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:27.361 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-03-30T00:51:27.444 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T00:51:27.444 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T00:51:27.444 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 454 Links: 1 2024-03-30T00:51:27.444 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T00:51:27.444 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T00:51:27.444 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-30 00:50:45.368363548 +0000 2024-03-30T00:51:27.444 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-30 00:44:12.598301950 +0000 2024-03-30T00:51:27.444 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-30 00:44:12.598301950 +0000 2024-03-30T00:51:27.444 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-30 00:44:12.572302370 +0000 2024-03-30T00:51:27.445 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T00:51:27.546 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-30T00:51:27.546 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-30T00:51:27.546 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000311608 s, 1.6 MB/s 2024-03-30T00:51:27.549 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T00:51:27.592 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-03-30T00:51:27.675 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T00:51:27.675 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T00:51:27.676 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 460 Links: 1 2024-03-30T00:51:27.676 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T00:51:27.676 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T00:51:27.676 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-30 00:50:45.432362511 +0000 2024-03-30T00:51:27.676 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-30 00:44:13.073294286 +0000 2024-03-30T00:51:27.676 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-30 00:44:13.073294286 +0000 2024-03-30T00:51:27.676 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-30 00:44:13.049294674 +0000 2024-03-30T00:51:27.676 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T00:51:27.776 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-30T00:51:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-30T00:51:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000286998 s, 1.8 MB/s 2024-03-30T00:51:27.779 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T00:51:27.823 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-03-30T00:51:27.907 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T00:51:27.907 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T00:51:27.907 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 466 Links: 1 2024-03-30T00:51:27.907 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T00:51:27.908 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T00:51:27.908 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-30 00:50:45.493361524 +0000 2024-03-30T00:51:27.908 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-30 00:44:13.540286752 +0000 2024-03-30T00:51:27.908 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-30 00:44:13.540286752 +0000 2024-03-30T00:51:27.908 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-30 00:44:13.517287123 +0000 2024-03-30T00:51:27.908 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T00:51:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-30T00:51:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-30T00:51:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000299855 s, 1.7 MB/s 2024-03-30T00:51:28.012 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T00:51:28.055 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-30T00:51:28.055 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T00:51:28.076 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T00:51:28.076 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_1 2024-03-30T00:51:28.137 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T00:51:28.138 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T00:51:28.138 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 179570 Links: 1 2024-03-30T00:51:28.138 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T00:51:28.138 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T00:51:28.138 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-30 00:51:14.517789509 +0000 2024-03-30T00:51:28.138 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-30 00:42:54.666301868 +0000 2024-03-30T00:51:28.138 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-30 00:42:54.666301868 +0000 2024-03-30T00:51:28.138 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2024-03-30T00:51:28.138 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T00:51:28.208 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-30T00:51:28.208 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-30T00:51:28.208 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000345543 s, 1.5 MB/s 2024-03-30T00:51:28.209 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T00:51:28.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:28 smithi181 ceph-mon[47622]: Deploying daemon mgr.z on smithi181 2024-03-30T00:51:28.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:28 smithi181 ceph-mon[47622]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:28.253 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:28 smithi181 systemd[1]: Starting Ceph mgr.z for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:51:28.270 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_2 2024-03-30T00:51:28.329 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T00:51:28.330 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T00:51:28.330 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 177767 Links: 1 2024-03-30T00:51:28.330 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T00:51:28.330 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T00:51:28.330 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-30 00:51:14.517789509 +0000 2024-03-30T00:51:28.330 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-30 00:42:55.075294591 +0000 2024-03-30T00:51:28.330 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-30 00:42:55.075294591 +0000 2024-03-30T00:51:28.330 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2024-03-30T00:51:28.330 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T00:51:28.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:28 smithi130 ceph-mon[47626]: Deploying daemon mgr.z on smithi181 2024-03-30T00:51:28.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:28 smithi130 ceph-mon[47626]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:28.372 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-30T00:51:28.372 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-30T00:51:28.372 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000230137 s, 2.2 MB/s 2024-03-30T00:51:28.373 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T00:51:28.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:28 smithi116 ceph-mon[38086]: Deploying daemon mgr.z on smithi181 2024-03-30T00:51:28.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:28 smithi116 ceph-mon[38086]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:28.434 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_3 2024-03-30T00:51:28.496 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T00:51:28.496 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T00:51:28.496 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 181387 Links: 1 2024-03-30T00:51:28.496 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T00:51:28.496 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T00:51:28.496 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-30 00:51:14.517789509 +0000 2024-03-30T00:51:28.496 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-30 00:42:55.424288382 +0000 2024-03-30T00:51:28.496 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-30 00:42:55.424288382 +0000 2024-03-30T00:51:28.497 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2024-03-30T00:51:28.497 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T00:51:28.564 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-30T00:51:28.564 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-30T00:51:28.564 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000174697 s, 2.9 MB/s 2024-03-30T00:51:28.565 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T00:51:28.624 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_4 2024-03-30T00:51:28.682 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T00:51:28.683 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T00:51:28.683 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 180617 Links: 1 2024-03-30T00:51:28.683 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T00:51:28.683 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T00:51:28.683 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-30 00:51:14.517789509 +0000 2024-03-30T00:51:28.683 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-30 00:42:55.794281800 +0000 2024-03-30T00:51:28.683 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-30 00:42:55.794281800 +0000 2024-03-30T00:51:28.683 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2024-03-30T00:51:28.683 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T00:51:28.750 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-30T00:51:28.750 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-30T00:51:28.751 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000215701 s, 2.4 MB/s 2024-03-30T00:51:28.752 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T00:51:28.752 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:28 smithi181 bash[49468]: 5de8b2a516707edea2778d731d1dacfade5992e2164d364663480a3675ace619 2024-03-30T00:51:28.752 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:28 smithi181 systemd[1]: Started Ceph mgr.z for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:51:28.752 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:28 smithi181 ceph-mgr[49506]: -- 172.21.15.181:0/4055160813 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556fbe03e1a0 con 0x556fbe015000 2024-03-30T00:51:28.811 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-30T00:51:28.811 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T00:51:28.829 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T00:51:28.829 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-03-30T00:51:28.887 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T00:51:28.887 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T00:51:28.887 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 179487 Links: 1 2024-03-30T00:51:28.887 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T00:51:28.888 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T00:51:28.888 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-03-30 00:51:28.015662910 +0000 2024-03-30T00:51:28.888 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-03-30 00:43:02.655331100 +0000 2024-03-30T00:51:28.888 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-03-30 00:43:02.655331100 +0000 2024-03-30T00:51:28.888 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-03-30T00:51:28.888 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T00:51:28.939 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:28 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:28.837+0000 7fdcb0f72200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T00:51:28.958 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-03-30T00:51:28.958 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-03-30T00:51:28.958 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000227704 s, 2.2 MB/s 2024-03-30T00:51:28.959 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T00:51:29.018 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-03-30T00:51:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T00:51:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T00:51:29.076 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 176099 Links: 1 2024-03-30T00:51:29.076 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T00:51:29.076 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T00:51:29.076 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-03-30 00:51:28.015662910 +0000 2024-03-30T00:51:29.076 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-03-30 00:43:03.013324148 +0000 2024-03-30T00:51:29.076 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-03-30 00:43:03.013324148 +0000 2024-03-30T00:51:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-03-30T00:51:29.077 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T00:51:29.117 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:28 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:28.933+0000 7fdcb0f72200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T00:51:29.117 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:29 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:29.024+0000 7fdcb0f72200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T00:51:29.117 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:29 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:29.111+0000 7fdcb0f72200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T00:51:29.142 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-03-30T00:51:29.143 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-03-30T00:51:29.143 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.00022367 s, 2.3 MB/s 2024-03-30T00:51:29.144 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T00:51:29.202 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-03-30T00:51:29.259 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T00:51:29.260 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T00:51:29.260 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 179608 Links: 1 2024-03-30T00:51:29.260 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T00:51:29.260 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T00:51:29.260 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-03-30 00:51:28.016662890 +0000 2024-03-30T00:51:29.260 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-03-30 00:43:03.359317430 +0000 2024-03-30T00:51:29.260 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-03-30 00:43:03.359317430 +0000 2024-03-30T00:51:29.260 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-03-30T00:51:29.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T00:51:29.325 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-03-30T00:51:29.326 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-03-30T00:51:29.326 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000185865 s, 2.8 MB/s 2024-03-30T00:51:29.327 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T00:51:29.384 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-03-30T00:51:29.442 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T00:51:29.442 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T00:51:29.442 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 174935 Links: 1 2024-03-30T00:51:29.442 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T00:51:29.442 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T00:51:29.443 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-03-30 00:51:28.016662890 +0000 2024-03-30T00:51:29.443 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-03-30 00:43:03.708310653 +0000 2024-03-30T00:51:29.443 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-03-30 00:43:03.708310653 +0000 2024-03-30T00:51:29.443 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-03-30T00:51:29.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T00:51:29.508 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-03-30T00:51:29.508 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-03-30T00:51:29.508 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000237011 s, 2.2 MB/s 2024-03-30T00:51:29.509 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T00:51:29.591 INFO:tasks.cephadm:Deploying osd.0 on smithi116 with /dev/vg_nvme/lv_4... 2024-03-30T00:51:29.592 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T00:51:29.683 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:29 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:29.683 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:29 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:29.683 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:29 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:29.683 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:29 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T00:51:29.683 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:29 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T00:51:29.683 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:29 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T00:51:29.683 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:29 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:29.683 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:29 smithi130 ceph-mon[47626]: Deploying daemon mgr.y on smithi130 2024-03-30T00:51:29.684 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:29 smithi130 ceph-mon[47626]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:29.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:29 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:29.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:29 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:29.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:29 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:29.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:29 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T00:51:29.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:29 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T00:51:29.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:29 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T00:51:29.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:29 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:29.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:29 smithi181 ceph-mon[47622]: Deploying daemon mgr.y on smithi130 2024-03-30T00:51:29.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:29 smithi181 ceph-mon[47622]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:29.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:29 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:29.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:29 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:29.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:29 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:29.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:29 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T00:51:29.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:29 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T00:51:29.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:29 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T00:51:29.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:29 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:29.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:29 smithi116 ceph-mon[38086]: Deploying daemon mgr.y on smithi130 2024-03-30T00:51:29.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:29 smithi116 ceph-mon[38086]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:29.944 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:51:30.044 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:29 smithi130 systemd[1]: Starting Ceph mgr.y for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:51:30.105 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:29 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:29.768+0000 7fdcb0f72200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T00:51:30.106 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:30.016+0000 7fdcb0f72200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T00:51:30.106 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:30.099+0000 7fdcb0f72200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T00:51:30.344 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:30 smithi130 ceph-mgr[49127]: -- 172.21.15.130:0/3405504781 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a4893a61a0 con 0x55a48937c800 2024-03-30T00:51:30.345 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:30 smithi130 bash[49090]: 15e936174e548c45acf179eee6f4692db670fef55438407b88e87d48abe80766 2024-03-30T00:51:30.345 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:30 smithi130 systemd[1]: Started Ceph mgr.y for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:51:30.371 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:30.205+0000 7fdcb0f72200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T00:51:30.722 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:30.612+0000 7f86c9ac4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T00:51:30.752 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:30.364+0000 7fdcb0f72200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T00:51:30.752 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:30.491+0000 7fdcb0f72200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T00:51:30.752 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:30.577+0000 7fdcb0f72200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T00:51:30.912 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:30.715+0000 7f86c9ac4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T00:51:30.912 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:30.808+0000 7f86c9ac4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T00:51:30.912 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:30.905+0000 7f86c9ac4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T00:51:31.252 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:30.995+0000 7fdcb0f72200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T00:51:31.573 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T00:51:31.573 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T00:51:31.573 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-30T00:51:31.573 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-30T00:51:31.573 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0482391 s, 217 MB/s 2024-03-30T00:51:31.574 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-30T00:51:31.602 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:31 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:31.403+0000 7fdcb0f72200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T00:51:31.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:31.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:31.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:31.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:31.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:31.611 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:31.612 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:31.612 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:31.612 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:31.612 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:31.831 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:31 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:31.574+0000 7f86c9ac4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T00:51:31.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:31.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:31.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:31.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:31.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:31.946 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi116:vg_nvme/lv_4 2024-03-30T00:51:32.002 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:31 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:31.595+0000 7fdcb0f72200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T00:51:32.002 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:31 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:31.691+0000 7fdcb0f72200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T00:51:32.094 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:31 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:31.825+0000 7f86c9ac4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T00:51:32.095 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:31 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:31.909+0000 7f86c9ac4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T00:51:32.095 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:32.017+0000 7f86c9ac4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T00:51:32.445 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:32 smithi116 ceph-mon[38086]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:32.445 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.445 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.445 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.445 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.445 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:32.445 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:32.446 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.446 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T00:51:32.446 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T00:51:32.446 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:32.594 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:32.175+0000 7f86c9ac4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T00:51:32.595 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:32.301+0000 7f86c9ac4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T00:51:32.595 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:32.386+0000 7f86c9ac4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T00:51:32.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-mon[47626]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:32.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:32.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:32.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T00:51:32.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T00:51:32.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:32.651 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:51:32.694 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:32 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:32.446+0000 7fdcb0f72200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T00:51:32.695 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:32 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:32.541+0000 7fdcb0f72200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T00:51:32.695 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:32 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:32.688+0000 7fdcb0f72200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T00:51:32.695 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:32 smithi181 ceph-mon[47622]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:32.695 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.695 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.695 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.695 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.695 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:32.695 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:32.695 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:32.696 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T00:51:32.696 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T00:51:32.696 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:33.095 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:32 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:32.759+0000 7f86c9ac4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T00:51:33.406 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:33 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:33.148+0000 7f86c9ac4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T00:51:33.406 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:33 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:33.341+0000 7f86c9ac4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T00:51:33.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:33 smithi116 ceph-mon[38086]: Reconfiguring mgr.x (unknown last config time)... 2024-03-30T00:51:33.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:33 smithi116 ceph-mon[38086]: Reconfiguring daemon mgr.x on smithi116 2024-03-30T00:51:33.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:33 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:33.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:33 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:33.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:33 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:33.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:33 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:33.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:33 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:33.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:33 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:33.690 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:33 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:33.313+0000 7fdcb0f72200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T00:51:33.690 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:33 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:33.492+0000 7fdcb0f72200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T00:51:33.690 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:33 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:33.684+0000 7fdcb0f72200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T00:51:33.690 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:33 smithi181 ceph-mon[47622]: Reconfiguring mgr.x (unknown last config time)... 2024-03-30T00:51:33.690 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:33 smithi181 ceph-mon[47622]: Reconfiguring daemon mgr.x on smithi116 2024-03-30T00:51:33.690 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:33 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:33.691 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:33 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:33.691 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:33 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:33.691 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:33 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:33.691 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:33 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:33.691 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:33 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:33.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.712+0000 7f8084a00700 1 -- 172.21.15.116:0/2090299074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8080101e20 msgr2=0x7f8080102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:33.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.712+0000 7f8084a00700 1 --2- 172.21.15.116:0/2090299074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8080101e20 0x7f8080102280 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f806c0099c0 tx=0x7f806c009cd0 comp rx=0 tx=0).stop 2024-03-30T00:51:33.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.712+0000 7f8084a00700 1 -- 172.21.15.116:0/2090299074 shutdown_connections 2024-03-30T00:51:33.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.712+0000 7f8084a00700 1 --2- 172.21.15.116:0/2090299074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80801027c0 0x7f8080109070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.712+0000 7f8084a00700 1 --2- 172.21.15.116:0/2090299074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8080100c20 0x7f8080101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.712+0000 7f8084a00700 1 --2- 172.21.15.116:0/2090299074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8080101e20 0x7f8080102280 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.713+0000 7f8084a00700 1 -- 172.21.15.116:0/2090299074 >> 172.21.15.116:0/2090299074 conn(0x7f80800fc1c0 msgr2=0x7f80800fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.713+0000 7f8084a00700 1 -- 172.21.15.116:0/2090299074 shutdown_connections 2024-03-30T00:51:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.713+0000 7f8084a00700 1 -- 172.21.15.116:0/2090299074 wait complete. 2024-03-30T00:51:33.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.714+0000 7f8084a00700 1 Processor -- start 2024-03-30T00:51:33.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.714+0000 7f8084a00700 1 -- start start 2024-03-30T00:51:33.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.714+0000 7f8084a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8080100c20 0x7f8080198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:33.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.714+0000 7f8084a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8080101e20 0x7f80801992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:33.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.714+0000 7f8084a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80801027c0 0x7f80801a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:33.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.714+0000 7f8084a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80801a0870 con 0x7f80801027c0 2024-03-30T00:51:33.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.714+0000 7f8084a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80801a09b0 con 0x7f8080100c20 2024-03-30T00:51:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.714+0000 7f8084a00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80801a0b70 con 0x7f8080101e20 2024-03-30T00:51:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.714+0000 7f807f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8080100c20 0x7f8080198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.714+0000 7f807f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8080100c20 0x7f8080198d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:33004/0 (socket says 172.21.15.116:33004) 2024-03-30T00:51:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.714+0000 7f807f400700 1 -- 172.21.15.116:0/57839556 learned_addr learned my addr 172.21.15.116:0/57839556 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:51:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.714+0000 7f807ea00700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8080101e20 0x7f80801992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.714+0000 7f807fe00700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80801027c0 0x7f80801a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.715+0000 7f807f400700 1 -- 172.21.15.116:0/57839556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8080101e20 msgr2=0x7f80801992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.715+0000 7f807f400700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8080101e20 0x7f80801992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.715+0000 7f807f400700 1 -- 172.21.15.116:0/57839556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80801027c0 msgr2=0x7f80801a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.715+0000 7f807f400700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80801027c0 0x7f80801a02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.715+0000 7f807f400700 1 -- 172.21.15.116:0/57839556 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806c0095f0 con 0x7f8080100c20 2024-03-30T00:51:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.715+0000 7f807fe00700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80801027c0 0x7f80801a02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:51:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.715+0000 7f807ea00700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8080101e20 0x7f80801992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:51:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.715+0000 7f807f400700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8080100c20 0x7f8080198d70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f807000e970 tx=0x7f807000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.715+0000 7f8077e00700 1 -- 172.21.15.116:0/57839556 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8070009790 con 0x7f8080100c20 2024-03-30T00:51:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.715+0000 7f8077e00700 1 -- 172.21.15.116:0/57839556 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f807000eec0 con 0x7f8080100c20 2024-03-30T00:51:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.715+0000 7f8077e00700 1 -- 172.21.15.116:0/57839556 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8070003cf0 con 0x7f8080100c20 2024-03-30T00:51:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.715+0000 7f8084a00700 1 -- 172.21.15.116:0/57839556 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80801a0e50 con 0x7f8080100c20 2024-03-30T00:51:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.715+0000 7f8084a00700 1 -- 172.21.15.116:0/57839556 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80801a1210 con 0x7f8080100c20 2024-03-30T00:51:33.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.716+0000 7f8077e00700 1 -- 172.21.15.116:0/57839556 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 13) v1 ==== 45590+0+0 (secure 0 0 0) 0x7f8070004020 con 0x7f8080100c20 2024-03-30T00:51:33.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.716+0000 7f8084a00700 1 -- 172.21.15.116:0/57839556 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f808004ea80 con 0x7f8080100c20 2024-03-30T00:51:33.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.717+0000 7f8077e00700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f80640389d0 0x7f806403ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:33.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.717+0000 7f8077e00700 1 -- 172.21.15.116:0/57839556 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8070014070 con 0x7f8080100c20 2024-03-30T00:51:33.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.717+0000 7f807ea00700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f80640389d0 0x7f806403ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:33.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.717+0000 7f807ea00700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f80640389d0 0x7f806403ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f806c006010 tx=0x7f806c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:33.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.722+0000 7f8077e00700 1 -- 172.21.15.116:0/57839556 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f807000fd50 con 0x7f8080100c20 2024-03-30T00:51:33.844 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:33 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:33.433+0000 7f86c9ac4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T00:51:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:33 smithi130 ceph-mon[47626]: Reconfiguring mgr.x (unknown last config time)... 2024-03-30T00:51:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:33 smithi130 ceph-mon[47626]: Reconfiguring daemon mgr.x on smithi116 2024-03-30T00:51:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:33 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:33 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:33 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:33 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:33 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:33 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:33.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:33.932+0000 7f8084a00700 1 -- 172.21.15.116:0/57839556 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8080074700 con 0x7f80640389d0 2024-03-30T00:51:34.594 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:34 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:34.193+0000 7f86c9ac4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T00:51:34.595 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:34 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:34.290+0000 7f86c9ac4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T00:51:34.595 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:34 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:34.437+0000 7f86c9ac4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T00:51:34.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:34 smithi130 ceph-mon[47626]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:34.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:51:34.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:51:34.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:34.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:34 smithi116 ceph-mon[38086]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:34.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:51:34.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:51:34.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:34.752 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:34 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:34.609+0000 7fdcb0f72200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T00:51:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:34 smithi181 ceph-mon[47622]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:51:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:51:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:35.173 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:34 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:34.892+0000 7fdcb0f72200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T00:51:35.173 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:34 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:34.977+0000 7fdcb0f72200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T00:51:35.344 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:35 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:35.060+0000 7f86c9ac4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T00:51:35.345 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:35 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:35.240+0000 7f86c9ac4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T00:51:35.502 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 00:51:35 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z[49483]: 2024-03-30T00:51:35.166+0000 7fdcb0f72200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T00:51:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:35 smithi181 ceph-mon[47622]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:51:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:35 smithi181 ceph-mon[47622]: Standby manager daemon z started 2024-03-30T00:51:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:35 smithi181 ceph-mon[47622]: from='mgr.? 172.21.15.181:0/698915853' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-30T00:51:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:35 smithi181 ceph-mon[47622]: from='mgr.? 172.21.15.181:0/698915853' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T00:51:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:35 smithi181 ceph-mon[47622]: from='mgr.? 172.21.15.181:0/698915853' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-30T00:51:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:35 smithi181 ceph-mon[47622]: from='mgr.? 172.21.15.181:0/698915853' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T00:51:35.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:35 smithi116 ceph-mon[38086]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:51:35.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:35.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:35 smithi116 ceph-mon[38086]: Standby manager daemon z started 2024-03-30T00:51:35.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:35 smithi116 ceph-mon[38086]: from='mgr.? 172.21.15.181:0/698915853' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-30T00:51:35.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:35 smithi116 ceph-mon[38086]: from='mgr.? 172.21.15.181:0/698915853' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T00:51:35.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:35 smithi116 ceph-mon[38086]: from='mgr.? 172.21.15.181:0/698915853' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-30T00:51:35.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:35 smithi116 ceph-mon[38086]: from='mgr.? 172.21.15.181:0/698915853' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T00:51:35.844 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:35 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:35.433+0000 7f86c9ac4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T00:51:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:35 smithi130 ceph-mon[47626]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:51:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:35 smithi130 ceph-mon[47626]: Standby manager daemon z started 2024-03-30T00:51:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:35 smithi130 ceph-mon[47626]: from='mgr.? 172.21.15.181:0/698915853' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-30T00:51:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:35 smithi130 ceph-mon[47626]: from='mgr.? 172.21.15.181:0/698915853' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T00:51:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:35 smithi130 ceph-mon[47626]: from='mgr.? 172.21.15.181:0/698915853' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-30T00:51:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:35 smithi130 ceph-mon[47626]: from='mgr.? 172.21.15.181:0/698915853' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T00:51:36.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:36.107+0000 7f8077e00700 1 -- 172.21.15.116:0/57839556 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mgrmap(e 14) v1 ==== 90700+0+0 (secure 0 0 0) 0x7f8070021ce0 con 0x7f8080100c20 2024-03-30T00:51:36.648 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:36 smithi130 ceph-mon[47626]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:36.648 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:36 smithi130 ceph-mon[47626]: mgrmap e14: x(active, since 61s), standbys: z 2024-03-30T00:51:36.648 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:36 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-30T00:51:36.648 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:36 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1772904700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec18ae5d-60d8-48ae-85ab-06e1c3294a17"}]: dispatch 2024-03-30T00:51:36.649 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:36 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1772904700' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec18ae5d-60d8-48ae-85ab-06e1c3294a17"}]': finished 2024-03-30T00:51:36.649 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:36 smithi130 ceph-mon[47626]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T00:51:36.649 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:36 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:36.649 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:36 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:36.359+0000 7f86c9ac4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T00:51:36.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:36 smithi116 ceph-mon[38086]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:36.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:36 smithi116 ceph-mon[38086]: mgrmap e14: x(active, since 61s), standbys: z 2024-03-30T00:51:36.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-30T00:51:36.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:36 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1772904700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec18ae5d-60d8-48ae-85ab-06e1c3294a17"}]: dispatch 2024-03-30T00:51:36.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:36 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1772904700' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec18ae5d-60d8-48ae-85ab-06e1c3294a17"}]': finished 2024-03-30T00:51:36.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:36 smithi116 ceph-mon[38086]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T00:51:36.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:36.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:36 smithi181 ceph-mon[47622]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:36.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:36 smithi181 ceph-mon[47622]: mgrmap e14: x(active, since 61s), standbys: z 2024-03-30T00:51:36.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:36 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-30T00:51:36.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:36 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1772904700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec18ae5d-60d8-48ae-85ab-06e1c3294a17"}]: dispatch 2024-03-30T00:51:36.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:36 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1772904700' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec18ae5d-60d8-48ae-85ab-06e1c3294a17"}]': finished 2024-03-30T00:51:36.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:36 smithi181 ceph-mon[47622]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T00:51:36.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:36 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:36.926 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:36 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:36.641+0000 7f86c9ac4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T00:51:36.926 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:36 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:36.731+0000 7f86c9ac4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T00:51:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:37.419+0000 7f8077e00700 1 -- 172.21.15.116:0/57839556 <== mon.2 v2:172.21.15.130:3300/0 8 ==== mgrmap(e 15) v1 ==== 135810+0+0 (secure 0 0 0) 0x7f8070018430 con 0x7f8080100c20 2024-03-30T00:51:37.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:37 smithi116 ceph-mon[38086]: Standby manager daemon y started 2024-03-30T00:51:37.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:37 smithi116 ceph-mon[38086]: from='mgr.? 172.21.15.130:0/377518587' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-30T00:51:37.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:37 smithi116 ceph-mon[38086]: from='mgr.? 172.21.15.130:0/377518587' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T00:51:37.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:37 smithi116 ceph-mon[38086]: from='mgr.? 172.21.15.130:0/377518587' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-30T00:51:37.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:37 smithi116 ceph-mon[38086]: from='mgr.? 172.21.15.130:0/377518587' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T00:51:37.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:37 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2778772247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:51:37.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:37 smithi181 ceph-mon[47622]: Standby manager daemon y started 2024-03-30T00:51:37.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:37 smithi181 ceph-mon[47622]: from='mgr.? 172.21.15.130:0/377518587' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-30T00:51:37.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:37 smithi181 ceph-mon[47622]: from='mgr.? 172.21.15.130:0/377518587' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T00:51:37.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:37 smithi181 ceph-mon[47622]: from='mgr.? 172.21.15.130:0/377518587' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-30T00:51:37.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:37 smithi181 ceph-mon[47622]: from='mgr.? 172.21.15.130:0/377518587' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T00:51:37.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:37 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2778772247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:51:37.844 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 00:51:36 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y[49104]: 2024-03-30T00:51:36.919+0000 7f86c9ac4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T00:51:37.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:37 smithi130 ceph-mon[47626]: Standby manager daemon y started 2024-03-30T00:51:37.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:37 smithi130 ceph-mon[47626]: from='mgr.? 172.21.15.130:0/377518587' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-30T00:51:37.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:37 smithi130 ceph-mon[47626]: from='mgr.? 172.21.15.130:0/377518587' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T00:51:37.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:37 smithi130 ceph-mon[47626]: from='mgr.? 172.21.15.130:0/377518587' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-30T00:51:37.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:37 smithi130 ceph-mon[47626]: from='mgr.? 172.21.15.130:0/377518587' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T00:51:37.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:37 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2778772247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:51:38.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:38 smithi181 ceph-mon[47622]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:38.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:38 smithi181 ceph-mon[47622]: mgrmap e15: x(active, since 62s), standbys: z, y 2024-03-30T00:51:38.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:38 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T00:51:38.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:38 smithi130 ceph-mon[47626]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:38.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:38 smithi130 ceph-mon[47626]: mgrmap e15: x(active, since 62s), standbys: z, y 2024-03-30T00:51:38.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:38 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T00:51:38.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:38 smithi116 ceph-mon[38086]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:38.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:38 smithi116 ceph-mon[38086]: mgrmap e15: x(active, since 62s), standbys: z, y 2024-03-30T00:51:38.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:38 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T00:51:40.485 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:40 smithi116 ceph-mon[38086]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:40.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:40 smithi181 ceph-mon[47622]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:40.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:40 smithi130 ceph-mon[47626]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:42.593 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:42 smithi116 ceph-mon[38086]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:42.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:42 smithi181 ceph-mon[47622]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:42.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:42 smithi130 ceph-mon[47626]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:43.543 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:43 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T00:51:43.543 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:43 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:43.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:43 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T00:51:43.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:43 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:43.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:43 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T00:51:43.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:43 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:44.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:44 smithi116 ceph-mon[38086]: Deploying daemon osd.0 on smithi116 2024-03-30T00:51:44.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:44 smithi116 ceph-mon[38086]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:44.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:44 smithi181 ceph-mon[47622]: Deploying daemon osd.0 on smithi116 2024-03-30T00:51:44.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:44 smithi181 ceph-mon[47622]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:44.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:44 smithi130 ceph-mon[47626]: Deploying daemon osd.0 on smithi116 2024-03-30T00:51:44.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:44 smithi130 ceph-mon[47626]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:46.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:46 smithi181 ceph-mon[47622]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:46.785 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:46 smithi116 ceph-mon[38086]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:46.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:46 smithi130 ceph-mon[47626]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:48.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:48 smithi116 ceph-mon[38086]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:48.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:48 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:48.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:48 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:48.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:48 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:48.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:48 smithi181 ceph-mon[47622]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:48.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:48 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:48.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:48 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:48.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:48 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:48.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:48 smithi130 ceph-mon[47626]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:48.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:48 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:48.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:48 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:48.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:48 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:50.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:49.995+0000 7f8077e00700 1 -- 172.21.15.116:0/57839556 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8080074700 con 0x7f80640389d0 2024-03-30T00:51:50.001 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 0 on host 'smithi116' 2024-03-30T00:51:50.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:49.997+0000 7f8084a00700 1 -- 172.21.15.116:0/57839556 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f80640389d0 msgr2=0x7f806403ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:50.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:49.997+0000 7f8084a00700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f80640389d0 0x7f806403ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f806c006010 tx=0x7f806c00b540 comp rx=0 tx=0).stop 2024-03-30T00:51:50.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:49.997+0000 7f8084a00700 1 -- 172.21.15.116:0/57839556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8080100c20 msgr2=0x7f8080198d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:50.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:49.997+0000 7f8084a00700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8080100c20 0x7f8080198d70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f807000e970 tx=0x7f807000ec80 comp rx=0 tx=0).stop 2024-03-30T00:51:50.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:49.997+0000 7f8084a00700 1 -- 172.21.15.116:0/57839556 shutdown_connections 2024-03-30T00:51:50.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:49.997+0000 7f8084a00700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f80640389d0 0x7f806403ae90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:50.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:49.997+0000 7f8084a00700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80801027c0 0x7f80801a02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:50.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:49.997+0000 7f8084a00700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8080100c20 0x7f8080198d70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:50.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:49.997+0000 7f8084a00700 1 --2- 172.21.15.116:0/57839556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8080101e20 0x7f80801992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:50.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:49.997+0000 7f8084a00700 1 -- 172.21.15.116:0/57839556 >> 172.21.15.116:0/57839556 conn(0x7f80800fc1c0 msgr2=0x7f8080072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:50.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:49.997+0000 7f8084a00700 1 -- 172.21.15.116:0/57839556 shutdown_connections 2024-03-30T00:51:50.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:51:49.997+0000 7f8084a00700 1 -- 172.21.15.116:0/57839556 wait complete. 2024-03-30T00:51:50.206 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.206 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.206 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:50 smithi116 ceph-mon[38086]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:50.206 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:50.206 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:50.206 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.206 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.206 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.207 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:50.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:50 smithi130 ceph-mon[47626]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:50.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:50.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:50.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:50.370 DEBUG:teuthology.orchestra.run.smithi116:osd.0> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.0.service 2024-03-30T00:51:50.375 INFO:tasks.cephadm:Deploying osd.1 on smithi130 with /dev/vg_nvme/lv_4... 2024-03-30T00:51:50.375 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T00:51:50.455 INFO:journalctl@ceph.osd.0.smithi116.stdout:-- Logs begin at Sat 2024-03-30 00:33:55 UTC. -- 2024-03-30T00:51:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:50 smithi181 ceph-mon[47622]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:50.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:51:50.602 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T00:51:51.674 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 00:51:51 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-0[55654]: 2024-03-30T00:51:51.220+0000 7f5ebb432640 -1 osd.0 0 log_to_monitors true 2024-03-30T00:51:52.039 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T00:51:52.039 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T00:51:52.040 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-03-30T00:51:52.040 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-03-30T00:51:52.040 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111484 s, 941 MB/s 2024-03-30T00:51:52.040 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-03-30T00:51:52.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:52 smithi130 ceph-mon[47626]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:52.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:52 smithi130 ceph-mon[47626]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T00:51:52.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:52 smithi116 ceph-mon[38086]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:52.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:52 smithi116 ceph-mon[38086]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T00:51:52.428 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi130:vg_nvme/lv_4 2024-03-30T00:51:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:52 smithi181 ceph-mon[47622]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:52 smithi181 ceph-mon[47622]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T00:51:52.642 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T00:51:53.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:53 smithi130 ceph-mon[47626]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T00:51:53.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:53 smithi130 ceph-mon[47626]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T00:51:53.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:53.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:53 smithi130 ceph-mon[47626]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:51:53.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:53.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:53.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:51:53.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:53.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:53.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:53.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:53 smithi116 ceph-mon[38086]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T00:51:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:53 smithi116 ceph-mon[38086]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T00:51:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:53 smithi116 ceph-mon[38086]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:51:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:51:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:53.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:53.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:53 smithi181 ceph-mon[47622]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T00:51:53.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:53 smithi181 ceph-mon[47622]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T00:51:53.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:53.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:53 smithi181 ceph-mon[47622]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:51:53.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:53.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:53.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:51:53.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:53.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:53.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:51:53.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:51:53.628 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.621+0000 7f1146e3c700 1 -- 172.21.15.130:0/4136484945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11401041b0 msgr2=0x7f11401065a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:53.629 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.621+0000 7f1146e3c700 1 --2- 172.21.15.130:0/4136484945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11401041b0 0x7f11401065a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f112c009990 tx=0x7f112c009ca0 comp rx=0 tx=0).stop 2024-03-30T00:51:53.629 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.621+0000 7f1146e3c700 1 -- 172.21.15.130:0/4136484945 shutdown_connections 2024-03-30T00:51:53.629 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.621+0000 7f1146e3c700 1 --2- 172.21.15.130:0/4136484945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1140106ae0 0x7f1140108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:53.629 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.621+0000 7f1146e3c700 1 --2- 172.21.15.130:0/4136484945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1140101880 0x7f1140103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:53.629 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.621+0000 7f1146e3c700 1 --2- 172.21.15.130:0/4136484945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11401041b0 0x7f11401065a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:53.629 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.621+0000 7f1146e3c700 1 -- 172.21.15.130:0/4136484945 >> 172.21.15.130:0/4136484945 conn(0x7f11400fb2b0 msgr2=0x7f11400fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:51:53.629 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.621+0000 7f1146e3c700 1 -- 172.21.15.130:0/4136484945 shutdown_connections 2024-03-30T00:51:53.629 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.621+0000 7f1146e3c700 1 -- 172.21.15.130:0/4136484945 wait complete. 2024-03-30T00:51:53.629 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.621+0000 7f1146e3c700 1 Processor -- start 2024-03-30T00:51:53.629 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.621+0000 7f1146e3c700 1 -- start start 2024-03-30T00:51:53.629 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f1146e3c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1140101880 0x7f1140196ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:53.629 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f1146e3c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11401041b0 0x7f11401971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:53.630 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f1146e3c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1140106ae0 0x7f114019e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:53.630 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f1146e3c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114019e7a0 con 0x7f1140101880 2024-03-30T00:51:53.630 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f1146e3c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114019e8e0 con 0x7f11401041b0 2024-03-30T00:51:53.630 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f1146e3c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114019eaa0 con 0x7f1140106ae0 2024-03-30T00:51:53.630 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f113ffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11401041b0 0x7f11401971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:53.630 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f113ffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11401041b0 0x7f11401971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:44208/0 (socket says 172.21.15.130:44208) 2024-03-30T00:51:53.630 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f113ffff700 1 -- 172.21.15.130:0/1617420658 learned_addr learned my addr 172.21.15.130:0/1617420658 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-03-30T00:51:53.630 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f1144bd8700 1 --2- 172.21.15.130:0/1617420658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1140101880 0x7f1140196ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:53.630 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f11453d9700 1 --2- 172.21.15.130:0/1617420658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1140106ae0 0x7f114019e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:53.630 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f113ffff700 1 -- 172.21.15.130:0/1617420658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1140106ae0 msgr2=0x7f114019e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:53.630 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f113ffff700 1 --2- 172.21.15.130:0/1617420658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1140106ae0 0x7f114019e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:53.630 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f113ffff700 1 -- 172.21.15.130:0/1617420658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1140101880 msgr2=0x7f1140196ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:51:53.630 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f113ffff700 1 --2- 172.21.15.130:0/1617420658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1140101880 0x7f1140196ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:51:53.631 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f113ffff700 1 -- 172.21.15.130:0/1617420658 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c0095f0 con 0x7f11401041b0 2024-03-30T00:51:53.631 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f113ffff700 1 --2- 172.21.15.130:0/1617420658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11401041b0 0x7f11401971e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f112c0095c0 tx=0x7f112c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:53.631 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f113dffb700 1 -- 172.21.15.130:0/1617420658 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c01d070 con 0x7f11401041b0 2024-03-30T00:51:53.631 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f113dffb700 1 -- 172.21.15.130:0/1617420658 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f112c005030 con 0x7f11401041b0 2024-03-30T00:51:53.631 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f1146e3c700 1 -- 172.21.15.130:0/1617420658 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114019ed20 con 0x7f11401041b0 2024-03-30T00:51:53.631 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.622+0000 7f113dffb700 1 -- 172.21.15.130:0/1617420658 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c004b00 con 0x7f11401041b0 2024-03-30T00:51:53.631 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.623+0000 7f1146e3c700 1 -- 172.21.15.130:0/1617420658 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f114019f130 con 0x7f11401041b0 2024-03-30T00:51:53.632 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.624+0000 7f1146e3c700 1 -- 172.21.15.130:0/1617420658 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11400fce30 con 0x7f11401041b0 2024-03-30T00:51:53.632 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.625+0000 7f113dffb700 1 -- 172.21.15.130:0/1617420658 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 15) v1 ==== 135810+0+0 (secure 0 0 0) 0x7f112c005410 con 0x7f11401041b0 2024-03-30T00:51:53.632 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.625+0000 7f113dffb700 1 --2- 172.21.15.130:0/1617420658 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f11300a1090 0x7f11300a3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:51:53.632 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.626+0000 7f113dffb700 1 -- 172.21.15.130:0/1617420658 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1487+0+0 (secure 0 0 0) 0x7f112c0cbbc0 con 0x7f11401041b0 2024-03-30T00:51:53.632 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.626+0000 7f1144bd8700 1 --2- 172.21.15.130:0/1617420658 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f11300a1090 0x7f11300a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:51:53.632 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.626+0000 7f1144bd8700 1 --2- 172.21.15.130:0/1617420658 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f11300a1090 0x7f11300a3550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1140198130 tx=0x7f113400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:51:53.635 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.630+0000 7f113dffb700 1 -- 172.21.15.130:0/1617420658 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f112c096f70 con 0x7f11401041b0 2024-03-30T00:51:53.836 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:51:53.829+0000 7f1146e3c700 1 -- 172.21.15.130:0/1617420658 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f114002cd70 con 0x7f11300a1090 2024-03-30T00:51:53.924 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 00:51:53 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-0[55654]: 2024-03-30T00:51:53.633+0000 7f5eafa00700 -1 osd.0 0 waiting for initial osdmap 2024-03-30T00:51:53.924 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 00:51:53 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-0[55654]: 2024-03-30T00:51:53.646+0000 7f5ea5800700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T00:51:54.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:54 smithi116 ceph-mon[38086]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:54.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:54 smithi116 ceph-mon[38086]: Detected new or changed devices on smithi116 2024-03-30T00:51:54.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:54 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi116 to 17270M 2024-03-30T00:51:54.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:54 smithi116 ceph-mon[38086]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-30T00:51:54.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:54 smithi116 ceph-mon[38086]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T00:51:54.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:54 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:54.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:54 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:54.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:54 smithi116 ceph-mon[38086]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' 2024-03-30T00:51:54.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:54 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:51:54.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:54 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:51:54.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:54 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:54.473 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:54 smithi130 ceph-mon[47626]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:54.473 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:54 smithi130 ceph-mon[47626]: Detected new or changed devices on smithi116 2024-03-30T00:51:54.473 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:54 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi116 to 17270M 2024-03-30T00:51:54.473 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:54 smithi130 ceph-mon[47626]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-30T00:51:54.473 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:54 smithi130 ceph-mon[47626]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T00:51:54.473 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:54 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:54.473 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:54 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:54.473 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:54 smithi130 ceph-mon[47626]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' 2024-03-30T00:51:54.473 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:54 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:51:54.473 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:54 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:51:54.473 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:54 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:54.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:54 smithi181 ceph-mon[47622]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:54.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:54 smithi181 ceph-mon[47622]: Detected new or changed devices on smithi116 2024-03-30T00:51:54.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:54 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi116 to 17270M 2024-03-30T00:51:54.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:54 smithi181 ceph-mon[47622]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-30T00:51:54.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:54 smithi181 ceph-mon[47622]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T00:51:54.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:54 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:54.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:54 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:54.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:54 smithi181 ceph-mon[47622]: from='osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018]' entity='osd.0' 2024-03-30T00:51:54.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:54 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:51:54.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:54 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:51:54.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:54 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:51:55.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:55 smithi116 ceph-mon[38086]: purged_snaps scrub starts 2024-03-30T00:51:55.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:55 smithi116 ceph-mon[38086]: purged_snaps scrub ok 2024-03-30T00:51:55.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:55 smithi116 ceph-mon[38086]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:51:55.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:55.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:55 smithi116 ceph-mon[38086]: osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018] boot 2024-03-30T00:51:55.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:55 smithi116 ceph-mon[38086]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T00:51:55.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:55.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:55 smithi181 ceph-mon[47622]: purged_snaps scrub starts 2024-03-30T00:51:55.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:55 smithi181 ceph-mon[47622]: purged_snaps scrub ok 2024-03-30T00:51:55.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:55 smithi181 ceph-mon[47622]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:51:55.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:55 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:55.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:55 smithi181 ceph-mon[47622]: osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018] boot 2024-03-30T00:51:55.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:55 smithi181 ceph-mon[47622]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T00:51:55.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:55 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:55.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:55 smithi130 ceph-mon[47626]: purged_snaps scrub starts 2024-03-30T00:51:55.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:55 smithi130 ceph-mon[47626]: purged_snaps scrub ok 2024-03-30T00:51:55.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:55 smithi130 ceph-mon[47626]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:51:55.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:55 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:55.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:55 smithi130 ceph-mon[47626]: osd.0 [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018] boot 2024-03-30T00:51:55.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:55 smithi130 ceph-mon[47626]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T00:51:55.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:55 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T00:51:56.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:56 smithi116 ceph-mon[38086]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:56.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:56 smithi116 ceph-mon[38086]: from='client.? 172.21.15.130:0/3680151896' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0047501-7c23-42fe-9e48-a949d3523427"}]: dispatch 2024-03-30T00:51:56.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:56 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0047501-7c23-42fe-9e48-a949d3523427"}]: dispatch 2024-03-30T00:51:56.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:56 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0047501-7c23-42fe-9e48-a949d3523427"}]': finished 2024-03-30T00:51:56.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:56 smithi116 ceph-mon[38086]: osdmap e9: 2 total, 1 up, 2 in 2024-03-30T00:51:56.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:56 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:51:56.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:56 smithi181 ceph-mon[47622]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:56.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:56 smithi181 ceph-mon[47622]: from='client.? 172.21.15.130:0/3680151896' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0047501-7c23-42fe-9e48-a949d3523427"}]: dispatch 2024-03-30T00:51:56.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:56 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0047501-7c23-42fe-9e48-a949d3523427"}]: dispatch 2024-03-30T00:51:56.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:56 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0047501-7c23-42fe-9e48-a949d3523427"}]': finished 2024-03-30T00:51:56.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:56 smithi181 ceph-mon[47622]: osdmap e9: 2 total, 1 up, 2 in 2024-03-30T00:51:56.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:56 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:51:56.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:56 smithi130 ceph-mon[47626]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T00:51:56.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:56 smithi130 ceph-mon[47626]: from='client.? 172.21.15.130:0/3680151896' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0047501-7c23-42fe-9e48-a949d3523427"}]: dispatch 2024-03-30T00:51:56.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:56 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0047501-7c23-42fe-9e48-a949d3523427"}]: dispatch 2024-03-30T00:51:56.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:56 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0047501-7c23-42fe-9e48-a949d3523427"}]': finished 2024-03-30T00:51:56.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:56 smithi130 ceph-mon[47626]: osdmap e9: 2 total, 1 up, 2 in 2024-03-30T00:51:56.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:56 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:51:57.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:57 smithi116 ceph-mon[38086]: from='client.? 172.21.15.130:0/459198221' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:51:57.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:57 smithi181 ceph-mon[47622]: from='client.? 172.21.15.130:0/459198221' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:51:57.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:57 smithi130 ceph-mon[47626]: from='client.? 172.21.15.130:0/459198221' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:51:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:51:58 smithi116 ceph-mon[38086]: pgmap v40: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:51:58.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:51:58 smithi181 ceph-mon[47622]: pgmap v40: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:51:58.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:51:58 smithi130 ceph-mon[47626]: pgmap v40: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:00.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:00 smithi116 ceph-mon[38086]: pgmap v41: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:00.427 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:00 smithi130 ceph-mon[47626]: pgmap v41: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:00.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:00 smithi181 ceph-mon[47622]: pgmap v41: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:02.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:02 smithi116 ceph-mon[38086]: pgmap v42: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:02.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T00:52:02.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:02.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:02 smithi181 ceph-mon[47622]: pgmap v42: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:02.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T00:52:02.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:02.522 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:02 smithi130 ceph-mon[47626]: pgmap v42: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:02.522 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T00:52:02.522 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:03.288 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:03 smithi130 ceph-mon[47626]: Deploying daemon osd.1 on smithi130 2024-03-30T00:52:03.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:03 smithi116 ceph-mon[38086]: Deploying daemon osd.1 on smithi130 2024-03-30T00:52:03.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:03 smithi181 ceph-mon[47622]: Deploying daemon osd.1 on smithi130 2024-03-30T00:52:04.316 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:04 smithi130 ceph-mon[47626]: pgmap v43: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:04.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:04 smithi116 ceph-mon[38086]: pgmap v43: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:04.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:04 smithi181 ceph-mon[47622]: pgmap v43: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:06.238 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:06 smithi130 ceph-mon[47626]: pgmap v44: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:06.238 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:06.238 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:06.238 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:06.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:06 smithi116 ceph-mon[38086]: pgmap v44: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:06.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:06.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:06.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:06.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:06 smithi181 ceph-mon[47622]: pgmap v44: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:06.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:06.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:06.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:07.747 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 1 on host 'smithi130' 2024-03-30T00:52:07.747 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:07.739+0000 7f113dffb700 1 -- 172.21.15.130:0/1617420658 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f114002cd70 con 0x7f11300a1090 2024-03-30T00:52:07.748 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:07.742+0000 7f1146e3c700 1 -- 172.21.15.130:0/1617420658 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f11300a1090 msgr2=0x7f11300a3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:07.749 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:07.742+0000 7f1146e3c700 1 --2- 172.21.15.130:0/1617420658 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f11300a1090 0x7f11300a3550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1140198130 tx=0x7f113400a380 comp rx=0 tx=0).stop 2024-03-30T00:52:07.749 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:07.742+0000 7f1146e3c700 1 -- 172.21.15.130:0/1617420658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11401041b0 msgr2=0x7f11401971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:07.749 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:07.742+0000 7f1146e3c700 1 --2- 172.21.15.130:0/1617420658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11401041b0 0x7f11401971e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f112c0095c0 tx=0x7f112c00f690 comp rx=0 tx=0).stop 2024-03-30T00:52:07.749 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:07.742+0000 7f1146e3c700 1 -- 172.21.15.130:0/1617420658 shutdown_connections 2024-03-30T00:52:07.749 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:07.742+0000 7f1146e3c700 1 --2- 172.21.15.130:0/1617420658 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f11300a1090 0x7f11300a3550 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:07.749 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:07.742+0000 7f1146e3c700 1 --2- 172.21.15.130:0/1617420658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1140106ae0 0x7f114019e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:07.749 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:07.742+0000 7f1146e3c700 1 --2- 172.21.15.130:0/1617420658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1140101880 0x7f1140196ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:07.749 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:07.742+0000 7f1146e3c700 1 --2- 172.21.15.130:0/1617420658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11401041b0 0x7f11401971e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:07.749 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:07.742+0000 7f1146e3c700 1 -- 172.21.15.130:0/1617420658 >> 172.21.15.130:0/1617420658 conn(0x7f11400fb2b0 msgr2=0x7f1140102320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:52:07.749 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:07.742+0000 7f1146e3c700 1 -- 172.21.15.130:0/1617420658 shutdown_connections 2024-03-30T00:52:07.749 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:07.742+0000 7f1146e3c700 1 -- 172.21.15.130:0/1617420658 wait complete. 2024-03-30T00:52:07.905 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:07.905 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:07.905 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:07 smithi130 ceph-mon[47626]: pgmap v45: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:07.905 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:07.905 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:07.905 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:07.905 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:07.905 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:07.906 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:08.118 DEBUG:teuthology.orchestra.run.smithi130:osd.1> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.1.service 2024-03-30T00:52:08.120 INFO:tasks.cephadm:Deploying osd.2 on smithi181 with /dev/vg_nvme/lv_4... 2024-03-30T00:52:08.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T00:52:08.152 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:08.152 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:08.152 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:07 smithi181 ceph-mon[47622]: pgmap v45: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:08.152 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:08.152 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:08.152 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:08.152 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:08.152 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:08.152 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:08.153 INFO:journalctl@ceph.osd.1.smithi130.stdout:-- Logs begin at Sat 2024-03-30 00:27:58 UTC. -- 2024-03-30T00:52:08.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:08.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:08.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:07 smithi116 ceph-mon[38086]: pgmap v45: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:08.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:08.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:08.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:08.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:08.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:08.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:08.344 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T00:52:09.658 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T00:52:09.658 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T00:52:09.658 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-03-30T00:52:09.659 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-03-30T00:52:09.659 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113818 s, 921 MB/s 2024-03-30T00:52:09.659 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-03-30T00:52:09.845 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 00:52:09 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-1[52636]: 2024-03-30T00:52:09.561+0000 7f5027afd640 -1 osd.1 0 log_to_monitors true 2024-03-30T00:52:09.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi181:vg_nvme/lv_4 2024-03-30T00:52:10.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:10 smithi181 ceph-mon[47622]: pgmap v46: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:10.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:10 smithi181 ceph-mon[47622]: from='osd.1 [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T00:52:10.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:10 smithi181 ceph-mon[47622]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T00:52:10.252 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T00:52:10.291 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:10 smithi130 ceph-mon[47626]: pgmap v46: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:10.291 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:10 smithi130 ceph-mon[47626]: from='osd.1 [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T00:52:10.291 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:10 smithi130 ceph-mon[47626]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T00:52:10.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:10 smithi116 ceph-mon[38086]: pgmap v46: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:10.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:10 smithi116 ceph-mon[38086]: from='osd.1 [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T00:52:10.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:10 smithi116 ceph-mon[38086]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T00:52:11.250 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.241+0000 7ff595c35700 1 -- 172.21.15.181:0/2066736982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff590101890 msgr2=0x7ff590108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:11.250 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.241+0000 7ff595c35700 1 --2- 172.21.15.181:0/2066736982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff590101890 0x7ff590108140 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff58800b0a0 tx=0x7ff58800b3b0 comp rx=0 tx=0).stop 2024-03-30T00:52:11.250 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.242+0000 7ff595c35700 1 -- 172.21.15.181:0/2066736982 shutdown_connections 2024-03-30T00:52:11.250 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.242+0000 7ff595c35700 1 --2- 172.21.15.181:0/2066736982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff590101890 0x7ff590108140 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:11.250 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.242+0000 7ff595c35700 1 --2- 172.21.15.181:0/2066736982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5900ffcf0 0x7ff590100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:11.250 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.242+0000 7ff595c35700 1 --2- 172.21.15.181:0/2066736982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff590100ef0 0x7ff590101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:11.250 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.242+0000 7ff595c35700 1 -- 172.21.15.181:0/2066736982 >> 172.21.15.181:0/2066736982 conn(0x7ff5900fb2b0 msgr2=0x7ff5900fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:52:11.251 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.242+0000 7ff595c35700 1 -- 172.21.15.181:0/2066736982 shutdown_connections 2024-03-30T00:52:11.251 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.242+0000 7ff595c35700 1 -- 172.21.15.181:0/2066736982 wait complete. 2024-03-30T00:52:11.251 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.242+0000 7ff595c35700 1 Processor -- start 2024-03-30T00:52:11.251 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.242+0000 7ff595c35700 1 -- start start 2024-03-30T00:52:11.251 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.242+0000 7ff595c35700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5900ffcf0 0x7ff590103c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:52:11.251 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.242+0000 7ff595c35700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff590100ef0 0x7ff5901022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:52:11.251 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.242+0000 7ff595c35700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff590101890 0x7ff590102810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:52:11.251 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff595c35700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff590102fb0 con 0x7ff5900ffcf0 2024-03-30T00:52:11.251 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff595c35700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff590103120 con 0x7ff590101890 2024-03-30T00:52:11.251 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff595c35700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5901a0e10 con 0x7ff590100ef0 2024-03-30T00:52:11.251 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff58effd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff590100ef0 0x7ff5901022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:52:11.252 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff58effd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff590100ef0 0x7ff5901022d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50134/0 (socket says 172.21.15.181:50134) 2024-03-30T00:52:11.252 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff58effd700 1 -- 172.21.15.181:0/1332400573 learned_addr learned my addr 172.21.15.181:0/1332400573 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-03-30T00:52:11.252 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff58ffff700 1 --2- 172.21.15.181:0/1332400573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff590101890 0x7ff590102810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:52:11.252 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff58effd700 1 -- 172.21.15.181:0/1332400573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff590101890 msgr2=0x7ff590102810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:11.252 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff58effd700 1 --2- 172.21.15.181:0/1332400573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff590101890 0x7ff590102810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:11.252 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff58effd700 1 -- 172.21.15.181:0/1332400573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5900ffcf0 msgr2=0x7ff590103c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:11.252 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff58effd700 1 --2- 172.21.15.181:0/1332400573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5900ffcf0 0x7ff590103c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:11.252 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff58effd700 1 -- 172.21.15.181:0/1332400573 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff588009d00 con 0x7ff590100ef0 2024-03-30T00:52:11.252 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff58effd700 1 --2- 172.21.15.181:0/1332400573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff590100ef0 0x7ff5901022d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff58400c9b0 tx=0x7ff58400cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:52:11.252 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff58cff9700 1 -- 172.21.15.181:0/1332400573 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584007b00 con 0x7ff590100ef0 2024-03-30T00:52:11.252 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff58cff9700 1 -- 172.21.15.181:0/1332400573 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff584007c60 con 0x7ff590100ef0 2024-03-30T00:52:11.252 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff595c35700 1 -- 172.21.15.181:0/1332400573 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5901a10a0 con 0x7ff590100ef0 2024-03-30T00:52:11.253 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff58cff9700 1 -- 172.21.15.181:0/1332400573 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584018660 con 0x7ff590100ef0 2024-03-30T00:52:11.253 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.243+0000 7ff595c35700 1 -- 172.21.15.181:0/1332400573 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5901a15f0 con 0x7ff590100ef0 2024-03-30T00:52:11.253 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.244+0000 7ff595c35700 1 -- 172.21.15.181:0/1332400573 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59004ea80 con 0x7ff590100ef0 2024-03-30T00:52:11.254 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.245+0000 7ff58cff9700 1 -- 172.21.15.181:0/1332400573 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 15) v1 ==== 135810+0+0 (secure 0 0 0) 0x7ff584018880 con 0x7ff590100ef0 2024-03-30T00:52:11.254 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.246+0000 7ff58cff9700 1 --2- 172.21.15.181:0/1332400573 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff5780a1190 0x7ff5780a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:52:11.254 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.246+0000 7ff58cff9700 1 -- 172.21.15.181:0/1332400573 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 2029+0+0 (secure 0 0 0) 0x7ff5840cafd0 con 0x7ff590100ef0 2024-03-30T00:52:11.254 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.246+0000 7ff58f7fe700 1 --2- 172.21.15.181:0/1332400573 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff5780a1190 0x7ff5780a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:52:11.254 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.246+0000 7ff58f7fe700 1 --2- 172.21.15.181:0/1332400573 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff5780a1190 0x7ff5780a3650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff590100d50 tx=0x7ff5800096c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:52:11.256 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.250+0000 7ff58cff9700 1 -- 172.21.15.181:0/1332400573 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff584096030 con 0x7ff590100ef0 2024-03-30T00:52:11.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:11 smithi116 ceph-mon[38086]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T00:52:11.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:11 smithi116 ceph-mon[38086]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T00:52:11.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:11 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:52:11.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:11 smithi116 ceph-mon[38086]: from='osd.1 [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:52:11.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:11 smithi116 ceph-mon[38086]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:52:11.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:11 smithi116 ceph-mon[38086]: Detected new or changed devices on smithi130 2024-03-30T00:52:11.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:11 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:11.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:11 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:11.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:11 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:52:11.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:11 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi130 to 17113M 2024-03-30T00:52:11.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:11 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:11.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:11 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:11.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:11 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:11.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:11 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:11.455 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:11.449+0000 7ff595c35700 1 -- 172.21.15.181:0/1332400573 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5900fd3f0 con 0x7ff5780a1190 2024-03-30T00:52:11.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:11 smithi181 ceph-mon[47622]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T00:52:11.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:11 smithi181 ceph-mon[47622]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T00:52:11.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:11 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:52:11.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:11 smithi181 ceph-mon[47622]: from='osd.1 [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:52:11.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:11 smithi181 ceph-mon[47622]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:52:11.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:11 smithi181 ceph-mon[47622]: Detected new or changed devices on smithi130 2024-03-30T00:52:11.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:11 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:11.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:11 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:11.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:11 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:52:11.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:11 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi130 to 17113M 2024-03-30T00:52:11.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:11 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:11.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:11 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:11.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:11 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:11.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:11 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:11.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-mon[47626]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T00:52:11.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-mon[47626]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T00:52:11.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:52:11.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-mon[47626]: from='osd.1 [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:52:11.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-mon[47626]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:52:11.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-mon[47626]: Detected new or changed devices on smithi130 2024-03-30T00:52:11.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:11.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:11.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:52:11.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi130 to 17113M 2024-03-30T00:52:11.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:11.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:11.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:11.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:11.596 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-1[52636]: 2024-03-30T00:52:11.120+0000 7f501d161700 -1 osd.1 0 waiting for initial osdmap 2024-03-30T00:52:11.596 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 00:52:11 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-1[52636]: 2024-03-30T00:52:11.124+0000 7f5018f56700 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T00:52:12.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:12 smithi116 ceph-mon[38086]: pgmap v48: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:12.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:12 smithi116 ceph-mon[38086]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-30T00:52:12.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:12 smithi116 ceph-mon[38086]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T00:52:12.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:52:12.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:52:12.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:12 smithi116 ceph-mon[38086]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:52:12.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:52:12.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:52:12.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:12.477 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:12 smithi181 ceph-mon[47622]: pgmap v48: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:12.477 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:12 smithi181 ceph-mon[47622]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-30T00:52:12.477 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:12 smithi181 ceph-mon[47622]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T00:52:12.477 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:52:12.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:52:12.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:12 smithi181 ceph-mon[47622]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:52:12.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:52:12.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:52:12.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:12.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:12 smithi130 ceph-mon[47626]: pgmap v48: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:12 smithi130 ceph-mon[47626]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-30T00:52:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:12 smithi130 ceph-mon[47626]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T00:52:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:12 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:52:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:12 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:52:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:12 smithi130 ceph-mon[47626]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:52:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:12 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:52:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:12 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:52:12.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:12 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:13.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:13 smithi116 ceph-mon[38086]: purged_snaps scrub starts 2024-03-30T00:52:13.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:13 smithi116 ceph-mon[38086]: purged_snaps scrub ok 2024-03-30T00:52:13.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:13 smithi116 ceph-mon[38086]: osd.1 [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112] boot 2024-03-30T00:52:13.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:13 smithi116 ceph-mon[38086]: osdmap e12: 2 total, 2 up, 2 in 2024-03-30T00:52:13.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:13 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:52:13.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:13 smithi181 ceph-mon[47622]: purged_snaps scrub starts 2024-03-30T00:52:13.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:13 smithi181 ceph-mon[47622]: purged_snaps scrub ok 2024-03-30T00:52:13.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:13 smithi181 ceph-mon[47622]: osd.1 [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112] boot 2024-03-30T00:52:13.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:13 smithi181 ceph-mon[47622]: osdmap e12: 2 total, 2 up, 2 in 2024-03-30T00:52:13.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:13 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:52:13.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:13 smithi130 ceph-mon[47626]: purged_snaps scrub starts 2024-03-30T00:52:13.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:13 smithi130 ceph-mon[47626]: purged_snaps scrub ok 2024-03-30T00:52:13.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:13 smithi130 ceph-mon[47626]: osd.1 [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112] boot 2024-03-30T00:52:13.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:13 smithi130 ceph-mon[47626]: osdmap e12: 2 total, 2 up, 2 in 2024-03-30T00:52:13.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:13 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T00:52:14.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:14 smithi116 ceph-mon[38086]: pgmap v51: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:14.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:14 smithi116 ceph-mon[38086]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T00:52:14.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:14 smithi116 ceph-mon[38086]: from='client.? 172.21.15.181:0/1471020707' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7679f78-2053-4395-952b-efbdabfc12fd"}]: dispatch 2024-03-30T00:52:14.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:14 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7679f78-2053-4395-952b-efbdabfc12fd"}]: dispatch 2024-03-30T00:52:14.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:14 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7679f78-2053-4395-952b-efbdabfc12fd"}]': finished 2024-03-30T00:52:14.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:14 smithi116 ceph-mon[38086]: osdmap e14: 3 total, 2 up, 3 in 2024-03-30T00:52:14.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:14 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:14.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:14 smithi116 ceph-mon[38086]: from='client.? 172.21.15.181:0/1278833984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:52:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:14 smithi181 ceph-mon[47622]: pgmap v51: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:14 smithi181 ceph-mon[47622]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T00:52:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:14 smithi181 ceph-mon[47622]: from='client.? 172.21.15.181:0/1471020707' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7679f78-2053-4395-952b-efbdabfc12fd"}]: dispatch 2024-03-30T00:52:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:14 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7679f78-2053-4395-952b-efbdabfc12fd"}]: dispatch 2024-03-30T00:52:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:14 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7679f78-2053-4395-952b-efbdabfc12fd"}]': finished 2024-03-30T00:52:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:14 smithi181 ceph-mon[47622]: osdmap e14: 3 total, 2 up, 3 in 2024-03-30T00:52:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:14 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:14 smithi181 ceph-mon[47622]: from='client.? 172.21.15.181:0/1278833984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:52:14.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:14 smithi130 ceph-mon[47626]: pgmap v51: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-03-30T00:52:14.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:14 smithi130 ceph-mon[47626]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T00:52:14.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:14 smithi130 ceph-mon[47626]: from='client.? 172.21.15.181:0/1471020707' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7679f78-2053-4395-952b-efbdabfc12fd"}]: dispatch 2024-03-30T00:52:14.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:14 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7679f78-2053-4395-952b-efbdabfc12fd"}]: dispatch 2024-03-30T00:52:14.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:14 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7679f78-2053-4395-952b-efbdabfc12fd"}]': finished 2024-03-30T00:52:14.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:14 smithi130 ceph-mon[47626]: osdmap e14: 3 total, 2 up, 3 in 2024-03-30T00:52:14.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:14.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:14 smithi130 ceph-mon[47626]: from='client.? 172.21.15.181:0/1278833984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:52:16.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:16 smithi116 ceph-mon[38086]: pgmap v54: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:16.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:16 smithi181 ceph-mon[47622]: pgmap v54: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:16.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:16 smithi130 ceph-mon[47626]: pgmap v54: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:18.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:18 smithi116 ceph-mon[38086]: pgmap v55: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:18.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:18 smithi181 ceph-mon[47622]: pgmap v55: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:18.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:18 smithi130 ceph-mon[47626]: pgmap v55: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:19.412 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:19 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T00:52:19.412 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:19 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:19.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:19 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T00:52:19.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:19 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:19.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:19 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T00:52:19.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:19 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:20.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:20 smithi181 ceph-mon[47622]: Deploying daemon osd.2 on smithi181 2024-03-30T00:52:20.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:20 smithi181 ceph-mon[47622]: pgmap v56: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:20.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:20 smithi116 ceph-mon[38086]: Deploying daemon osd.2 on smithi181 2024-03-30T00:52:20.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:20 smithi116 ceph-mon[38086]: pgmap v56: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:20.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:20 smithi130 ceph-mon[47626]: Deploying daemon osd.2 on smithi181 2024-03-30T00:52:20.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:20 smithi130 ceph-mon[47626]: pgmap v56: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:22.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:22 smithi116 ceph-mon[38086]: pgmap v57: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:22.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:22 smithi181 ceph-mon[47622]: pgmap v57: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:22.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:22 smithi130 ceph-mon[47626]: pgmap v57: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:23.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:23.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:23.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:23.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:23.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:23.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:23.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:23.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:23.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:23.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:23.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:23.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:23.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:23.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:23.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:23.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:23.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:23.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:23.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:23.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:23.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:23.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:23.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:23.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:24.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:24 smithi116 ceph-mon[38086]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:24.712 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:24.705+0000 7ff58cff9700 1 -- 172.21.15.181:0/1332400573 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff5900fd3f0 con 0x7ff5780a1190 2024-03-30T00:52:24.712 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 2 on host 'smithi181' 2024-03-30T00:52:24.715 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:24.708+0000 7ff595c35700 1 -- 172.21.15.181:0/1332400573 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff5780a1190 msgr2=0x7ff5780a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:24.715 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:24.708+0000 7ff595c35700 1 --2- 172.21.15.181:0/1332400573 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff5780a1190 0x7ff5780a3650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff590100d50 tx=0x7ff5800096c0 comp rx=0 tx=0).stop 2024-03-30T00:52:24.715 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:24.708+0000 7ff595c35700 1 -- 172.21.15.181:0/1332400573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff590100ef0 msgr2=0x7ff5901022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:24.715 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:24.708+0000 7ff595c35700 1 --2- 172.21.15.181:0/1332400573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff590100ef0 0x7ff5901022d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff58400c9b0 tx=0x7ff58400cd70 comp rx=0 tx=0).stop 2024-03-30T00:52:24.715 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:24.708+0000 7ff595c35700 1 -- 172.21.15.181:0/1332400573 shutdown_connections 2024-03-30T00:52:24.715 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:24.708+0000 7ff595c35700 1 --2- 172.21.15.181:0/1332400573 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff5780a1190 0x7ff5780a3650 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:24.715 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:24.708+0000 7ff595c35700 1 --2- 172.21.15.181:0/1332400573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff590101890 0x7ff590102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:24.715 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:24.708+0000 7ff595c35700 1 --2- 172.21.15.181:0/1332400573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5900ffcf0 0x7ff590103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:24.715 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:24.708+0000 7ff595c35700 1 --2- 172.21.15.181:0/1332400573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff590100ef0 0x7ff5901022d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:24.715 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:24.708+0000 7ff595c35700 1 -- 172.21.15.181:0/1332400573 >> 172.21.15.181:0/1332400573 conn(0x7ff5900fb2b0 msgr2=0x7ff5900fcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:52:24.715 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:24.708+0000 7ff595c35700 1 -- 172.21.15.181:0/1332400573 shutdown_connections 2024-03-30T00:52:24.715 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:52:24.708+0000 7ff595c35700 1 -- 172.21.15.181:0/1332400573 wait complete. 2024-03-30T00:52:24.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:24 smithi181 ceph-mon[47622]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:24 smithi130 ceph-mon[47626]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:25.143 DEBUG:teuthology.orchestra.run.smithi181:osd.2> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.2.service 2024-03-30T00:52:25.145 INFO:tasks.cephadm:Deploying osd.3 on smithi116 with /dev/vg_nvme/lv_3... 2024-03-30T00:52:25.146 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T00:52:25.175 INFO:journalctl@ceph.osd.2.smithi181.stdout:-- Logs begin at Sat 2024-03-30 00:27:51 UTC. -- 2024-03-30T00:52:25.508 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:52:25.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:25.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:25.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:25.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:25 smithi116 ceph-mon[38086]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:25.955 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:25.955 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:25.955 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:25.956 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:25 smithi181 ceph-mon[47622]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:26.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:25 smithi130 ceph-mon[47626]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:26.673 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 00:52:26 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-2[53393]: 2024-03-30T00:52:26.359+0000 7f2f8d50b640 -1 osd.2 0 log_to_monitors true 2024-03-30T00:52:26.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:26 smithi116 ceph-mon[38086]: from='osd.2 [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T00:52:26.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:26 smithi116 ceph-mon[38086]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T00:52:27.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:26 smithi181 ceph-mon[47622]: from='osd.2 [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T00:52:27.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:26 smithi181 ceph-mon[47622]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T00:52:27.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:26 smithi130 ceph-mon[47626]: from='osd.2 [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T00:52:27.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:26 smithi130 ceph-mon[47626]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T00:52:27.131 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0430987 s, 243 MB/s 2024-03-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-30T00:52:27.528 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi116:vg_nvme/lv_3 2024-03-30T00:52:27.879 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:52:27.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T00:52:27.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T00:52:27.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:27.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: from='osd.2 [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:52:27.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:52:27.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:27.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: Detected new or changed devices on smithi181 2024-03-30T00:52:27.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:27.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:27.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:52:27.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi181 to 17113M 2024-03-30T00:52:27.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:27.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:27.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:27.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:27 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:28.002 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-2[53393]: 2024-03-30T00:52:27.684+0000 7f2f82b6f700 -1 osd.2 0 waiting for initial osdmap 2024-03-30T00:52:28.002 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-2[53393]: 2024-03-30T00:52:27.688+0000 7f2f7e964700 -1 osd.2 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T00:52:28.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T00:52:28.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T00:52:28.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:28.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: from='osd.2 [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:52:28.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:52:28.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:28.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: Detected new or changed devices on smithi181 2024-03-30T00:52:28.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:28.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:28.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:52:28.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi181 to 17113M 2024-03-30T00:52:28.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:28.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:28.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:28.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:27 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:28.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T00:52:28.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T00:52:28.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:28.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: from='osd.2 [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:52:28.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:52:28.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:28.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: Detected new or changed devices on smithi181 2024-03-30T00:52:28.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:28.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:28.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:52:28.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi181 to 17113M 2024-03-30T00:52:28.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:28.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:28.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:28.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:27 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:28.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.936+0000 7f4521a00700 1 -- 172.21.15.116:0/4107494592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f451c106ae0 msgr2=0x7f451c108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:28.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.936+0000 7f4521a00700 1 --2- 172.21.15.116:0/4107494592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f451c106ae0 0x7f451c108f50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f451400b0a0 tx=0x7f451400b3b0 comp rx=0 tx=0).stop 2024-03-30T00:52:28.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.936+0000 7f4521a00700 1 -- 172.21.15.116:0/4107494592 shutdown_connections 2024-03-30T00:52:28.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.936+0000 7f4521a00700 1 --2- 172.21.15.116:0/4107494592 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f451c101880 0x7f451c103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:28.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.936+0000 7f4521a00700 1 --2- 172.21.15.116:0/4107494592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f451c106ae0 0x7f451c108f50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:28.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.936+0000 7f4521a00700 1 --2- 172.21.15.116:0/4107494592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f451c1041b0 0x7f451c1065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:28.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.936+0000 7f4521a00700 1 -- 172.21.15.116:0/4107494592 >> 172.21.15.116:0/4107494592 conn(0x7f451c0fb2b0 msgr2=0x7f451c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:52:28.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.936+0000 7f4521a00700 1 -- 172.21.15.116:0/4107494592 shutdown_connections 2024-03-30T00:52:28.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.936+0000 7f4521a00700 1 -- 172.21.15.116:0/4107494592 wait complete. 2024-03-30T00:52:28.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.937+0000 7f4521a00700 1 Processor -- start 2024-03-30T00:52:28.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.937+0000 7f4521a00700 1 -- start start 2024-03-30T00:52:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f4521a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f451c101880 0x7f451c199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:52:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f4521a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f451c1041b0 0x7f451c1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:52:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f4521a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f451c106ae0 0x7f451c1a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:52:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f4521a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451c1a0ba0 con 0x7f451c101880 2024-03-30T00:52:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f4521a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451c1a0ce0 con 0x7f451c1041b0 2024-03-30T00:52:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f4521a00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451c1a0e70 con 0x7f451c106ae0 2024-03-30T00:52:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f451aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f451c101880 0x7f451c199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:52:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f451aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f451c101880 0x7f451c199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32892/0 (socket says 172.21.15.116:32892) 2024-03-30T00:52:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f451aa00700 1 -- 172.21.15.116:0/1197595741 learned_addr learned my addr 172.21.15.116:0/1197595741 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:52:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f451a000700 1 --2- 172.21.15.116:0/1197595741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f451c1041b0 0x7f451c1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:52:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f451b400700 1 --2- 172.21.15.116:0/1197595741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f451c106ae0 0x7f451c1a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:52:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f451aa00700 1 -- 172.21.15.116:0/1197595741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f451c106ae0 msgr2=0x7f451c1a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f451aa00700 1 --2- 172.21.15.116:0/1197595741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f451c106ae0 0x7f451c1a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:28.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f451aa00700 1 -- 172.21.15.116:0/1197595741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f451c1041b0 msgr2=0x7f451c1995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:28.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f451aa00700 1 --2- 172.21.15.116:0/1197595741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f451c1041b0 0x7f451c1995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:28.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.938+0000 7f451aa00700 1 -- 172.21.15.116:0/1197595741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4514009d00 con 0x7f451c101880 2024-03-30T00:52:28.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.939+0000 7f451aa00700 1 --2- 172.21.15.116:0/1197595741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f451c101880 0x7f451c199060 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f450c00e9d0 tx=0x7f450c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:52:28.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.939+0000 7f4513400700 1 -- 172.21.15.116:0/1197595741 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c009790 con 0x7f451c101880 2024-03-30T00:52:28.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.939+0000 7f4513400700 1 -- 172.21.15.116:0/1197595741 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f450c00ee70 con 0x7f451c101880 2024-03-30T00:52:28.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.939+0000 7f4513400700 1 -- 172.21.15.116:0/1197595741 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c010ea0 con 0x7f451c101880 2024-03-30T00:52:28.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.939+0000 7f4521a00700 1 -- 172.21.15.116:0/1197595741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451c1a1150 con 0x7f451c101880 2024-03-30T00:52:28.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.939+0000 7f4521a00700 1 -- 172.21.15.116:0/1197595741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f451c1a1600 con 0x7f451c101880 2024-03-30T00:52:28.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.940+0000 7f4521a00700 1 -- 172.21.15.116:0/1197595741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f451c108660 con 0x7f451c101880 2024-03-30T00:52:28.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.942+0000 7f4513400700 1 -- 172.21.15.116:0/1197595741 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 15) v1 ==== 135810+0+0 (secure 0 0 0) 0x7f450c00cbd0 con 0x7f451c101880 2024-03-30T00:52:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.944+0000 7f4513400700 1 --2- 172.21.15.116:0/1197595741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f45000a1110 0x7f45000a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:52:28.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.944+0000 7f4513400700 1 -- 172.21.15.116:0/1197595741 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 2851+0+0 (secure 0 0 0) 0x7f450c014070 con 0x7f451c101880 2024-03-30T00:52:28.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.944+0000 7f451a000700 1 --2- 172.21.15.116:0/1197595741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f45000a1110 0x7f45000a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:52:28.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.945+0000 7f451a000700 1 --2- 172.21.15.116:0/1197595741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f45000a1110 0x7f45000a35d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4508009c60 tx=0x7f4508009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:52:28.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:28.948+0000 7f4513400700 1 -- 172.21.15.116:0/1197595741 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f450c0529f0 con 0x7f451c101880 2024-03-30T00:52:29.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:28 smithi181 ceph-mon[47622]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-30T00:52:29.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:28 smithi181 ceph-mon[47622]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T00:52:29.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:28 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:29.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:28 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:29.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:28 smithi130 ceph-mon[47626]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-30T00:52:29.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:28 smithi130 ceph-mon[47626]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T00:52:29.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:28 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:29.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:28 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:29.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:29.161+0000 7f4521a00700 1 -- 172.21.15.116:0/1197595741 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f451c02cd20 con 0x7f45000a1110 2024-03-30T00:52:29.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:28 smithi116 ceph-mon[38086]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-30T00:52:29.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:28 smithi116 ceph-mon[38086]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T00:52:29.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:28 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:29.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:28 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:30.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:29 smithi181 ceph-mon[47622]: purged_snaps scrub starts 2024-03-30T00:52:30.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:29 smithi181 ceph-mon[47622]: purged_snaps scrub ok 2024-03-30T00:52:30.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:29 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:30.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:29 smithi181 ceph-mon[47622]: osd.2 [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060] boot 2024-03-30T00:52:30.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:29 smithi181 ceph-mon[47622]: osdmap e17: 3 total, 3 up, 3 in 2024-03-30T00:52:30.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:29 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:30.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:29 smithi181 ceph-mon[47622]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:30.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:29 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T00:52:30.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:29 smithi181 ceph-mon[47622]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:52:30.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:29 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:52:30.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:29 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:52:30.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:29 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:30.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:29 smithi130 ceph-mon[47626]: purged_snaps scrub starts 2024-03-30T00:52:30.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:29 smithi130 ceph-mon[47626]: purged_snaps scrub ok 2024-03-30T00:52:30.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:29 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:30.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:29 smithi130 ceph-mon[47626]: osd.2 [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060] boot 2024-03-30T00:52:30.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:29 smithi130 ceph-mon[47626]: osdmap e17: 3 total, 3 up, 3 in 2024-03-30T00:52:30.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:29 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:30.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:29 smithi130 ceph-mon[47626]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:30.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:29 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T00:52:30.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:29 smithi130 ceph-mon[47626]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:52:30.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:29 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:52:30.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:29 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:52:30.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:29 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:30.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:29 smithi116 ceph-mon[38086]: purged_snaps scrub starts 2024-03-30T00:52:30.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:29 smithi116 ceph-mon[38086]: purged_snaps scrub ok 2024-03-30T00:52:30.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:29 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:30.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:29 smithi116 ceph-mon[38086]: osd.2 [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060] boot 2024-03-30T00:52:30.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:29 smithi116 ceph-mon[38086]: osdmap e17: 3 total, 3 up, 3 in 2024-03-30T00:52:30.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:29 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T00:52:30.158 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:29 smithi116 ceph-mon[38086]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T00:52:30.159 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:29 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T00:52:30.159 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:29 smithi116 ceph-mon[38086]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:52:30.159 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:29 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:52:30.159 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:29 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:52:30.159 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:29 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:30.871 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:30 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T00:52:30.871 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:30 smithi116 ceph-mon[38086]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T00:52:30.871 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:30 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T00:52:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:30 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T00:52:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:30 smithi181 ceph-mon[47622]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T00:52:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:30 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T00:52:31.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:30 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T00:52:31.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:30 smithi130 ceph-mon[47626]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T00:52:31.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:30 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T00:52:31.674 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 00:52:31 smithi116 sudo[61379]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T00:52:31.674 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 00:52:31 smithi116 sudo[61379]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T00:52:31.674 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 00:52:31 smithi116 sudo[61379]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T00:52:31.674 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 00:52:31 smithi116 sudo[61379]: pam_unix(sudo:session): session closed for user root 2024-03-30T00:52:31.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:31.732+0000 7f4513400700 1 -- 172.21.15.116:0/1197595741 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f450c096540 con 0x7f451c101880 2024-03-30T00:52:32.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T00:52:32.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:31 smithi181 ceph-mon[47622]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T00:52:32.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:31 smithi181 ceph-mon[47622]: pgmap v67: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:32.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T00:52:32.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:31 smithi130 ceph-mon[47626]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T00:52:32.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:31 smithi130 ceph-mon[47626]: pgmap v67: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:32.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T00:52:32.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:31 smithi116 ceph-mon[38086]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T00:52:32.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:31 smithi116 ceph-mon[38086]: pgmap v67: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:32.174 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 00:52:31 smithi116 sudo[61415]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T00:52:32.175 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 00:52:31 smithi116 sudo[61415]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T00:52:32.175 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 00:52:31 smithi116 sudo[61415]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T00:52:32.175 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 00:52:31 smithi116 sudo[61415]: pam_unix(sudo:session): session closed for user root 2024-03-30T00:52:32.595 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 00:52:32 smithi130 sudo[58081]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T00:52:32.595 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 00:52:32 smithi130 sudo[58081]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T00:52:32.595 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 00:52:32 smithi130 sudo[58081]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T00:52:32.595 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 00:52:32 smithi130 sudo[58081]: pam_unix(sudo:session): session closed for user root 2024-03-30T00:52:32.595 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 00:52:32 smithi130 sudo[58084]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T00:52:32.595 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 00:52:32 smithi130 sudo[58084]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T00:52:32.595 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 00:52:32 smithi130 sudo[58084]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T00:52:32.595 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 00:52:32 smithi130 sudo[58084]: pam_unix(sudo:session): session closed for user root 2024-03-30T00:52:32.977 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:32 smithi181 ceph-mon[47622]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T00:52:32.977 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:32 smithi181 ceph-mon[47622]: mgrmap e16: x(active, since 116s), standbys: z, y 2024-03-30T00:52:32.977 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:32 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1864713757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec0571fc-3fb7-436a-82e6-7544e2491baf"}]: dispatch 2024-03-30T00:52:32.977 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:32 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec0571fc-3fb7-436a-82e6-7544e2491baf"}]: dispatch 2024-03-30T00:52:32.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:32 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec0571fc-3fb7-436a-82e6-7544e2491baf"}]': finished 2024-03-30T00:52:32.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:32 smithi181 ceph-mon[47622]: osdmap e21: 4 total, 3 up, 4 in 2024-03-30T00:52:32.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:32.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:32.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:32.978 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 00:52:32 smithi181 sudo[58652]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T00:52:32.978 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 00:52:32 smithi181 sudo[58652]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T00:52:32.978 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 00:52:32 smithi181 sudo[58652]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T00:52:32.978 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 00:52:32 smithi181 sudo[58652]: pam_unix(sudo:session): session closed for user root 2024-03-30T00:52:32.978 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 00:52:32 smithi181 sudo[58655]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T00:52:32.978 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 00:52:32 smithi181 sudo[58655]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T00:52:32.979 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 00:52:32 smithi181 sudo[58655]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T00:52:32.979 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 00:52:32 smithi181 sudo[58655]: pam_unix(sudo:session): session closed for user root 2024-03-30T00:52:33.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:32 smithi130 ceph-mon[47626]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T00:52:33.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:32 smithi130 ceph-mon[47626]: mgrmap e16: x(active, since 116s), standbys: z, y 2024-03-30T00:52:33.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:32 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1864713757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec0571fc-3fb7-436a-82e6-7544e2491baf"}]: dispatch 2024-03-30T00:52:33.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:32 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec0571fc-3fb7-436a-82e6-7544e2491baf"}]: dispatch 2024-03-30T00:52:33.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:32 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec0571fc-3fb7-436a-82e6-7544e2491baf"}]': finished 2024-03-30T00:52:33.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:32 smithi130 ceph-mon[47626]: osdmap e21: 4 total, 3 up, 4 in 2024-03-30T00:52:33.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:33.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:33.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:33.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:32 smithi116 ceph-mon[38086]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T00:52:33.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:32 smithi116 ceph-mon[38086]: mgrmap e16: x(active, since 116s), standbys: z, y 2024-03-30T00:52:33.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:32 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1864713757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec0571fc-3fb7-436a-82e6-7544e2491baf"}]: dispatch 2024-03-30T00:52:33.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:32 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec0571fc-3fb7-436a-82e6-7544e2491baf"}]: dispatch 2024-03-30T00:52:33.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:32 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec0571fc-3fb7-436a-82e6-7544e2491baf"}]': finished 2024-03-30T00:52:33.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:32 smithi116 ceph-mon[38086]: osdmap e21: 4 total, 3 up, 4 in 2024-03-30T00:52:33.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:33.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:33.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:33.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:33 smithi116 sudo[61690]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T00:52:33.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:33 smithi116 sudo[61690]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T00:52:33.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:33 smithi116 sudo[61690]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T00:52:33.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:33 smithi116 sudo[61690]: pam_unix(sudo:session): session closed for user root 2024-03-30T00:52:34.336 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:34 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2263338353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:52:34.336 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:34.337 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:34 smithi181 ceph-mon[47622]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T00:52:34.337 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:34 smithi181 sudo[58658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T00:52:34.337 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:34 smithi181 sudo[58658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T00:52:34.337 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:34 smithi181 sudo[58658]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T00:52:34.337 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:34 smithi181 sudo[58658]: pam_unix(sudo:session): session closed for user root 2024-03-30T00:52:34.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:34 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2263338353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:52:34.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:34.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:34 smithi130 ceph-mon[47626]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T00:52:34.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:34 smithi116 sudo[61882]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T00:52:34.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:34 smithi116 sudo[61882]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T00:52:34.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:34 smithi116 sudo[61882]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T00:52:34.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:34 smithi116 sudo[61882]: pam_unix(sudo:session): session closed for user root 2024-03-30T00:52:34.427 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:34 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2263338353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:52:34.427 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:34.427 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:34 smithi116 ceph-mon[38086]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T00:52:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:34 smithi181 sudo[58661]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T00:52:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:34 smithi181 sudo[58661]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T00:52:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:34 smithi181 sudo[58661]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T00:52:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:34 smithi181 sudo[58661]: pam_unix(sudo:session): session closed for user root 2024-03-30T00:52:35.090 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:34 smithi130 sudo[58087]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T00:52:35.090 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:34 smithi130 sudo[58087]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T00:52:35.091 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:34 smithi130 sudo[58087]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T00:52:35.091 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:34 smithi130 sudo[58087]: pam_unix(sudo:session): session closed for user root 2024-03-30T00:52:35.091 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 sudo[58090]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T00:52:35.091 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 sudo[58090]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T00:52:35.091 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 sudo[58090]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T00:52:35.091 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 sudo[58090]: pam_unix(sudo:session): session closed for user root 2024-03-30T00:52:35.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:35.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T00:52:35.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:52:35.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:52:35.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:52:35.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T00:52:35.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:52:35.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:52:35.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:52:35.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T00:52:35.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T00:52:35.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:52:35.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:52:35.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:52:35.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:35 smithi130 ceph-mon[47626]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T00:52:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T00:52:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:52:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:52:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:52:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T00:52:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:52:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:52:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:52:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T00:52:35.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T00:52:35.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:52:35.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:52:35.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:52:35.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:35 smithi116 ceph-mon[38086]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T00:52:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T00:52:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:52:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:52:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:52:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T00:52:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:52:35.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:52:35.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:52:35.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T00:52:35.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T00:52:35.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T00:52:35.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T00:52:35.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T00:52:35.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:35 smithi181 ceph-mon[47622]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T00:52:36.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:36 smithi116 ceph-mon[38086]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:36.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:36 smithi130 ceph-mon[47626]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:36.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:36 smithi181 ceph-mon[47622]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:38.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:38 smithi130 ceph-mon[47626]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:38.346 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:38 smithi116 ceph-mon[38086]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:38.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:38 smithi181 ceph-mon[47622]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:39.350 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:39 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T00:52:39.351 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:39 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:39.351 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:39 smithi116 ceph-mon[38086]: Deploying daemon osd.3 on smithi116 2024-03-30T00:52:39.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:39 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T00:52:39.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:39 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:39.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:39 smithi181 ceph-mon[47622]: Deploying daemon osd.3 on smithi116 2024-03-30T00:52:39.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:39 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T00:52:39.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:39 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:39.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:39 smithi130 ceph-mon[47626]: Deploying daemon osd.3 on smithi116 2024-03-30T00:52:40.361 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:40 smithi116 ceph-mon[38086]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-30T00:52:40.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:40 smithi181 ceph-mon[47622]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-30T00:52:40.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:40 smithi130 ceph-mon[47626]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-30T00:52:42.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:42 smithi116 ceph-mon[38086]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-30T00:52:42.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:42 smithi181 ceph-mon[47622]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-30T00:52:42.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:42 smithi130 ceph-mon[47626]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-30T00:52:44.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:44 smithi116 ceph-mon[38086]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-30T00:52:44.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:44 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:44.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:44 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:44.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:44 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:44.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:44 smithi130 ceph-mon[47626]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-30T00:52:44.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:44 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:44.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:44 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:44.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:44 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:44.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:44 smithi181 ceph-mon[47622]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-30T00:52:44.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:44 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:44.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:44 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:44.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:44 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:46.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:45.995+0000 7f4513400700 1 -- 172.21.15.116:0/1197595741 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f451c02cd20 con 0x7f45000a1110 2024-03-30T00:52:46.002 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 3 on host 'smithi116' 2024-03-30T00:52:46.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:45.998+0000 7f4521a00700 1 -- 172.21.15.116:0/1197595741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f45000a1110 msgr2=0x7f45000a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:46.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:45.998+0000 7f4521a00700 1 --2- 172.21.15.116:0/1197595741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f45000a1110 0x7f45000a35d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4508009c60 tx=0x7f4508009260 comp rx=0 tx=0).stop 2024-03-30T00:52:46.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:45.998+0000 7f4521a00700 1 -- 172.21.15.116:0/1197595741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f451c101880 msgr2=0x7f451c199060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:46.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:45.998+0000 7f4521a00700 1 --2- 172.21.15.116:0/1197595741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f451c101880 0x7f451c199060 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f450c00e9d0 tx=0x7f450c00ece0 comp rx=0 tx=0).stop 2024-03-30T00:52:46.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:45.999+0000 7f4521a00700 1 -- 172.21.15.116:0/1197595741 shutdown_connections 2024-03-30T00:52:46.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:45.999+0000 7f4521a00700 1 --2- 172.21.15.116:0/1197595741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f45000a1110 0x7f45000a35d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:46.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:45.999+0000 7f4521a00700 1 --2- 172.21.15.116:0/1197595741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f451c106ae0 0x7f451c1a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:46.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:45.999+0000 7f4521a00700 1 --2- 172.21.15.116:0/1197595741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f451c101880 0x7f451c199060 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:46.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:45.999+0000 7f4521a00700 1 --2- 172.21.15.116:0/1197595741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f451c1041b0 0x7f451c1995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:46.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:45.999+0000 7f4521a00700 1 -- 172.21.15.116:0/1197595741 >> 172.21.15.116:0/1197595741 conn(0x7f451c0fb2b0 msgr2=0x7f451c101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:52:46.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:45.999+0000 7f4521a00700 1 -- 172.21.15.116:0/1197595741 shutdown_connections 2024-03-30T00:52:46.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:52:45.999+0000 7f4521a00700 1 -- 172.21.15.116:0/1197595741 wait complete. 2024-03-30T00:52:46.197 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:46 smithi116 ceph-mon[38086]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-30T00:52:46.198 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.198 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.198 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:46.198 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:46.198 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.198 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.198 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.198 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:46.358 DEBUG:teuthology.orchestra.run.smithi116:osd.3> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.3.service 2024-03-30T00:52:46.362 INFO:tasks.cephadm:Deploying osd.4 on smithi130 with /dev/vg_nvme/lv_3... 2024-03-30T00:52:46.362 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T00:52:46.395 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:46 smithi130 ceph-mon[47626]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-30T00:52:46.395 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.395 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.395 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:46.395 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:46.396 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.396 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.396 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.396 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:46.432 INFO:journalctl@ceph.osd.3.smithi116.stdout:-- Logs begin at Sat 2024-03-30 00:33:55 UTC. -- 2024-03-30T00:52:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:46 smithi181 ceph-mon[47622]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-30T00:52:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:46.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:52:46.594 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T00:52:47.674 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 30 00:52:47 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-3[63681]: 2024-03-30T00:52:47.352+0000 7fe1cfdc6640 -1 osd.3 0 log_to_monitors true 2024-03-30T00:52:47.969 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T00:52:47.970 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T00:52:47.970 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-03-30T00:52:47.970 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-03-30T00:52:47.970 INFO:teuthology.orchestra.run.smithi130.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123197 s, 851 MB/s 2024-03-30T00:52:47.970 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-03-30T00:52:48.293 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:48 smithi130 ceph-mon[47626]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-30T00:52:48.294 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:48 smithi130 ceph-mon[47626]: from='osd.3 [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T00:52:48.294 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:48 smithi130 ceph-mon[47626]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T00:52:48.395 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi130:vg_nvme/lv_3 2024-03-30T00:52:48.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:48 smithi116 ceph-mon[38086]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-30T00:52:48.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:48 smithi116 ceph-mon[38086]: from='osd.3 [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T00:52:48.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:48 smithi116 ceph-mon[38086]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T00:52:48.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:48 smithi181 ceph-mon[47622]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-30T00:52:48.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:48 smithi181 ceph-mon[47622]: from='osd.3 [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T00:52:48.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:48 smithi181 ceph-mon[47622]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T00:52:48.650 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T00:52:49.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:49 smithi130 ceph-mon[47626]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T00:52:49.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:49 smithi130 ceph-mon[47626]: osdmap e22: 4 total, 3 up, 4 in 2024-03-30T00:52:49.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:49.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:49 smithi130 ceph-mon[47626]: from='osd.3 [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:52:49.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:49 smithi130 ceph-mon[47626]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:52:49.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:49.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:49.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:52:49.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:52:49.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:49.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:49.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:49.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:49.424 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-3[63681]: 2024-03-30T00:52:49.142+0000 7fe1c4400700 -1 osd.3 0 waiting for initial osdmap 2024-03-30T00:52:49.424 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-3[63681]: 2024-03-30T00:52:49.148+0000 7fe1bf200700 -1 osd.3 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T00:52:49.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-mon[38086]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T00:52:49.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-mon[38086]: osdmap e22: 4 total, 3 up, 4 in 2024-03-30T00:52:49.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:49.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-mon[38086]: from='osd.3 [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:52:49.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-mon[38086]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:52:49.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:49.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:49.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:52:49.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:52:49.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:49.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:49.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:49.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:49 smithi181 ceph-mon[47622]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T00:52:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:49 smithi181 ceph-mon[47622]: osdmap e22: 4 total, 3 up, 4 in 2024-03-30T00:52:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:49 smithi181 ceph-mon[47622]: from='osd.3 [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:52:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:49 smithi181 ceph-mon[47622]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:52:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:49.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:49.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:52:49.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:52:49.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:49.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:49.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:52:49.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:52:49.554 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.547+0000 7fde7332d700 1 -- 172.21.15.130:0/1929792810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6c101890 msgr2=0x7fde6c108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:49.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.547+0000 7fde7332d700 1 --2- 172.21.15.130:0/1929792810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6c101890 0x7fde6c108140 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fde6400b0d0 tx=0x7fde6400b3e0 comp rx=0 tx=0).stop 2024-03-30T00:52:49.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.547+0000 7fde7332d700 1 -- 172.21.15.130:0/1929792810 shutdown_connections 2024-03-30T00:52:49.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.547+0000 7fde7332d700 1 --2- 172.21.15.130:0/1929792810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6c101890 0x7fde6c108140 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:49.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.547+0000 7fde7332d700 1 --2- 172.21.15.130:0/1929792810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde6c0ffcf0 0x7fde6c100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:49.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.547+0000 7fde7332d700 1 --2- 172.21.15.130:0/1929792810 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde6c100ef0 0x7fde6c101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:49.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.547+0000 7fde7332d700 1 -- 172.21.15.130:0/1929792810 >> 172.21.15.130:0/1929792810 conn(0x7fde6c0fb2b0 msgr2=0x7fde6c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:52:49.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.547+0000 7fde7332d700 1 -- 172.21.15.130:0/1929792810 shutdown_connections 2024-03-30T00:52:49.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.547+0000 7fde7332d700 1 -- 172.21.15.130:0/1929792810 wait complete. 2024-03-30T00:52:49.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde7332d700 1 Processor -- start 2024-03-30T00:52:49.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde7332d700 1 -- start start 2024-03-30T00:52:49.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde7332d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde6c0ffcf0 0x7fde6c103c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:52:49.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde7332d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde6c100ef0 0x7fde6c1022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:52:49.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde7332d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6c101890 0x7fde6c102810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:52:49.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde7332d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c102fb0 con 0x7fde6c0ffcf0 2024-03-30T00:52:49.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde7332d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c103120 con 0x7fde6c101890 2024-03-30T00:52:49.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde7332d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c1a0e60 con 0x7fde6c100ef0 2024-03-30T00:52:49.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde718ca700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6c101890 0x7fde6c102810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:52:49.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde718ca700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6c101890 0x7fde6c102810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:49228/0 (socket says 172.21.15.130:49228) 2024-03-30T00:52:49.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde718ca700 1 -- 172.21.15.130:0/2607266809 learned_addr learned my addr 172.21.15.130:0/2607266809 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-03-30T00:52:49.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde708c8700 1 --2- 172.21.15.130:0/2607266809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde6c100ef0 0x7fde6c1022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:52:49.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde710c9700 1 --2- 172.21.15.130:0/2607266809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde6c0ffcf0 0x7fde6c103c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:52:49.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde718ca700 1 -- 172.21.15.130:0/2607266809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde6c100ef0 msgr2=0x7fde6c1022d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:49.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde718ca700 1 --2- 172.21.15.130:0/2607266809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde6c100ef0 0x7fde6c1022d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:49.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.548+0000 7fde718ca700 1 -- 172.21.15.130:0/2607266809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde6c0ffcf0 msgr2=0x7fde6c103c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:52:49.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.549+0000 7fde718ca700 1 --2- 172.21.15.130:0/2607266809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde6c0ffcf0 0x7fde6c103c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:52:49.557 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.549+0000 7fde718ca700 1 -- 172.21.15.130:0/2607266809 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde64009d00 con 0x7fde6c101890 2024-03-30T00:52:49.557 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.549+0000 7fde718ca700 1 --2- 172.21.15.130:0/2607266809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6c101890 0x7fde6c102810 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fde64007b60 tx=0x7fde64008ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:52:49.557 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.549+0000 7fde627fc700 1 -- 172.21.15.130:0/2607266809 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6400e040 con 0x7fde6c101890 2024-03-30T00:52:49.557 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.549+0000 7fde627fc700 1 -- 172.21.15.130:0/2607266809 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde6401d930 con 0x7fde6c101890 2024-03-30T00:52:49.557 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.549+0000 7fde7332d700 1 -- 172.21.15.130:0/2607266809 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6c1a1090 con 0x7fde6c101890 2024-03-30T00:52:49.557 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.549+0000 7fde627fc700 1 -- 172.21.15.130:0/2607266809 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6401ba00 con 0x7fde6c101890 2024-03-30T00:52:49.557 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.549+0000 7fde7332d700 1 -- 172.21.15.130:0/2607266809 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6c1a1500 con 0x7fde6c101890 2024-03-30T00:52:49.558 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.550+0000 7fde7332d700 1 -- 172.21.15.130:0/2607266809 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6c04ea80 con 0x7fde6c101890 2024-03-30T00:52:49.558 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.551+0000 7fde627fc700 1 -- 172.21.15.130:0/2607266809 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fde6401daa0 con 0x7fde6c101890 2024-03-30T00:52:49.558 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.551+0000 7fde627fc700 1 --2- 172.21.15.130:0/2607266809 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fde580a1120 0x7fde580a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:52:49.558 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.551+0000 7fde627fc700 1 -- 172.21.15.130:0/2607266809 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 3389+0+0 (secure 0 0 0) 0x7fde640cc9d0 con 0x7fde6c101890 2024-03-30T00:52:49.558 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.552+0000 7fde710c9700 1 --2- 172.21.15.130:0/2607266809 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fde580a1120 0x7fde580a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:52:49.558 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.552+0000 7fde710c9700 1 --2- 172.21.15.130:0/2607266809 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fde580a1120 0x7fde580a35e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fde6c100d50 tx=0x7fde5c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:52:49.561 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.556+0000 7fde627fc700 1 -- 172.21.15.130:0/2607266809 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fde64097530 con 0x7fde6c101890 2024-03-30T00:52:49.761 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:52:49.756+0000 7fde7332d700 1 -- 172.21.15.130:0/2607266809 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fde6c0fd490 con 0x7fde580a1120 2024-03-30T00:52:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:50 smithi116 ceph-mon[38086]: Detected new or changed devices on smithi116 2024-03-30T00:52:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:50 smithi116 ceph-mon[38086]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:50 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi116 to 8635M 2024-03-30T00:52:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:50 smithi116 ceph-mon[38086]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-30T00:52:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:50 smithi116 ceph-mon[38086]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T00:52:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:52:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:52:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:50.439 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:50 smithi130 ceph-mon[47626]: Detected new or changed devices on smithi116 2024-03-30T00:52:50.440 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:50 smithi130 ceph-mon[47626]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:50.440 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:50 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi116 to 8635M 2024-03-30T00:52:50.440 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:50 smithi130 ceph-mon[47626]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-30T00:52:50.440 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:50 smithi130 ceph-mon[47626]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T00:52:50.440 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:50.440 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:50.440 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:52:50.440 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:52:50.440 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:50 smithi181 ceph-mon[47622]: Detected new or changed devices on smithi116 2024-03-30T00:52:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:50 smithi181 ceph-mon[47622]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:50 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi116 to 8635M 2024-03-30T00:52:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:50 smithi181 ceph-mon[47622]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-30T00:52:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:50 smithi181 ceph-mon[47622]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T00:52:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:52:50.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:52:50.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:51.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:51 smithi116 ceph-mon[38086]: purged_snaps scrub starts 2024-03-30T00:52:51.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:51 smithi116 ceph-mon[38086]: purged_snaps scrub ok 2024-03-30T00:52:51.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:51 smithi116 ceph-mon[38086]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:52:51.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:51 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:51.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:51 smithi116 ceph-mon[38086]: osd.3 [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072] boot 2024-03-30T00:52:51.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:51 smithi116 ceph-mon[38086]: osdmap e24: 4 total, 4 up, 4 in 2024-03-30T00:52:51.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:51 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:51.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:51 smithi181 ceph-mon[47622]: purged_snaps scrub starts 2024-03-30T00:52:51.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:51 smithi181 ceph-mon[47622]: purged_snaps scrub ok 2024-03-30T00:52:51.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:51 smithi181 ceph-mon[47622]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:52:51.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:51 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:51.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:51 smithi181 ceph-mon[47622]: osd.3 [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072] boot 2024-03-30T00:52:51.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:51 smithi181 ceph-mon[47622]: osdmap e24: 4 total, 4 up, 4 in 2024-03-30T00:52:51.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:51 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:51 smithi130 ceph-mon[47626]: purged_snaps scrub starts 2024-03-30T00:52:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:51 smithi130 ceph-mon[47626]: purged_snaps scrub ok 2024-03-30T00:52:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:51 smithi130 ceph-mon[47626]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:52:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:51 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:51 smithi130 ceph-mon[47626]: osd.3 [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072] boot 2024-03-30T00:52:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:51 smithi130 ceph-mon[47626]: osdmap e24: 4 total, 4 up, 4 in 2024-03-30T00:52:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:51 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T00:52:52.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:52 smithi116 ceph-mon[38086]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:52.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:52 smithi116 ceph-mon[38086]: osdmap e25: 4 total, 4 up, 4 in 2024-03-30T00:52:52.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:52 smithi116 ceph-mon[38086]: from='client.? 172.21.15.130:0/1019651359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e7952d5-9d10-42ba-bea0-93dda26eff04"}]: dispatch 2024-03-30T00:52:52.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:52 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e7952d5-9d10-42ba-bea0-93dda26eff04"}]: dispatch 2024-03-30T00:52:52.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:52 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e7952d5-9d10-42ba-bea0-93dda26eff04"}]': finished 2024-03-30T00:52:52.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:52 smithi116 ceph-mon[38086]: osdmap e26: 5 total, 4 up, 5 in 2024-03-30T00:52:52.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:52 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:52:52.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:52 smithi116 ceph-mon[38086]: osdmap e27: 5 total, 4 up, 5 in 2024-03-30T00:52:52.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:52 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:52:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:52 smithi181 ceph-mon[47622]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:52 smithi181 ceph-mon[47622]: osdmap e25: 4 total, 4 up, 4 in 2024-03-30T00:52:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:52 smithi181 ceph-mon[47622]: from='client.? 172.21.15.130:0/1019651359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e7952d5-9d10-42ba-bea0-93dda26eff04"}]: dispatch 2024-03-30T00:52:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:52 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e7952d5-9d10-42ba-bea0-93dda26eff04"}]: dispatch 2024-03-30T00:52:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:52 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e7952d5-9d10-42ba-bea0-93dda26eff04"}]': finished 2024-03-30T00:52:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:52 smithi181 ceph-mon[47622]: osdmap e26: 5 total, 4 up, 5 in 2024-03-30T00:52:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:52 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:52:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:52 smithi181 ceph-mon[47622]: osdmap e27: 5 total, 4 up, 5 in 2024-03-30T00:52:52.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:52 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:52:52.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:52 smithi130 ceph-mon[47626]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T00:52:52.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:52 smithi130 ceph-mon[47626]: osdmap e25: 4 total, 4 up, 4 in 2024-03-30T00:52:52.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:52 smithi130 ceph-mon[47626]: from='client.? 172.21.15.130:0/1019651359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e7952d5-9d10-42ba-bea0-93dda26eff04"}]: dispatch 2024-03-30T00:52:52.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:52 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e7952d5-9d10-42ba-bea0-93dda26eff04"}]: dispatch 2024-03-30T00:52:52.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:52 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e7952d5-9d10-42ba-bea0-93dda26eff04"}]': finished 2024-03-30T00:52:52.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:52 smithi130 ceph-mon[47626]: osdmap e26: 5 total, 4 up, 5 in 2024-03-30T00:52:52.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:52 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:52:52.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:52 smithi130 ceph-mon[47626]: osdmap e27: 5 total, 4 up, 5 in 2024-03-30T00:52:52.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:52 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:52:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:53 smithi116 ceph-mon[38086]: from='client.? 172.21.15.130:0/1510808882' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:52:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:53 smithi116 ceph-mon[38086]: osdmap e28: 5 total, 4 up, 5 in 2024-03-30T00:52:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:52:53.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:53 smithi181 ceph-mon[47622]: from='client.? 172.21.15.130:0/1510808882' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:52:53.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:53 smithi181 ceph-mon[47622]: osdmap e28: 5 total, 4 up, 5 in 2024-03-30T00:52:53.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:52:53.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:53 smithi130 ceph-mon[47626]: from='client.? 172.21.15.130:0/1510808882' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:52:53.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:53 smithi130 ceph-mon[47626]: osdmap e28: 5 total, 4 up, 5 in 2024-03-30T00:52:53.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:52:54.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:54 smithi116 ceph-mon[38086]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-30T00:52:54.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:54 smithi181 ceph-mon[47622]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-30T00:52:54.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:54 smithi130 ceph-mon[47626]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-30T00:52:56.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:56 smithi130 ceph-mon[47626]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T00:52:56.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:56 smithi116 ceph-mon[38086]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T00:52:56.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:56 smithi181 ceph-mon[47622]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T00:52:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:58 smithi116 ceph-mon[38086]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-30T00:52:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:58 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T00:52:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:58 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:52:58 smithi116 ceph-mon[38086]: Deploying daemon osd.4 on smithi130 2024-03-30T00:52:58.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:58 smithi181 ceph-mon[47622]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-30T00:52:58.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:58 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T00:52:58.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:58 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:58.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:52:58 smithi181 ceph-mon[47622]: Deploying daemon osd.4 on smithi130 2024-03-30T00:52:58.556 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:58 smithi130 ceph-mon[47626]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-30T00:52:58.556 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:58 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T00:52:58.556 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:58 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:52:58.556 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:52:58 smithi130 ceph-mon[47626]: Deploying daemon osd.4 on smithi130 2024-03-30T00:53:00.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:00 smithi116 ceph-mon[38086]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-30T00:53:00.424 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:00 smithi130 ceph-mon[47626]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-30T00:53:00.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:00 smithi181 ceph-mon[47622]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-30T00:53:02.400 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:02 smithi130 ceph-mon[47626]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-30T00:53:02.400 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:02.401 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:02.401 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:02.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:02 smithi116 ceph-mon[38086]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-30T00:53:02.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:02.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:02.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:02.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:02 smithi181 ceph-mon[47622]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-30T00:53:02.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:02.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:02.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:03.341 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:03.342 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:03.342 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:03.342 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:03.342 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:03.417 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 4 on host 'smithi130' 2024-03-30T00:53:03.417 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:03.410+0000 7fde627fc700 1 -- 172.21.15.130:0/2607266809 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fde6c0fd490 con 0x7fde580a1120 2024-03-30T00:53:03.419 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:03.412+0000 7fde7332d700 1 -- 172.21.15.130:0/2607266809 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fde580a1120 msgr2=0x7fde580a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:03.419 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:03.412+0000 7fde7332d700 1 --2- 172.21.15.130:0/2607266809 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fde580a1120 0x7fde580a35e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fde6c100d50 tx=0x7fde5c009260 comp rx=0 tx=0).stop 2024-03-30T00:53:03.419 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:03.412+0000 7fde7332d700 1 -- 172.21.15.130:0/2607266809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6c101890 msgr2=0x7fde6c102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:03.419 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:03.412+0000 7fde7332d700 1 --2- 172.21.15.130:0/2607266809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6c101890 0x7fde6c102810 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fde64007b60 tx=0x7fde64008ec0 comp rx=0 tx=0).stop 2024-03-30T00:53:03.419 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:03.412+0000 7fde7332d700 1 -- 172.21.15.130:0/2607266809 shutdown_connections 2024-03-30T00:53:03.419 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:03.412+0000 7fde7332d700 1 --2- 172.21.15.130:0/2607266809 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fde580a1120 0x7fde580a35e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:03.419 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:03.412+0000 7fde7332d700 1 --2- 172.21.15.130:0/2607266809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6c101890 0x7fde6c102810 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:03.419 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:03.412+0000 7fde7332d700 1 --2- 172.21.15.130:0/2607266809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde6c0ffcf0 0x7fde6c103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:03.419 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:03.412+0000 7fde7332d700 1 --2- 172.21.15.130:0/2607266809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde6c100ef0 0x7fde6c1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:03.419 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:03.412+0000 7fde7332d700 1 -- 172.21.15.130:0/2607266809 >> 172.21.15.130:0/2607266809 conn(0x7fde6c0fb2b0 msgr2=0x7fde6c0fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:53:03.419 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:03.412+0000 7fde7332d700 1 -- 172.21.15.130:0/2607266809 shutdown_connections 2024-03-30T00:53:03.420 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:03.412+0000 7fde7332d700 1 -- 172.21.15.130:0/2607266809 wait complete. 2024-03-30T00:53:03.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:03.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:03.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:03.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:03.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:03.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:03.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:03.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:03.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:03.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:03.815 DEBUG:teuthology.orchestra.run.smithi130:osd.4> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.4.service 2024-03-30T00:53:03.817 INFO:tasks.cephadm:Deploying osd.5 on smithi181 with /dev/vg_nvme/lv_3... 2024-03-30T00:53:03.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T00:53:03.848 INFO:journalctl@ceph.osd.4.smithi130.stdout:-- Logs begin at Sat 2024-03-30 00:27:58 UTC. -- 2024-03-30T00:53:04.048 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T00:53:04.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:04 smithi181 ceph-mon[47622]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T00:53:04.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:04 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:04.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:04 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:04.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:04 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:04.563 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:04 smithi130 ceph-mon[47626]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T00:53:04.563 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:04 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:04.563 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:04 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:04.563 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:04 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:04.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:04 smithi116 ceph-mon[38086]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T00:53:04.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:04 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:04.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:04 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:04.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:04 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:05.385 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T00:53:05.385 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T00:53:05.385 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-03-30T00:53:05.385 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-03-30T00:53:05.386 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113159 s, 927 MB/s 2024-03-30T00:53:05.386 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-03-30T00:53:05.406 INFO:journalctl@ceph.osd.4.smithi130.stdout:Mar 30 00:53:05 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-4[61134]: 2024-03-30T00:53:05.119+0000 7f44d2457640 -1 osd.4 0 log_to_monitors true 2024-03-30T00:53:05.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:05 smithi116 ceph-mon[38086]: from='osd.4 [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T00:53:05.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:05 smithi116 ceph-mon[38086]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T00:53:05.714 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:05 smithi130 ceph-mon[47626]: from='osd.4 [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T00:53:05.714 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:05 smithi130 ceph-mon[47626]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T00:53:05.715 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:05 smithi181 ceph-mon[47622]: from='osd.4 [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T00:53:05.715 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:05 smithi181 ceph-mon[47622]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T00:53:05.736 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi181:vg_nvme/lv_3 2024-03-30T00:53:05.953 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T00:53:06.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:06 smithi181 ceph-mon[47622]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-30T00:53:06.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:06 smithi181 ceph-mon[47622]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T00:53:06.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:06 smithi181 ceph-mon[47622]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T00:53:06.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:06.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:06 smithi181 ceph-mon[47622]: from='osd.4 [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:53:06.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:06 smithi181 ceph-mon[47622]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:53:06.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:06.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:06.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:06.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:06.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:06.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:06.671 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:06.672 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:06.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:06 smithi116 ceph-mon[38086]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-30T00:53:06.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:06 smithi116 ceph-mon[38086]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T00:53:06.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:06 smithi116 ceph-mon[38086]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T00:53:06.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:06.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:06 smithi116 ceph-mon[38086]: from='osd.4 [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:53:06.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:06 smithi116 ceph-mon[38086]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:53:06.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:06.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:06.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:06.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:06.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:06.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:06.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:06.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:06.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-mon[47626]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-30T00:53:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-mon[47626]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T00:53:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-mon[47626]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T00:53:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-mon[47626]: from='osd.4 [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:53:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-mon[47626]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:53:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:06.846 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.840+0000 7fcc2296c700 1 -- 172.21.15.181:0/1230794074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc1c102150 msgr2=0x7fcc1c1025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:06.846 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.840+0000 7fcc2296c700 1 --2- 172.21.15.181:0/1230794074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc1c102150 0x7fcc1c1025b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009990 tx=0x7fcc0c009ca0 comp rx=0 tx=0).stop 2024-03-30T00:53:06.846 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.840+0000 7fcc2296c700 1 -- 172.21.15.181:0/1230794074 shutdown_connections 2024-03-30T00:53:06.846 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.840+0000 7fcc2296c700 1 --2- 172.21.15.181:0/1230794074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc1c102af0 0x7fcc1c1093a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:06.846 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.840+0000 7fcc2296c700 1 --2- 172.21.15.181:0/1230794074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc1c0ff0d0 0x7fcc1c0ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:06.846 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.840+0000 7fcc2296c700 1 --2- 172.21.15.181:0/1230794074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc1c102150 0x7fcc1c1025b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:06.846 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.840+0000 7fcc2296c700 1 -- 172.21.15.181:0/1230794074 >> 172.21.15.181:0/1230794074 conn(0x7fcc1c0faa10 msgr2=0x7fcc1c0fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:53:06.846 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.840+0000 7fcc2296c700 1 -- 172.21.15.181:0/1230794074 shutdown_connections 2024-03-30T00:53:06.847 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.840+0000 7fcc2296c700 1 -- 172.21.15.181:0/1230794074 wait complete. 2024-03-30T00:53:06.849 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.840+0000 7fcc2296c700 1 Processor -- start 2024-03-30T00:53:06.849 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.840+0000 7fcc2296c700 1 -- start start 2024-03-30T00:53:06.849 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc2296c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc1c0ff0d0 0x7fcc1c196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:53:06.849 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc2296c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc1c102150 0x7fcc1c197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:53:06.849 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc2296c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc1c102af0 0x7fcc1c19e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:53:06.849 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc2296c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c19e900 con 0x7fcc1c102af0 2024-03-30T00:53:06.849 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc2296c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c19ea40 con 0x7fcc1c102150 2024-03-30T00:53:06.849 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc2296c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c19ec00 con 0x7fcc1c0ff0d0 2024-03-30T00:53:06.849 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc1bfff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc1c0ff0d0 0x7fcc1c196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:53:06.850 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc1bfff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc1c0ff0d0 0x7fcc1c196e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:40242/0 (socket says 172.21.15.181:40242) 2024-03-30T00:53:06.850 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc1bfff700 1 -- 172.21.15.181:0/64311265 learned_addr learned my addr 172.21.15.181:0/64311265 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-03-30T00:53:06.850 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc1b7fe700 1 --2- 172.21.15.181:0/64311265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc1c102150 0x7fcc1c197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:53:06.850 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc20f09700 1 --2- 172.21.15.181:0/64311265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc1c102af0 0x7fcc1c19e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:53:06.850 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc1bfff700 1 -- 172.21.15.181:0/64311265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc1c102150 msgr2=0x7fcc1c197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:06.850 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc1bfff700 1 --2- 172.21.15.181:0/64311265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc1c102150 0x7fcc1c197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:06.850 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc1bfff700 1 -- 172.21.15.181:0/64311265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc1c102af0 msgr2=0x7fcc1c19e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:06.850 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc1bfff700 1 --2- 172.21.15.181:0/64311265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc1c102af0 0x7fcc1c19e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:06.850 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc1bfff700 1 -- 172.21.15.181:0/64311265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c0095f0 con 0x7fcc1c0ff0d0 2024-03-30T00:53:06.850 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.841+0000 7fcc1bfff700 1 --2- 172.21.15.181:0/64311265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc1c0ff0d0 0x7fcc1c196e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000b500 tx=0x7fcc1000b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:53:06.851 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.842+0000 7fcc197fa700 1 -- 172.21.15.181:0/64311265 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10011830 con 0x7fcc1c0ff0d0 2024-03-30T00:53:06.851 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.842+0000 7fcc197fa700 1 -- 172.21.15.181:0/64311265 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc10011e50 con 0x7fcc1c0ff0d0 2024-03-30T00:53:06.851 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.842+0000 7fcc2296c700 1 -- 172.21.15.181:0/64311265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1c19eee0 con 0x7fcc1c0ff0d0 2024-03-30T00:53:06.851 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.842+0000 7fcc197fa700 1 -- 172.21.15.181:0/64311265 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1000f2e0 con 0x7fcc1c0ff0d0 2024-03-30T00:53:06.851 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.842+0000 7fcc2296c700 1 -- 172.21.15.181:0/64311265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1c19f300 con 0x7fcc1c0ff0d0 2024-03-30T00:53:06.851 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.843+0000 7fcc2296c700 1 -- 172.21.15.181:0/64311265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1c108430 con 0x7fcc1c0ff0d0 2024-03-30T00:53:06.852 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.844+0000 7fcc197fa700 1 -- 172.21.15.181:0/64311265 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fcc10010580 con 0x7fcc1c0ff0d0 2024-03-30T00:53:06.852 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.845+0000 7fcc197fa700 1 --2- 172.21.15.181:0/64311265 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcc040a11a0 0x7fcc040a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:53:06.853 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.845+0000 7fcc197fa700 1 -- 172.21.15.181:0/64311265 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3821+0+0 (secure 0 0 0) 0x7fcc100cb650 con 0x7fcc1c0ff0d0 2024-03-30T00:53:06.853 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.845+0000 7fcc1b7fe700 1 --2- 172.21.15.181:0/64311265 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcc040a11a0 0x7fcc040a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:53:06.853 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.846+0000 7fcc1b7fe700 1 --2- 172.21.15.181:0/64311265 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcc040a11a0 0x7fcc040a3660 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00b2e0 tx=0x7fcc0c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:53:06.855 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:06.849+0000 7fcc197fa700 1 -- 172.21.15.181:0/64311265 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcc10095f50 con 0x7fcc1c0ff0d0 2024-03-30T00:53:07.058 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:07.052+0000 7fcc2296c700 1 -- 172.21.15.181:0/64311265 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc1c02cdc0 con 0x7fcc040a11a0 2024-03-30T00:53:07.344 INFO:journalctl@ceph.osd.4.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-4[61134]: 2024-03-30T00:53:06.882+0000 7f44c7abb700 -1 osd.4 0 waiting for initial osdmap 2024-03-30T00:53:07.345 INFO:journalctl@ceph.osd.4.smithi130.stdout:Mar 30 00:53:06 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-4[61134]: 2024-03-30T00:53:06.888+0000 7f44bf8a8700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T00:53:07.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:07 smithi116 ceph-mon[38086]: Detected new or changed devices on smithi130 2024-03-30T00:53:07.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:07 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi130 to 8556M 2024-03-30T00:53:07.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:07 smithi116 ceph-mon[38086]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-30T00:53:07.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:07 smithi116 ceph-mon[38086]: osdmap e30: 5 total, 4 up, 5 in 2024-03-30T00:53:07.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:07.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:07.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:07 smithi116 ceph-mon[38086]: from='osd.4 ' entity='osd.4' 2024-03-30T00:53:07.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:53:07.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:53:07.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:07 smithi181 ceph-mon[47622]: Detected new or changed devices on smithi130 2024-03-30T00:53:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:07 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi130 to 8556M 2024-03-30T00:53:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:07 smithi181 ceph-mon[47622]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-30T00:53:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:07 smithi181 ceph-mon[47622]: osdmap e30: 5 total, 4 up, 5 in 2024-03-30T00:53:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:07 smithi181 ceph-mon[47622]: from='osd.4 ' entity='osd.4' 2024-03-30T00:53:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:53:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:53:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:07.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:07 smithi130 ceph-mon[47626]: Detected new or changed devices on smithi130 2024-03-30T00:53:07.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:07 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi130 to 8556M 2024-03-30T00:53:07.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:07 smithi130 ceph-mon[47626]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-30T00:53:07.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:07 smithi130 ceph-mon[47626]: osdmap e30: 5 total, 4 up, 5 in 2024-03-30T00:53:07.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:07.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:07.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:07 smithi130 ceph-mon[47626]: from='osd.4 ' entity='osd.4' 2024-03-30T00:53:07.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:53:07.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:53:07.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:08 smithi116 ceph-mon[38086]: purged_snaps scrub starts 2024-03-30T00:53:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:08 smithi116 ceph-mon[38086]: purged_snaps scrub ok 2024-03-30T00:53:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:08 smithi116 ceph-mon[38086]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:53:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:08 smithi116 ceph-mon[38086]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-30T00:53:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:08 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:08 smithi116 ceph-mon[38086]: osd.4 [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957] boot 2024-03-30T00:53:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:08 smithi116 ceph-mon[38086]: osdmap e31: 5 total, 5 up, 5 in 2024-03-30T00:53:08.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:08 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:08 smithi181 ceph-mon[47622]: purged_snaps scrub starts 2024-03-30T00:53:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:08 smithi181 ceph-mon[47622]: purged_snaps scrub ok 2024-03-30T00:53:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:08 smithi181 ceph-mon[47622]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:53:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:08 smithi181 ceph-mon[47622]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-30T00:53:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:08 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:08 smithi181 ceph-mon[47622]: osd.4 [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957] boot 2024-03-30T00:53:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:08 smithi181 ceph-mon[47622]: osdmap e31: 5 total, 5 up, 5 in 2024-03-30T00:53:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:08 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:08.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:08 smithi130 ceph-mon[47626]: purged_snaps scrub starts 2024-03-30T00:53:08.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:08 smithi130 ceph-mon[47626]: purged_snaps scrub ok 2024-03-30T00:53:08.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:08 smithi130 ceph-mon[47626]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:53:08.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:08 smithi130 ceph-mon[47626]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-30T00:53:08.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:08 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:08.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:08 smithi130 ceph-mon[47626]: osd.4 [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957] boot 2024-03-30T00:53:08.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:08 smithi130 ceph-mon[47626]: osdmap e31: 5 total, 5 up, 5 in 2024-03-30T00:53:08.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:08 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T00:53:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:09 smithi181 ceph-mon[47622]: from='client.? 172.21.15.181:0/634136110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d94364f4-bdb6-4c7f-9ed0-293887f07a8b"}]: dispatch 2024-03-30T00:53:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:09 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d94364f4-bdb6-4c7f-9ed0-293887f07a8b"}]: dispatch 2024-03-30T00:53:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:09 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d94364f4-bdb6-4c7f-9ed0-293887f07a8b"}]': finished 2024-03-30T00:53:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:09 smithi181 ceph-mon[47622]: osdmap e32: 6 total, 5 up, 6 in 2024-03-30T00:53:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:09 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:09.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:09 smithi130 ceph-mon[47626]: from='client.? 172.21.15.181:0/634136110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d94364f4-bdb6-4c7f-9ed0-293887f07a8b"}]: dispatch 2024-03-30T00:53:09.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:09 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d94364f4-bdb6-4c7f-9ed0-293887f07a8b"}]: dispatch 2024-03-30T00:53:09.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:09 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d94364f4-bdb6-4c7f-9ed0-293887f07a8b"}]': finished 2024-03-30T00:53:09.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:09 smithi130 ceph-mon[47626]: osdmap e32: 6 total, 5 up, 6 in 2024-03-30T00:53:09.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:09 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:09.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:09 smithi116 ceph-mon[38086]: from='client.? 172.21.15.181:0/634136110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d94364f4-bdb6-4c7f-9ed0-293887f07a8b"}]: dispatch 2024-03-30T00:53:09.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:09 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d94364f4-bdb6-4c7f-9ed0-293887f07a8b"}]: dispatch 2024-03-30T00:53:09.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:09 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d94364f4-bdb6-4c7f-9ed0-293887f07a8b"}]': finished 2024-03-30T00:53:09.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:09 smithi116 ceph-mon[38086]: osdmap e32: 6 total, 5 up, 6 in 2024-03-30T00:53:09.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:09 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:10 smithi181 ceph-mon[47622]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:10 smithi181 ceph-mon[47622]: from='client.? 172.21.15.181:0/2171390810' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:53:10.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:10 smithi130 ceph-mon[47626]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:10.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:10 smithi130 ceph-mon[47626]: from='client.? 172.21.15.181:0/2171390810' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:53:10.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:10 smithi116 ceph-mon[38086]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:10.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:10 smithi116 ceph-mon[38086]: from='client.? 172.21.15.181:0/2171390810' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:53:12.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:12 smithi181 ceph-mon[47622]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:12.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:12 smithi130 ceph-mon[47626]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:12.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:12 smithi116 ceph-mon[38086]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:14.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:14 smithi181 ceph-mon[47622]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:14.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:14 smithi130 ceph-mon[47626]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:14.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:14 smithi116 ceph-mon[38086]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:15.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:15 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T00:53:15.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:15 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:15.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:15 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T00:53:15.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:15 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:15.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:15 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T00:53:15.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:15 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:16.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:16 smithi181 ceph-mon[47622]: Deploying daemon osd.5 on smithi181 2024-03-30T00:53:16.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:16 smithi181 ceph-mon[47622]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:16.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:16 smithi130 ceph-mon[47626]: Deploying daemon osd.5 on smithi181 2024-03-30T00:53:16.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:16 smithi130 ceph-mon[47626]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:16.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:16 smithi116 ceph-mon[38086]: Deploying daemon osd.5 on smithi181 2024-03-30T00:53:16.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:16 smithi116 ceph-mon[38086]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:18.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:18 smithi181 ceph-mon[47622]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:18.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:18 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:18.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:18 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:18.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:18 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:18.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:18 smithi130 ceph-mon[47626]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:18.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:18 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:18.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:18 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:18.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:18 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:18.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:18 smithi116 ceph-mon[38086]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:18.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:18 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:18.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:18 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:18.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:18 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:20.525 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 5 on host 'smithi181' 2024-03-30T00:53:20.525 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:20.518+0000 7fcc197fa700 1 -- 172.21.15.181:0/64311265 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcc1c02cdc0 con 0x7fcc040a11a0 2024-03-30T00:53:20.526 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:20.520+0000 7fcc2296c700 1 -- 172.21.15.181:0/64311265 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcc040a11a0 msgr2=0x7fcc040a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:20.526 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:20.520+0000 7fcc2296c700 1 --2- 172.21.15.181:0/64311265 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcc040a11a0 0x7fcc040a3660 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00b2e0 tx=0x7fcc0c005fd0 comp rx=0 tx=0).stop 2024-03-30T00:53:20.526 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:20.520+0000 7fcc2296c700 1 -- 172.21.15.181:0/64311265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc1c0ff0d0 msgr2=0x7fcc1c196e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:20.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:20.520+0000 7fcc2296c700 1 --2- 172.21.15.181:0/64311265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc1c0ff0d0 0x7fcc1c196e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000b500 tx=0x7fcc1000b810 comp rx=0 tx=0).stop 2024-03-30T00:53:20.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:20.520+0000 7fcc2296c700 1 -- 172.21.15.181:0/64311265 shutdown_connections 2024-03-30T00:53:20.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:20.520+0000 7fcc2296c700 1 --2- 172.21.15.181:0/64311265 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcc040a11a0 0x7fcc040a3660 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:20.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:20.520+0000 7fcc2296c700 1 --2- 172.21.15.181:0/64311265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc1c0ff0d0 0x7fcc1c196e00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:20.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:20.520+0000 7fcc2296c700 1 --2- 172.21.15.181:0/64311265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc1c102af0 0x7fcc1c19e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:20.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:20.520+0000 7fcc2296c700 1 --2- 172.21.15.181:0/64311265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc1c102150 0x7fcc1c197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:20.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:20.520+0000 7fcc2296c700 1 -- 172.21.15.181:0/64311265 >> 172.21.15.181:0/64311265 conn(0x7fcc1c0faa10 msgr2=0x7fcc1c105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:53:20.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:20.520+0000 7fcc2296c700 1 -- 172.21.15.181:0/64311265 shutdown_connections 2024-03-30T00:53:20.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:53:20.520+0000 7fcc2296c700 1 -- 172.21.15.181:0/64311265 wait complete. 2024-03-30T00:53:20.685 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:20 smithi181 ceph-mon[47622]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:20.686 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:20.686 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:20.686 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:20.686 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:20.686 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:20.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:20 smithi130 ceph-mon[47626]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:20 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:20 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:20 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:20 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:20 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:20.914 DEBUG:teuthology.orchestra.run.smithi181:osd.5> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.5.service 2024-03-30T00:53:20.917 INFO:tasks.cephadm:Deploying osd.6 on smithi116 with /dev/vg_nvme/lv_2... 2024-03-30T00:53:20.917 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T00:53:20.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:20 smithi116 ceph-mon[38086]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:20.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:20.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:20.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:20.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:20.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:20.946 INFO:journalctl@ceph.osd.5.smithi181.stdout:-- Logs begin at Sat 2024-03-30 00:27:51 UTC. -- 2024-03-30T00:53:21.280 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:53:21.710 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:21.711 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:21.711 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:21.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:21.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:21.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:21.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:21.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:21.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:22.615 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 00:53:22 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T00:53:22.314+0000 7fe4b4dc8640 -1 osd.5 0 log_to_monitors true 2024-03-30T00:53:22.615 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:22 smithi181 ceph-mon[47622]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:22.615 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:22 smithi181 ceph-mon[47622]: from='osd.5 [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T00:53:22.615 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:22 smithi181 ceph-mon[47622]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T00:53:22.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:22 smithi130 ceph-mon[47626]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:22.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:22 smithi130 ceph-mon[47626]: from='osd.5 [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T00:53:22.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:22 smithi130 ceph-mon[47626]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T00:53:22.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:22 smithi116 ceph-mon[38086]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:22.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:22 smithi116 ceph-mon[38086]: from='osd.5 [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T00:53:22.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:22 smithi116 ceph-mon[38086]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T00:53:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T00:53:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T00:53:22.936 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-30T00:53:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-30T00:53:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0297778 s, 352 MB/s 2024-03-30T00:53:22.936 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 2024-03-30T00:53:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-30T00:53:23.355 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi116:vg_nvme/lv_2 2024-03-30T00:53:23.746 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:53:24.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: Detected new or changed devices on smithi181 2024-03-30T00:53:24.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:24.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:24.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:24.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:24.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi181 to 8556M 2024-03-30T00:53:24.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:24.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T00:53:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: osdmap e33: 6 total, 5 up, 6 in 2024-03-30T00:53:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: from='osd.5 [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:53:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:23 smithi116 ceph-mon[38086]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:53:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: Detected new or changed devices on smithi181 2024-03-30T00:53:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi181 to 8556M 2024-03-30T00:53:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:24.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:24.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T00:53:24.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: osdmap e33: 6 total, 5 up, 6 in 2024-03-30T00:53:24.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:24.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: from='osd.5 [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:53:24.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:23 smithi181 ceph-mon[47622]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:53:24.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: Detected new or changed devices on smithi181 2024-03-30T00:53:24.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:24.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:24.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:24.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:24.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi181 to 8556M 2024-03-30T00:53:24.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:24.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:24.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:24.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:24.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:24.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T00:53:24.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: osdmap e33: 6 total, 5 up, 6 in 2024-03-30T00:53:24.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:24.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: from='osd.5 [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:53:24.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:23 smithi130 ceph-mon[47626]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:53:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.812+0000 7fe939600700 1 -- 172.21.15.116:0/924783278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe934102920 msgr2=0x7fe934109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.812+0000 7fe939600700 1 --2- 172.21.15.116:0/924783278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe934102920 0x7fe934109240 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe92c00b0a0 tx=0x7fe92c00b3b0 comp rx=0 tx=0).stop 2024-03-30T00:53:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.812+0000 7fe939600700 1 -- 172.21.15.116:0/924783278 shutdown_connections 2024-03-30T00:53:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.812+0000 7fe939600700 1 --2- 172.21.15.116:0/924783278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe934100e00 0x7fe934101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.812+0000 7fe939600700 1 --2- 172.21.15.116:0/924783278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe934102920 0x7fe934109240 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.812+0000 7fe939600700 1 --2- 172.21.15.116:0/924783278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe934101f60 0x7fe9341023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.812+0000 7fe939600700 1 -- 172.21.15.116:0/924783278 >> 172.21.15.116:0/924783278 conn(0x7fe9340fc3c0 msgr2=0x7fe9340fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:53:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.812+0000 7fe939600700 1 -- 172.21.15.116:0/924783278 shutdown_connections 2024-03-30T00:53:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.812+0000 7fe939600700 1 -- 172.21.15.116:0/924783278 wait complete. 2024-03-30T00:53:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.813+0000 7fe939600700 1 Processor -- start 2024-03-30T00:53:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.813+0000 7fe939600700 1 -- start start 2024-03-30T00:53:24.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.813+0000 7fe939600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe934100e00 0x7fe934199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:53:24.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.813+0000 7fe939600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe934101f60 0x7fe9341995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:53:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.813+0000 7fe939600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe934102920 0x7fe9341a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:53:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.813+0000 7fe939600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9341a0ba0 con 0x7fe934102920 2024-03-30T00:53:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.813+0000 7fe939600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9341a0ce0 con 0x7fe934100e00 2024-03-30T00:53:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.813+0000 7fe939600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9341a0e70 con 0x7fe934101f60 2024-03-30T00:53:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.814+0000 7fe932000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe934101f60 0x7fe9341995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:53:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.814+0000 7fe932a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe934100e00 0x7fe934199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:53:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.814+0000 7fe932000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe934101f60 0x7fe9341995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:56032/0 (socket says 172.21.15.116:56032) 2024-03-30T00:53:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.814+0000 7fe932000700 1 -- 172.21.15.116:0/1486623159 learned_addr learned my addr 172.21.15.116:0/1486623159 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:53:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.814+0000 7fe933400700 1 --2- 172.21.15.116:0/1486623159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe934102920 0x7fe9341a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:53:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.814+0000 7fe932000700 1 -- 172.21.15.116:0/1486623159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe934100e00 msgr2=0x7fe934199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.814+0000 7fe932000700 1 --2- 172.21.15.116:0/1486623159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe934100e00 0x7fe934199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.814+0000 7fe932000700 1 -- 172.21.15.116:0/1486623159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe934102920 msgr2=0x7fe9341a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.814+0000 7fe932000700 1 --2- 172.21.15.116:0/1486623159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe934102920 0x7fe9341a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.814+0000 7fe932000700 1 -- 172.21.15.116:0/1486623159 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c009d00 con 0x7fe934101f60 2024-03-30T00:53:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.814+0000 7fe932000700 1 --2- 172.21.15.116:0/1486623159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe934101f60 0x7fe9341995a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe92000e9d0 tx=0x7fe92000ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:53:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.814+0000 7fe92b400700 1 -- 172.21.15.116:0/1486623159 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920009790 con 0x7fe934101f60 2024-03-30T00:53:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.814+0000 7fe939600700 1 -- 172.21.15.116:0/1486623159 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9341a1150 con 0x7fe934101f60 2024-03-30T00:53:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.815+0000 7fe92b400700 1 -- 172.21.15.116:0/1486623159 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe92000ee70 con 0x7fe934101f60 2024-03-30T00:53:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.815+0000 7fe939600700 1 -- 172.21.15.116:0/1486623159 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9341a1650 con 0x7fe934101f60 2024-03-30T00:53:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.815+0000 7fe92b400700 1 -- 172.21.15.116:0/1486623159 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920010ea0 con 0x7fe934101f60 2024-03-30T00:53:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.815+0000 7fe939600700 1 -- 172.21.15.116:0/1486623159 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93404ea80 con 0x7fe934101f60 2024-03-30T00:53:24.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.817+0000 7fe92b400700 1 -- 172.21.15.116:0/1486623159 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fe92000cbd0 con 0x7fe934101f60 2024-03-30T00:53:24.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.818+0000 7fe92b400700 1 --2- 172.21.15.116:0/1486623159 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe9180a1120 0x7fe9180a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:53:24.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.819+0000 7fe92b400700 1 -- 172.21.15.116:0/1486623159 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4253+0+0 (secure 0 0 0) 0x7fe920014070 con 0x7fe934101f60 2024-03-30T00:53:24.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.819+0000 7fe932a00700 1 --2- 172.21.15.116:0/1486623159 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe9180a1120 0x7fe9180a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:53:24.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.819+0000 7fe932a00700 1 --2- 172.21.15.116:0/1486623159 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe9180a1120 0x7fe9180a35e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe924009c60 tx=0x7fe924009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:53:24.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:24.821+0000 7fe92b400700 1 -- 172.21.15.116:0/1486623159 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe920096190 con 0x7fe934101f60 2024-03-30T00:53:25.002 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 00:53:24 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T00:53:24.545+0000 7fe4aa42c700 -1 osd.5 0 waiting for initial osdmap 2024-03-30T00:53:25.002 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 00:53:24 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T00:53:24.552+0000 7fe4a2219700 -1 osd.5 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T00:53:25.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:25.032+0000 7fe939600700 1 -- 172.21.15.116:0/1486623159 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe934074700 con 0x7fe9180a1120 2024-03-30T00:53:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:25 smithi116 ceph-mon[38086]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-30T00:53:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:25 smithi116 ceph-mon[38086]: osdmap e34: 6 total, 5 up, 6 in 2024-03-30T00:53:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:25 smithi116 ceph-mon[38086]: from='osd.5 ' entity='osd.5' 2024-03-30T00:53:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:53:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:53:25.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:25.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:25 smithi181 ceph-mon[47622]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-30T00:53:25.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:25 smithi181 ceph-mon[47622]: osdmap e34: 6 total, 5 up, 6 in 2024-03-30T00:53:25.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:25.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:25.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:25 smithi181 ceph-mon[47622]: from='osd.5 ' entity='osd.5' 2024-03-30T00:53:25.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:53:25.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:53:25.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:25.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:25 smithi130 ceph-mon[47626]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-30T00:53:25.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:25 smithi130 ceph-mon[47626]: osdmap e34: 6 total, 5 up, 6 in 2024-03-30T00:53:25.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:25.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:25.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:25 smithi130 ceph-mon[47626]: from='osd.5 ' entity='osd.5' 2024-03-30T00:53:25.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:53:25.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:53:25.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:26 smithi116 ceph-mon[38086]: purged_snaps scrub starts 2024-03-30T00:53:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:26 smithi116 ceph-mon[38086]: purged_snaps scrub ok 2024-03-30T00:53:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:26 smithi116 ceph-mon[38086]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:53:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:26 smithi116 ceph-mon[38086]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:26 smithi116 ceph-mon[38086]: osd.5 [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027] boot 2024-03-30T00:53:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:26 smithi116 ceph-mon[38086]: osdmap e35: 6 total, 6 up, 6 in 2024-03-30T00:53:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:26 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:26 smithi116 ceph-mon[38086]: osdmap e36: 6 total, 6 up, 6 in 2024-03-30T00:53:26.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:26 smithi181 ceph-mon[47622]: purged_snaps scrub starts 2024-03-30T00:53:26.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:26 smithi181 ceph-mon[47622]: purged_snaps scrub ok 2024-03-30T00:53:26.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:26 smithi181 ceph-mon[47622]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:53:26.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:26 smithi181 ceph-mon[47622]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:26.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:26 smithi181 ceph-mon[47622]: osd.5 [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027] boot 2024-03-30T00:53:26.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:26 smithi181 ceph-mon[47622]: osdmap e35: 6 total, 6 up, 6 in 2024-03-30T00:53:26.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:26 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:26.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:26 smithi181 ceph-mon[47622]: osdmap e36: 6 total, 6 up, 6 in 2024-03-30T00:53:26.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:26 smithi130 ceph-mon[47626]: purged_snaps scrub starts 2024-03-30T00:53:26.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:26 smithi130 ceph-mon[47626]: purged_snaps scrub ok 2024-03-30T00:53:26.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:26 smithi130 ceph-mon[47626]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:53:26.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:26 smithi130 ceph-mon[47626]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-30T00:53:26.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:26 smithi130 ceph-mon[47626]: osd.5 [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027] boot 2024-03-30T00:53:26.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:26 smithi130 ceph-mon[47626]: osdmap e35: 6 total, 6 up, 6 in 2024-03-30T00:53:26.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:26 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T00:53:26.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:26 smithi130 ceph-mon[47626]: osdmap e36: 6 total, 6 up, 6 in 2024-03-30T00:53:28.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:28 smithi116 ceph-mon[38086]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:28.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:28 smithi116 ceph-mon[38086]: osdmap e37: 6 total, 6 up, 6 in 2024-03-30T00:53:28.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:28 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2713970188' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9866aade-0505-4465-87bc-91fa24eba9c7"}]: dispatch 2024-03-30T00:53:28.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:28 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2713970188' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9866aade-0505-4465-87bc-91fa24eba9c7"}]': finished 2024-03-30T00:53:28.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:28 smithi116 ceph-mon[38086]: osdmap e38: 7 total, 6 up, 7 in 2024-03-30T00:53:28.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:28 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:28.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:28 smithi181 ceph-mon[47622]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:28.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:28 smithi181 ceph-mon[47622]: osdmap e37: 6 total, 6 up, 6 in 2024-03-30T00:53:28.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:28 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2713970188' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9866aade-0505-4465-87bc-91fa24eba9c7"}]: dispatch 2024-03-30T00:53:28.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:28 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2713970188' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9866aade-0505-4465-87bc-91fa24eba9c7"}]': finished 2024-03-30T00:53:28.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:28 smithi181 ceph-mon[47622]: osdmap e38: 7 total, 6 up, 7 in 2024-03-30T00:53:28.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:28 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:28.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:28 smithi130 ceph-mon[47626]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:28 smithi130 ceph-mon[47626]: osdmap e37: 6 total, 6 up, 6 in 2024-03-30T00:53:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:28 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2713970188' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9866aade-0505-4465-87bc-91fa24eba9c7"}]: dispatch 2024-03-30T00:53:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:28 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2713970188' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9866aade-0505-4465-87bc-91fa24eba9c7"}]': finished 2024-03-30T00:53:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:28 smithi130 ceph-mon[47626]: osdmap e38: 7 total, 6 up, 7 in 2024-03-30T00:53:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:28 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:29.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:29 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2274483911' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:53:29.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:29 smithi116 ceph-mon[38086]: osdmap e39: 7 total, 6 up, 7 in 2024-03-30T00:53:29.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:29 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:29.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:29 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2274483911' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:53:29.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:29 smithi181 ceph-mon[47622]: osdmap e39: 7 total, 6 up, 7 in 2024-03-30T00:53:29.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:29 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:29.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:29 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2274483911' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:53:29.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:29 smithi130 ceph-mon[47626]: osdmap e39: 7 total, 6 up, 7 in 2024-03-30T00:53:29.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:29 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:30.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:30 smithi116 ceph-mon[38086]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-03-30T00:53:30.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:30 smithi181 ceph-mon[47622]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-03-30T00:53:30.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:30 smithi130 ceph-mon[47626]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 198 MiB used, 536 GiB / 536 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-03-30T00:53:32.576 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:32 smithi116 ceph-mon[38086]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-03-30T00:53:32.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:32 smithi181 ceph-mon[47622]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-03-30T00:53:32.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:32 smithi130 ceph-mon[47626]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-03-30T00:53:34.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:34 smithi116 ceph-mon[38086]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T00:53:34.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T00:53:34.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:34 smithi181 ceph-mon[47622]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T00:53:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T00:53:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:34.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:34 smithi130 ceph-mon[47626]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T00:53:34.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T00:53:34.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:35.604 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:35 smithi116 ceph-mon[38086]: Deploying daemon osd.6 on smithi116 2024-03-30T00:53:35.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:35 smithi181 ceph-mon[47622]: Deploying daemon osd.6 on smithi116 2024-03-30T00:53:35.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:35 smithi130 ceph-mon[47626]: Deploying daemon osd.6 on smithi116 2024-03-30T00:53:36.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:36 smithi116 ceph-mon[38086]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T00:53:36.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:36 smithi181 ceph-mon[47622]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T00:53:36.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:36 smithi130 ceph-mon[47626]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T00:53:38.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:38 smithi116 ceph-mon[38086]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-30T00:53:38.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:38 smithi181 ceph-mon[47622]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-30T00:53:38.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:38 smithi130 ceph-mon[47626]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-30T00:53:39.801 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:39 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:39.801 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:39 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:39.801 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:39 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:39.801 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:39 smithi116 ceph-mon[38086]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:40.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:39 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:40.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:39 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:40.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:39 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:40.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:39 smithi181 ceph-mon[47622]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:40.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:39 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:40.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:39 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:40.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:39 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:40.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:39 smithi130 ceph-mon[47626]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:41.229+0000 7fe92b400700 1 -- 172.21.15.116:0/1486623159 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe934074700 con 0x7fe9180a1120 2024-03-30T00:53:41.236 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 6 on host 'smithi116' 2024-03-30T00:53:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:41.231+0000 7fe939600700 1 -- 172.21.15.116:0/1486623159 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe9180a1120 msgr2=0x7fe9180a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:41.231+0000 7fe939600700 1 --2- 172.21.15.116:0/1486623159 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe9180a1120 0x7fe9180a35e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe924009c60 tx=0x7fe924009260 comp rx=0 tx=0).stop 2024-03-30T00:53:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:41.231+0000 7fe939600700 1 -- 172.21.15.116:0/1486623159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe934101f60 msgr2=0x7fe9341995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:41.231+0000 7fe939600700 1 --2- 172.21.15.116:0/1486623159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe934101f60 0x7fe9341995a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe92000e9d0 tx=0x7fe92000ece0 comp rx=0 tx=0).stop 2024-03-30T00:53:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:41.232+0000 7fe939600700 1 -- 172.21.15.116:0/1486623159 shutdown_connections 2024-03-30T00:53:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:41.232+0000 7fe939600700 1 --2- 172.21.15.116:0/1486623159 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe9180a1120 0x7fe9180a35e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:41.232+0000 7fe939600700 1 --2- 172.21.15.116:0/1486623159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe934102920 0x7fe9341a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:41.232+0000 7fe939600700 1 --2- 172.21.15.116:0/1486623159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe934100e00 0x7fe934199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:41.232+0000 7fe939600700 1 --2- 172.21.15.116:0/1486623159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe934101f60 0x7fe9341995a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:41.232+0000 7fe939600700 1 -- 172.21.15.116:0/1486623159 >> 172.21.15.116:0/1486623159 conn(0x7fe9340fc3c0 msgr2=0x7fe934072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:53:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:41.232+0000 7fe939600700 1 -- 172.21.15.116:0/1486623159 shutdown_connections 2024-03-30T00:53:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:53:41.232+0000 7fe939600700 1 -- 172.21.15.116:0/1486623159 wait complete. 2024-03-30T00:53:41.616 DEBUG:teuthology.orchestra.run.smithi116:osd.6> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.6.service 2024-03-30T00:53:41.620 INFO:tasks.cephadm:Deploying osd.7 on smithi130 with /dev/vg_nvme/lv_2... 2024-03-30T00:53:41.621 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T00:53:41.693 INFO:journalctl@ceph.osd.6.smithi116.stdout:-- Logs begin at Sat 2024-03-30 00:33:55 UTC. -- 2024-03-30T00:53:41.851 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T00:53:41.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:41.856 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:41.856 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:41.856 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:41.856 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:41.856 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:41 smithi116 ceph-mon[38086]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:41.856 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:41.856 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:41.856 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:41.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:41.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:41.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:41.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:41.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:41.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:41 smithi130 ceph-mon[47626]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:41.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:41.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:41.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:41 smithi181 ceph-mon[47622]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:42.924 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 00:53:42 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T00:53:42.679+0000 7f746fa67640 -1 osd.6 0 log_to_monitors true 2024-03-30T00:53:43.207 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T00:53:43.207 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T00:53:43.207 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-03-30T00:53:43.207 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-03-30T00:53:43.207 INFO:teuthology.orchestra.run.smithi130.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109326 s, 959 MB/s 2024-03-30T00:53:43.207 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 2024-03-30T00:53:43.207 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-03-30T00:53:43.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:43 smithi116 ceph-mon[38086]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T00:53:43.440 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:43 smithi130 ceph-mon[47626]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T00:53:43.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:43 smithi181 ceph-mon[47622]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T00:53:43.644 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi130:vg_nvme/lv_2 2024-03-30T00:53:43.862 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T00:53:44.256 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:44 smithi130 ceph-mon[47626]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:44.257 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:44 smithi130 ceph-mon[47626]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T00:53:44.257 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:44 smithi130 ceph-mon[47626]: osdmap e40: 7 total, 6 up, 7 in 2024-03-30T00:53:44.257 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:44 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:44.257 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:44 smithi130 ceph-mon[47626]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:53:44.388 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:44 smithi116 ceph-mon[38086]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:44.388 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:44 smithi116 ceph-mon[38086]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T00:53:44.388 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:44 smithi116 ceph-mon[38086]: osdmap e40: 7 total, 6 up, 7 in 2024-03-30T00:53:44.388 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:44 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:44.388 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:44 smithi116 ceph-mon[38086]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:53:44.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:44 smithi181 ceph-mon[47622]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:44.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:44 smithi181 ceph-mon[47622]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T00:53:44.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:44 smithi181 ceph-mon[47622]: osdmap e40: 7 total, 6 up, 7 in 2024-03-30T00:53:44.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:44 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:44.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:44 smithi181 ceph-mon[47622]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:53:44.802 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.796+0000 7f9ed1743700 1 -- 172.21.15.130:0/328189864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ecc102980 msgr2=0x7f9ecc109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:44.802 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.796+0000 7f9ed1743700 1 --2- 172.21.15.130:0/328189864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ecc102980 0x7f9ecc109230 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400b0d0 tx=0x7f9ec400b3e0 comp rx=0 tx=0).stop 2024-03-30T00:53:44.802 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.796+0000 7f9ed1743700 1 -- 172.21.15.130:0/328189864 shutdown_connections 2024-03-30T00:53:44.802 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.796+0000 7f9ed1743700 1 --2- 172.21.15.130:0/328189864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ecc102980 0x7f9ecc109230 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:44.802 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.796+0000 7f9ed1743700 1 --2- 172.21.15.130:0/328189864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ecc100de0 0x7f9ecc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:44.802 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.796+0000 7f9ed1743700 1 --2- 172.21.15.130:0/328189864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ecc101fe0 0x7f9ecc102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:44.802 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.796+0000 7f9ed1743700 1 -- 172.21.15.130:0/328189864 >> 172.21.15.130:0/328189864 conn(0x7f9ecc0fc3c0 msgr2=0x7f9ecc0fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:53:44.802 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.796+0000 7f9ed1743700 1 -- 172.21.15.130:0/328189864 shutdown_connections 2024-03-30T00:53:44.802 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.796+0000 7f9ed1743700 1 -- 172.21.15.130:0/328189864 wait complete. 2024-03-30T00:53:44.804 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.797+0000 7f9ed1743700 1 Processor -- start 2024-03-30T00:53:44.804 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.797+0000 7f9ed1743700 1 -- start start 2024-03-30T00:53:44.805 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.797+0000 7f9ed1743700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ecc100de0 0x7f9ecc074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:53:44.805 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.797+0000 7f9ed1743700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ecc101fe0 0x7f9ecc0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:53:44.805 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.797+0000 7f9ed1743700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ecc102980 0x7f9ecc0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:53:44.805 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.797+0000 7f9ed1743700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc073e80 con 0x7f9ecc100de0 2024-03-30T00:53:44.805 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.797+0000 7f9ed1743700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc073ff0 con 0x7f9ecc101fe0 2024-03-30T00:53:44.805 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.797+0000 7f9ed1743700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc1a0ef0 con 0x7f9ecc102980 2024-03-30T00:53:44.805 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.797+0000 7f9eca7fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ecc101fe0 0x7f9ecc0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:53:44.805 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.797+0000 7f9eca7fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ecc101fe0 0x7f9ecc0731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:59120/0 (socket says 172.21.15.130:59120) 2024-03-30T00:53:44.805 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.797+0000 7f9eca7fc700 1 -- 172.21.15.130:0/227622426 learned_addr learned my addr 172.21.15.130:0/227622426 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-03-30T00:53:44.805 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.797+0000 7f9ecaffd700 1 --2- 172.21.15.130:0/227622426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ecc100de0 0x7f9ecc074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:53:44.805 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.798+0000 7f9ecb7fe700 1 --2- 172.21.15.130:0/227622426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ecc102980 0x7f9ecc0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:53:44.805 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.798+0000 7f9eca7fc700 1 -- 172.21.15.130:0/227622426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ecc102980 msgr2=0x7f9ecc0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:44.805 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.798+0000 7f9eca7fc700 1 --2- 172.21.15.130:0/227622426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ecc102980 0x7f9ecc0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:44.806 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.798+0000 7f9eca7fc700 1 -- 172.21.15.130:0/227622426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ecc100de0 msgr2=0x7f9ecc074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:44.806 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.798+0000 7f9eca7fc700 1 --2- 172.21.15.130:0/227622426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ecc100de0 0x7f9ecc074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:44.806 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.798+0000 7f9eca7fc700 1 -- 172.21.15.130:0/227622426 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec4009d00 con 0x7f9ecc101fe0 2024-03-30T00:53:44.806 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.798+0000 7f9eca7fc700 1 --2- 172.21.15.130:0/227622426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ecc101fe0 0x7f9ecc0731a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400c920 tx=0x7f9eb400cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:53:44.806 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.798+0000 7f9ec3fff700 1 -- 172.21.15.130:0/227622426 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4007b00 con 0x7f9ecc101fe0 2024-03-30T00:53:44.806 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.798+0000 7f9ec3fff700 1 -- 172.21.15.130:0/227622426 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9eb4004cf0 con 0x7f9ecc101fe0 2024-03-30T00:53:44.806 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.798+0000 7f9ec3fff700 1 -- 172.21.15.130:0/227622426 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4005610 con 0x7f9ecc101fe0 2024-03-30T00:53:44.806 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.798+0000 7f9ed1743700 1 -- 172.21.15.130:0/227622426 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc1a1180 con 0x7f9ecc101fe0 2024-03-30T00:53:44.806 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.798+0000 7f9ed1743700 1 -- 172.21.15.130:0/227622426 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc1a1680 con 0x7f9ecc101fe0 2024-03-30T00:53:44.806 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.799+0000 7f9ed1743700 1 -- 172.21.15.130:0/227622426 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc108270 con 0x7f9ecc101fe0 2024-03-30T00:53:44.807 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.800+0000 7f9ec3fff700 1 -- 172.21.15.130:0/227622426 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f9eb4004750 con 0x7f9ecc101fe0 2024-03-30T00:53:44.807 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.800+0000 7f9ec3fff700 1 --2- 172.21.15.130:0/227622426 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9eb80a11a0 0x7f9eb80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:53:44.807 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.801+0000 7f9ec3fff700 1 -- 172.21.15.130:0/227622426 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4685+0+0 (secure 0 0 0) 0x7f9eb40cba50 con 0x7f9ecc101fe0 2024-03-30T00:53:44.807 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.801+0000 7f9ecaffd700 1 --2- 172.21.15.130:0/227622426 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9eb80a11a0 0x7f9eb80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:53:44.807 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.801+0000 7f9ecaffd700 1 --2- 172.21.15.130:0/227622426 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9eb80a11a0 0x7f9eb80a3660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc006fd0 tx=0x7f9ebc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:53:44.809 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:44.804+0000 7f9ec3fff700 1 -- 172.21.15.130:0/227622426 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9eb4096090 con 0x7f9ecc101fe0 2024-03-30T00:53:44.924 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 00:53:44 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T00:53:44.656+0000 7f7464000700 -1 osd.6 0 waiting for initial osdmap 2024-03-30T00:53:44.924 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 00:53:44 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T00:53:44.677+0000 7f7459e00700 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T00:53:45.008 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:45.003+0000 7f9ed1743700 1 -- 172.21.15.130:0/227622426 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ecc02cd20 con 0x7f9eb80a11a0 2024-03-30T00:53:45.381 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-30T00:53:45.381 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: osdmap e41: 7 total, 6 up, 7 in 2024-03-30T00:53:45.381 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:45.381 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: Detected new or changed devices on smithi116 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi116 to 5756M 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:53:45.382 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:45 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:45.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-30T00:53:45.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: osdmap e41: 7 total, 6 up, 7 in 2024-03-30T00:53:45.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:45.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:45.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: Detected new or changed devices on smithi116 2024-03-30T00:53:45.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:45.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:45.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:45.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:45.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:45.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi116 to 5756M 2024-03-30T00:53:45.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:45.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:45.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:45.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:45.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' 2024-03-30T00:53:45.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:53:45.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:53:45.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:45 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:45.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-30T00:53:45.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: osdmap e41: 7 total, 6 up, 7 in 2024-03-30T00:53:45.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:45.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:45.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: Detected new or changed devices on smithi116 2024-03-30T00:53:45.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:45.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:45.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:45.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:45.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:53:45.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi116 to 5756M 2024-03-30T00:53:45.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:45.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:45.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:45.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:45.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130]' entity='osd.6' 2024-03-30T00:53:45.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:53:45.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:53:45.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:45 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:46 smithi116 ceph-mon[38086]: purged_snaps scrub starts 2024-03-30T00:53:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:46 smithi116 ceph-mon[38086]: purged_snaps scrub ok 2024-03-30T00:53:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:46 smithi116 ceph-mon[38086]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:53:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:46 smithi116 ceph-mon[38086]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:46 smithi116 ceph-mon[38086]: osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130] boot 2024-03-30T00:53:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:46 smithi116 ceph-mon[38086]: osdmap e42: 7 total, 7 up, 7 in 2024-03-30T00:53:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:46 smithi181 ceph-mon[47622]: purged_snaps scrub starts 2024-03-30T00:53:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:46 smithi181 ceph-mon[47622]: purged_snaps scrub ok 2024-03-30T00:53:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:46 smithi181 ceph-mon[47622]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:53:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:46 smithi181 ceph-mon[47622]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:46 smithi181 ceph-mon[47622]: osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130] boot 2024-03-30T00:53:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:46 smithi181 ceph-mon[47622]: osdmap e42: 7 total, 7 up, 7 in 2024-03-30T00:53:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:46.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:46 smithi130 ceph-mon[47626]: purged_snaps scrub starts 2024-03-30T00:53:46.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:46 smithi130 ceph-mon[47626]: purged_snaps scrub ok 2024-03-30T00:53:46.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:46 smithi130 ceph-mon[47626]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:53:46.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:46 smithi130 ceph-mon[47626]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-30T00:53:46.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:46.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:46 smithi130 ceph-mon[47626]: osd.6 [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130] boot 2024-03-30T00:53:46.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:46 smithi130 ceph-mon[47626]: osdmap e42: 7 total, 7 up, 7 in 2024-03-30T00:53:46.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T00:53:47.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:47 smithi116 ceph-mon[38086]: from='client.? 172.21.15.130:0/3146528385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11c9be8d-f9e3-4e20-8557-cb9153a894ed"}]: dispatch 2024-03-30T00:53:47.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:47 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11c9be8d-f9e3-4e20-8557-cb9153a894ed"}]: dispatch 2024-03-30T00:53:47.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:47 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11c9be8d-f9e3-4e20-8557-cb9153a894ed"}]': finished 2024-03-30T00:53:47.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:47 smithi116 ceph-mon[38086]: osdmap e43: 8 total, 7 up, 8 in 2024-03-30T00:53:47.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:47 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:53:47.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:47 smithi181 ceph-mon[47622]: from='client.? 172.21.15.130:0/3146528385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11c9be8d-f9e3-4e20-8557-cb9153a894ed"}]: dispatch 2024-03-30T00:53:47.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:47 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11c9be8d-f9e3-4e20-8557-cb9153a894ed"}]: dispatch 2024-03-30T00:53:47.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:47 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11c9be8d-f9e3-4e20-8557-cb9153a894ed"}]': finished 2024-03-30T00:53:47.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:47 smithi181 ceph-mon[47622]: osdmap e43: 8 total, 7 up, 8 in 2024-03-30T00:53:47.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:47 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:53:47.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:47 smithi130 ceph-mon[47626]: from='client.? 172.21.15.130:0/3146528385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11c9be8d-f9e3-4e20-8557-cb9153a894ed"}]: dispatch 2024-03-30T00:53:47.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:47 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11c9be8d-f9e3-4e20-8557-cb9153a894ed"}]: dispatch 2024-03-30T00:53:47.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:47 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11c9be8d-f9e3-4e20-8557-cb9153a894ed"}]': finished 2024-03-30T00:53:47.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:47 smithi130 ceph-mon[47626]: osdmap e43: 8 total, 7 up, 8 in 2024-03-30T00:53:47.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:47 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:53:48.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:48 smithi116 ceph-mon[38086]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:48.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:48 smithi116 ceph-mon[38086]: from='client.? 172.21.15.130:0/2269730815' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:53:48.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:48 smithi181 ceph-mon[47622]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:48.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:48 smithi181 ceph-mon[47622]: from='client.? 172.21.15.130:0/2269730815' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:53:48.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:48 smithi130 ceph-mon[47626]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:48.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:48 smithi130 ceph-mon[47626]: from='client.? 172.21.15.130:0/2269730815' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:53:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:50 smithi116 ceph-mon[38086]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:50 smithi181 ceph-mon[47622]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:50.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:50 smithi130 ceph-mon[47626]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 203 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:52.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:52 smithi116 ceph-mon[38086]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:52.500 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:52 smithi130 ceph-mon[47626]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:52.501 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:52 smithi181 ceph-mon[47622]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T00:53:53.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:53.456 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T00:53:53.457 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:53.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T00:53:53.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:54.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:54 smithi130 ceph-mon[47626]: Deploying daemon osd.7 on smithi130 2024-03-30T00:53:54.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:54 smithi130 ceph-mon[47626]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:54.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:54 smithi116 ceph-mon[38086]: Deploying daemon osd.7 on smithi130 2024-03-30T00:53:54.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:54 smithi116 ceph-mon[38086]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:54.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:54 smithi181 ceph-mon[47622]: Deploying daemon osd.7 on smithi130 2024-03-30T00:53:54.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:54 smithi181 ceph-mon[47622]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:56.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:56 smithi116 ceph-mon[38086]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:56.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:56 smithi181 ceph-mon[47622]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:56.504 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:56 smithi130 ceph-mon[47626]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:58.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:57 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:58.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:57 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:58.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:57 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:58.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:57 smithi181 ceph-mon[47622]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:58.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:57 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:58.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:57 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:58.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:57 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:58.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:57 smithi130 ceph-mon[47626]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:58.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:57 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:58.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:57 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:58.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:57 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:58.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:57 smithi116 ceph-mon[38086]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:53:59.013 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:59.006+0000 7f9ec3fff700 1 -- 172.21.15.130:0/227622426 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9ecc02cd20 con 0x7f9eb80a11a0 2024-03-30T00:53:59.013 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 7 on host 'smithi130' 2024-03-30T00:53:59.015 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:59.008+0000 7f9ed1743700 1 -- 172.21.15.130:0/227622426 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9eb80a11a0 msgr2=0x7f9eb80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:59.015 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:59.008+0000 7f9ed1743700 1 --2- 172.21.15.130:0/227622426 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9eb80a11a0 0x7f9eb80a3660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc006fd0 tx=0x7f9ebc008040 comp rx=0 tx=0).stop 2024-03-30T00:53:59.015 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:59.008+0000 7f9ed1743700 1 -- 172.21.15.130:0/227622426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ecc101fe0 msgr2=0x7f9ecc0731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:53:59.015 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:59.008+0000 7f9ed1743700 1 --2- 172.21.15.130:0/227622426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ecc101fe0 0x7f9ecc0731a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400c920 tx=0x7f9eb400cce0 comp rx=0 tx=0).stop 2024-03-30T00:53:59.015 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:59.008+0000 7f9ed1743700 1 -- 172.21.15.130:0/227622426 shutdown_connections 2024-03-30T00:53:59.015 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:59.008+0000 7f9ed1743700 1 --2- 172.21.15.130:0/227622426 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9eb80a11a0 0x7f9eb80a3660 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:59.015 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:59.008+0000 7f9ed1743700 1 --2- 172.21.15.130:0/227622426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ecc102980 0x7f9ecc0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:59.015 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:59.008+0000 7f9ed1743700 1 --2- 172.21.15.130:0/227622426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ecc100de0 0x7f9ecc074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:59.015 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:59.008+0000 7f9ed1743700 1 --2- 172.21.15.130:0/227622426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ecc101fe0 0x7f9ecc0731a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:53:59.015 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:59.008+0000 7f9ed1743700 1 -- 172.21.15.130:0/227622426 >> 172.21.15.130:0/227622426 conn(0x7f9ecc0fc3c0 msgr2=0x7f9ecc1053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:53:59.016 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:59.008+0000 7f9ed1743700 1 -- 172.21.15.130:0/227622426 shutdown_connections 2024-03-30T00:53:59.016 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:53:59.008+0000 7f9ed1743700 1 -- 172.21.15.130:0/227622426 wait complete. 2024-03-30T00:53:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:59 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:59 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:59 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:59 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:59 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:59 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:59 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:53:59 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:59.495 DEBUG:teuthology.orchestra.run.smithi130:osd.7> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.7.service 2024-03-30T00:53:59.497 INFO:tasks.cephadm:Deploying osd.8 on smithi181 with /dev/vg_nvme/lv_2... 2024-03-30T00:53:59.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T00:53:59.540 INFO:journalctl@ceph.osd.7.smithi130.stdout:-- Logs begin at Sat 2024-03-30 00:27:58 UTC. -- 2024-03-30T00:53:59.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:59 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:59 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:59 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:59.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:59 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:59.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:59 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:59 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:59 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:53:59 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:53:59.739 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T00:53:59.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:59 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:59 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:59 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:53:59.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:59 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:53:59.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:59 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:59 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:59 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:53:59.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:53:59 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:00.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:00 smithi181 ceph-mon[47622]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:54:00.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:00 smithi130 ceph-mon[47626]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:54:00.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:00 smithi116 ceph-mon[38086]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:54:01.053 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T00:54:01.053 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T00:54:01.053 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-03-30T00:54:01.053 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-03-30T00:54:01.053 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112509 s, 932 MB/s 2024-03-30T00:54:01.053 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-03-30T00:54:01.094 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 00:54:00 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T00:54:00.694+0000 7fd318c46640 -1 osd.7 0 log_to_monitors true 2024-03-30T00:54:01.449 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:01 smithi130 ceph-mon[47626]: from='osd.7 [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T00:54:01.449 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:01 smithi130 ceph-mon[47626]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T00:54:01.457 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi181:vg_nvme/lv_2 2024-03-30T00:54:01.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:01 smithi116 ceph-mon[38086]: from='osd.7 [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T00:54:01.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:01 smithi116 ceph-mon[38086]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T00:54:01.676 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T00:54:01.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:01 smithi181 ceph-mon[47622]: from='osd.7 [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T00:54:01.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:01 smithi181 ceph-mon[47622]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T00:54:02.372 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:54:02.372 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T00:54:02.372 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: osdmap e44: 8 total, 7 up, 8 in 2024-03-30T00:54:02.372 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:02.373 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: from='osd.7 [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:54:02.373 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:54:02.373 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: Detected new or changed devices on smithi130 2024-03-30T00:54:02.373 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:02.373 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:02.373 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:02.373 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:02.373 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:02.373 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi130 to 5704M 2024-03-30T00:54:02.373 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:02.373 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:02.373 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:02.374 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:02.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.633+0000 7ff9d6948700 1 -- 172.21.15.181:0/1279776983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9d0100ef0 msgr2=0x7ff9d0101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:02.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.633+0000 7ff9d6948700 1 --2- 172.21.15.181:0/1279776983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9d0100ef0 0x7ff9d0101350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009990 tx=0x7ff9c0009ca0 comp rx=0 tx=0).stop 2024-03-30T00:54:02.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.633+0000 7ff9d6948700 1 -- 172.21.15.181:0/1279776983 shutdown_connections 2024-03-30T00:54:02.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.633+0000 7ff9d6948700 1 --2- 172.21.15.181:0/1279776983 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9d0101890 0x7ff9d0108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:02.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.633+0000 7ff9d6948700 1 --2- 172.21.15.181:0/1279776983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9d00ffcf0 0x7ff9d0100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:02.642 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.633+0000 7ff9d6948700 1 --2- 172.21.15.181:0/1279776983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9d0100ef0 0x7ff9d0101350 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:02.642 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.633+0000 7ff9d6948700 1 -- 172.21.15.181:0/1279776983 >> 172.21.15.181:0/1279776983 conn(0x7ff9d00fb2b0 msgr2=0x7ff9d00fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:54:02.642 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.633+0000 7ff9d6948700 1 -- 172.21.15.181:0/1279776983 shutdown_connections 2024-03-30T00:54:02.642 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.633+0000 7ff9d6948700 1 -- 172.21.15.181:0/1279776983 wait complete. 2024-03-30T00:54:02.642 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.633+0000 7ff9d6948700 1 Processor -- start 2024-03-30T00:54:02.642 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.633+0000 7ff9d6948700 1 -- start start 2024-03-30T00:54:02.642 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9d6948700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9d00ffcf0 0x7ff9d0103c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:02.642 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9d6948700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9d0100ef0 0x7ff9d01022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:02.642 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9d6948700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9d0101890 0x7ff9d0102810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:02.642 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9d6948700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0102fb0 con 0x7ff9d0101890 2024-03-30T00:54:02.642 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9d6948700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0103120 con 0x7ff9d00ffcf0 2024-03-30T00:54:02.642 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9d6948700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d01a0e10 con 0x7ff9d0100ef0 2024-03-30T00:54:02.642 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9cf7fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9d0100ef0 0x7ff9d01022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9cf7fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9d0100ef0 0x7ff9d01022d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:57882/0 (socket says 172.21.15.181:57882) 2024-03-30T00:54:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9cf7fe700 1 -- 172.21.15.181:0/2718379334 learned_addr learned my addr 172.21.15.181:0/2718379334 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-03-30T00:54:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9cffff700 1 --2- 172.21.15.181:0/2718379334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9d00ffcf0 0x7ff9d0103c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9cf7fe700 1 -- 172.21.15.181:0/2718379334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9d00ffcf0 msgr2=0x7ff9d0103c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9cf7fe700 1 --2- 172.21.15.181:0/2718379334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9d00ffcf0 0x7ff9d0103c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9cf7fe700 1 -- 172.21.15.181:0/2718379334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9d0101890 msgr2=0x7ff9d0102810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9cf7fe700 1 --2- 172.21.15.181:0/2718379334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9d0101890 0x7ff9d0102810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9cf7fe700 1 -- 172.21.15.181:0/2718379334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c00095f0 con 0x7ff9d0100ef0 2024-03-30T00:54:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9cf7fe700 1 --2- 172.21.15.181:0/2718379334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9d0100ef0 0x7ff9d01022d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000b5c0 tx=0x7ff9c000f7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:54:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9cd7fa700 1 -- 172.21.15.181:0/2718379334 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c001d070 con 0x7ff9d0100ef0 2024-03-30T00:54:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9d6948700 1 -- 172.21.15.181:0/2718379334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d01a1040 con 0x7ff9d0100ef0 2024-03-30T00:54:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9cd7fa700 1 -- 172.21.15.181:0/2718379334 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9c0003e20 con 0x7ff9d0100ef0 2024-03-30T00:54:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9cd7fa700 1 -- 172.21.15.181:0/2718379334 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0017650 con 0x7ff9d0100ef0 2024-03-30T00:54:02.643 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.634+0000 7ff9d6948700 1 -- 172.21.15.181:0/2718379334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d01a14b0 con 0x7ff9d0100ef0 2024-03-30T00:54:02.644 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.635+0000 7ff9d6948700 1 -- 172.21.15.181:0/2718379334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d004ea80 con 0x7ff9d0100ef0 2024-03-30T00:54:02.645 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.636+0000 7ff9cd7fa700 1 -- 172.21.15.181:0/2718379334 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7ff9c0003960 con 0x7ff9d0100ef0 2024-03-30T00:54:02.645 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.637+0000 7ff9cd7fa700 1 --2- 172.21.15.181:0/2718379334 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff9b80a1120 0x7ff9b80a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:02.645 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.637+0000 7ff9cd7fa700 1 -- 172.21.15.181:0/2718379334 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5117+0+0 (secure 0 0 0) 0x7ff9c00cd060 con 0x7ff9d0100ef0 2024-03-30T00:54:02.645 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.637+0000 7ff9cffff700 1 --2- 172.21.15.181:0/2718379334 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff9b80a1120 0x7ff9b80a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:02.645 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.638+0000 7ff9cffff700 1 --2- 172.21.15.181:0/2718379334 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff9b80a1120 0x7ff9b80a35e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4006fd0 tx=0x7ff9c4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:54:02.648 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.642+0000 7ff9cd7fa700 1 -- 172.21.15.181:0/2718379334 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff9c0097500 con 0x7ff9d0100ef0 2024-03-30T00:54:02.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:54:02.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T00:54:02.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: osdmap e44: 8 total, 7 up, 8 in 2024-03-30T00:54:02.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:02.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: from='osd.7 [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:54:02.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:54:02.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: Detected new or changed devices on smithi130 2024-03-30T00:54:02.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:02.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:02.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:02.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:02.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:02.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi130 to 5704M 2024-03-30T00:54:02.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:02.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:02.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:02.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:02.845 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T00:54:02.371+0000 7fd30e2aa700 -1 osd.7 0 waiting for initial osdmap 2024-03-30T00:54:02.845 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T00:54:02.382+0000 7fd30a09f700 -1 osd.7 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T00:54:02.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:54:02.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T00:54:02.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: osdmap e44: 8 total, 7 up, 8 in 2024-03-30T00:54:02.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:02.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: from='osd.7 [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:54:02.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:54:02.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: Detected new or changed devices on smithi130 2024-03-30T00:54:02.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:02.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:02.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:02.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:02.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:02.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi130 to 5704M 2024-03-30T00:54:02.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:02.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:02.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:02.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:02.848 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:02.842+0000 7ff9d6948700 1 -- 172.21.15.181:0/2718379334 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9d0069710 con 0x7ff9b80a1120 2024-03-30T00:54:03.477 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:03 smithi181 ceph-mon[47622]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-30T00:54:03.477 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:03 smithi181 ceph-mon[47622]: osdmap e45: 8 total, 7 up, 8 in 2024-03-30T00:54:03.477 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:03.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:03.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:54:03.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:54:03.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:03.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:03 smithi116 ceph-mon[38086]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-30T00:54:03.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:03 smithi116 ceph-mon[38086]: osdmap e45: 8 total, 7 up, 8 in 2024-03-30T00:54:03.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:03.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:03.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:54:03.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:54:03.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:03.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:03 smithi130 ceph-mon[47626]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-30T00:54:03.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:03 smithi130 ceph-mon[47626]: osdmap e45: 8 total, 7 up, 8 in 2024-03-30T00:54:03.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:03.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:03.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:54:03.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:54:03.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:04.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:04 smithi116 ceph-mon[38086]: purged_snaps scrub starts 2024-03-30T00:54:04.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:04 smithi116 ceph-mon[38086]: purged_snaps scrub ok 2024-03-30T00:54:04.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:04 smithi116 ceph-mon[38086]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:54:04.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:04 smithi116 ceph-mon[38086]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:54:04.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:04 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:04.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:04 smithi116 ceph-mon[38086]: osd.7 [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701] boot 2024-03-30T00:54:04.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:04 smithi116 ceph-mon[38086]: osdmap e46: 8 total, 8 up, 8 in 2024-03-30T00:54:04.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:04 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:04.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:04 smithi181 ceph-mon[47622]: purged_snaps scrub starts 2024-03-30T00:54:04.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:04 smithi181 ceph-mon[47622]: purged_snaps scrub ok 2024-03-30T00:54:04.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:04 smithi181 ceph-mon[47622]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:54:04.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:04 smithi181 ceph-mon[47622]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:54:04.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:04 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:04.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:04 smithi181 ceph-mon[47622]: osd.7 [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701] boot 2024-03-30T00:54:04.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:04 smithi181 ceph-mon[47622]: osdmap e46: 8 total, 8 up, 8 in 2024-03-30T00:54:04.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:04 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:04.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:04 smithi130 ceph-mon[47626]: purged_snaps scrub starts 2024-03-30T00:54:04.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:04 smithi130 ceph-mon[47626]: purged_snaps scrub ok 2024-03-30T00:54:04.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:04 smithi130 ceph-mon[47626]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:54:04.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:04 smithi130 ceph-mon[47626]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-30T00:54:04.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:04 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:04.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:04 smithi130 ceph-mon[47626]: osd.7 [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701] boot 2024-03-30T00:54:04.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:04 smithi130 ceph-mon[47626]: osdmap e46: 8 total, 8 up, 8 in 2024-03-30T00:54:04.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:04 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T00:54:05.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:05 smithi116 ceph-mon[38086]: osdmap e47: 8 total, 8 up, 8 in 2024-03-30T00:54:05.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:05 smithi116 ceph-mon[38086]: from='client.? 172.21.15.181:0/174236298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78423339-0c27-4d28-bde2-440127f4730d"}]: dispatch 2024-03-30T00:54:05.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:05 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78423339-0c27-4d28-bde2-440127f4730d"}]: dispatch 2024-03-30T00:54:05.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:05 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78423339-0c27-4d28-bde2-440127f4730d"}]': finished 2024-03-30T00:54:05.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:05 smithi116 ceph-mon[38086]: osdmap e48: 9 total, 8 up, 9 in 2024-03-30T00:54:05.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:05 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:05.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:05 smithi116 ceph-mon[38086]: from='client.? 172.21.15.181:0/398873174' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:54:05.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:05 smithi181 ceph-mon[47622]: osdmap e47: 8 total, 8 up, 8 in 2024-03-30T00:54:05.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:05 smithi181 ceph-mon[47622]: from='client.? 172.21.15.181:0/174236298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78423339-0c27-4d28-bde2-440127f4730d"}]: dispatch 2024-03-30T00:54:05.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:05 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78423339-0c27-4d28-bde2-440127f4730d"}]: dispatch 2024-03-30T00:54:05.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:05 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78423339-0c27-4d28-bde2-440127f4730d"}]': finished 2024-03-30T00:54:05.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:05 smithi181 ceph-mon[47622]: osdmap e48: 9 total, 8 up, 9 in 2024-03-30T00:54:05.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:05 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:05.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:05 smithi181 ceph-mon[47622]: from='client.? 172.21.15.181:0/398873174' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:54:05.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:05 smithi130 ceph-mon[47626]: osdmap e47: 8 total, 8 up, 8 in 2024-03-30T00:54:05.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:05 smithi130 ceph-mon[47626]: from='client.? 172.21.15.181:0/174236298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78423339-0c27-4d28-bde2-440127f4730d"}]: dispatch 2024-03-30T00:54:05.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:05 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78423339-0c27-4d28-bde2-440127f4730d"}]: dispatch 2024-03-30T00:54:05.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:05 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78423339-0c27-4d28-bde2-440127f4730d"}]': finished 2024-03-30T00:54:05.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:05 smithi130 ceph-mon[47626]: osdmap e48: 9 total, 8 up, 9 in 2024-03-30T00:54:05.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:05 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:05.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:05 smithi130 ceph-mon[47626]: from='client.? 172.21.15.181:0/398873174' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:54:06.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:06 smithi116 ceph-mon[38086]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:06.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:06 smithi181 ceph-mon[47622]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:06 smithi130 ceph-mon[47626]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:07.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:07 smithi116 ceph-mon[38086]: osdmap e49: 9 total, 8 up, 9 in 2024-03-30T00:54:07.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:07.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:07 smithi181 ceph-mon[47622]: osdmap e49: 9 total, 8 up, 9 in 2024-03-30T00:54:07.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:07.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:07 smithi130 ceph-mon[47626]: osdmap e49: 9 total, 8 up, 9 in 2024-03-30T00:54:07.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:08.613 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:08 smithi181 ceph-mon[47622]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T00:54:08.614 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:08 smithi181 ceph-mon[47622]: osdmap e50: 9 total, 8 up, 9 in 2024-03-30T00:54:08.614 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:08 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:08.614 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:08 smithi181 ceph-mon[47622]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-30T00:54:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:08 smithi116 ceph-mon[38086]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T00:54:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:08 smithi116 ceph-mon[38086]: osdmap e50: 9 total, 8 up, 9 in 2024-03-30T00:54:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:08 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:08 smithi116 ceph-mon[38086]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-30T00:54:08.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:08 smithi130 ceph-mon[47626]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T00:54:08.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:08 smithi130 ceph-mon[47626]: osdmap e50: 9 total, 8 up, 9 in 2024-03-30T00:54:08.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:08 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:08.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:08 smithi130 ceph-mon[47626]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-30T00:54:10.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:10 smithi181 ceph-mon[47622]: pgmap v147: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T00:54:10.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:10 smithi116 ceph-mon[38086]: pgmap v147: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T00:54:10.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:10 smithi130 ceph-mon[47626]: pgmap v147: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T00:54:11.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:11 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-30T00:54:11.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:11 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:11.692 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:11 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-30T00:54:11.692 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:11 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:11.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:11 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-30T00:54:11.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:11 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:12.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:12 smithi116 ceph-mon[38086]: Deploying daemon osd.8 on smithi181 2024-03-30T00:54:12.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:12 smithi116 ceph-mon[38086]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:12.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:12 smithi116 ceph-mon[38086]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-30T00:54:12.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:12 smithi116 ceph-mon[38086]: Cluster is now healthy 2024-03-30T00:54:12.700 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:12 smithi181 ceph-mon[47622]: Deploying daemon osd.8 on smithi181 2024-03-30T00:54:12.701 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:12 smithi181 ceph-mon[47622]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:12.701 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:12 smithi181 ceph-mon[47622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-30T00:54:12.701 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:12 smithi181 ceph-mon[47622]: Cluster is now healthy 2024-03-30T00:54:12.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:12 smithi130 ceph-mon[47626]: Deploying daemon osd.8 on smithi181 2024-03-30T00:54:12.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:12 smithi130 ceph-mon[47626]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:12.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:12 smithi130 ceph-mon[47626]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-30T00:54:12.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:12 smithi130 ceph-mon[47626]: Cluster is now healthy 2024-03-30T00:54:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:14 smithi181 ceph-mon[47622]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:14 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:14 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:14 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:14.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:14 smithi116 ceph-mon[38086]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:14.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:14 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:14.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:14 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:14.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:14 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:14.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:14 smithi130 ceph-mon[47626]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:14.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:14.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:14.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:14 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:16.505 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 8 on host 'smithi181' 2024-03-30T00:54:16.505 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:16.498+0000 7ff9cd7fa700 1 -- 172.21.15.181:0/2718379334 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff9d0069710 con 0x7ff9b80a1120 2024-03-30T00:54:16.507 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:16.501+0000 7ff9d6948700 1 -- 172.21.15.181:0/2718379334 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff9b80a1120 msgr2=0x7ff9b80a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:16.507 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:16.501+0000 7ff9d6948700 1 --2- 172.21.15.181:0/2718379334 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff9b80a1120 0x7ff9b80a35e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4006fd0 tx=0x7ff9c4008040 comp rx=0 tx=0).stop 2024-03-30T00:54:16.507 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:16.501+0000 7ff9d6948700 1 -- 172.21.15.181:0/2718379334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9d0100ef0 msgr2=0x7ff9d01022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:16.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:16.501+0000 7ff9d6948700 1 --2- 172.21.15.181:0/2718379334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9d0100ef0 0x7ff9d01022d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000b5c0 tx=0x7ff9c000f7b0 comp rx=0 tx=0).stop 2024-03-30T00:54:16.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:16.501+0000 7ff9d6948700 1 -- 172.21.15.181:0/2718379334 shutdown_connections 2024-03-30T00:54:16.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:16.501+0000 7ff9d6948700 1 --2- 172.21.15.181:0/2718379334 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff9b80a1120 0x7ff9b80a35e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:16.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:16.501+0000 7ff9d6948700 1 --2- 172.21.15.181:0/2718379334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9d0101890 0x7ff9d0102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:16.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:16.501+0000 7ff9d6948700 1 --2- 172.21.15.181:0/2718379334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9d00ffcf0 0x7ff9d0103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:16.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:16.501+0000 7ff9d6948700 1 --2- 172.21.15.181:0/2718379334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9d0100ef0 0x7ff9d01022d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:16.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:16.501+0000 7ff9d6948700 1 -- 172.21.15.181:0/2718379334 >> 172.21.15.181:0/2718379334 conn(0x7ff9d00fb2b0 msgr2=0x7ff9d0069000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:54:16.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:16.501+0000 7ff9d6948700 1 -- 172.21.15.181:0/2718379334 shutdown_connections 2024-03-30T00:54:16.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:16.501+0000 7ff9d6948700 1 -- 172.21.15.181:0/2718379334 wait complete. 2024-03-30T00:54:16.666 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:16 smithi181 ceph-mon[47622]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:16.666 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:16 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:16.666 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:16 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:16.667 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:16 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:16.667 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:16 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:16.667 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:16 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:16.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:16 smithi116 ceph-mon[38086]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:16.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:16 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:16.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:16 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:16.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:16 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:16.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:16 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:16.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:16 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:16.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:16 smithi130 ceph-mon[47626]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:16.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:16 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:16.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:16 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:16.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:16 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:16.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:16 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:16.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:16 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:16.989 DEBUG:teuthology.orchestra.run.smithi181:osd.8> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.8.service 2024-03-30T00:54:16.991 INFO:tasks.cephadm:Deploying osd.9 on smithi116 with /dev/vg_nvme/lv_1... 2024-03-30T00:54:16.991 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-30T00:54:17.022 INFO:journalctl@ceph.osd.8.smithi181.stdout:-- Logs begin at Sat 2024-03-30 00:27:51 UTC. -- 2024-03-30T00:54:17.374 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:54:17.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:17 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:17.603 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:17 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:17.603 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:17 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:17.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:17 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:17.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:17 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:17.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:17 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:17.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:17 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:17.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:17 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:17.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:17 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:18.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:18 smithi116 ceph-mon[38086]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:18.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:18 smithi116 ceph-mon[38086]: from='osd.8 [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-30T00:54:18.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:18 smithi116 ceph-mon[38086]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-30T00:54:18.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:18 smithi181 ceph-mon[47622]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:18.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:18 smithi181 ceph-mon[47622]: from='osd.8 [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-30T00:54:18.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:18 smithi181 ceph-mon[47622]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-30T00:54:18.753 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 00:54:18 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T00:54:18.269+0000 7f32f9074640 -1 osd.8 0 log_to_monitors true 2024-03-30T00:54:18.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:18 smithi130 ceph-mon[47626]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:18.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:18 smithi130 ceph-mon[47626]: from='osd.8 [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-30T00:54:18.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:18 smithi130 ceph-mon[47626]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-30T00:54:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-30T00:54:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-30T00:54:19.059 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-30T00:54:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-30T00:54:19.059 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.037415 s, 280 MB/s 2024-03-30T00:54:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-30T00:54:19.480 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi116:vg_nvme/lv_1 2024-03-30T00:54:19.752 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T00:54:19.473+0000 7f32ee6d8700 -1 osd.8 0 waiting for initial osdmap 2024-03-30T00:54:19.752 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T00:54:19.483+0000 7f32ea4cd700 -1 osd.8 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T00:54:19.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-mon[47622]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-30T00:54:19.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-mon[47622]: osdmap e51: 9 total, 8 up, 9 in 2024-03-30T00:54:19.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:19.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-mon[47622]: from='osd.8 [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:54:19.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-mon[47622]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:54:19.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:19.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:19.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:19.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:19.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:19.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:19.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:19.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:19.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:19 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:54:19.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:19 smithi130 ceph-mon[47626]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-30T00:54:19.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:19 smithi130 ceph-mon[47626]: osdmap e51: 9 total, 8 up, 9 in 2024-03-30T00:54:19.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:19 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:19.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:19 smithi130 ceph-mon[47626]: from='osd.8 [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:54:19.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:19 smithi130 ceph-mon[47626]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:54:19.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:19 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:19.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:19 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:19.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:19 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:19.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:19 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:19.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:19 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:19.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:19 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:19.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:19 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:19.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:19 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:19.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:19 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:19.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:19 smithi116 ceph-mon[38086]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-30T00:54:19.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:19 smithi116 ceph-mon[38086]: osdmap e51: 9 total, 8 up, 9 in 2024-03-30T00:54:19.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:19 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:19.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:19 smithi116 ceph-mon[38086]: from='osd.8 [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:54:19.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:19 smithi116 ceph-mon[38086]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:54:19.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:19 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:19.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:19 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:19.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:19 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:19.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:19 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:19.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:19 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:19.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:19 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:19.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:19 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:19.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:19 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:19.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:19 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:20.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:20 smithi116 ceph-mon[38086]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:20.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:20 smithi116 ceph-mon[38086]: Detected new or changed devices on smithi181 2024-03-30T00:54:20.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:20 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi181 to 5704M 2024-03-30T00:54:20.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:20 smithi116 ceph-mon[38086]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-30T00:54:20.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:20 smithi116 ceph-mon[38086]: osdmap e52: 9 total, 8 up, 9 in 2024-03-30T00:54:20.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:20.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:20 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:20.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:20 smithi181 ceph-mon[47622]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:20.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:20 smithi181 ceph-mon[47622]: Detected new or changed devices on smithi181 2024-03-30T00:54:20.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:20 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi181 to 5704M 2024-03-30T00:54:20.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:20 smithi181 ceph-mon[47622]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-30T00:54:20.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:20 smithi181 ceph-mon[47622]: osdmap e52: 9 total, 8 up, 9 in 2024-03-30T00:54:20.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:20.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:20 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:20 smithi130 ceph-mon[47626]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-30T00:54:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:20 smithi130 ceph-mon[47626]: Detected new or changed devices on smithi181 2024-03-30T00:54:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:20 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi181 to 5704M 2024-03-30T00:54:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:20 smithi130 ceph-mon[47626]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-30T00:54:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:20 smithi130 ceph-mon[47626]: osdmap e52: 9 total, 8 up, 9 in 2024-03-30T00:54:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:20 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:20 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:20.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.911+0000 7fb679400700 1 -- 172.21.15.116:0/3987621844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674100e00 msgr2=0x7fb674101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:20.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.911+0000 7fb679400700 1 --2- 172.21.15.116:0/3987621844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674100e00 0x7fb674101220 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb664009990 tx=0x7fb664009ca0 comp rx=0 tx=0).stop 2024-03-30T00:54:20.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.911+0000 7fb679400700 1 -- 172.21.15.116:0/3987621844 shutdown_connections 2024-03-30T00:54:20.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.911+0000 7fb679400700 1 --2- 172.21.15.116:0/3987621844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb674102920 0x7fb674109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:20.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.911+0000 7fb679400700 1 --2- 172.21.15.116:0/3987621844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674100e00 0x7fb674101220 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:20.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.911+0000 7fb679400700 1 --2- 172.21.15.116:0/3987621844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb674101f60 0x7fb6741023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:20.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.911+0000 7fb679400700 1 -- 172.21.15.116:0/3987621844 >> 172.21.15.116:0/3987621844 conn(0x7fb6740fc3c0 msgr2=0x7fb6740fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:54:20.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.911+0000 7fb679400700 1 -- 172.21.15.116:0/3987621844 shutdown_connections 2024-03-30T00:54:20.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.911+0000 7fb679400700 1 -- 172.21.15.116:0/3987621844 wait complete. 2024-03-30T00:54:20.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.912+0000 7fb679400700 1 Processor -- start 2024-03-30T00:54:20.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.912+0000 7fb679400700 1 -- start start 2024-03-30T00:54:20.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.912+0000 7fb679400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb674100e00 0x7fb674198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:20.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.912+0000 7fb679400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674101f60 0x7fb6741994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:20.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.912+0000 7fb679400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb674102920 0x7fb6741a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:20.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.912+0000 7fb679400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741a0a90 con 0x7fb674101f60 2024-03-30T00:54:20.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.912+0000 7fb679400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741a0bd0 con 0x7fb674100e00 2024-03-30T00:54:20.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.912+0000 7fb679400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741a0d60 con 0x7fb674102920 2024-03-30T00:54:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.913+0000 7fb671e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674101f60 0x7fb6741994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.913+0000 7fb672800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb674100e00 0x7fb674198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.913+0000 7fb672800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb674100e00 0x7fb674198fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:37236/0 (socket says 172.21.15.116:37236) 2024-03-30T00:54:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.913+0000 7fb672800700 1 -- 172.21.15.116:0/4162543297 learned_addr learned my addr 172.21.15.116:0/4162543297 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:54:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.913+0000 7fb673200700 1 --2- 172.21.15.116:0/4162543297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb674102920 0x7fb6741a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.913+0000 7fb671e00700 1 -- 172.21.15.116:0/4162543297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb674102920 msgr2=0x7fb6741a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.913+0000 7fb671e00700 1 --2- 172.21.15.116:0/4162543297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb674102920 0x7fb6741a04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.913+0000 7fb671e00700 1 -- 172.21.15.116:0/4162543297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb674100e00 msgr2=0x7fb674198fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.913+0000 7fb671e00700 1 --2- 172.21.15.116:0/4162543297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb674100e00 0x7fb674198fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.913+0000 7fb671e00700 1 -- 172.21.15.116:0/4162543297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6640095f0 con 0x7fb674101f60 2024-03-30T00:54:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.913+0000 7fb671e00700 1 --2- 172.21.15.116:0/4162543297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674101f60 0x7fb6741994e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb66000ea20 tx=0x7fb66000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:54:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.914+0000 7fb66b400700 1 -- 172.21.15.116:0/4162543297 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66000cbd0 con 0x7fb674101f60 2024-03-30T00:54:20.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.914+0000 7fb66b400700 1 -- 172.21.15.116:0/4162543297 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb66000ee60 con 0x7fb674101f60 2024-03-30T00:54:20.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.914+0000 7fb679400700 1 -- 172.21.15.116:0/4162543297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6741a0fe0 con 0x7fb674101f60 2024-03-30T00:54:20.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.914+0000 7fb66b400700 1 -- 172.21.15.116:0/4162543297 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb660018680 con 0x7fb674101f60 2024-03-30T00:54:20.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.914+0000 7fb679400700 1 -- 172.21.15.116:0/4162543297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6741a14b0 con 0x7fb674101f60 2024-03-30T00:54:20.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.915+0000 7fb679400700 1 -- 172.21.15.116:0/4162543297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67404ea80 con 0x7fb674101f60 2024-03-30T00:54:20.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.917+0000 7fb66b400700 1 -- 172.21.15.116:0/4162543297 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fb660010430 con 0x7fb674101f60 2024-03-30T00:54:20.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.918+0000 7fb66b400700 1 --2- 172.21.15.116:0/4162543297 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb6580a1120 0x7fb6580a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:20.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.918+0000 7fb66b400700 1 -- 172.21.15.116:0/4162543297 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5862+0+0 (secure 0 0 0) 0x7fb660010e40 con 0x7fb674101f60 2024-03-30T00:54:20.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.918+0000 7fb672800700 1 --2- 172.21.15.116:0/4162543297 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb6580a1120 0x7fb6580a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:20.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.919+0000 7fb672800700 1 --2- 172.21.15.116:0/4162543297 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb6580a1120 0x7fb6580a35e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb66400b2e0 tx=0x7fb66400b6f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:54:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:20.921+0000 7fb66b400700 1 -- 172.21.15.116:0/4162543297 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb6600964e0 con 0x7fb674101f60 2024-03-30T00:54:21.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:21.132+0000 7fb679400700 1 -- 172.21.15.116:0/4162543297 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb674074700 con 0x7fb6580a1120 2024-03-30T00:54:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:21 smithi181 ceph-mon[47622]: purged_snaps scrub starts 2024-03-30T00:54:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:21 smithi181 ceph-mon[47622]: purged_snaps scrub ok 2024-03-30T00:54:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:21 smithi181 ceph-mon[47622]: osd.8 [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389] boot 2024-03-30T00:54:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:21 smithi181 ceph-mon[47622]: osdmap e53: 9 total, 9 up, 9 in 2024-03-30T00:54:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:54:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:54:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:21 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:21.756 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:21 smithi116 ceph-mon[38086]: purged_snaps scrub starts 2024-03-30T00:54:21.756 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:21 smithi116 ceph-mon[38086]: purged_snaps scrub ok 2024-03-30T00:54:21.756 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:21 smithi116 ceph-mon[38086]: osd.8 [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389] boot 2024-03-30T00:54:21.756 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:21 smithi116 ceph-mon[38086]: osdmap e53: 9 total, 9 up, 9 in 2024-03-30T00:54:21.756 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:21.756 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:54:21.756 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:54:21.756 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:21 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:21.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:21 smithi130 ceph-mon[47626]: purged_snaps scrub starts 2024-03-30T00:54:21.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:21 smithi130 ceph-mon[47626]: purged_snaps scrub ok 2024-03-30T00:54:21.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:21 smithi130 ceph-mon[47626]: osd.8 [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389] boot 2024-03-30T00:54:21.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:21 smithi130 ceph-mon[47626]: osdmap e53: 9 total, 9 up, 9 in 2024-03-30T00:54:21.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T00:54:21.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:54:21.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:54:21.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:21 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:22.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:22 smithi116 ceph-mon[38086]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:22.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:22 smithi116 ceph-mon[38086]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:54:22.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:22 smithi116 ceph-mon[38086]: osdmap e54: 9 total, 9 up, 9 in 2024-03-30T00:54:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:22 smithi181 ceph-mon[47622]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:22 smithi181 ceph-mon[47622]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:54:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:22 smithi181 ceph-mon[47622]: osdmap e54: 9 total, 9 up, 9 in 2024-03-30T00:54:22.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:22 smithi130 ceph-mon[47626]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:22.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:22 smithi130 ceph-mon[47626]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:54:22.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:22 smithi130 ceph-mon[47626]: osdmap e54: 9 total, 9 up, 9 in 2024-03-30T00:54:23.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:23 smithi181 ceph-mon[47622]: osdmap e55: 9 total, 9 up, 9 in 2024-03-30T00:54:23.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:23 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3180212768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce69b948-3a4c-4e7e-905f-109698ca78e3"}]: dispatch 2024-03-30T00:54:23.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:23 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3180212768' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce69b948-3a4c-4e7e-905f-109698ca78e3"}]': finished 2024-03-30T00:54:23.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:23 smithi181 ceph-mon[47622]: osdmap e56: 10 total, 9 up, 10 in 2024-03-30T00:54:23.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:23.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:23 smithi130 ceph-mon[47626]: osdmap e55: 9 total, 9 up, 9 in 2024-03-30T00:54:23.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:23 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3180212768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce69b948-3a4c-4e7e-905f-109698ca78e3"}]: dispatch 2024-03-30T00:54:23.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:23 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3180212768' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce69b948-3a4c-4e7e-905f-109698ca78e3"}]': finished 2024-03-30T00:54:23.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:23 smithi130 ceph-mon[47626]: osdmap e56: 10 total, 9 up, 10 in 2024-03-30T00:54:23.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:23.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:23 smithi116 ceph-mon[38086]: osdmap e55: 9 total, 9 up, 9 in 2024-03-30T00:54:23.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:23 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3180212768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce69b948-3a4c-4e7e-905f-109698ca78e3"}]: dispatch 2024-03-30T00:54:23.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:23 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3180212768' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce69b948-3a4c-4e7e-905f-109698ca78e3"}]': finished 2024-03-30T00:54:23.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:23 smithi116 ceph-mon[38086]: osdmap e56: 10 total, 9 up, 10 in 2024-03-30T00:54:23.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:24 smithi181 ceph-mon[47622]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:24 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/986862949' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:54:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:24 smithi181 ceph-mon[47622]: osdmap e57: 10 total, 9 up, 10 in 2024-03-30T00:54:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:24 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:24 smithi130 ceph-mon[47626]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:24.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:24 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/986862949' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:54:24.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:24 smithi130 ceph-mon[47626]: osdmap e57: 10 total, 9 up, 10 in 2024-03-30T00:54:24.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:24 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:24 smithi116 ceph-mon[38086]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:24 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/986862949' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:54:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:24 smithi116 ceph-mon[38086]: osdmap e57: 10 total, 9 up, 10 in 2024-03-30T00:54:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:24 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:26.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:26 smithi181 ceph-mon[47622]: pgmap v162: 1 pgs: 1 peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T00:54:26.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:26 smithi130 ceph-mon[47626]: pgmap v162: 1 pgs: 1 peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T00:54:26.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:26 smithi116 ceph-mon[38086]: pgmap v162: 1 pgs: 1 peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T00:54:28.660 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:28 smithi116 ceph-mon[38086]: pgmap v163: 1 pgs: 1 peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-30T00:54:28.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:28 smithi130 ceph-mon[47626]: pgmap v163: 1 pgs: 1 peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-30T00:54:29.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:28 smithi181 ceph-mon[47622]: pgmap v163: 1 pgs: 1 peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-30T00:54:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:30 smithi130 ceph-mon[47626]: pgmap v164: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-30T00:54:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:30 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-30T00:54:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:30 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:30.860 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:30 smithi116 ceph-mon[38086]: pgmap v164: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-30T00:54:30.860 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:30 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-30T00:54:30.860 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:30 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:30 smithi181 ceph-mon[47622]: pgmap v164: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-30T00:54:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:30 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-30T00:54:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:30 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:31.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:31 smithi116 ceph-mon[38086]: Deploying daemon osd.9 on smithi116 2024-03-30T00:54:31.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:31 smithi130 ceph-mon[47626]: Deploying daemon osd.9 on smithi116 2024-03-30T00:54:32.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:31 smithi181 ceph-mon[47622]: Deploying daemon osd.9 on smithi116 2024-03-30T00:54:32.661 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:32 smithi116 ceph-mon[38086]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T00:54:32.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:32 smithi130 ceph-mon[47626]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T00:54:33.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:32 smithi181 ceph-mon[47622]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T00:54:34.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:34 smithi130 ceph-mon[47626]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-30T00:54:34.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:34 smithi116 ceph-mon[38086]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-30T00:54:35.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:34 smithi181 ceph-mon[47622]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-30T00:54:35.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:35.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:35.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:35.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:35 smithi116 ceph-mon[38086]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:36.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:36.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:36.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:36.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:35 smithi130 ceph-mon[47626]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:36.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:36.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:36.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:36.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:35 smithi181 ceph-mon[47622]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:37.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:37 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:37.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:37 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:37.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:37 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:37.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:37 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:37.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:37 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:37.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:37.738+0000 7fb66b400700 1 -- 172.21.15.116:0/4162543297 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb674074700 con 0x7fb6580a1120 2024-03-30T00:54:37.745 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 9 on host 'smithi116' 2024-03-30T00:54:37.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:37.740+0000 7fb679400700 1 -- 172.21.15.116:0/4162543297 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb6580a1120 msgr2=0x7fb6580a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:37.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:37.740+0000 7fb679400700 1 --2- 172.21.15.116:0/4162543297 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb6580a1120 0x7fb6580a35e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb66400b2e0 tx=0x7fb66400b6f0 comp rx=0 tx=0).stop 2024-03-30T00:54:37.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:37.740+0000 7fb679400700 1 -- 172.21.15.116:0/4162543297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674101f60 msgr2=0x7fb6741994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:37.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:37.740+0000 7fb679400700 1 --2- 172.21.15.116:0/4162543297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674101f60 0x7fb6741994e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb66000ea20 tx=0x7fb66000ed30 comp rx=0 tx=0).stop 2024-03-30T00:54:37.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:37.740+0000 7fb679400700 1 -- 172.21.15.116:0/4162543297 shutdown_connections 2024-03-30T00:54:37.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:37.740+0000 7fb679400700 1 --2- 172.21.15.116:0/4162543297 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb6580a1120 0x7fb6580a35e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:37.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:37.740+0000 7fb679400700 1 --2- 172.21.15.116:0/4162543297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb674102920 0x7fb6741a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:37.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:37.740+0000 7fb679400700 1 --2- 172.21.15.116:0/4162543297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb674100e00 0x7fb674198fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:37.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:37.740+0000 7fb679400700 1 --2- 172.21.15.116:0/4162543297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674101f60 0x7fb6741994e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:37.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:37.740+0000 7fb679400700 1 -- 172.21.15.116:0/4162543297 >> 172.21.15.116:0/4162543297 conn(0x7fb6740fc3c0 msgr2=0x7fb674072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:54:37.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:37.740+0000 7fb679400700 1 -- 172.21.15.116:0/4162543297 shutdown_connections 2024-03-30T00:54:37.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:54:37.741+0000 7fb679400700 1 -- 172.21.15.116:0/4162543297 wait complete. 2024-03-30T00:54:37.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:37 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:37.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:37 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:37.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:37 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:37.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:37 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:37.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:37 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:38.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:37 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:38.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:37 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:38.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:37 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:38.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:37 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:38.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:37 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:38.065 DEBUG:teuthology.orchestra.run.smithi116:osd.9> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.9.service 2024-03-30T00:54:38.069 INFO:tasks.cephadm:Deploying osd.10 on smithi130 with /dev/vg_nvme/lv_1... 2024-03-30T00:54:38.070 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-30T00:54:38.150 INFO:journalctl@ceph.osd.9.smithi116.stdout:-- Logs begin at Sat 2024-03-30 00:33:55 UTC. -- 2024-03-30T00:54:38.302 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T00:54:38.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:38 smithi116 ceph-mon[38086]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:38.659 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:38 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:38.659 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:38 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:38.659 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:38 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:38.756 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:38 smithi130 ceph-mon[47626]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:38.756 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:38 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:38.756 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:38 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:38.756 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:38 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:39.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:38 smithi181 ceph-mon[47622]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:39.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:38 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:39.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:38 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:39.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:38 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:39.174 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 00:54:38 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T00:54:38.910+0000 7fee75cf1640 -1 osd.9 0 log_to_monitors true 2024-03-30T00:54:39.688 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-30T00:54:39.688 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-30T00:54:39.688 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-03-30T00:54:39.688 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-03-30T00:54:39.688 INFO:teuthology.orchestra.run.smithi130.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.010993 s, 954 MB/s 2024-03-30T00:54:39.688 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-03-30T00:54:39.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:39 smithi130 ceph-mon[47626]: from='osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-30T00:54:39.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:39 smithi116 ceph-mon[38086]: from='osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-30T00:54:40.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:39 smithi181 ceph-mon[47622]: from='osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-30T00:54:40.128 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi130:vg_nvme/lv_1 2024-03-30T00:54:40.345 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T00:54:40.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 00:54:40 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T00:54:40.535+0000 7fee6a200700 -1 osd.9 0 waiting for initial osdmap 2024-03-30T00:54:40.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 00:54:40 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T00:54:40.549+0000 7fee65000700 -1 osd.9 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T00:54:40.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:40 smithi116 ceph-mon[38086]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:40.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:40 smithi116 ceph-mon[38086]: from='osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-30T00:54:40.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:40 smithi116 ceph-mon[38086]: osdmap e58: 10 total, 9 up, 10 in 2024-03-30T00:54:40.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:40 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:40.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:40 smithi116 ceph-mon[38086]: from='osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:54:40.781 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:40 smithi130 ceph-mon[47626]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:40.781 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:40 smithi130 ceph-mon[47626]: from='osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-30T00:54:40.781 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:40 smithi130 ceph-mon[47626]: osdmap e58: 10 total, 9 up, 10 in 2024-03-30T00:54:40.781 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:40 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:40.781 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:40 smithi130 ceph-mon[47626]: from='osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:54:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:40 smithi181 ceph-mon[47622]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:40 smithi181 ceph-mon[47622]: from='osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-30T00:54:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:40 smithi181 ceph-mon[47622]: osdmap e58: 10 total, 9 up, 10 in 2024-03-30T00:54:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:40 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:40 smithi181 ceph-mon[47622]: from='osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-30T00:54:41.278 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.270+0000 7f3b0dda7700 1 -- 172.21.15.130:0/3331836192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b08102920 msgr2=0x7f3b08109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:41.278 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.270+0000 7f3b0dda7700 1 --2- 172.21.15.130:0/3331836192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b08102920 0x7f3b08109240 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000b0a0 tx=0x7f3b0000b3b0 comp rx=0 tx=0).stop 2024-03-30T00:54:41.278 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.270+0000 7f3b0dda7700 1 -- 172.21.15.130:0/3331836192 shutdown_connections 2024-03-30T00:54:41.278 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.270+0000 7f3b0dda7700 1 --2- 172.21.15.130:0/3331836192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b08102920 0x7f3b08109240 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:41.278 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.270+0000 7f3b0dda7700 1 --2- 172.21.15.130:0/3331836192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b08100e00 0x7f3b08101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:41.278 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.270+0000 7f3b0dda7700 1 --2- 172.21.15.130:0/3331836192 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b08101f60 0x7f3b081023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:41.278 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.270+0000 7f3b0dda7700 1 -- 172.21.15.130:0/3331836192 >> 172.21.15.130:0/3331836192 conn(0x7f3b080fc3a0 msgr2=0x7f3b080fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:54:41.278 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.270+0000 7f3b0dda7700 1 -- 172.21.15.130:0/3331836192 shutdown_connections 2024-03-30T00:54:41.278 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.270+0000 7f3b0dda7700 1 -- 172.21.15.130:0/3331836192 wait complete. 2024-03-30T00:54:41.278 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.270+0000 7f3b0dda7700 1 Processor -- start 2024-03-30T00:54:41.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b0dda7700 1 -- start start 2024-03-30T00:54:41.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b0dda7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b08100e00 0x7f3b08196e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:41.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b0dda7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b08101f60 0x7f3b08197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:41.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b0dda7700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b08102920 0x7f3b0819e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:41.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b0dda7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0819e980 con 0x7f3b08100e00 2024-03-30T00:54:41.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b0dda7700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0819eac0 con 0x7f3b08102920 2024-03-30T00:54:41.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b0dda7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0819ec50 con 0x7f3b08101f60 2024-03-30T00:54:41.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b07fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b08102920 0x7f3b0819e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:41.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b07fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b08102920 0x7f3b0819e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:53368/0 (socket says 172.21.15.130:53368) 2024-03-30T00:54:41.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b07fff700 1 -- 172.21.15.130:0/97869517 learned_addr learned my addr 172.21.15.130:0/97869517 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-03-30T00:54:41.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b077fe700 1 --2- 172.21.15.130:0/97869517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b08100e00 0x7f3b08196e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:41.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b06ffd700 1 --2- 172.21.15.130:0/97869517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b08101f60 0x7f3b08197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:41.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b07fff700 1 -- 172.21.15.130:0/97869517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b08101f60 msgr2=0x7f3b08197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:41.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b07fff700 1 --2- 172.21.15.130:0/97869517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b08101f60 0x7f3b08197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:41.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b07fff700 1 -- 172.21.15.130:0/97869517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b08100e00 msgr2=0x7f3b08196e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:41.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b07fff700 1 --2- 172.21.15.130:0/97869517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b08100e00 0x7f3b08196e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:41.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b07fff700 1 -- 172.21.15.130:0/97869517 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b00009d00 con 0x7f3b08102920 2024-03-30T00:54:41.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.271+0000 7f3b07fff700 1 --2- 172.21.15.130:0/97869517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b08102920 0x7f3b0819e3b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000d010 tx=0x7f3b00004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:54:41.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.272+0000 7f3b04ff9700 1 -- 172.21.15.130:0/97869517 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0000e040 con 0x7f3b08102920 2024-03-30T00:54:41.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.272+0000 7f3b04ff9700 1 -- 172.21.15.130:0/97869517 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b0001db50 con 0x7f3b08102920 2024-03-30T00:54:41.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.272+0000 7f3b0dda7700 1 -- 172.21.15.130:0/97869517 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b0819eed0 con 0x7f3b08102920 2024-03-30T00:54:41.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.272+0000 7f3b04ff9700 1 -- 172.21.15.130:0/97869517 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0001bb70 con 0x7f3b08102920 2024-03-30T00:54:41.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.272+0000 7f3b0dda7700 1 -- 172.21.15.130:0/97869517 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b0819f290 con 0x7f3b08102920 2024-03-30T00:54:41.281 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.273+0000 7f3b0dda7700 1 -- 172.21.15.130:0/97869517 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b08108220 con 0x7f3b08102920 2024-03-30T00:54:41.282 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.274+0000 7f3b04ff9700 1 -- 172.21.15.130:0/97869517 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f3b0001dcc0 con 0x7f3b08102920 2024-03-30T00:54:41.282 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.274+0000 7f3b04ff9700 1 --2- 172.21.15.130:0/97869517 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3af00a1120 0x7f3af00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:41.282 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.274+0000 7f3b04ff9700 1 -- 172.21.15.130:0/97869517 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5981+0+0 (secure 0 0 0) 0x7f3b000ce140 con 0x7f3b08102920 2024-03-30T00:54:41.282 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.274+0000 7f3b077fe700 1 --2- 172.21.15.130:0/97869517 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3af00a1120 0x7f3af00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:41.282 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.275+0000 7f3b077fe700 1 --2- 172.21.15.130:0/97869517 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3af00a1120 0x7f3af00a35e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009a10 tx=0x7f3afc009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:54:41.283 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.278+0000 7f3b04ff9700 1 -- 172.21.15.130:0/97869517 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3b00098280 con 0x7f3b08102920 2024-03-30T00:54:41.483 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:41.477+0000 7f3b0dda7700 1 -- 172.21.15.130:0/97869517 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b0802cdc0 con 0x7f3af00a1120 2024-03-30T00:54:41.828 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-30T00:54:41.828 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: osdmap e59: 10 total, 9 up, 10 in 2024-03-30T00:54:41.828 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:41.828 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:41.829 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:41.829 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:41.829 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:41.829 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:41.829 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:41.829 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:41.829 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:41.829 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:41.829 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:41.829 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:41.829 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:54:41.829 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:54:41.829 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:41.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-30T00:54:41.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: osdmap e59: 10 total, 9 up, 10 in 2024-03-30T00:54:41.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:41.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:41.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:41.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:41.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:41.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:41.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:41.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:41.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:41.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:41.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:41.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:41.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:54:41.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:54:41.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-30T00:54:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: osdmap e59: 10 total, 9 up, 10 in 2024-03-30T00:54:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:42.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:42.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:42.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:42.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:42.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:42.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:54:42.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:54:42.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:42.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:42 smithi130 ceph-mon[47626]: purged_snaps scrub starts 2024-03-30T00:54:42.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:42 smithi130 ceph-mon[47626]: purged_snaps scrub ok 2024-03-30T00:54:42.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:42 smithi130 ceph-mon[47626]: Detected new or changed devices on smithi116 2024-03-30T00:54:42.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:42 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi116 to 4317M 2024-03-30T00:54:42.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:42 smithi130 ceph-mon[47626]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:42.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:42 smithi130 ceph-mon[47626]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:54:42.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:42 smithi130 ceph-mon[47626]: osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178] boot 2024-03-30T00:54:42.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:42 smithi130 ceph-mon[47626]: osdmap e60: 10 total, 10 up, 10 in 2024-03-30T00:54:42.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:42 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:42.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:42 smithi116 ceph-mon[38086]: purged_snaps scrub starts 2024-03-30T00:54:42.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:42 smithi116 ceph-mon[38086]: purged_snaps scrub ok 2024-03-30T00:54:42.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:42 smithi116 ceph-mon[38086]: Detected new or changed devices on smithi116 2024-03-30T00:54:42.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:42 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi116 to 4317M 2024-03-30T00:54:42.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:42 smithi116 ceph-mon[38086]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:42.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:42 smithi116 ceph-mon[38086]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:54:42.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:42 smithi116 ceph-mon[38086]: osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178] boot 2024-03-30T00:54:42.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:42 smithi116 ceph-mon[38086]: osdmap e60: 10 total, 10 up, 10 in 2024-03-30T00:54:42.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:42 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:42 smithi181 ceph-mon[47622]: purged_snaps scrub starts 2024-03-30T00:54:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:42 smithi181 ceph-mon[47622]: purged_snaps scrub ok 2024-03-30T00:54:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:42 smithi181 ceph-mon[47622]: Detected new or changed devices on smithi116 2024-03-30T00:54:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:42 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi116 to 4317M 2024-03-30T00:54:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:42 smithi181 ceph-mon[47622]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-30T00:54:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:42 smithi181 ceph-mon[47622]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:54:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:42 smithi181 ceph-mon[47622]: osd.9 [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178] boot 2024-03-30T00:54:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:42 smithi181 ceph-mon[47622]: osdmap e60: 10 total, 10 up, 10 in 2024-03-30T00:54:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:42 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T00:54:44.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:43 smithi130 ceph-mon[47626]: osdmap e61: 10 total, 10 up, 10 in 2024-03-30T00:54:44.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:43 smithi130 ceph-mon[47626]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:44.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:43 smithi130 ceph-mon[47626]: from='client.? 172.21.15.130:0/1224566838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "575cf370-93a6-4985-b490-b962f2870aa1"}]: dispatch 2024-03-30T00:54:44.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:43 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "575cf370-93a6-4985-b490-b962f2870aa1"}]: dispatch 2024-03-30T00:54:44.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:43 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "575cf370-93a6-4985-b490-b962f2870aa1"}]': finished 2024-03-30T00:54:44.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:43 smithi130 ceph-mon[47626]: osdmap e62: 11 total, 10 up, 11 in 2024-03-30T00:54:44.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:43 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:54:44.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:43 smithi116 ceph-mon[38086]: osdmap e61: 10 total, 10 up, 10 in 2024-03-30T00:54:44.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:43 smithi116 ceph-mon[38086]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:44.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:43 smithi116 ceph-mon[38086]: from='client.? 172.21.15.130:0/1224566838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "575cf370-93a6-4985-b490-b962f2870aa1"}]: dispatch 2024-03-30T00:54:44.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:43 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "575cf370-93a6-4985-b490-b962f2870aa1"}]: dispatch 2024-03-30T00:54:44.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:43 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "575cf370-93a6-4985-b490-b962f2870aa1"}]': finished 2024-03-30T00:54:44.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:43 smithi116 ceph-mon[38086]: osdmap e62: 11 total, 10 up, 11 in 2024-03-30T00:54:44.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:43 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:54:44.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:43 smithi181 ceph-mon[47622]: osdmap e61: 10 total, 10 up, 10 in 2024-03-30T00:54:44.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:43 smithi181 ceph-mon[47622]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:44.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:43 smithi181 ceph-mon[47622]: from='client.? 172.21.15.130:0/1224566838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "575cf370-93a6-4985-b490-b962f2870aa1"}]: dispatch 2024-03-30T00:54:44.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:43 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "575cf370-93a6-4985-b490-b962f2870aa1"}]: dispatch 2024-03-30T00:54:44.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:43 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "575cf370-93a6-4985-b490-b962f2870aa1"}]': finished 2024-03-30T00:54:44.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:43 smithi181 ceph-mon[47622]: osdmap e62: 11 total, 10 up, 11 in 2024-03-30T00:54:44.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:43 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:54:45.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:44 smithi130 ceph-mon[47626]: from='client.? 172.21.15.130:0/1461214292' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:54:45.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:44 smithi116 ceph-mon[38086]: from='client.? 172.21.15.130:0/1461214292' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:54:45.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:44 smithi181 ceph-mon[47622]: from='client.? 172.21.15.130:0/1461214292' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:54:46.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:45 smithi130 ceph-mon[47626]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:46.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:45 smithi116 ceph-mon[38086]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:46.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:45 smithi181 ceph-mon[47622]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:48.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:48 smithi130 ceph-mon[47626]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:48.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:48 smithi116 ceph-mon[38086]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:48.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:48 smithi181 ceph-mon[47622]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:50.255 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:50 smithi130 ceph-mon[47626]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:50.255 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-30T00:54:50.255 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:50.255 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:50 smithi130 ceph-mon[47626]: Deploying daemon osd.10 on smithi130 2024-03-30T00:54:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:50 smithi116 ceph-mon[38086]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-30T00:54:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:50.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:50 smithi116 ceph-mon[38086]: Deploying daemon osd.10 on smithi130 2024-03-30T00:54:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:50 smithi181 ceph-mon[47622]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-30T00:54:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:50 smithi181 ceph-mon[47622]: Deploying daemon osd.10 on smithi130 2024-03-30T00:54:52.428 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:52 smithi116 ceph-mon[38086]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:52 smithi181 ceph-mon[47622]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:52.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:52 smithi130 ceph-mon[47626]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:54.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:54.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:54.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:54.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:53 smithi130 ceph-mon[47626]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:54.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:54.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:54.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:54.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:53 smithi116 ceph-mon[38086]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:54.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:54.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:54.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:54.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:53 smithi181 ceph-mon[47622]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:54.986 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:54.979+0000 7f3b04ff9700 1 -- 172.21.15.130:0/97869517 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f3b0802cdc0 con 0x7f3af00a1120 2024-03-30T00:54:54.987 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 10 on host 'smithi130' 2024-03-30T00:54:54.988 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:54.982+0000 7f3b0dda7700 1 -- 172.21.15.130:0/97869517 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3af00a1120 msgr2=0x7f3af00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:54.988 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:54.982+0000 7f3b0dda7700 1 --2- 172.21.15.130:0/97869517 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3af00a1120 0x7f3af00a35e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009a10 tx=0x7f3afc009190 comp rx=0 tx=0).stop 2024-03-30T00:54:54.988 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:54.982+0000 7f3b0dda7700 1 -- 172.21.15.130:0/97869517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b08102920 msgr2=0x7f3b0819e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:54.989 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:54.982+0000 7f3b0dda7700 1 --2- 172.21.15.130:0/97869517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b08102920 0x7f3b0819e3b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000d010 tx=0x7f3b00004d00 comp rx=0 tx=0).stop 2024-03-30T00:54:54.989 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:54.982+0000 7f3b0dda7700 1 -- 172.21.15.130:0/97869517 shutdown_connections 2024-03-30T00:54:54.989 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:54.982+0000 7f3b0dda7700 1 --2- 172.21.15.130:0/97869517 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3af00a1120 0x7f3af00a35e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:54.989 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:54.982+0000 7f3b0dda7700 1 --2- 172.21.15.130:0/97869517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b08102920 0x7f3b0819e3b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:54.989 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:54.982+0000 7f3b0dda7700 1 --2- 172.21.15.130:0/97869517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b08100e00 0x7f3b08196e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:54.989 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:54.982+0000 7f3b0dda7700 1 --2- 172.21.15.130:0/97869517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b08101f60 0x7f3b08197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:54.989 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:54.982+0000 7f3b0dda7700 1 -- 172.21.15.130:0/97869517 >> 172.21.15.130:0/97869517 conn(0x7f3b080fc3a0 msgr2=0x7f3b08105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:54:54.989 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:54.982+0000 7f3b0dda7700 1 -- 172.21.15.130:0/97869517 shutdown_connections 2024-03-30T00:54:54.989 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-30T00:54:54.982+0000 7f3b0dda7700 1 -- 172.21.15.130:0/97869517 wait complete. 2024-03-30T00:54:55.391 DEBUG:teuthology.orchestra.run.smithi130:osd.10> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.10.service 2024-03-30T00:54:55.393 INFO:tasks.cephadm:Deploying osd.11 on smithi181 with /dev/vg_nvme/lv_1... 2024-03-30T00:54:55.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-30T00:54:55.428 INFO:journalctl@ceph.osd.10.smithi130.stdout:-- Logs begin at Sat 2024-03-30 00:27:58 UTC. -- 2024-03-30T00:54:55.625 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T00:54:55.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:55 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:55 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:55 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:55.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:55 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:55.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:55 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:55 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:55 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:55 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:55.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:55 smithi181 ceph-mon[47622]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:55.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:55 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:55 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:55 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:55.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:55 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:55.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:55 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:55 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:55 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:55 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:55.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:55 smithi130 ceph-mon[47626]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:55.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:55.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:55.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:55.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:55 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:54:55.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:55 smithi116 ceph-mon[38086]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:56.951 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-30T00:54:56.951 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-30T00:54:56.951 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-03-30T00:54:56.951 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-03-30T00:54:56.952 INFO:teuthology.orchestra.run.smithi181.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112185 s, 935 MB/s 2024-03-30T00:54:56.952 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 2024-03-30T00:54:56.952 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-03-30T00:54:57.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:56 smithi181 ceph-mon[47622]: from='osd.10 [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-30T00:54:57.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:56 smithi181 ceph-mon[47622]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-30T00:54:57.261 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:56 smithi130 ceph-mon[47626]: from='osd.10 [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-30T00:54:57.261 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:56 smithi130 ceph-mon[47626]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-30T00:54:57.261 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 00:54:56 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T00:54:56.895+0000 7f3240dda640 -1 osd.10 0 log_to_monitors true 2024-03-30T00:54:57.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi181:vg_nvme/lv_1 2024-03-30T00:54:57.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:56 smithi116 ceph-mon[38086]: from='osd.10 [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-30T00:54:57.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:56 smithi116 ceph-mon[38086]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-30T00:54:57.499 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T00:54:58.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-30T00:54:58.302 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: osdmap e63: 11 total, 10 up, 11 in 2024-03-30T00:54:58.303 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:54:58.303 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: from='osd.10 [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:54:58.303 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:54:58.303 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:58.303 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:58.303 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:58.303 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:58.303 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:58.303 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:58.304 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:58.304 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:58.304 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:58.304 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:58.304 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:57 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:58.344 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T00:54:57.997+0000 7f323643e700 -1 osd.10 0 waiting for initial osdmap 2024-03-30T00:54:58.344 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 00:54:58 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T00:54:58.012+0000 7f3232233700 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T00:54:58.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-30T00:54:58.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: osdmap e63: 11 total, 10 up, 11 in 2024-03-30T00:54:58.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:54:58.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: from='osd.10 [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:54:58.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:54:58.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:58.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:58.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:58.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:58.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:58.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:58.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:58.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:58.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:58.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:58.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:57 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-30T00:54:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: osdmap e63: 11 total, 10 up, 11 in 2024-03-30T00:54:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:54:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: from='osd.10 [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:54:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-30T00:54:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:54:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:58.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:58.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:54:58.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:58.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:58.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:54:58.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:57 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:54:58.498 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.492+0000 7f982a16a700 1 -- 172.21.15.181:0/112218617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9824100ef0 msgr2=0x7f9824101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:58.499 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.492+0000 7f982a16a700 1 --2- 172.21.15.181:0/112218617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9824100ef0 0x7f9824101350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9818009990 tx=0x7f9818009ca0 comp rx=0 tx=0).stop 2024-03-30T00:54:58.499 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.492+0000 7f982a16a700 1 -- 172.21.15.181:0/112218617 shutdown_connections 2024-03-30T00:54:58.499 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.492+0000 7f982a16a700 1 --2- 172.21.15.181:0/112218617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9824101890 0x7f9824108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:58.499 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.492+0000 7f982a16a700 1 --2- 172.21.15.181:0/112218617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98240ffcf0 0x7f9824100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:58.499 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.492+0000 7f982a16a700 1 --2- 172.21.15.181:0/112218617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9824100ef0 0x7f9824101350 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:58.499 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.492+0000 7f982a16a700 1 -- 172.21.15.181:0/112218617 >> 172.21.15.181:0/112218617 conn(0x7f98240fb2b0 msgr2=0x7f98240fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:54:58.499 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.492+0000 7f982a16a700 1 -- 172.21.15.181:0/112218617 shutdown_connections 2024-03-30T00:54:58.499 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.492+0000 7f982a16a700 1 -- 172.21.15.181:0/112218617 wait complete. 2024-03-30T00:54:58.499 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.492+0000 7f982a16a700 1 Processor -- start 2024-03-30T00:54:58.499 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.492+0000 7f982a16a700 1 -- start start 2024-03-30T00:54:58.501 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f982a16a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98240ffcf0 0x7f9824069b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:58.501 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f982a16a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9824100ef0 0x7f982419ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:58.501 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f982a16a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9824101890 0x7f982419e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:58.501 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f982a16a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982419e7f0 con 0x7f9824100ef0 2024-03-30T00:54:58.501 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f982a16a700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982419e960 con 0x7f9824101890 2024-03-30T00:54:58.501 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f982a16a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982419eb20 con 0x7f98240ffcf0 2024-03-30T00:54:58.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f98237fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98240ffcf0 0x7f9824069b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:58.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f98237fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98240ffcf0 0x7f9824069b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50326/0 (socket says 172.21.15.181:50326) 2024-03-30T00:54:58.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f98237fe700 1 -- 172.21.15.181:0/1548194201 learned_addr learned my addr 172.21.15.181:0/1548194201 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-03-30T00:54:58.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f9823fff700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9824101890 0x7f982419e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:58.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f9822ffd700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9824100ef0 0x7f982419ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:58.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f98237fe700 1 -- 172.21.15.181:0/1548194201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9824101890 msgr2=0x7f982419e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:58.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f98237fe700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9824101890 0x7f982419e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:58.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f98237fe700 1 -- 172.21.15.181:0/1548194201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9824100ef0 msgr2=0x7f982419ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:54:58.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f98237fe700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9824100ef0 0x7f982419ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:54:58.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f98237fe700 1 -- 172.21.15.181:0/1548194201 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98180095f0 con 0x7f98240ffcf0 2024-03-30T00:54:58.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f9822ffd700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9824100ef0 0x7f982419ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T00:54:58.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f98237fe700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98240ffcf0 0x7f9824069b10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f981400d980 tx=0x7f981400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:54:58.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f9820ff9700 1 -- 172.21.15.181:0/1548194201 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9814004020 con 0x7f98240ffcf0 2024-03-30T00:54:58.503 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f9820ff9700 1 -- 172.21.15.181:0/1548194201 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9814009730 con 0x7f98240ffcf0 2024-03-30T00:54:58.503 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f982a16a700 1 -- 172.21.15.181:0/1548194201 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982419ee00 con 0x7f98240ffcf0 2024-03-30T00:54:58.503 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f9820ff9700 1 -- 172.21.15.181:0/1548194201 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9814003a80 con 0x7f98240ffcf0 2024-03-30T00:54:58.503 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.493+0000 7f982a16a700 1 -- 172.21.15.181:0/1548194201 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982419f270 con 0x7f98240ffcf0 2024-03-30T00:54:58.503 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.494+0000 7f982a16a700 1 -- 172.21.15.181:0/1548194201 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982404eaf0 con 0x7f98240ffcf0 2024-03-30T00:54:58.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.495+0000 7f9820ff9700 1 -- 172.21.15.181:0/1548194201 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f98140098a0 con 0x7f98240ffcf0 2024-03-30T00:54:58.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.496+0000 7f9820ff9700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f980c0a11a0 0x7f980c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:54:58.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.496+0000 7f9820ff9700 1 -- 172.21.15.181:0/1548194201 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f98140cbfe0 con 0x7f98240ffcf0 2024-03-30T00:54:58.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.497+0000 7f9822ffd700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f980c0a11a0 0x7f980c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:54:58.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.497+0000 7f9822ffd700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f980c0a11a0 0x7f980c0a3660 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9818000c00 tx=0x7f9818005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:54:58.506 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.500+0000 7f9820ff9700 1 -- 172.21.15.181:0/1548194201 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9814095f20 con 0x7f98240ffcf0 2024-03-30T00:54:58.708 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:54:58.702+0000 7f982a16a700 1 -- 172.21.15.181:0/1548194201 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f982404ea80 con 0x7f980c0a11a0 2024-03-30T00:54:59.326 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:59 smithi181 ceph-mon[47622]: Detected new or changed devices on smithi130 2024-03-30T00:54:59.326 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:59 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi130 to 4278M 2024-03-30T00:54:59.326 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:59 smithi181 ceph-mon[47622]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-30T00:54:59.326 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:59 smithi181 ceph-mon[47622]: osdmap e64: 11 total, 10 up, 11 in 2024-03-30T00:54:59.326 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:59 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:54:59.327 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:59 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:54:59.327 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:59 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:54:59.327 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:59 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:54:59.327 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:54:59 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:59.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:59 smithi130 ceph-mon[47626]: Detected new or changed devices on smithi130 2024-03-30T00:54:59.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:59 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi130 to 4278M 2024-03-30T00:54:59.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:59 smithi130 ceph-mon[47626]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-30T00:54:59.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:59 smithi130 ceph-mon[47626]: osdmap e64: 11 total, 10 up, 11 in 2024-03-30T00:54:59.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:59 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:54:59.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:59 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:54:59.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:59 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:54:59.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:59 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:54:59.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:54:59 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:54:59.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:59 smithi116 ceph-mon[38086]: Detected new or changed devices on smithi130 2024-03-30T00:54:59.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:59 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi130 to 4278M 2024-03-30T00:54:59.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:59 smithi116 ceph-mon[38086]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-30T00:54:59.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:59 smithi116 ceph-mon[38086]: osdmap e64: 11 total, 10 up, 11 in 2024-03-30T00:54:59.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:59 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:54:59.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:59 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:54:59.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:59 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T00:54:59.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:59 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T00:54:59.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:54:59 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:00.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:00 smithi130 ceph-mon[47626]: purged_snaps scrub starts 2024-03-30T00:55:00.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:00 smithi130 ceph-mon[47626]: purged_snaps scrub ok 2024-03-30T00:55:00.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:00 smithi130 ceph-mon[47626]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:55:00.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:00 smithi130 ceph-mon[47626]: osd.10 [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262] boot 2024-03-30T00:55:00.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:00 smithi130 ceph-mon[47626]: osdmap e65: 11 total, 11 up, 11 in 2024-03-30T00:55:00.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:00 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:55:00.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:00 smithi130 ceph-mon[47626]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:55:00.348 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:00 smithi130 ceph-mon[47626]: osdmap e66: 11 total, 11 up, 11 in 2024-03-30T00:55:00.435 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:00 smithi116 ceph-mon[38086]: purged_snaps scrub starts 2024-03-30T00:55:00.435 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:00 smithi116 ceph-mon[38086]: purged_snaps scrub ok 2024-03-30T00:55:00.435 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:00 smithi116 ceph-mon[38086]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:55:00.435 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:00 smithi116 ceph-mon[38086]: osd.10 [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262] boot 2024-03-30T00:55:00.435 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:00 smithi116 ceph-mon[38086]: osdmap e65: 11 total, 11 up, 11 in 2024-03-30T00:55:00.435 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:00 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:55:00.436 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:00 smithi116 ceph-mon[38086]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:55:00.436 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:00 smithi116 ceph-mon[38086]: osdmap e66: 11 total, 11 up, 11 in 2024-03-30T00:55:00.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:00 smithi181 ceph-mon[47622]: purged_snaps scrub starts 2024-03-30T00:55:00.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:00 smithi181 ceph-mon[47622]: purged_snaps scrub ok 2024-03-30T00:55:00.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:00 smithi181 ceph-mon[47622]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:55:00.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:00 smithi181 ceph-mon[47622]: osd.10 [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262] boot 2024-03-30T00:55:00.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:00 smithi181 ceph-mon[47622]: osdmap e65: 11 total, 11 up, 11 in 2024-03-30T00:55:00.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:00 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T00:55:00.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:00 smithi181 ceph-mon[47622]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-03-30T00:55:00.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:00 smithi181 ceph-mon[47622]: osdmap e66: 11 total, 11 up, 11 in 2024-03-30T00:55:01.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:01 smithi130 ceph-mon[47626]: from='client.? 172.21.15.181:0/3672559541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09ffa044-3256-41a3-98d1-ddb3ed32ecc7"}]: dispatch 2024-03-30T00:55:01.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:01 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09ffa044-3256-41a3-98d1-ddb3ed32ecc7"}]: dispatch 2024-03-30T00:55:01.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:01 smithi130 ceph-mon[47626]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09ffa044-3256-41a3-98d1-ddb3ed32ecc7"}]': finished 2024-03-30T00:55:01.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:01 smithi130 ceph-mon[47626]: osdmap e67: 12 total, 11 up, 12 in 2024-03-30T00:55:01.353 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:01 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:01.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:01 smithi116 ceph-mon[38086]: from='client.? 172.21.15.181:0/3672559541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09ffa044-3256-41a3-98d1-ddb3ed32ecc7"}]: dispatch 2024-03-30T00:55:01.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:01 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09ffa044-3256-41a3-98d1-ddb3ed32ecc7"}]: dispatch 2024-03-30T00:55:01.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:01 smithi116 ceph-mon[38086]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09ffa044-3256-41a3-98d1-ddb3ed32ecc7"}]': finished 2024-03-30T00:55:01.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:01 smithi116 ceph-mon[38086]: osdmap e67: 12 total, 11 up, 12 in 2024-03-30T00:55:01.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:01 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:01.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:01 smithi181 ceph-mon[47622]: from='client.? 172.21.15.181:0/3672559541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09ffa044-3256-41a3-98d1-ddb3ed32ecc7"}]: dispatch 2024-03-30T00:55:01.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:01 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09ffa044-3256-41a3-98d1-ddb3ed32ecc7"}]: dispatch 2024-03-30T00:55:01.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:01 smithi181 ceph-mon[47622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09ffa044-3256-41a3-98d1-ddb3ed32ecc7"}]': finished 2024-03-30T00:55:01.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:01 smithi181 ceph-mon[47622]: osdmap e67: 12 total, 11 up, 12 in 2024-03-30T00:55:01.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:01 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:02.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:02 smithi130 ceph-mon[47626]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-30T00:55:02.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:02 smithi130 ceph-mon[47626]: from='client.? 172.21.15.181:0/2321287666' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:55:02.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:02 smithi130 ceph-mon[47626]: osdmap e68: 12 total, 11 up, 12 in 2024-03-30T00:55:02.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:02.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:02 smithi116 ceph-mon[38086]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-30T00:55:02.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:02 smithi116 ceph-mon[38086]: from='client.? 172.21.15.181:0/2321287666' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:55:02.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:02 smithi116 ceph-mon[38086]: osdmap e68: 12 total, 11 up, 12 in 2024-03-30T00:55:02.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:02.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:02 smithi181 ceph-mon[47622]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-03-30T00:55:02.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:02 smithi181 ceph-mon[47622]: from='client.? 172.21.15.181:0/2321287666' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T00:55:02.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:02 smithi181 ceph-mon[47622]: osdmap e68: 12 total, 11 up, 12 in 2024-03-30T00:55:02.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:03.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:03 smithi181 ceph-mon[47622]: osdmap e69: 12 total, 11 up, 12 in 2024-03-30T00:55:03.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:03.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:03 smithi130 ceph-mon[47626]: osdmap e69: 12 total, 11 up, 12 in 2024-03-30T00:55:03.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:03.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:03 smithi116 ceph-mon[38086]: osdmap e69: 12 total, 11 up, 12 in 2024-03-30T00:55:03.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:04.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:04 smithi181 ceph-mon[47622]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-30T00:55:04.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:04 smithi130 ceph-mon[47626]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-30T00:55:04.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:04 smithi116 ceph-mon[38086]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-30T00:55:06.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:06 smithi181 ceph-mon[47622]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-30T00:55:06.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-30T00:55:06.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:06.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:06 smithi130 ceph-mon[47626]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-30T00:55:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-30T00:55:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:06.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:06 smithi116 ceph-mon[38086]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-30T00:55:06.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-30T00:55:06.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:07.630 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:07 smithi181 ceph-mon[47622]: Deploying daemon osd.11 on smithi181 2024-03-30T00:55:07.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:07 smithi130 ceph-mon[47626]: Deploying daemon osd.11 on smithi181 2024-03-30T00:55:07.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:07 smithi116 ceph-mon[38086]: Deploying daemon osd.11 on smithi181 2024-03-30T00:55:08.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:08 smithi130 ceph-mon[47626]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-30T00:55:08.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:08 smithi116 ceph-mon[38086]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-30T00:55:09.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:08 smithi181 ceph-mon[47622]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-30T00:55:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:10 smithi181 ceph-mon[47622]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T00:55:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:10 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:10 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:10 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:10.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:10 smithi130 ceph-mon[47626]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T00:55:10.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:10 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:10.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:10 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:10.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:10 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:10.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:10 smithi116 ceph-mon[38086]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T00:55:10.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:10 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:10.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:10 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:10.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:10 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:11.810 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:11.804+0000 7f9820ff9700 1 -- 172.21.15.181:0/1548194201 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f982404ea80 con 0x7f980c0a11a0 2024-03-30T00:55:11.811 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 11 on host 'smithi181' 2024-03-30T00:55:11.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:11.807+0000 7f982a16a700 1 -- 172.21.15.181:0/1548194201 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f980c0a11a0 msgr2=0x7f980c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:11.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:11.807+0000 7f982a16a700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f980c0a11a0 0x7f980c0a3660 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9818000c00 tx=0x7f9818005fb0 comp rx=0 tx=0).stop 2024-03-30T00:55:11.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:11.807+0000 7f982a16a700 1 -- 172.21.15.181:0/1548194201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98240ffcf0 msgr2=0x7f9824069b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:11.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:11.807+0000 7f982a16a700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98240ffcf0 0x7f9824069b10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f981400d980 tx=0x7f981400dc90 comp rx=0 tx=0).stop 2024-03-30T00:55:11.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:11.807+0000 7f982a16a700 1 -- 172.21.15.181:0/1548194201 shutdown_connections 2024-03-30T00:55:11.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:11.807+0000 7f982a16a700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f980c0a11a0 0x7f980c0a3660 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:11.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:11.807+0000 7f982a16a700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98240ffcf0 0x7f9824069b10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:11.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:11.807+0000 7f982a16a700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9824101890 0x7f982419e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:11.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:11.807+0000 7f982a16a700 1 --2- 172.21.15.181:0/1548194201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9824100ef0 0x7f982419ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:11.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:11.807+0000 7f982a16a700 1 -- 172.21.15.181:0/1548194201 >> 172.21.15.181:0/1548194201 conn(0x7f98240fb2b0 msgr2=0x7f9824104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:11.814 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:11.807+0000 7f982a16a700 1 -- 172.21.15.181:0/1548194201 shutdown_connections 2024-03-30T00:55:11.814 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:11.807+0000 7f982a16a700 1 -- 172.21.15.181:0/1548194201 wait complete. 2024-03-30T00:55:12.209 DEBUG:teuthology.orchestra.run.smithi181:osd.11> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.11.service 2024-03-30T00:55:12.212 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-30T00:55:12.212 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-03-30T00:55:12.242 INFO:journalctl@ceph.osd.11.smithi181.stdout:-- Logs begin at Sat 2024-03-30 00:27:51 UTC. -- 2024-03-30T00:55:12.416 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:12 smithi181 ceph-mon[47622]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-30T00:55:12.416 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.416 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.416 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:12.416 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:12.416 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.416 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.417 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.417 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:12 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:12.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:12 smithi116 ceph-mon[38086]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-30T00:55:12.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:12.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:12.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:12 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:12.573 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:55:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:12 smithi130 ceph-mon[47626]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-30T00:55:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:12 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:12 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:12 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:12 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:12 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:12 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:12 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:12.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:12 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:13.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.711+0000 7f2f7c400700 1 -- 172.21.15.116:0/3758916682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f74108cd0 msgr2=0x7f2f7410b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:13.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.711+0000 7f2f7c400700 1 --2- 172.21.15.116:0/3758916682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f74108cd0 0x7f2f7410b160 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009f90 tx=0x7f2f6c00b330 comp rx=0 tx=0).stop 2024-03-30T00:55:13.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.711+0000 7f2f7c400700 1 -- 172.21.15.116:0/3758916682 shutdown_connections 2024-03-30T00:55:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.711+0000 7f2f7c400700 1 --2- 172.21.15.116:0/3758916682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f74108cd0 0x7f2f7410b160 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.711+0000 7f2f7c400700 1 --2- 172.21.15.116:0/3758916682 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f74074de0 0x7f2f74073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.711+0000 7f2f7c400700 1 --2- 172.21.15.116:0/3758916682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f74073810 0x7f2f74073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.711+0000 7f2f7c400700 1 -- 172.21.15.116:0/3758916682 >> 172.21.15.116:0/3758916682 conn(0x7f2f740fc3c0 msgr2=0x7f2f740fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.711+0000 7f2f7c400700 1 -- 172.21.15.116:0/3758916682 shutdown_connections 2024-03-30T00:55:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.711+0000 7f2f7c400700 1 -- 172.21.15.116:0/3758916682 wait complete. 2024-03-30T00:55:13.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.712+0000 7f2f7c400700 1 Processor -- start 2024-03-30T00:55:13.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.712+0000 7f2f7c400700 1 -- start start 2024-03-30T00:55:13.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f7c400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f74073810 0x7f2f74199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:13.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f7c400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f74074de0 0x7f2f741995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:13.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f7c400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f74108cd0 0x7f2f741a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:13.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f7c400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f741a0bc0 con 0x7f2f74108cd0 2024-03-30T00:55:13.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f7c400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f741a0d00 con 0x7f2f74074de0 2024-03-30T00:55:13.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f7c400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f741a0e90 con 0x7f2f74073810 2024-03-30T00:55:13.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f79a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f74073810 0x7f2f74199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:13.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f79a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f74073810 0x7f2f74199080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:49642/0 (socket says 172.21.15.116:49642) 2024-03-30T00:55:13.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f79a00700 1 -- 172.21.15.116:0/1293400920 learned_addr learned my addr 172.21.15.116:0/1293400920 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:13.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f7a400700 1 --2- 172.21.15.116:0/1293400920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f74108cd0 0x7f2f741a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:13.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f79000700 1 --2- 172.21.15.116:0/1293400920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f74074de0 0x7f2f741995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:13.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f79a00700 1 -- 172.21.15.116:0/1293400920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f74074de0 msgr2=0x7f2f741995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:13.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f79a00700 1 --2- 172.21.15.116:0/1293400920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f74074de0 0x7f2f741995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:13.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f79a00700 1 -- 172.21.15.116:0/1293400920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f74108cd0 msgr2=0x7f2f741a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:13.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f79a00700 1 --2- 172.21.15.116:0/1293400920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f74108cd0 0x7f2f741a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:13.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.713+0000 7f2f79a00700 1 -- 172.21.15.116:0/1293400920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c009c40 con 0x7f2f74073810 2024-03-30T00:55:13.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.714+0000 7f2f79a00700 1 --2- 172.21.15.116:0/1293400920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f74073810 0x7f2f74199080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800e9a0 tx=0x7f2f6800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:13.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.714+0000 7f2f72000700 1 -- 172.21.15.116:0/1293400920 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68009790 con 0x7f2f74073810 2024-03-30T00:55:13.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.714+0000 7f2f72000700 1 -- 172.21.15.116:0/1293400920 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f68005020 con 0x7f2f74073810 2024-03-30T00:55:13.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.714+0000 7f2f72000700 1 -- 172.21.15.116:0/1293400920 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6800ccc0 con 0x7f2f74073810 2024-03-30T00:55:13.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.714+0000 7f2f7c400700 1 -- 172.21.15.116:0/1293400920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f741a1170 con 0x7f2f74073810 2024-03-30T00:55:13.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.714+0000 7f2f7c400700 1 -- 172.21.15.116:0/1293400920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f741a1620 con 0x7f2f74073810 2024-03-30T00:55:13.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.715+0000 7f2f7c400700 1 -- 172.21.15.116:0/1293400920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f74107a90 con 0x7f2f74073810 2024-03-30T00:55:13.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.717+0000 7f2f72000700 1 -- 172.21.15.116:0/1293400920 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f2f68005490 con 0x7f2f74073810 2024-03-30T00:55:13.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.718+0000 7f2f72000700 1 --2- 172.21.15.116:0/1293400920 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2f5c0a11a0 0x7f2f5c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:13.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.719+0000 7f2f72000700 1 -- 172.21.15.116:0/1293400920 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2f68014070 con 0x7f2f74073810 2024-03-30T00:55:13.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.719+0000 7f2f79000700 1 --2- 172.21.15.116:0/1293400920 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2f5c0a11a0 0x7f2f5c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:13.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.719+0000 7f2f79000700 1 --2- 172.21.15.116:0/1293400920 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2f5c0a11a0 0x7f2f5c0a3660 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2f74074b10 tx=0x7f2f60009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:13.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.722+0000 7f2f72000700 1 -- 172.21.15.116:0/1293400920 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2f68096370 con 0x7f2f74073810 2024-03-30T00:55:13.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.924+0000 7f2f7c400700 1 -- 172.21.15.116:0/1293400920 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f2f7404ea80 con 0x7f2f74073810 2024-03-30T00:55:13.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.925+0000 7f2f72000700 1 -- 172.21.15.116:0/1293400920 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v69) v1 ==== 74+0+133 (secure 0 0 0) 0x7f2f68099990 con 0x7f2f74073810 2024-03-30T00:55:13.937 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:13.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.927+0000 7f2f7c400700 1 -- 172.21.15.116:0/1293400920 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2f5c0a11a0 msgr2=0x7f2f5c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:13.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.927+0000 7f2f7c400700 1 --2- 172.21.15.116:0/1293400920 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2f5c0a11a0 0x7f2f5c0a3660 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2f74074b10 tx=0x7f2f60009310 comp rx=0 tx=0).stop 2024-03-30T00:55:13.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.927+0000 7f2f7c400700 1 -- 172.21.15.116:0/1293400920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f74073810 msgr2=0x7f2f74199080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:13.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.927+0000 7f2f7c400700 1 --2- 172.21.15.116:0/1293400920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f74073810 0x7f2f74199080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800e9a0 tx=0x7f2f6800ecb0 comp rx=0 tx=0).stop 2024-03-30T00:55:13.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.927+0000 7f2f7c400700 1 -- 172.21.15.116:0/1293400920 shutdown_connections 2024-03-30T00:55:13.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.927+0000 7f2f7c400700 1 --2- 172.21.15.116:0/1293400920 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2f5c0a11a0 0x7f2f5c0a3660 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:13.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.927+0000 7f2f7c400700 1 --2- 172.21.15.116:0/1293400920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f74073810 0x7f2f74199080 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:13.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.927+0000 7f2f7c400700 1 --2- 172.21.15.116:0/1293400920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f74108cd0 0x7f2f741a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:13.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.927+0000 7f2f7c400700 1 --2- 172.21.15.116:0/1293400920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f74074de0 0x7f2f741995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:13.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.927+0000 7f2f7c400700 1 -- 172.21.15.116:0/1293400920 >> 172.21.15.116:0/1293400920 conn(0x7f2f740fc3c0 msgr2=0x7f2f74104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:13.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.927+0000 7f2f7c400700 1 -- 172.21.15.116:0/1293400920 shutdown_connections 2024-03-30T00:55:13.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:13.927+0000 7f2f7c400700 1 -- 172.21.15.116:0/1293400920 wait complete. 2024-03-30T00:55:14.002 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 00:55:13 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T00:55:13.704+0000 7f7be11b7640 -1 osd.11 0 log_to_monitors true 2024-03-30T00:55:14.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:14 smithi116 ceph-mon[38086]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-30T00:55:14.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:14 smithi116 ceph-mon[38086]: from='osd.11 [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-30T00:55:14.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:14 smithi116 ceph-mon[38086]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-30T00:55:14.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:14 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1293400920' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T00:55:14.321 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":69,"num_osds":12,"num_up_osds":11,"osd_up_since":1711760098,"num_in_osds":12,"osd_in_since":1711760100,"num_remapped_pgs":0} 2024-03-30T00:55:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:14 smithi181 ceph-mon[47622]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-30T00:55:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:14 smithi181 ceph-mon[47622]: from='osd.11 [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-30T00:55:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:14 smithi181 ceph-mon[47622]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-30T00:55:14.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:14 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1293400920' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T00:55:14.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:14 smithi130 ceph-mon[47626]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-30T00:55:14.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:14 smithi130 ceph-mon[47626]: from='osd.11 [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-30T00:55:14.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:14 smithi130 ceph-mon[47626]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-30T00:55:14.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:14 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1293400920' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T00:55:15.322 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-03-30T00:55:15.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T00:55:15.190+0000 7f7bd681b700 -1 osd.11 0 waiting for initial osdmap 2024-03-30T00:55:15.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T00:55:15.208+0000 7f7bd2610700 -1 osd.11 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T00:55:15.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-30T00:55:15.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: osdmap e70: 12 total, 11 up, 12 in 2024-03-30T00:55:15.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:15.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: from='osd.11 [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:55:15.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:55:15.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: Detected new or changed devices on smithi181 2024-03-30T00:55:15.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:15.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:15.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:55:15.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:55:15.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:55:15.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:55:15.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi181 to 4278M 2024-03-30T00:55:15.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:15.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:15.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:15.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:15 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:15.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-30T00:55:15.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: osdmap e70: 12 total, 11 up, 12 in 2024-03-30T00:55:15.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:15.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: from='osd.11 [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:55:15.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:55:15.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: Detected new or changed devices on smithi181 2024-03-30T00:55:15.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:15.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:15.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:55:15.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:55:15.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:55:15.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:55:15.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi181 to 4278M 2024-03-30T00:55:15.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:15.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:15.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:15.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:15 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:15.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-30T00:55:15.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: osdmap e70: 12 total, 11 up, 12 in 2024-03-30T00:55:15.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:15.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: from='osd.11 [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:55:15.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-30T00:55:15.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: Detected new or changed devices on smithi181 2024-03-30T00:55:15.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:15.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:15.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:55:15.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:55:15.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:55:15.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T00:55:15.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi181 to 4278M 2024-03-30T00:55:15.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:15.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:15.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:15.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:15 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:15.678 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:55:16.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:16 smithi181 ceph-mon[47622]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-30T00:55:16.622 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:16 smithi181 ceph-mon[47622]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-30T00:55:16.623 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:16 smithi181 ceph-mon[47622]: osdmap e71: 12 total, 11 up, 12 in 2024-03-30T00:55:16.623 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:16 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:16.623 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:16 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:16.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:16 smithi116 ceph-mon[38086]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-30T00:55:16.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:16 smithi116 ceph-mon[38086]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-30T00:55:16.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:16 smithi116 ceph-mon[38086]: osdmap e71: 12 total, 11 up, 12 in 2024-03-30T00:55:16.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:16 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:16.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:16 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:16.624 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:16 smithi130 ceph-mon[47626]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-30T00:55:16.625 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:16 smithi130 ceph-mon[47626]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-30T00:55:16.625 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:16 smithi130 ceph-mon[47626]: osdmap e71: 12 total, 11 up, 12 in 2024-03-30T00:55:16.625 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:16 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:16.625 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:16 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:16.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.836+0000 7f7039800700 1 -- 172.21.15.116:0/1122441999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70341041b0 msgr2=0x7f70341065a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:16.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.836+0000 7f7039800700 1 --2- 172.21.15.116:0/1122441999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70341041b0 0x7f70341065a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7020009990 tx=0x7f7020009ca0 comp rx=0 tx=0).stop 2024-03-30T00:55:16.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.836+0000 7f7039800700 1 -- 172.21.15.116:0/1122441999 shutdown_connections 2024-03-30T00:55:16.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.836+0000 7f7039800700 1 --2- 172.21.15.116:0/1122441999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7034101880 0x7f7034103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:16.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.836+0000 7f7039800700 1 --2- 172.21.15.116:0/1122441999 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7034106ae0 0x7f7034108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:16.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.836+0000 7f7039800700 1 --2- 172.21.15.116:0/1122441999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70341041b0 0x7f70341065a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:16.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.836+0000 7f7039800700 1 -- 172.21.15.116:0/1122441999 >> 172.21.15.116:0/1122441999 conn(0x7f70340fb2b0 msgr2=0x7f70340fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:16.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.836+0000 7f7039800700 1 -- 172.21.15.116:0/1122441999 shutdown_connections 2024-03-30T00:55:16.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.836+0000 7f7039800700 1 -- 172.21.15.116:0/1122441999 wait complete. 2024-03-30T00:55:16.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.837+0000 7f7039800700 1 Processor -- start 2024-03-30T00:55:16.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.837+0000 7f7039800700 1 -- start start 2024-03-30T00:55:16.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7039800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7034101880 0x7f7034101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:16.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7039800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70341041b0 0x7f70340ff830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:16.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7039800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7034106ae0 0x7f70340ffd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:16.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7039800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7034100510 con 0x7f70341041b0 2024-03-30T00:55:16.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7039800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7034100680 con 0x7f7034101880 2024-03-30T00:55:16.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7039800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70341a0dc0 con 0x7f7034106ae0 2024-03-30T00:55:16.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7032000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70341041b0 0x7f70340ff830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:16.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7032000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70341041b0 0x7f70340ff830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38906/0 (socket says 172.21.15.116:38906) 2024-03-30T00:55:16.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7032000700 1 -- 172.21.15.116:0/4220991502 learned_addr learned my addr 172.21.15.116:0/4220991502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:16.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7032a00700 1 --2- 172.21.15.116:0/4220991502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7034101880 0x7f7034101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:16.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7033400700 1 --2- 172.21.15.116:0/4220991502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7034106ae0 0x7f70340ffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:16.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7032000700 1 -- 172.21.15.116:0/4220991502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7034106ae0 msgr2=0x7f70340ffd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:16.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7032000700 1 --2- 172.21.15.116:0/4220991502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7034106ae0 0x7f70340ffd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:16.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7032000700 1 -- 172.21.15.116:0/4220991502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7034101880 msgr2=0x7f7034101190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:16.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7032000700 1 --2- 172.21.15.116:0/4220991502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7034101880 0x7f7034101190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:16.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.838+0000 7f7032000700 1 -- 172.21.15.116:0/4220991502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70200095f0 con 0x7f70341041b0 2024-03-30T00:55:16.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.839+0000 7f7032000700 1 --2- 172.21.15.116:0/4220991502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70341041b0 0x7f70340ff830 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f702000b680 tx=0x7f7020009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:16.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.839+0000 7f702b400700 1 -- 172.21.15.116:0/4220991502 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702001d070 con 0x7f70341041b0 2024-03-30T00:55:16.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.839+0000 7f702b400700 1 -- 172.21.15.116:0/4220991502 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7020022470 con 0x7f70341041b0 2024-03-30T00:55:16.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.839+0000 7f7039800700 1 -- 172.21.15.116:0/4220991502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70341a0ff0 con 0x7f70341041b0 2024-03-30T00:55:16.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.839+0000 7f702b400700 1 -- 172.21.15.116:0/4220991502 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7020017650 con 0x7f70341041b0 2024-03-30T00:55:16.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.839+0000 7f7039800700 1 -- 172.21.15.116:0/4220991502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70341a1400 con 0x7f70341041b0 2024-03-30T00:55:16.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.840+0000 7f7039800700 1 -- 172.21.15.116:0/4220991502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70340fce30 con 0x7f70341041b0 2024-03-30T00:55:16.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.842+0000 7f702b400700 1 -- 172.21.15.116:0/4220991502 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f70200225e0 con 0x7f70341041b0 2024-03-30T00:55:16.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.843+0000 7f702b400700 1 --2- 172.21.15.116:0/4220991502 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f70180a1220 0x7f70180a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:16.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.843+0000 7f702b400700 1 -- 172.21.15.116:0/4220991502 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f70200cd720 con 0x7f70341041b0 2024-03-30T00:55:16.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.843+0000 7f7032a00700 1 --2- 172.21.15.116:0/4220991502 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f70180a1220 0x7f70180a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:16.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.844+0000 7f7032a00700 1 --2- 172.21.15.116:0/4220991502 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f70180a1220 0x7f70180a36e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7024009c60 tx=0x7f7024009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:16.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:16.847+0000 7f702b400700 1 -- 172.21.15.116:0/4220991502 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7020097340 con 0x7f70341041b0 2024-03-30T00:55:17.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:17.047+0000 7f7039800700 1 -- 172.21.15.116:0/4220991502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f703404ea80 con 0x7f70341041b0 2024-03-30T00:55:17.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:17.048+0000 7f702b400700 1 -- 172.21.15.116:0/4220991502 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) v1 ==== 74+0+133 (secure 0 0 0) 0x7f702009a960 con 0x7f70341041b0 2024-03-30T00:55:17.054 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:17.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:17.050+0000 7f7039800700 1 -- 172.21.15.116:0/4220991502 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f70180a1220 msgr2=0x7f70180a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:17.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:17.050+0000 7f7039800700 1 --2- 172.21.15.116:0/4220991502 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f70180a1220 0x7f70180a36e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7024009c60 tx=0x7f7024009260 comp rx=0 tx=0).stop 2024-03-30T00:55:17.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:17.050+0000 7f7039800700 1 -- 172.21.15.116:0/4220991502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70341041b0 msgr2=0x7f70340ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:17.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:17.050+0000 7f7039800700 1 --2- 172.21.15.116:0/4220991502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70341041b0 0x7f70340ff830 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f702000b680 tx=0x7f7020009580 comp rx=0 tx=0).stop 2024-03-30T00:55:17.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:17.050+0000 7f7039800700 1 -- 172.21.15.116:0/4220991502 shutdown_connections 2024-03-30T00:55:17.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:17.050+0000 7f7039800700 1 --2- 172.21.15.116:0/4220991502 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f70180a1220 0x7f70180a36e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:17.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:17.050+0000 7f7039800700 1 --2- 172.21.15.116:0/4220991502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7034106ae0 0x7f70340ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:17.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:17.050+0000 7f7039800700 1 --2- 172.21.15.116:0/4220991502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7034101880 0x7f7034101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:17.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:17.050+0000 7f7039800700 1 --2- 172.21.15.116:0/4220991502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70341041b0 0x7f70340ff830 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:17.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:17.050+0000 7f7039800700 1 -- 172.21.15.116:0/4220991502 >> 172.21.15.116:0/4220991502 conn(0x7f70340fb2b0 msgr2=0x7f7034103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:17.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:17.050+0000 7f7039800700 1 -- 172.21.15.116:0/4220991502 shutdown_connections 2024-03-30T00:55:17.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:17.050+0000 7f7039800700 1 -- 172.21.15.116:0/4220991502 wait complete. 2024-03-30T00:55:17.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:17 smithi116 ceph-mon[38086]: purged_snaps scrub starts 2024-03-30T00:55:17.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:17 smithi116 ceph-mon[38086]: purged_snaps scrub ok 2024-03-30T00:55:17.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:17 smithi116 ceph-mon[38086]: osd.11 [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272] boot 2024-03-30T00:55:17.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:17 smithi116 ceph-mon[38086]: osdmap e72: 12 total, 12 up, 12 in 2024-03-30T00:55:17.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:17 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:17.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:17 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/4220991502' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T00:55:17.492 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1711760116,"num_in_osds":12,"osd_in_since":1711760100,"num_remapped_pgs":0} 2024-03-30T00:55:17.493 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T00:55:17.501 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:17 smithi181 ceph-mon[47622]: purged_snaps scrub starts 2024-03-30T00:55:17.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:17 smithi181 ceph-mon[47622]: purged_snaps scrub ok 2024-03-30T00:55:17.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:17 smithi181 ceph-mon[47622]: osd.11 [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272] boot 2024-03-30T00:55:17.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:17 smithi181 ceph-mon[47622]: osdmap e72: 12 total, 12 up, 12 in 2024-03-30T00:55:17.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:17 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:17.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:17 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/4220991502' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T00:55:17.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:17 smithi130 ceph-mon[47626]: purged_snaps scrub starts 2024-03-30T00:55:17.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:17 smithi130 ceph-mon[47626]: purged_snaps scrub ok 2024-03-30T00:55:17.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:17 smithi130 ceph-mon[47626]: osd.11 [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272] boot 2024-03-30T00:55:17.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:17 smithi130 ceph-mon[47626]: osdmap e72: 12 total, 12 up, 12 in 2024-03-30T00:55:17.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:17 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T00:55:17.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:17 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/4220991502' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T00:55:17.847 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:55:18.293 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:18 smithi116 ceph-mon[38086]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-30T00:55:18.293 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:18 smithi116 ceph-mon[38086]: osdmap e73: 12 total, 12 up, 12 in 2024-03-30T00:55:18.501 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:18 smithi181 ceph-mon[47622]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-30T00:55:18.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:18 smithi181 ceph-mon[47622]: osdmap e73: 12 total, 12 up, 12 in 2024-03-30T00:55:18.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:18 smithi130 ceph-mon[47626]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-30T00:55:18.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:18 smithi130 ceph-mon[47626]: osdmap e73: 12 total, 12 up, 12 in 2024-03-30T00:55:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.910+0000 7f8b48c00700 1 -- 172.21.15.116:0/692969989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b44073900 msgr2=0x7f8b4410d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.910+0000 7f8b48c00700 1 --2- 172.21.15.116:0/692969989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b44073900 0x7f8b4410d2d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8b30009990 tx=0x7f8b30009ca0 comp rx=0 tx=0).stop 2024-03-30T00:55:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.910+0000 7f8b48c00700 1 -- 172.21.15.116:0/692969989 shutdown_connections 2024-03-30T00:55:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.910+0000 7f8b48c00700 1 --2- 172.21.15.116:0/692969989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b4410d810 0x7f8b4410fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.910+0000 7f8b48c00700 1 --2- 172.21.15.116:0/692969989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b44072fe0 0x7f8b440733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.910+0000 7f8b48c00700 1 --2- 172.21.15.116:0/692969989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b44073900 0x7f8b4410d2d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.910+0000 7f8b48c00700 1 -- 172.21.15.116:0/692969989 >> 172.21.15.116:0/692969989 conn(0x7f8b44077fa0 msgr2=0x7f8b440783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.910+0000 7f8b48c00700 1 -- 172.21.15.116:0/692969989 shutdown_connections 2024-03-30T00:55:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.910+0000 7f8b48c00700 1 -- 172.21.15.116:0/692969989 wait complete. 2024-03-30T00:55:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.911+0000 7f8b48c00700 1 Processor -- start 2024-03-30T00:55:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.911+0000 7f8b48c00700 1 -- start start 2024-03-30T00:55:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.911+0000 7f8b48c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b44072fe0 0x7f8b441994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.911+0000 7f8b48c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b44073900 0x7f8b44199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b48c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b4410d810 0x7f8b4419dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b48c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4419e390 con 0x7f8b44072fe0 2024-03-30T00:55:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b48c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4419e500 con 0x7f8b44073900 2024-03-30T00:55:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b48c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4419e6c0 con 0x7f8b4410d810 2024-03-30T00:55:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b41e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b44072fe0 0x7f8b441994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b41e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b44072fe0 0x7f8b441994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38922/0 (socket says 172.21.15.116:38922) 2024-03-30T00:55:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b41e00700 1 -- 172.21.15.116:0/3119151011 learned_addr learned my addr 172.21.15.116:0/3119151011 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b42800700 1 --2- 172.21.15.116:0/3119151011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b4410d810 0x7f8b4419dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b41400700 1 --2- 172.21.15.116:0/3119151011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b44073900 0x7f8b44199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b41e00700 1 -- 172.21.15.116:0/3119151011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b4410d810 msgr2=0x7f8b4419dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b41e00700 1 --2- 172.21.15.116:0/3119151011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b4410d810 0x7f8b4419dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b41e00700 1 -- 172.21.15.116:0/3119151011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b44073900 msgr2=0x7f8b44199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b41e00700 1 --2- 172.21.15.116:0/3119151011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b44073900 0x7f8b44199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b41e00700 1 -- 172.21.15.116:0/3119151011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b300095f0 con 0x7f8b44072fe0 2024-03-30T00:55:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.912+0000 7f8b41e00700 1 --2- 172.21.15.116:0/3119151011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b44072fe0 0x7f8b441994e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400e9a0 tx=0x7f8b3400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.913+0000 7f8b3aa00700 1 -- 172.21.15.116:0/3119151011 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34009790 con 0x7f8b44072fe0 2024-03-30T00:55:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.913+0000 7f8b3aa00700 1 -- 172.21.15.116:0/3119151011 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b34010430 con 0x7f8b44072fe0 2024-03-30T00:55:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.913+0000 7f8b48c00700 1 -- 172.21.15.116:0/3119151011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b4419e9a0 con 0x7f8b44072fe0 2024-03-30T00:55:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.913+0000 7f8b3aa00700 1 -- 172.21.15.116:0/3119151011 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34018680 con 0x7f8b44072fe0 2024-03-30T00:55:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.913+0000 7f8b48c00700 1 -- 172.21.15.116:0/3119151011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b441a6200 con 0x7f8b44072fe0 2024-03-30T00:55:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.914+0000 7f8b48c00700 1 -- 172.21.15.116:0/3119151011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b44108a20 con 0x7f8b44072fe0 2024-03-30T00:55:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.916+0000 7f8b3aa00700 1 -- 172.21.15.116:0/3119151011 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f8b340105a0 con 0x7f8b44072fe0 2024-03-30T00:55:18.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.917+0000 7f8b3aa00700 1 --2- 172.21.15.116:0/3119151011 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8b280a1120 0x7f8b280a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:18.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.917+0000 7f8b3aa00700 1 -- 172.21.15.116:0/3119151011 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8b34014070 con 0x7f8b44072fe0 2024-03-30T00:55:18.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.917+0000 7f8b41400700 1 --2- 172.21.15.116:0/3119151011 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8b280a1120 0x7f8b280a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:18.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.918+0000 7f8b41400700 1 --2- 172.21.15.116:0/3119151011 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8b280a1120 0x7f8b280a35e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8b4419a9f0 tx=0x7f8b30005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:18.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:18.921+0000 7f8b3aa00700 1 -- 172.21.15.116:0/3119151011 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8b340529f0 con 0x7f8b44072fe0 2024-03-30T00:55:19.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:19.122+0000 7f8b48c00700 1 -- 172.21.15.116:0/3119151011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b4404eab0 con 0x7f8b44072fe0 2024-03-30T00:55:19.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:19.123+0000 7f8b3aa00700 1 -- 172.21.15.116:0/3119151011 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+18490 (secure 0 0 0) 0x7f8b34096220 con 0x7f8b44072fe0 2024-03-30T00:55:19.130 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:19.130 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":73,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:55:17.191932+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:55:19.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:19.125+0000 7f8b48c00700 1 -- 172.21.15.116:0/3119151011 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8b280a1120 msgr2=0x7f8b280a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:19.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:19.125+0000 7f8b48c00700 1 --2- 172.21.15.116:0/3119151011 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8b280a1120 0x7f8b280a35e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8b4419a9f0 tx=0x7f8b30005fd0 comp rx=0 tx=0).stop 2024-03-30T00:55:19.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:19.125+0000 7f8b48c00700 1 -- 172.21.15.116:0/3119151011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b44072fe0 msgr2=0x7f8b441994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:19.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:19.125+0000 7f8b48c00700 1 --2- 172.21.15.116:0/3119151011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b44072fe0 0x7f8b441994e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400e9a0 tx=0x7f8b3400ecb0 comp rx=0 tx=0).stop 2024-03-30T00:55:19.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:19.125+0000 7f8b48c00700 1 -- 172.21.15.116:0/3119151011 shutdown_connections 2024-03-30T00:55:19.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:19.125+0000 7f8b48c00700 1 --2- 172.21.15.116:0/3119151011 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8b280a1120 0x7f8b280a35e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:19.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:19.125+0000 7f8b48c00700 1 --2- 172.21.15.116:0/3119151011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b4410d810 0x7f8b4419dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:19.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:19.125+0000 7f8b48c00700 1 --2- 172.21.15.116:0/3119151011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b44072fe0 0x7f8b441994e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:19.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:19.125+0000 7f8b48c00700 1 --2- 172.21.15.116:0/3119151011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b44073900 0x7f8b44199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:19.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:19.125+0000 7f8b48c00700 1 -- 172.21.15.116:0/3119151011 >> 172.21.15.116:0/3119151011 conn(0x7f8b44077fa0 msgr2=0x7f8b440fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:19.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:19.125+0000 7f8b48c00700 1 -- 172.21.15.116:0/3119151011 shutdown_connections 2024-03-30T00:55:19.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:19.125+0000 7f8b48c00700 1 -- 172.21.15.116:0/3119151011 wait complete. 2024-03-30T00:55:19.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:19 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3119151011' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:19.494 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T00:52:29.602428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-30T00:55:19.495 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-03-30T00:55:19.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:19 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3119151011' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:19.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:19 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3119151011' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:19.847 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:55:20.286 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:20 smithi116 ceph-mon[38086]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:20.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:20 smithi181 ceph-mon[47622]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:20.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:20 smithi130 ceph-mon[47626]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.870+0000 7f7ce9000700 1 -- 172.21.15.116:0/1384344663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce4106ef0 msgr2=0x7f7ce41072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.870+0000 7f7ce9000700 1 --2- 172.21.15.116:0/1384344663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce4106ef0 0x7f7ce41072d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009920 tx=0x7f7cd4009c30 comp rx=0 tx=0).stop 2024-03-30T00:55:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.870+0000 7f7ce9000700 1 -- 172.21.15.116:0/1384344663 shutdown_connections 2024-03-30T00:55:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.870+0000 7f7ce9000700 1 --2- 172.21.15.116:0/1384344663 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ce4101820 0x7f7ce410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.870+0000 7f7ce9000700 1 --2- 172.21.15.116:0/1384344663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce4106ef0 0x7f7ce41072d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.870+0000 7f7ce9000700 1 --2- 172.21.15.116:0/1384344663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ce4100e60 0x7f7ce41012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.870+0000 7f7ce9000700 1 -- 172.21.15.116:0/1384344663 >> 172.21.15.116:0/1384344663 conn(0x7f7ce4077fa0 msgr2=0x7f7ce40783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.870+0000 7f7ce9000700 1 -- 172.21.15.116:0/1384344663 shutdown_connections 2024-03-30T00:55:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.870+0000 7f7ce9000700 1 -- 172.21.15.116:0/1384344663 wait complete. 2024-03-30T00:55:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.871+0000 7f7ce9000700 1 Processor -- start 2024-03-30T00:55:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.871+0000 7f7ce9000700 1 -- start start 2024-03-30T00:55:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7ce9000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ce4100e60 0x7f7ce40730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7ce9000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce4101820 0x7f7ce40735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7ce9000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ce4106ef0 0x7f7ce4073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7ce9000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce41a56c0 con 0x7f7ce4101820 2024-03-30T00:55:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7ce9000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce41a5830 con 0x7f7ce4100e60 2024-03-30T00:55:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7ce9000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce41a59f0 con 0x7f7ce4106ef0 2024-03-30T00:55:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7cdbe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce4101820 0x7f7ce40735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7cdbe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce4101820 0x7f7ce40735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38944/0 (socket says 172.21.15.116:38944) 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7cdbe00700 1 -- 172.21.15.116:0/3396154694 learned_addr learned my addr 172.21.15.116:0/3396154694 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7ce1e00700 1 --2- 172.21.15.116:0/3396154694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ce4100e60 0x7f7ce40730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7ce2800700 1 --2- 172.21.15.116:0/3396154694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ce4106ef0 0x7f7ce4073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7cdbe00700 1 -- 172.21.15.116:0/3396154694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ce4106ef0 msgr2=0x7f7ce4073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7cdbe00700 1 --2- 172.21.15.116:0/3396154694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ce4106ef0 0x7f7ce4073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7cdbe00700 1 -- 172.21.15.116:0/3396154694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ce4100e60 msgr2=0x7f7ce40730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7cdbe00700 1 --2- 172.21.15.116:0/3396154694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ce4100e60 0x7f7ce40730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.872+0000 7f7cdbe00700 1 -- 172.21.15.116:0/3396154694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc009d00 con 0x7f7ce4101820 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.873+0000 7f7cdbe00700 1 --2- 172.21.15.116:0/3396154694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce4101820 0x7f7ce40735f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00e9a0 tx=0x7f7ccc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.873+0000 7f7cdaa00700 1 -- 172.21.15.116:0/3396154694 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc009790 con 0x7f7ce4101820 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.873+0000 7f7cdaa00700 1 -- 172.21.15.116:0/3396154694 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ccc00cbd0 con 0x7f7ce4101820 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.873+0000 7f7cdaa00700 1 -- 172.21.15.116:0/3396154694 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc018680 con 0x7f7ce4101820 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.873+0000 7f7ce9000700 1 -- 172.21.15.116:0/3396154694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd4009520 con 0x7f7ce4101820 2024-03-30T00:55:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.873+0000 7f7ce9000700 1 -- 172.21.15.116:0/3396154694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce41a5ff0 con 0x7f7ce4101820 2024-03-30T00:55:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.874+0000 7f7ce9000700 1 -- 172.21.15.116:0/3396154694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce41092d0 con 0x7f7ce4101820 2024-03-30T00:55:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.876+0000 7f7cdaa00700 1 -- 172.21.15.116:0/3396154694 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f7ccc00cd40 con 0x7f7ce4101820 2024-03-30T00:55:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.877+0000 7f7cdaa00700 1 --2- 172.21.15.116:0/3396154694 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7cc80a1120 0x7f7cc80a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.877+0000 7f7cdaa00700 1 -- 172.21.15.116:0/3396154694 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7ccc014070 con 0x7f7ce4101820 2024-03-30T00:55:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.877+0000 7f7ce1e00700 1 --2- 172.21.15.116:0/3396154694 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7cc80a1120 0x7f7cc80a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.878+0000 7f7ce1e00700 1 --2- 172.21.15.116:0/3396154694 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7cc80a1120 0x7f7cc80a35e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7cd400c010 tx=0x7f7cd4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:20.879+0000 7f7cdaa00700 1 -- 172.21.15.116:0/3396154694 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7ccc0965c0 con 0x7f7ce4101820 2024-03-30T00:55:21.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:21.081+0000 7f7ce9000700 1 -- 172.21.15.116:0/3396154694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ce4074590 con 0x7f7ce4101820 2024-03-30T00:55:21.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:21.082+0000 7f7cdaa00700 1 -- 172.21.15.116:0/3396154694 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ccc099be0 con 0x7f7ce4101820 2024-03-30T00:55:21.088 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-30T00:55:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:21.084+0000 7f7ce9000700 1 -- 172.21.15.116:0/3396154694 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7cc80a1120 msgr2=0x7f7cc80a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:21.084+0000 7f7ce9000700 1 --2- 172.21.15.116:0/3396154694 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7cc80a1120 0x7f7cc80a35e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7cd400c010 tx=0x7f7cd4019040 comp rx=0 tx=0).stop 2024-03-30T00:55:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:21.084+0000 7f7ce9000700 1 -- 172.21.15.116:0/3396154694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce4101820 msgr2=0x7f7ce40735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:21.084+0000 7f7ce9000700 1 --2- 172.21.15.116:0/3396154694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce4101820 0x7f7ce40735f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00e9a0 tx=0x7f7ccc00ecb0 comp rx=0 tx=0).stop 2024-03-30T00:55:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:21.084+0000 7f7ce9000700 1 -- 172.21.15.116:0/3396154694 shutdown_connections 2024-03-30T00:55:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:21.084+0000 7f7ce9000700 1 --2- 172.21.15.116:0/3396154694 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7cc80a1120 0x7f7cc80a35e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:21.084+0000 7f7ce9000700 1 --2- 172.21.15.116:0/3396154694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ce4106ef0 0x7f7ce4073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:21.084+0000 7f7ce9000700 1 --2- 172.21.15.116:0/3396154694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ce4100e60 0x7f7ce40730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:21.084+0000 7f7ce9000700 1 --2- 172.21.15.116:0/3396154694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ce4101820 0x7f7ce40735f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:21.084+0000 7f7ce9000700 1 -- 172.21.15.116:0/3396154694 >> 172.21.15.116:0/3396154694 conn(0x7f7ce4077fa0 msgr2=0x7f7ce40ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:21.084+0000 7f7ce9000700 1 -- 172.21.15.116:0/3396154694 shutdown_connections 2024-03-30T00:55:21.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:21.084+0000 7f7ce9000700 1 -- 172.21.15.116:0/3396154694 wait complete. 2024-03-30T00:55:21.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:21 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3396154694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:55:21.462 INFO:tasks.cephadm:Adding mds.a on smithi116 2024-03-30T00:55:21.463 INFO:tasks.cephadm:Adding mds.d on smithi116 2024-03-30T00:55:21.463 INFO:tasks.cephadm:Adding mds.g on smithi116 2024-03-30T00:55:21.463 INFO:tasks.cephadm:Adding mds.j on smithi116 2024-03-30T00:55:21.463 INFO:tasks.cephadm:Adding mds.b on smithi130 2024-03-30T00:55:21.463 INFO:tasks.cephadm:Adding mds.e on smithi130 2024-03-30T00:55:21.463 INFO:tasks.cephadm:Adding mds.h on smithi130 2024-03-30T00:55:21.463 INFO:tasks.cephadm:Adding mds.k on smithi130 2024-03-30T00:55:21.463 INFO:tasks.cephadm:Adding mds.c on smithi181 2024-03-30T00:55:21.463 INFO:tasks.cephadm:Adding mds.f on smithi181 2024-03-30T00:55:21.463 INFO:tasks.cephadm:Adding mds.i on smithi181 2024-03-30T00:55:21.463 INFO:tasks.cephadm:Adding mds.l on smithi181 2024-03-30T00:55:21.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph orch apply mds all '12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l' 2024-03-30T00:55:21.497 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:21 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3396154694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:55:21.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:21 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3396154694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:55:21.700 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T00:55:22.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:22 smithi181 ceph-mon[47622]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:22.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:22 smithi130 ceph-mon[47626]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:22.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:22 smithi116 ceph-mon[38086]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:22.685 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.678+0000 7f8a77f65700 1 -- 172.21.15.181:0/970944194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a70101870 msgr2=0x7f8a70108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:22.685 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.678+0000 7f8a77f65700 1 --2- 172.21.15.181:0/970944194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a70101870 0x7f8a70108120 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800b0a0 tx=0x7f8a6800b3b0 comp rx=0 tx=0).stop 2024-03-30T00:55:22.685 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 -- 172.21.15.181:0/970944194 shutdown_connections 2024-03-30T00:55:22.685 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 --2- 172.21.15.181:0/970944194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a70101870 0x7f8a70108120 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:22.685 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 --2- 172.21.15.181:0/970944194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a700ffcd0 0x7f8a701000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:22.686 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 --2- 172.21.15.181:0/970944194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a70100ed0 0x7f8a70101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:22.686 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 -- 172.21.15.181:0/970944194 >> 172.21.15.181:0/970944194 conn(0x7f8a700fb270 msgr2=0x7f8a700fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:22.686 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 -- 172.21.15.181:0/970944194 shutdown_connections 2024-03-30T00:55:22.686 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 -- 172.21.15.181:0/970944194 wait complete. 2024-03-30T00:55:22.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 Processor -- start 2024-03-30T00:55:22.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 -- start start 2024-03-30T00:55:22.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a700ffcd0 0x7f8a70103c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:22.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a70100ed0 0x7f8a701022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:22.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a70101870 0x7f8a701027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:22.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a70102f90 con 0x7f8a700ffcd0 2024-03-30T00:55:22.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a70103100 con 0x7f8a70100ed0 2024-03-30T00:55:22.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.679+0000 7f8a77f65700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a701a0ea0 con 0x7f8a70101870 2024-03-30T00:55:22.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a76502700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a70101870 0x7f8a701027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:22.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a76502700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a70101870 0x7f8a701027f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:43044/0 (socket says 172.21.15.181:43044) 2024-03-30T00:55:22.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a76502700 1 -- 172.21.15.181:0/2210955113 learned_addr learned my addr 172.21.15.181:0/2210955113 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-03-30T00:55:22.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a75500700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a70100ed0 0x7f8a701022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a75d01700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a700ffcd0 0x7f8a70103c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a76502700 1 -- 172.21.15.181:0/2210955113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a70100ed0 msgr2=0x7f8a701022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a76502700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a70100ed0 0x7f8a701022b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a76502700 1 -- 172.21.15.181:0/2210955113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a700ffcd0 msgr2=0x7f8a70103c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a76502700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a700ffcd0 0x7f8a70103c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a76502700 1 -- 172.21.15.181:0/2210955113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a68009d00 con 0x7f8a70101870 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a75500700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a70100ed0 0x7f8a701022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a75d01700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a700ffcd0 0x7f8a70103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a76502700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a70101870 0x7f8a701027f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8a68004d00 tx=0x7f8a68004de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a66ffd700 1 -- 172.21.15.181:0/2210955113 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6800e040 con 0x7f8a70101870 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a66ffd700 1 -- 172.21.15.181:0/2210955113 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a68009440 con 0x7f8a70101870 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a77f65700 1 -- 172.21.15.181:0/2210955113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a701a10d0 con 0x7f8a70101870 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a66ffd700 1 -- 172.21.15.181:0/2210955113 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6801b980 con 0x7f8a70101870 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.680+0000 7f8a77f65700 1 -- 172.21.15.181:0/2210955113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a701a1540 con 0x7f8a70101870 2024-03-30T00:55:22.689 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.681+0000 7f8a77f65700 1 -- 172.21.15.181:0/2210955113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a7004ea80 con 0x7f8a70101870 2024-03-30T00:55:22.690 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.682+0000 7f8a66ffd700 1 -- 172.21.15.181:0/2210955113 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f8a6801dc50 con 0x7f8a70101870 2024-03-30T00:55:22.690 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.683+0000 7f8a66ffd700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8a5c0a1120 0x7f8a5c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:22.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.683+0000 7f8a66ffd700 1 -- 172.21.15.181:0/2210955113 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8a680cea80 con 0x7f8a70101870 2024-03-30T00:55:22.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.683+0000 7f8a75d01700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8a5c0a1120 0x7f8a5c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:22.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.684+0000 7f8a75d01700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8a5c0a1120 0x7f8a5c0a35e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8a60005fd0 tx=0x7f8a60005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:22.693 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.688+0000 7f8a66ffd700 1 -- 172.21.15.181:0/2210955113 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8a680986a0 con 0x7f8a70101870 2024-03-30T00:55:22.897 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.891+0000 7f8a77f65700 1 -- 172.21.15.181:0/2210955113 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a700fd470 con 0x7f8a5c0a1120 2024-03-30T00:55:22.907 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.900+0000 7f8a66ffd700 1 -- 172.21.15.181:0/2210955113 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f8a700fd470 con 0x7f8a5c0a1120 2024-03-30T00:55:22.907 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mds.all update... 2024-03-30T00:55:22.909 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.902+0000 7f8a77f65700 1 -- 172.21.15.181:0/2210955113 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8a5c0a1120 msgr2=0x7f8a5c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:22.909 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.902+0000 7f8a77f65700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8a5c0a1120 0x7f8a5c0a35e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8a60005fd0 tx=0x7f8a60005dc0 comp rx=0 tx=0).stop 2024-03-30T00:55:22.909 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.902+0000 7f8a77f65700 1 -- 172.21.15.181:0/2210955113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a70101870 msgr2=0x7f8a701027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:22.909 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.902+0000 7f8a77f65700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a70101870 0x7f8a701027f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8a68004d00 tx=0x7f8a68004de0 comp rx=0 tx=0).stop 2024-03-30T00:55:22.909 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.902+0000 7f8a77f65700 1 -- 172.21.15.181:0/2210955113 shutdown_connections 2024-03-30T00:55:22.909 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.902+0000 7f8a77f65700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8a5c0a1120 0x7f8a5c0a35e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:22.909 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.902+0000 7f8a77f65700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a70101870 0x7f8a701027f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:22.909 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.902+0000 7f8a77f65700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a700ffcd0 0x7f8a70103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:22.910 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.902+0000 7f8a77f65700 1 --2- 172.21.15.181:0/2210955113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a70100ed0 0x7f8a701022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:22.910 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.903+0000 7f8a77f65700 1 -- 172.21.15.181:0/2210955113 >> 172.21.15.181:0/2210955113 conn(0x7f8a700fb270 msgr2=0x7f8a700fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:22.910 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.903+0000 7f8a77f65700 1 -- 172.21.15.181:0/2210955113 shutdown_connections 2024-03-30T00:55:22.910 INFO:teuthology.orchestra.run.smithi181.stderr:2024-03-30T00:55:22.903+0000 7f8a77f65700 1 -- 172.21.15.181:0/2210955113 wait complete. 2024-03-30T00:55:23.278 DEBUG:teuthology.orchestra.run.smithi116:mds.a> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.a.service 2024-03-30T00:55:23.283 DEBUG:teuthology.orchestra.run.smithi116:mds.d> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.d.service 2024-03-30T00:55:23.294 DEBUG:teuthology.orchestra.run.smithi116:mds.g> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.g.service 2024-03-30T00:55:23.344 DEBUG:teuthology.orchestra.run.smithi116:mds.j> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.j.service 2024-03-30T00:55:23.355 INFO:journalctl@ceph.mds.a.smithi116.stdout:-- Logs begin at Sat 2024-03-30 00:33:55 UTC. -- 2024-03-30T00:55:23.359 DEBUG:teuthology.orchestra.run.smithi130:mds.b> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.b.service 2024-03-30T00:55:23.362 INFO:journalctl@ceph.mds.d.smithi116.stdout:-- Logs begin at Sat 2024-03-30 00:33:55 UTC. -- 2024-03-30T00:55:23.365 DEBUG:teuthology.orchestra.run.smithi130:mds.e> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.e.service 2024-03-30T00:55:23.403 INFO:journalctl@ceph.mds.b.smithi130.stdout:-- Logs begin at Sat 2024-03-30 00:27:58 UTC. -- 2024-03-30T00:55:23.406 DEBUG:teuthology.orchestra.run.smithi130:mds.h> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.h.service 2024-03-30T00:55:23.412 INFO:journalctl@ceph.mds.g.smithi116.stdout:-- Logs begin at Sat 2024-03-30 00:33:55 UTC. -- 2024-03-30T00:55:23.425 INFO:journalctl@ceph.mds.j.smithi116.stdout:-- Logs begin at Sat 2024-03-30 00:33:55 UTC. -- 2024-03-30T00:55:23.435 INFO:journalctl@ceph.mds.e.smithi130.stdout:-- Logs begin at Sat 2024-03-30 00:27:58 UTC. -- 2024-03-30T00:55:23.437 DEBUG:teuthology.orchestra.run.smithi130:mds.k> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.k.service 2024-03-30T00:55:23.468 INFO:journalctl@ceph.mds.h.smithi130.stdout:-- Logs begin at Sat 2024-03-30 00:27:58 UTC. -- 2024-03-30T00:55:23.475 DEBUG:teuthology.orchestra.run.smithi181:mds.c> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.c.service 2024-03-30T00:55:23.479 DEBUG:teuthology.orchestra.run.smithi181:mds.f> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.f.service 2024-03-30T00:55:23.496 INFO:journalctl@ceph.mds.k.smithi130.stdout:-- Logs begin at Sat 2024-03-30 00:27:58 UTC. -- 2024-03-30T00:55:23.509 INFO:journalctl@ceph.mds.c.smithi181.stdout:-- Logs begin at Sat 2024-03-30 00:27:51 UTC. -- 2024-03-30T00:55:23.511 DEBUG:teuthology.orchestra.run.smithi181:mds.i> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.i.service 2024-03-30T00:55:23.537 INFO:journalctl@ceph.mds.f.smithi181.stdout:-- Logs begin at Sat 2024-03-30 00:27:51 UTC. -- 2024-03-30T00:55:23.539 DEBUG:teuthology.orchestra.run.smithi181:mds.l> sudo journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.l.service 2024-03-30T00:55:23.565 INFO:journalctl@ceph.mds.i.smithi181.stdout:-- Logs begin at Sat 2024-03-30 00:27:51 UTC. -- 2024-03-30T00:55:23.568 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-30T00:55:23.569 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:55:23.593 INFO:journalctl@ceph.mds.l.smithi181.stdout:-- Logs begin at Sat 2024-03-30 00:27:51 UTC. -- 2024-03-30T00:55:23.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.765+0000 7f08bf800700 1 -- 172.21.15.116:0/2950371477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08b8100c80 msgr2=0x7f08b81010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.765+0000 7f08bf800700 1 --2- 172.21.15.116:0/2950371477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08b8100c80 0x7f08b81010a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0098f0 tx=0x7f08ac009c00 comp rx=0 tx=0).stop 2024-03-30T00:55:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.765+0000 7f08bf800700 1 -- 172.21.15.116:0/2950371477 shutdown_connections 2024-03-30T00:55:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.765+0000 7f08bf800700 1 --2- 172.21.15.116:0/2950371477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08b8100c80 0x7f08b81010a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.765+0000 7f08bf800700 1 --2- 172.21.15.116:0/2950371477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08b8102810 0x7f08b81090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.765+0000 7f08bf800700 1 --2- 172.21.15.116:0/2950371477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08b8101e70 0x7f08b81022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.765+0000 7f08bf800700 1 -- 172.21.15.116:0/2950371477 >> 172.21.15.116:0/2950371477 conn(0x7f08b80fc1f0 msgr2=0x7f08b80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:23.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.766+0000 7f08bf800700 1 -- 172.21.15.116:0/2950371477 shutdown_connections 2024-03-30T00:55:23.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.766+0000 7f08bf800700 1 -- 172.21.15.116:0/2950371477 wait complete. 2024-03-30T00:55:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.767+0000 7f08bf800700 1 Processor -- start 2024-03-30T00:55:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.767+0000 7f08bf800700 1 -- start start 2024-03-30T00:55:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.767+0000 7f08bf800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08b8100c80 0x7f08b8198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.767+0000 7f08bf800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08b8101e70 0x7f08b8199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.767+0000 7f08bf800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08b8102810 0x7f08b81a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.767+0000 7f08bf800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b81a09f0 con 0x7f08b8102810 2024-03-30T00:55:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.767+0000 7f08bf800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b81a0b30 con 0x7f08b8101e70 2024-03-30T00:55:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.767+0000 7f08bf800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b81a0cc0 con 0x7f08b8100c80 2024-03-30T00:55:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.767+0000 7f08bce00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08b8100c80 0x7f08b8198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.767+0000 7f08bce00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08b8100c80 0x7f08b8198ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:52470/0 (socket says 172.21.15.116:52470) 2024-03-30T00:55:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.767+0000 7f08bd800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08b8102810 0x7f08b81a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.767+0000 7f08bce00700 1 -- 172.21.15.116:0/2495295525 learned_addr learned my addr 172.21.15.116:0/2495295525 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.767+0000 7f08b7e00700 1 --2- 172.21.15.116:0/2495295525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08b8101e70 0x7f08b8199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.768+0000 7f08bce00700 1 -- 172.21.15.116:0/2495295525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08b8101e70 msgr2=0x7f08b8199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.768+0000 7f08bce00700 1 --2- 172.21.15.116:0/2495295525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08b8101e70 0x7f08b8199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.768+0000 7f08bce00700 1 -- 172.21.15.116:0/2495295525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08b8102810 msgr2=0x7f08b81a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.768+0000 7f08bce00700 1 --2- 172.21.15.116:0/2495295525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08b8102810 0x7f08b81a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.768+0000 7f08bce00700 1 -- 172.21.15.116:0/2495295525 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a40095f0 con 0x7f08b8100c80 2024-03-30T00:55:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.768+0000 7f08bce00700 1 --2- 172.21.15.116:0/2495295525 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08b8100c80 0x7f08b8198ec0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009810 tx=0x7f08ac00bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.768+0000 7f08b5600700 1 -- 172.21.15.116:0/2495295525 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac01c070 con 0x7f08b8100c80 2024-03-30T00:55:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.768+0000 7f08b5600700 1 -- 172.21.15.116:0/2495295525 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08ac00bba0 con 0x7f08b8100c80 2024-03-30T00:55:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.768+0000 7f08b5600700 1 -- 172.21.15.116:0/2495295525 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac003900 con 0x7f08b8100c80 2024-03-30T00:55:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.768+0000 7f08bf800700 1 -- 172.21.15.116:0/2495295525 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ac009520 con 0x7f08b8100c80 2024-03-30T00:55:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.768+0000 7f08bf800700 1 -- 172.21.15.116:0/2495295525 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b81a1180 con 0x7f08b8100c80 2024-03-30T00:55:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.769+0000 7f08bf800700 1 -- 172.21.15.116:0/2495295525 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b8067a10 con 0x7f08b8100c80 2024-03-30T00:55:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.771+0000 7f08b5600700 1 -- 172.21.15.116:0/2495295525 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f08ac020030 con 0x7f08b8100c80 2024-03-30T00:55:23.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.772+0000 7f08b5600700 1 --2- 172.21.15.116:0/2495295525 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f08a00a1120 0x7f08a00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:23.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.772+0000 7f08b5600700 1 -- 172.21.15.116:0/2495295525 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f08ac0cd360 con 0x7f08b8100c80 2024-03-30T00:55:23.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.773+0000 7f08b7e00700 1 --2- 172.21.15.116:0/2495295525 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f08a00a1120 0x7f08a00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:23.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.773+0000 7f08b7e00700 1 --2- 172.21.15.116:0/2495295525 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f08a00a1120 0x7f08a00a35e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f08a4005fd0 tx=0x7f08a4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:23.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:23.776+0000 7f08b5600700 1 -- 172.21.15.116:0/2495295525 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f08ac096f80 con 0x7f08b8100c80 2024-03-30T00:55:24.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.005+0000 7f08bf800700 1 -- 172.21.15.116:0/2495295525 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08b810c4e0 con 0x7f08b8100c80 2024-03-30T00:55:24.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.007+0000 7f08b5600700 1 -- 172.21.15.116:0/2495295525 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+18490 (secure 0 0 0) 0x7f08ac09a5a0 con 0x7f08b8100c80 2024-03-30T00:55:24.015 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:24.015 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":73,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:55:17.191932+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:55:24.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.009+0000 7f08bf800700 1 -- 172.21.15.116:0/2495295525 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f08a00a1120 msgr2=0x7f08a00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:24.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.009+0000 7f08bf800700 1 --2- 172.21.15.116:0/2495295525 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f08a00a1120 0x7f08a00a35e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f08a4005fd0 tx=0x7f08a4009310 comp rx=0 tx=0).stop 2024-03-30T00:55:24.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.009+0000 7f08bf800700 1 -- 172.21.15.116:0/2495295525 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08b8100c80 msgr2=0x7f08b8198ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:24.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.009+0000 7f08bf800700 1 --2- 172.21.15.116:0/2495295525 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08b8100c80 0x7f08b8198ec0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009810 tx=0x7f08ac00bf60 comp rx=0 tx=0).stop 2024-03-30T00:55:24.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.009+0000 7f08bf800700 1 -- 172.21.15.116:0/2495295525 shutdown_connections 2024-03-30T00:55:24.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.009+0000 7f08bf800700 1 --2- 172.21.15.116:0/2495295525 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f08a00a1120 0x7f08a00a35e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.009+0000 7f08bf800700 1 --2- 172.21.15.116:0/2495295525 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08b8100c80 0x7f08b8198ec0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.009+0000 7f08bf800700 1 --2- 172.21.15.116:0/2495295525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08b8102810 0x7f08b81a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.009+0000 7f08bf800700 1 --2- 172.21.15.116:0/2495295525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08b8101e70 0x7f08b8199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.009+0000 7f08bf800700 1 -- 172.21.15.116:0/2495295525 >> 172.21.15.116:0/2495295525 conn(0x7f08b80fc1f0 msgr2=0x7f08b8107610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:24.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.010+0000 7f08bf800700 1 -- 172.21.15.116:0/2495295525 shutdown_connections 2024-03-30T00:55:24.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.010+0000 7f08bf800700 1 -- 172.21.15.116:0/2495295525 wait complete. 2024-03-30T00:55:24.037 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T00:52:29.602428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-30T00:55:24.037 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T00:55:24.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:23 smithi116 ceph-mon[38086]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:55:24.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:23 smithi116 ceph-mon[38086]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2024-03-30T00:55:24.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:23 smithi116 ceph-mon[38086]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:24.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:24.209 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:23 smithi181 ceph-mon[47622]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:55:24.209 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:23 smithi181 ceph-mon[47622]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2024-03-30T00:55:24.209 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:24.209 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:24.210 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:23 smithi181 ceph-mon[47622]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:24.210 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:24.210 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:24.210 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:24.210 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:24.210 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:24.211 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.226+0000 7fa3cf600700 1 -- 172.21.15.116:0/119817401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c80737b0 msgr2=0x7fa3c8073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.226+0000 7fa3cf600700 1 --2- 172.21.15.116:0/119817401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c80737b0 0x7fa3c8073c30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009990 tx=0x7fa3b4009ca0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.226+0000 7fa3cf600700 1 -- 172.21.15.116:0/119817401 shutdown_connections 2024-03-30T00:55:24.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.226+0000 7fa3cf600700 1 --2- 172.21.15.116:0/119817401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3c8108c50 0x7fa3c810b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.226+0000 7fa3cf600700 1 --2- 172.21.15.116:0/119817401 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa3c8074d90 0x7fa3c80731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.226+0000 7fa3cf600700 1 --2- 172.21.15.116:0/119817401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c80737b0 0x7fa3c8073c30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.226+0000 7fa3cf600700 1 -- 172.21.15.116:0/119817401 >> 172.21.15.116:0/119817401 conn(0x7fa3c80fc2f0 msgr2=0x7fa3c80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:24.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.226+0000 7fa3cf600700 1 -- 172.21.15.116:0/119817401 shutdown_connections 2024-03-30T00:55:24.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.226+0000 7fa3cf600700 1 -- 172.21.15.116:0/119817401 wait complete. 2024-03-30T00:55:24.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.227+0000 7fa3cf600700 1 Processor -- start 2024-03-30T00:55:24.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.227+0000 7fa3cf600700 1 -- start start 2024-03-30T00:55:24.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.227+0000 7fa3cf600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c80737b0 0x7fa3c8198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:24.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.227+0000 7fa3cf600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa3c8074d90 0x7fa3c8199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:24.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.227+0000 7fa3cf600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3c8108c50 0x7fa3c81a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:24.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.227+0000 7fa3cf600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c81a0a50 con 0x7fa3c80737b0 2024-03-30T00:55:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.227+0000 7fa3cf600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c81a0b90 con 0x7fa3c8074d90 2024-03-30T00:55:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.227+0000 7fa3cf600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c81a0d20 con 0x7fa3c8108c50 2024-03-30T00:55:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.228+0000 7fa3ccc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c80737b0 0x7fa3c8198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.228+0000 7fa3ccc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c80737b0 0x7fa3c8198f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38984/0 (socket says 172.21.15.116:38984) 2024-03-30T00:55:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.228+0000 7fa3ccc00700 1 -- 172.21.15.116:0/852558979 learned_addr learned my addr 172.21.15.116:0/852558979 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.228+0000 7fa3c7e00700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa3c8074d90 0x7fa3c8199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.228+0000 7fa3cd600700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3c8108c50 0x7fa3c81a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.228+0000 7fa3ccc00700 1 -- 172.21.15.116:0/852558979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3c8108c50 msgr2=0x7fa3c81a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.228+0000 7fa3ccc00700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3c8108c50 0x7fa3c81a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.228+0000 7fa3ccc00700 1 -- 172.21.15.116:0/852558979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa3c8074d90 msgr2=0x7fa3c8199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.228+0000 7fa3ccc00700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa3c8074d90 0x7fa3c8199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.228+0000 7fa3ccc00700 1 -- 172.21.15.116:0/852558979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3bc009520 con 0x7fa3c80737b0 2024-03-30T00:55:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.228+0000 7fa3c7e00700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa3c8074d90 0x7fa3c8199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.228+0000 7fa3ccc00700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c80737b0 0x7fa3c8198f20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00eb30 tx=0x7fa3bc00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.229+0000 7fa3c5600700 1 -- 172.21.15.116:0/852558979 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc009880 con 0x7fa3c80737b0 2024-03-30T00:55:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.229+0000 7fa3c5600700 1 -- 172.21.15.116:0/852558979 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3bc0044e0 con 0x7fa3c80737b0 2024-03-30T00:55:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.229+0000 7fa3cf600700 1 -- 172.21.15.116:0/852558979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c0009c40 con 0x7fa3c80737b0 2024-03-30T00:55:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.229+0000 7fa3c5600700 1 -- 172.21.15.116:0/852558979 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc003d80 con 0x7fa3c80737b0 2024-03-30T00:55:24.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.229+0000 7fa3cf600700 1 -- 172.21.15.116:0/852558979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b40095f0 con 0x7fa3c80737b0 2024-03-30T00:55:24.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.230+0000 7fa3cf600700 1 -- 172.21.15.116:0/852558979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c8107a60 con 0x7fa3c80737b0 2024-03-30T00:55:24.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.232+0000 7fa3c5600700 1 -- 172.21.15.116:0/852558979 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fa3bc003fa0 con 0x7fa3c80737b0 2024-03-30T00:55:24.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.233+0000 7fa3c5600700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3b00a1120 0x7fa3b00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:24.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.233+0000 7fa3c5600700 1 -- 172.21.15.116:0/852558979 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa3bc014070 con 0x7fa3c80737b0 2024-03-30T00:55:24.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.233+0000 7fa3c7e00700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3b00a1120 0x7fa3b00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:24.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.233+0000 7fa3c7e00700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3b00a1120 0x7fa3b00a35e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa3b400b2e0 tx=0x7fa3b4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:24.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.237+0000 7fa3c5600700 1 -- 172.21.15.116:0/852558979 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa3bc0964a0 con 0x7fa3c80737b0 2024-03-30T00:55:24.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:23 smithi130 ceph-mon[47626]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:55:24.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:23 smithi130 ceph-mon[47626]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2024-03-30T00:55:24.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:24.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:24.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:23 smithi130 ceph-mon[47626]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:24.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:24.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:24.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:24.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:24.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:24.347 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:24.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.441+0000 7fa3cf600700 1 -- 172.21.15.116:0/852558979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa3c802ccb0 con 0x7fa3c80737b0 2024-03-30T00:55:24.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.442+0000 7fa3c5600700 1 -- 172.21.15.116:0/852558979 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa3bc099ac0 con 0x7fa3c80737b0 2024-03-30T00:55:24.448 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-30T00:55:24.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.444+0000 7fa3cf600700 1 -- 172.21.15.116:0/852558979 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3b00a1120 msgr2=0x7fa3b00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:24.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.444+0000 7fa3cf600700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3b00a1120 0x7fa3b00a35e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa3b400b2e0 tx=0x7fa3b4005fd0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.444+0000 7fa3cf600700 1 -- 172.21.15.116:0/852558979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c80737b0 msgr2=0x7fa3c8198f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:24.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.444+0000 7fa3cf600700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c80737b0 0x7fa3c8198f20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00eb30 tx=0x7fa3bc00ee40 comp rx=0 tx=0).stop 2024-03-30T00:55:24.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.444+0000 7fa3cf600700 1 -- 172.21.15.116:0/852558979 shutdown_connections 2024-03-30T00:55:24.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.444+0000 7fa3cf600700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3b00a1120 0x7fa3b00a35e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.444+0000 7fa3cf600700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3c8108c50 0x7fa3c81a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.444+0000 7fa3cf600700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c80737b0 0x7fa3c8198f20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.444+0000 7fa3cf600700 1 --2- 172.21.15.116:0/852558979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa3c8074d90 0x7fa3c8199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.444+0000 7fa3cf600700 1 -- 172.21.15.116:0/852558979 >> 172.21.15.116:0/852558979 conn(0x7fa3c80fc2f0 msgr2=0x7fa3c8104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:24.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.444+0000 7fa3cf600700 1 -- 172.21.15.116:0/852558979 shutdown_connections 2024-03-30T00:55:24.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.444+0000 7fa3cf600700 1 -- 172.21.15.116:0/852558979 wait complete. 2024-03-30T00:55:24.470 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:55:24.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.658+0000 7fa53ce00700 1 -- 172.21.15.116:0/4085424431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa538100c80 msgr2=0x7fa5381010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:24.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.658+0000 7fa53ce00700 1 --2- 172.21.15.116:0/4085424431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa538100c80 0x7fa5381010a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa5280099c0 tx=0x7fa528009cd0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.658+0000 7fa53ce00700 1 -- 172.21.15.116:0/4085424431 shutdown_connections 2024-03-30T00:55:24.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.658+0000 7fa53ce00700 1 --2- 172.21.15.116:0/4085424431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa538102810 0x7fa5381090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.658+0000 7fa53ce00700 1 --2- 172.21.15.116:0/4085424431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa538100c80 0x7fa5381010a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.658+0000 7fa53ce00700 1 --2- 172.21.15.116:0/4085424431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa538101e70 0x7fa5381022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.658+0000 7fa53ce00700 1 -- 172.21.15.116:0/4085424431 >> 172.21.15.116:0/4085424431 conn(0x7fa5380fc1f0 msgr2=0x7fa5380fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:24.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.658+0000 7fa53ce00700 1 -- 172.21.15.116:0/4085424431 shutdown_connections 2024-03-30T00:55:24.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.658+0000 7fa53ce00700 1 -- 172.21.15.116:0/4085424431 wait complete. 2024-03-30T00:55:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.659+0000 7fa53ce00700 1 Processor -- start 2024-03-30T00:55:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.660+0000 7fa53ce00700 1 -- start start 2024-03-30T00:55:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.660+0000 7fa53ce00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa538100c80 0x7fa538198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.660+0000 7fa53ce00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa538101e70 0x7fa5381993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.660+0000 7fa53ce00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa538102810 0x7fa5381a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.660+0000 7fa53ce00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5381a09e0 con 0x7fa538102810 2024-03-30T00:55:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.660+0000 7fa53ce00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5381a0b20 con 0x7fa538101e70 2024-03-30T00:55:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.660+0000 7fa53ce00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5381a0cb0 con 0x7fa538100c80 2024-03-30T00:55:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.660+0000 7fa536800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa538102810 0x7fa5381a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.660+0000 7fa535400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa538101e70 0x7fa5381993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.660+0000 7fa536800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa538102810 0x7fa5381a0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38994/0 (socket says 172.21.15.116:38994) 2024-03-30T00:55:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.660+0000 7fa536800700 1 -- 172.21.15.116:0/3053617447 learned_addr learned my addr 172.21.15.116:0/3053617447 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.660+0000 7fa535e00700 1 --2- 172.21.15.116:0/3053617447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa538100c80 0x7fa538198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.661+0000 7fa536800700 1 -- 172.21.15.116:0/3053617447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa538100c80 msgr2=0x7fa538198eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.661+0000 7fa536800700 1 --2- 172.21.15.116:0/3053617447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa538100c80 0x7fa538198eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.661+0000 7fa536800700 1 -- 172.21.15.116:0/3053617447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa538101e70 msgr2=0x7fa5381993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.661+0000 7fa536800700 1 --2- 172.21.15.116:0/3053617447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa538101e70 0x7fa5381993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.661+0000 7fa536800700 1 -- 172.21.15.116:0/3053617447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa524009520 con 0x7fa538102810 2024-03-30T00:55:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.661+0000 7fa535400700 1 --2- 172.21.15.116:0/3053617447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa538101e70 0x7fa5381993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.661+0000 7fa536800700 1 --2- 172.21.15.116:0/3053617447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa538102810 0x7fa5381a0410 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa53000b970 tx=0x7fa53000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.661+0000 7fa52ea00700 1 -- 172.21.15.116:0/3053617447 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53000c930 con 0x7fa538102810 2024-03-30T00:55:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.661+0000 7fa52ea00700 1 -- 172.21.15.116:0/3053617447 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa530014440 con 0x7fa538102810 2024-03-30T00:55:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.661+0000 7fa53ce00700 1 -- 172.21.15.116:0/3053617447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5280095f0 con 0x7fa538102810 2024-03-30T00:55:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.661+0000 7fa52ea00700 1 -- 172.21.15.116:0/3053617447 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5300075d0 con 0x7fa538102810 2024-03-30T00:55:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.661+0000 7fa53ce00700 1 -- 172.21.15.116:0/3053617447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5381a1280 con 0x7fa538102810 2024-03-30T00:55:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.662+0000 7fa53ce00700 1 -- 172.21.15.116:0/3053617447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa538067a10 con 0x7fa538102810 2024-03-30T00:55:24.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.664+0000 7fa52ea00700 1 -- 172.21.15.116:0/3053617447 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fa5300145b0 con 0x7fa538102810 2024-03-30T00:55:24.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.665+0000 7fa52ea00700 1 --2- 172.21.15.116:0/3053617447 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa51c0a1120 0x7fa51c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.665+0000 7fa52ea00700 1 -- 172.21.15.116:0/3053617447 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa530006e40 con 0x7fa538102810 2024-03-30T00:55:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.665+0000 7fa535e00700 1 --2- 172.21.15.116:0/3053617447 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa51c0a1120 0x7fa51c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.666+0000 7fa535e00700 1 --2- 172.21.15.116:0/3053617447 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa51c0a1120 0x7fa51c0a35e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa528009990 tx=0x7fa528005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.669+0000 7fa52ea00700 1 -- 172.21.15.116:0/3053617447 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa530095680 con 0x7fa538102810 2024-03-30T00:55:24.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.919+0000 7fa53ce00700 1 -- 172.21.15.116:0/3053617447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa53810c560 con 0x7fa538102810 2024-03-30T00:55:24.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.919+0000 7fa52ea00700 1 -- 172.21.15.116:0/3053617447 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7fa530098ca0 con 0x7fa538102810 2024-03-30T00:55:24.927 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:24.927 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-30T00:55:24.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.922+0000 7fa53ce00700 1 -- 172.21.15.116:0/3053617447 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa51c0a1120 msgr2=0x7fa51c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:24.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.922+0000 7fa53ce00700 1 --2- 172.21.15.116:0/3053617447 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa51c0a1120 0x7fa51c0a35e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa528009990 tx=0x7fa528005fd0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.922+0000 7fa53ce00700 1 -- 172.21.15.116:0/3053617447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa538102810 msgr2=0x7fa5381a0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:24.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.922+0000 7fa53ce00700 1 --2- 172.21.15.116:0/3053617447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa538102810 0x7fa5381a0410 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa53000b970 tx=0x7fa53000bc80 comp rx=0 tx=0).stop 2024-03-30T00:55:24.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.922+0000 7fa53ce00700 1 -- 172.21.15.116:0/3053617447 shutdown_connections 2024-03-30T00:55:24.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.922+0000 7fa53ce00700 1 --2- 172.21.15.116:0/3053617447 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa51c0a1120 0x7fa51c0a35e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.922+0000 7fa53ce00700 1 --2- 172.21.15.116:0/3053617447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa538100c80 0x7fa538198eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.922+0000 7fa53ce00700 1 --2- 172.21.15.116:0/3053617447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa538102810 0x7fa5381a0410 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.922+0000 7fa53ce00700 1 --2- 172.21.15.116:0/3053617447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa538101e70 0x7fa5381993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:24.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.922+0000 7fa53ce00700 1 -- 172.21.15.116:0/3053617447 >> 172.21.15.116:0/3053617447 conn(0x7fa5380fc1f0 msgr2=0x7fa538072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:24.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.922+0000 7fa53ce00700 1 -- 172.21.15.116:0/3053617447 shutdown_connections 2024-03-30T00:55:24.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:24.922+0000 7fa53ce00700 1 -- 172.21.15.116:0/3053617447 wait complete. 2024-03-30T00:55:24.932 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 1 2024-03-30T00:55:25.130 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 30 00:55:24 smithi181 systemd[1]: Starting Ceph mds.l for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:55:25.130 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:24 smithi181 ceph-mon[47622]: Deploying daemon mds.l on smithi181 2024-03-30T00:55:25.130 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:24 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2495295525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:25.130 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:24 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/852558979' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:55:25.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:24 smithi116 ceph-mon[38086]: Deploying daemon mds.l on smithi181 2024-03-30T00:55:25.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:24 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2495295525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:25.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:24 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/852558979' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:55:25.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:24 smithi130 ceph-mon[47626]: Deploying daemon mds.l on smithi181 2024-03-30T00:55:25.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:24 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2495295525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:25.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:24 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/852558979' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:55:25.502 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 30 00:55:25 smithi181 ceph-mds[84766]: -- 172.21.15.181:0/3803350332 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5629d4b7c1a0 con 0x5629d4b53000 2024-03-30T00:55:25.502 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 30 00:55:25 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-l[84743]: starting mds.l at 2024-03-30T00:55:25.502 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 30 00:55:25 smithi181 bash[84728]: 02f3dd0e14275ea5263ff7ce97c723a34b079b1e5224b01f55eb394187f08f89 2024-03-30T00:55:25.502 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 30 00:55:25 smithi181 systemd[1]: Started Ceph mds.l for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:55:26.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:25 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3053617447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:25 smithi130 ceph-mon[47626]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:25 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.181:6832/1642031781,v1:172.21.15.181:6833/1642031781] up:boot 2024-03-30T00:55:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:25 smithi130 ceph-mon[47626]: fsmap 1 up:standby 2024-03-30T00:55:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-30T00:55:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:25 smithi130 ceph-mon[47626]: Deploying daemon mds.k on smithi130 2024-03-30T00:55:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:25 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3053617447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:25 smithi116 ceph-mon[38086]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:25 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.181:6832/1642031781,v1:172.21.15.181:6833/1642031781] up:boot 2024-03-30T00:55:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:25 smithi116 ceph-mon[38086]: fsmap 1 up:standby 2024-03-30T00:55:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-30T00:55:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:26.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:26.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:26.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:26.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:25 smithi116 ceph-mon[38086]: Deploying daemon mds.k on smithi130 2024-03-30T00:55:26.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:25 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3053617447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:26.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:25 smithi181 ceph-mon[47622]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:26.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:25 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.181:6832/1642031781,v1:172.21.15.181:6833/1642031781] up:boot 2024-03-30T00:55:26.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:25 smithi181 ceph-mon[47622]: fsmap 1 up:standby 2024-03-30T00:55:26.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-30T00:55:26.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:26.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:26.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:26.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:26.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:26.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:26.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:25 smithi181 ceph-mon[47622]: Deploying daemon mds.k on smithi130 2024-03-30T00:55:26.845 INFO:journalctl@ceph.mds.k.smithi130.stdout:Mar 30 00:55:26 smithi130 systemd[1]: Starting Ceph mds.k for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:55:26.949 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:55:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.149+0000 7fb1bdc00700 1 -- 172.21.15.116:0/920295286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1b8100c80 msgr2=0x7fb1b81010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.149+0000 7fb1bdc00700 1 --2- 172.21.15.116:0/920295286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1b8100c80 0x7fb1b81010a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8009990 tx=0x7fb1a8009ca0 comp rx=0 tx=0).stop 2024-03-30T00:55:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.149+0000 7fb1bdc00700 1 -- 172.21.15.116:0/920295286 shutdown_connections 2024-03-30T00:55:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.149+0000 7fb1bdc00700 1 --2- 172.21.15.116:0/920295286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1b8100c80 0x7fb1b81010a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.149+0000 7fb1bdc00700 1 --2- 172.21.15.116:0/920295286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8102810 0x7fb1b81090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.149+0000 7fb1bdc00700 1 --2- 172.21.15.116:0/920295286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1b8101e70 0x7fb1b81022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.149+0000 7fb1bdc00700 1 -- 172.21.15.116:0/920295286 >> 172.21.15.116:0/920295286 conn(0x7fb1b80fc1f0 msgr2=0x7fb1b80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.149+0000 7fb1bdc00700 1 -- 172.21.15.116:0/920295286 shutdown_connections 2024-03-30T00:55:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.150+0000 7fb1bdc00700 1 -- 172.21.15.116:0/920295286 wait complete. 2024-03-30T00:55:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.150+0000 7fb1bdc00700 1 Processor -- start 2024-03-30T00:55:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1bdc00700 1 -- start start 2024-03-30T00:55:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1bdc00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1b8100c80 0x7fb1b8198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1bdc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8101e70 0x7fb1b8199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1bdc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1b8102810 0x7fb1b81a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1bdc00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b81a08f0 con 0x7fb1b8101e70 2024-03-30T00:55:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1bdc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b81a0a30 con 0x7fb1b8102810 2024-03-30T00:55:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1bdc00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b81a0bc0 con 0x7fb1b8100c80 2024-03-30T00:55:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1b6000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8101e70 0x7fb1b8199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1b6000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8101e70 0x7fb1b8199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37238/0 (socket says 172.21.15.116:37238) 2024-03-30T00:55:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1b6000700 1 -- 172.21.15.116:0/2276606898 learned_addr learned my addr 172.21.15.116:0/2276606898 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1b7400700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1b8102810 0x7fb1b81a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1b6a00700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1b8100c80 0x7fb1b8198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1b6000700 1 -- 172.21.15.116:0/2276606898 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1b8100c80 msgr2=0x7fb1b8198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1b6000700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1b8100c80 0x7fb1b8198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1b6000700 1 -- 172.21.15.116:0/2276606898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1b8102810 msgr2=0x7fb1b81a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1b6000700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1b8102810 0x7fb1b81a0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1b6000700 1 -- 172.21.15.116:0/2276606898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a80095f0 con 0x7fb1b8101e70 2024-03-30T00:55:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.151+0000 7fb1b6a00700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1b8100c80 0x7fb1b8198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:55:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.152+0000 7fb1b6000700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8101e70 0x7fb1b8199300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb1a400e9a0 tx=0x7fb1a400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.152+0000 7fb1af400700 1 -- 172.21.15.116:0/2276606898 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4009790 con 0x7fb1b8101e70 2024-03-30T00:55:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.152+0000 7fb1af400700 1 -- 172.21.15.116:0/2276606898 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb1a400cbd0 con 0x7fb1b8101e70 2024-03-30T00:55:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.152+0000 7fb1bdc00700 1 -- 172.21.15.116:0/2276606898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b81a0ea0 con 0x7fb1b8101e70 2024-03-30T00:55:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.152+0000 7fb1af400700 1 -- 172.21.15.116:0/2276606898 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4018680 con 0x7fb1b8101e70 2024-03-30T00:55:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.152+0000 7fb1bdc00700 1 -- 172.21.15.116:0/2276606898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1b81a1320 con 0x7fb1b8101e70 2024-03-30T00:55:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.153+0000 7fb1bdc00700 1 -- 172.21.15.116:0/2276606898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b8067a10 con 0x7fb1b8101e70 2024-03-30T00:55:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.155+0000 7fb1af400700 1 -- 172.21.15.116:0/2276606898 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fb1a400cd40 con 0x7fb1b8101e70 2024-03-30T00:55:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.156+0000 7fb1af400700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb19c0a1120 0x7fb19c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.156+0000 7fb1af400700 1 -- 172.21.15.116:0/2276606898 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb1a4014070 con 0x7fb1b8101e70 2024-03-30T00:55:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.156+0000 7fb1b6a00700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb19c0a1120 0x7fb19c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.157+0000 7fb1b6a00700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb19c0a1120 0x7fb19c0a35e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800b7b0 tx=0x7fb1a8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:27.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.162+0000 7fb1af400700 1 -- 172.21.15.116:0/2276606898 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb1a40529f0 con 0x7fb1b8101e70 2024-03-30T00:55:27.344 INFO:journalctl@ceph.mds.k.smithi130.stdout:Mar 30 00:55:27 smithi130 ceph-mds[83904]: -- 172.21.15.130:0/316124116 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557f99b501a0 con 0x557f99b26c00 2024-03-30T00:55:27.345 INFO:journalctl@ceph.mds.k.smithi130.stdout:Mar 30 00:55:27 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-k[83881]: starting mds.k at 2024-03-30T00:55:27.345 INFO:journalctl@ceph.mds.k.smithi130.stdout:Mar 30 00:55:27 smithi130 bash[83866]: 95b998cc012feae38627579bfd1ece8cd23c24c461e279d3f69c7eb94b9c95ac 2024-03-30T00:55:27.345 INFO:journalctl@ceph.mds.k.smithi130.stdout:Mar 30 00:55:27 smithi130 systemd[1]: Started Ceph mds.k for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:55:27.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.427+0000 7fb1bdc00700 1 -- 172.21.15.116:0/2276606898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1b810c4e0 con 0x7fb1b8101e70 2024-03-30T00:55:27.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.427+0000 7fb1af400700 1 -- 172.21.15.116:0/2276606898 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2020 (secure 0 0 0) 0x7fb1a40962c0 con 0x7fb1b8101e70 2024-03-30T00:55:27.434 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:27.434 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-03-30T00:55:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.430+0000 7fb1bdc00700 1 -- 172.21.15.116:0/2276606898 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb19c0a1120 msgr2=0x7fb19c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.430+0000 7fb1bdc00700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb19c0a1120 0x7fb19c0a35e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800b7b0 tx=0x7fb1a8005fb0 comp rx=0 tx=0).stop 2024-03-30T00:55:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.430+0000 7fb1bdc00700 1 -- 172.21.15.116:0/2276606898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8101e70 msgr2=0x7fb1b8199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.430+0000 7fb1bdc00700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8101e70 0x7fb1b8199300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb1a400e9a0 tx=0x7fb1a400ecb0 comp rx=0 tx=0).stop 2024-03-30T00:55:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.430+0000 7fb1bdc00700 1 -- 172.21.15.116:0/2276606898 shutdown_connections 2024-03-30T00:55:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.430+0000 7fb1bdc00700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb19c0a1120 0x7fb19c0a35e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.430+0000 7fb1bdc00700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1b8100c80 0x7fb1b8198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.430+0000 7fb1bdc00700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1b8102810 0x7fb1b81a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.430+0000 7fb1bdc00700 1 --2- 172.21.15.116:0/2276606898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8101e70 0x7fb1b8199300 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.430+0000 7fb1bdc00700 1 -- 172.21.15.116:0/2276606898 >> 172.21.15.116:0/2276606898 conn(0x7fb1b80fc1f0 msgr2=0x7fb1b8072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.430+0000 7fb1bdc00700 1 -- 172.21.15.116:0/2276606898 shutdown_connections 2024-03-30T00:55:27.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:27.430+0000 7fb1bdc00700 1 -- 172.21.15.116:0/2276606898 wait complete. 2024-03-30T00:55:27.438 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 3 2024-03-30T00:55:28.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:28 smithi116 ceph-mon[38086]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:28.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:28 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.130:6832/2156314548,v1:172.21.15.130:6833/2156314548] up:boot 2024-03-30T00:55:28.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:28 smithi116 ceph-mon[38086]: fsmap 2 up:standby 2024-03-30T00:55:28.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:28 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-30T00:55:28.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:28 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:28.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:28 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:28.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:28 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:28.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:28 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:28.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:28 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:28.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:28 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:28.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:28 smithi116 ceph-mon[38086]: Deploying daemon mds.j on smithi116 2024-03-30T00:55:28.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:28 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2276606898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:28.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:28 smithi181 ceph-mon[47622]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:28.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:28 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.130:6832/2156314548,v1:172.21.15.130:6833/2156314548] up:boot 2024-03-30T00:55:28.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:28 smithi181 ceph-mon[47622]: fsmap 2 up:standby 2024-03-30T00:55:28.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:28 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-30T00:55:28.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:28 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:28.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:28 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:28.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:28 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:28.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:28 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:28.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:28 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:28.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:28 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:28.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:28 smithi181 ceph-mon[47622]: Deploying daemon mds.j on smithi116 2024-03-30T00:55:28.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:28 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2276606898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:28.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:28 smithi130 ceph-mon[47626]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:28 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.130:6832/2156314548,v1:172.21.15.130:6833/2156314548] up:boot 2024-03-30T00:55:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:28 smithi130 ceph-mon[47626]: fsmap 2 up:standby 2024-03-30T00:55:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:28 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-30T00:55:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:28 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:28 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:28 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:28 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:28 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:28 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:28 smithi130 ceph-mon[47626]: Deploying daemon mds.j on smithi116 2024-03-30T00:55:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:28 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2276606898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:29.174 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 30 00:55:28 smithi116 systemd[1]: Starting Ceph mds.j for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:55:29.457 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:55:29.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.656+0000 7feb2a800700 1 -- 172.21.15.116:0/2164317936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb24104010 msgr2=0x7feb24106400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:29.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.656+0000 7feb2a800700 1 --2- 172.21.15.116:0/2164317936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb24104010 0x7feb24106400 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009990 tx=0x7feb0c009ca0 comp rx=0 tx=0).stop 2024-03-30T00:55:29.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.656+0000 7feb2a800700 1 -- 172.21.15.116:0/2164317936 shutdown_connections 2024-03-30T00:55:29.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.656+0000 7feb2a800700 1 --2- 172.21.15.116:0/2164317936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb241016e0 0x7feb24103ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:29.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.656+0000 7feb2a800700 1 --2- 172.21.15.116:0/2164317936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb24106940 0x7feb24108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:29.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.656+0000 7feb2a800700 1 --2- 172.21.15.116:0/2164317936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb24104010 0x7feb24106400 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:29.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.656+0000 7feb2a800700 1 -- 172.21.15.116:0/2164317936 >> 172.21.15.116:0/2164317936 conn(0x7feb240fb0d0 msgr2=0x7feb240fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:29.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.656+0000 7feb2a800700 1 -- 172.21.15.116:0/2164317936 shutdown_connections 2024-03-30T00:55:29.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.656+0000 7feb2a800700 1 -- 172.21.15.116:0/2164317936 wait complete. 2024-03-30T00:55:29.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.658+0000 7feb2a800700 1 Processor -- start 2024-03-30T00:55:29.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.658+0000 7feb2a800700 1 -- start start 2024-03-30T00:55:29.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.658+0000 7feb2a800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb241016e0 0x7feb24196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:29.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.658+0000 7feb2a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb24104010 0x7feb24197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:29.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.658+0000 7feb2a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb24106940 0x7feb2419e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:29.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.658+0000 7feb2a800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2419e730 con 0x7feb24106940 2024-03-30T00:55:29.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.658+0000 7feb2a800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2419e870 con 0x7feb24104010 2024-03-30T00:55:29.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.658+0000 7feb2a800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2419ea00 con 0x7feb241016e0 2024-03-30T00:55:29.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.658+0000 7feb23400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb241016e0 0x7feb24196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.658+0000 7feb23e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb24106940 0x7feb2419e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.658+0000 7feb23400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb241016e0 0x7feb24196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:51762/0 (socket says 172.21.15.116:51762) 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.658+0000 7feb23400700 1 -- 172.21.15.116:0/2259390657 learned_addr learned my addr 172.21.15.116:0/2259390657 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.659+0000 7feb1be00700 1 --2- 172.21.15.116:0/2259390657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb24104010 0x7feb24197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.659+0000 7feb23400700 1 -- 172.21.15.116:0/2259390657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb24104010 msgr2=0x7feb24197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.659+0000 7feb23400700 1 --2- 172.21.15.116:0/2259390657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb24104010 0x7feb24197140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.659+0000 7feb23400700 1 -- 172.21.15.116:0/2259390657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb24106940 msgr2=0x7feb2419e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.659+0000 7feb23400700 1 --2- 172.21.15.116:0/2259390657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb24106940 0x7feb2419e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.659+0000 7feb23400700 1 -- 172.21.15.116:0/2259390657 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c0095f0 con 0x7feb241016e0 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.659+0000 7feb23400700 1 --2- 172.21.15.116:0/2259390657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb241016e0 0x7feb24196c00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feb1400e9a0 tx=0x7feb1400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.659+0000 7feb20c00700 1 -- 172.21.15.116:0/2259390657 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14009790 con 0x7feb241016e0 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.659+0000 7feb20c00700 1 -- 172.21.15.116:0/2259390657 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb14010430 con 0x7feb241016e0 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.659+0000 7feb2a800700 1 -- 172.21.15.116:0/2259390657 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2419ece0 con 0x7feb241016e0 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.659+0000 7feb20c00700 1 -- 172.21.15.116:0/2259390657 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14018680 con 0x7feb241016e0 2024-03-30T00:55:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.660+0000 7feb2a800700 1 -- 172.21.15.116:0/2259390657 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2419f190 con 0x7feb241016e0 2024-03-30T00:55:29.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.661+0000 7feb2a800700 1 -- 172.21.15.116:0/2259390657 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb240fcc50 con 0x7feb241016e0 2024-03-30T00:55:29.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.662+0000 7feb20c00700 1 -- 172.21.15.116:0/2259390657 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7feb140105a0 con 0x7feb241016e0 2024-03-30T00:55:29.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.664+0000 7feb20c00700 1 --2- 172.21.15.116:0/2259390657 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb080a11a0 0x7feb080a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:29.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.664+0000 7feb20c00700 1 -- 172.21.15.116:0/2259390657 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7feb14014070 con 0x7feb241016e0 2024-03-30T00:55:29.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.664+0000 7feb1be00700 1 --2- 172.21.15.116:0/2259390657 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb080a11a0 0x7feb080a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:29.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.664+0000 7feb1be00700 1 --2- 172.21.15.116:0/2259390657 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb080a11a0 0x7feb080a3660 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00b2e0 tx=0x7feb0c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:29.673 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 30 00:55:29 smithi116 bash[87237]: e2ee95916bb9fe2003dbe54024b067c08c09fb6ddd900d0922c2de50e99ac5d2 2024-03-30T00:55:29.673 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 30 00:55:29 smithi116 systemd[1]: Started Ceph mds.j for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:55:29.673 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 30 00:55:29 smithi116 ceph-mds[87276]: -- 172.21.15.116:0/1879987026 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56135a1a31e0 con 0x56135a166800 2024-03-30T00:55:29.673 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 30 00:55:29 smithi116 ceph-mds[87276]: -- 172.21.15.116:0/1879987026 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56135a18e1a0 con 0x56135a166800 2024-03-30T00:55:29.674 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 30 00:55:29 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-j[87252]: starting mds.j at 2024-03-30T00:55:29.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.668+0000 7feb20c00700 1 -- 172.21.15.116:0/2259390657 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7feb140529f0 con 0x7feb241016e0 2024-03-30T00:55:29.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.935+0000 7feb2a800700 1 -- 172.21.15.116:0/2259390657 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb24067a10 con 0x7feb241016e0 2024-03-30T00:55:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.935+0000 7feb20c00700 1 -- 172.21.15.116:0/2259390657 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2020 (secure 0 0 0) 0x7feb24067a10 con 0x7feb241016e0 2024-03-30T00:55:29.942 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:29.942 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-03-30T00:55:29.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.938+0000 7feb2a800700 1 -- 172.21.15.116:0/2259390657 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb080a11a0 msgr2=0x7feb080a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:29.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.938+0000 7feb2a800700 1 --2- 172.21.15.116:0/2259390657 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb080a11a0 0x7feb080a3660 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00b2e0 tx=0x7feb0c005fb0 comp rx=0 tx=0).stop 2024-03-30T00:55:29.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.938+0000 7feb2a800700 1 -- 172.21.15.116:0/2259390657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb241016e0 msgr2=0x7feb24196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:29.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.938+0000 7feb2a800700 1 --2- 172.21.15.116:0/2259390657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb241016e0 0x7feb24196c00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feb1400e9a0 tx=0x7feb1400ecb0 comp rx=0 tx=0).stop 2024-03-30T00:55:29.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.938+0000 7feb2a800700 1 -- 172.21.15.116:0/2259390657 shutdown_connections 2024-03-30T00:55:29.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.938+0000 7feb2a800700 1 --2- 172.21.15.116:0/2259390657 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb080a11a0 0x7feb080a3660 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:29.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.938+0000 7feb2a800700 1 --2- 172.21.15.116:0/2259390657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb241016e0 0x7feb24196c00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:29.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.938+0000 7feb2a800700 1 --2- 172.21.15.116:0/2259390657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb24106940 0x7feb2419e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:29.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.938+0000 7feb2a800700 1 --2- 172.21.15.116:0/2259390657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb24104010 0x7feb24197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:29.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.938+0000 7feb2a800700 1 -- 172.21.15.116:0/2259390657 >> 172.21.15.116:0/2259390657 conn(0x7feb240fb0d0 msgr2=0x7feb241035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:29.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.938+0000 7feb2a800700 1 -- 172.21.15.116:0/2259390657 shutdown_connections 2024-03-30T00:55:29.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:29.938+0000 7feb2a800700 1 -- 172.21.15.116:0/2259390657 wait complete. 2024-03-30T00:55:29.946 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 3 2024-03-30T00:55:30.667 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:30 smithi116 ceph-mon[38086]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:30.667 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:30 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:30.667 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:30 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:30.667 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:30 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:30.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:30 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:30.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:30 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:30.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:30 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:30.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:30 smithi116 ceph-mon[38086]: Deploying daemon mds.d on smithi116 2024-03-30T00:55:30.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:30 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2259390657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:30.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:30 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.116:6834/459507942,v1:172.21.15.116:6835/459507942] up:boot 2024-03-30T00:55:30.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:30 smithi116 ceph-mon[38086]: fsmap 3 up:standby 2024-03-30T00:55:30.668 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:30 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-30T00:55:30.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:30 smithi181 ceph-mon[47622]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:30.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:30 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:30.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:30 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:30.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:30 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:30.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:30 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:30.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:30 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:30.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:30 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:30.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:30 smithi181 ceph-mon[47622]: Deploying daemon mds.d on smithi116 2024-03-30T00:55:30.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:30 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2259390657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:30.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:30 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.116:6834/459507942,v1:172.21.15.116:6835/459507942] up:boot 2024-03-30T00:55:30.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:30 smithi181 ceph-mon[47622]: fsmap 3 up:standby 2024-03-30T00:55:30.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:30 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-30T00:55:30.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:30 smithi130 ceph-mon[47626]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:30 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:30 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:30 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:30 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:30 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:30 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:30 smithi130 ceph-mon[47626]: Deploying daemon mds.d on smithi116 2024-03-30T00:55:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:30 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2259390657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:30 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.116:6834/459507942,v1:172.21.15.116:6835/459507942] up:boot 2024-03-30T00:55:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:30 smithi130 ceph-mon[47626]: fsmap 3 up:standby 2024-03-30T00:55:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:30 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-30T00:55:31.424 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 30 00:55:31 smithi116 systemd[1]: Starting Ceph mds.d for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:55:31.965 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:55:32.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.158+0000 7ffba7800700 1 -- 172.21.15.116:0/1374278752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffba0100c90 msgr2=0x7ffba01010b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:32.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.158+0000 7ffba7800700 1 --2- 172.21.15.116:0/1374278752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffba0100c90 0x7ffba01010b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffb940099c0 tx=0x7ffb94009cd0 comp rx=0 tx=0).stop 2024-03-30T00:55:32.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.158+0000 7ffba7800700 1 -- 172.21.15.116:0/1374278752 shutdown_connections 2024-03-30T00:55:32.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.158+0000 7ffba7800700 1 --2- 172.21.15.116:0/1374278752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba0102820 0x7ffba01090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:32.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.158+0000 7ffba7800700 1 --2- 172.21.15.116:0/1374278752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffba0100c90 0x7ffba01010b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:32.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.158+0000 7ffba7800700 1 --2- 172.21.15.116:0/1374278752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0101e80 0x7ffba01022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:32.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.158+0000 7ffba7800700 1 -- 172.21.15.116:0/1374278752 >> 172.21.15.116:0/1374278752 conn(0x7ffba00fc200 msgr2=0x7ffba00fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:32.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.158+0000 7ffba7800700 1 -- 172.21.15.116:0/1374278752 shutdown_connections 2024-03-30T00:55:32.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.158+0000 7ffba7800700 1 -- 172.21.15.116:0/1374278752 wait complete. 2024-03-30T00:55:32.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.159+0000 7ffba7800700 1 Processor -- start 2024-03-30T00:55:32.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.160+0000 7ffba7800700 1 -- start start 2024-03-30T00:55:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.160+0000 7ffba7800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0100c90 0x7ffba0196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.160+0000 7ffba7800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffba0101e80 0x7ffba01971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.160+0000 7ffba7800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba0102820 0x7ffba019e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.160+0000 7ffba7800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba019e770 con 0x7ffba0102820 2024-03-30T00:55:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.160+0000 7ffba7800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba019e8e0 con 0x7ffba0101e80 2024-03-30T00:55:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.160+0000 7ffba7800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba019eaa0 con 0x7ffba0100c90 2024-03-30T00:55:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffba5800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba0102820 0x7ffba019e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffb9fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffba0101e80 0x7ffba01971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffba4e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0100c90 0x7ffba0196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffb9fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffba0101e80 0x7ffba01971c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43812/0 (socket says 172.21.15.116:43812) 2024-03-30T00:55:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffb9fe00700 1 -- 172.21.15.116:0/1839681234 learned_addr learned my addr 172.21.15.116:0/1839681234 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffb9fe00700 1 -- 172.21.15.116:0/1839681234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0100c90 msgr2=0x7ffba0196c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffb9fe00700 1 --2- 172.21.15.116:0/1839681234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0100c90 0x7ffba0196c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffb9fe00700 1 -- 172.21.15.116:0/1839681234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba0102820 msgr2=0x7ffba019e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffb9fe00700 1 --2- 172.21.15.116:0/1839681234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba0102820 0x7ffba019e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffb9fe00700 1 -- 172.21.15.116:0/1839681234 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb98009c40 con 0x7ffba0101e80 2024-03-30T00:55:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffba4e00700 1 --2- 172.21.15.116:0/1839681234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0100c90 0x7ffba0196c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffb9fe00700 1 --2- 172.21.15.116:0/1839681234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffba0101e80 0x7ffba01971c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c00e9a0 tx=0x7ffb8c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffb9d600700 1 -- 172.21.15.116:0/1839681234 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c009790 con 0x7ffba0101e80 2024-03-30T00:55:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffb9d600700 1 -- 172.21.15.116:0/1839681234 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb8c00ce80 con 0x7ffba0101e80 2024-03-30T00:55:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffba7800700 1 -- 172.21.15.116:0/1839681234 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb940095f0 con 0x7ffba0101e80 2024-03-30T00:55:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.161+0000 7ffb9d600700 1 -- 172.21.15.116:0/1839681234 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c018680 con 0x7ffba0101e80 2024-03-30T00:55:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.162+0000 7ffba7800700 1 -- 172.21.15.116:0/1839681234 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba019f0d0 con 0x7ffba0101e80 2024-03-30T00:55:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.163+0000 7ffba7800700 1 -- 172.21.15.116:0/1839681234 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba0108110 con 0x7ffba0101e80 2024-03-30T00:55:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.164+0000 7ffb9d600700 1 -- 172.21.15.116:0/1839681234 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7ffb8c010430 con 0x7ffba0101e80 2024-03-30T00:55:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.166+0000 7ffb9d600700 1 --2- 172.21.15.116:0/1839681234 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ffb880a1120 0x7ffb880a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.167+0000 7ffba4e00700 1 --2- 172.21.15.116:0/1839681234 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ffb880a1120 0x7ffb880a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.167+0000 7ffb9d600700 1 -- 172.21.15.116:0/1839681234 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ffb8c014070 con 0x7ffba0101e80 2024-03-30T00:55:32.174 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 30 00:55:31 smithi116 ceph-mds[87662]: -- 172.21.15.116:0/1043737672 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d2b39c61a0 con 0x55d2b399e800 2024-03-30T00:55:32.174 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 30 00:55:31 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-d[87639]: starting mds.d at 2024-03-30T00:55:32.174 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 30 00:55:31 smithi116 bash[87624]: 3b7459ec2b364516ae305ad29d293aa8a4c096e416e2734fdbe8430e0f0c3b3a 2024-03-30T00:55:32.174 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 30 00:55:31 smithi116 systemd[1]: Started Ceph mds.d for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:55:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.167+0000 7ffba4e00700 1 --2- 172.21.15.116:0/1839681234 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ffb880a1120 0x7ffb880a35e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffb94003730 tx=0x7ffb9400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.170+0000 7ffb9d600700 1 -- 172.21.15.116:0/1839681234 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffb8c0529f0 con 0x7ffba0101e80 2024-03-30T00:55:32.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.421+0000 7ffba7800700 1 -- 172.21.15.116:0/1839681234 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffba0067a10 con 0x7ffba0101e80 2024-03-30T00:55:32.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.422+0000 7ffb9d600700 1 -- 172.21.15.116:0/1839681234 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7ffb8c099d50 con 0x7ffba0101e80 2024-03-30T00:55:32.429 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:32.429 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-03-30T00:55:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.424+0000 7ffba7800700 1 -- 172.21.15.116:0/1839681234 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ffb880a1120 msgr2=0x7ffb880a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.424+0000 7ffba7800700 1 --2- 172.21.15.116:0/1839681234 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ffb880a1120 0x7ffb880a35e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffb94003730 tx=0x7ffb9400b540 comp rx=0 tx=0).stop 2024-03-30T00:55:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.424+0000 7ffba7800700 1 -- 172.21.15.116:0/1839681234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffba0101e80 msgr2=0x7ffba01971c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.424+0000 7ffba7800700 1 --2- 172.21.15.116:0/1839681234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffba0101e80 0x7ffba01971c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c00e9a0 tx=0x7ffb8c00ecb0 comp rx=0 tx=0).stop 2024-03-30T00:55:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.424+0000 7ffba7800700 1 -- 172.21.15.116:0/1839681234 shutdown_connections 2024-03-30T00:55:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.424+0000 7ffba7800700 1 --2- 172.21.15.116:0/1839681234 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ffb880a1120 0x7ffb880a35e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.424+0000 7ffba7800700 1 --2- 172.21.15.116:0/1839681234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba0100c90 0x7ffba0196c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.424+0000 7ffba7800700 1 --2- 172.21.15.116:0/1839681234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba0102820 0x7ffba019e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.424+0000 7ffba7800700 1 --2- 172.21.15.116:0/1839681234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffba0101e80 0x7ffba01971c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.424+0000 7ffba7800700 1 -- 172.21.15.116:0/1839681234 >> 172.21.15.116:0/1839681234 conn(0x7ffba00fc200 msgr2=0x7ffba0105250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.425+0000 7ffba7800700 1 -- 172.21.15.116:0/1839681234 shutdown_connections 2024-03-30T00:55:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:32.425+0000 7ffba7800700 1 -- 172.21.15.116:0/1839681234 wait complete. 2024-03-30T00:55:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 4 2024-03-30T00:55:32.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:32 smithi130 ceph-mon[47626]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:32.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:32.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:32.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:32.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:32.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:32.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:32.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:32 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1839681234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:32.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:32 smithi181 ceph-mon[47622]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:32.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:32.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:32.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:32.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:32.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:32.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:32.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:32 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1839681234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:32 smithi116 ceph-mon[38086]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:32 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1839681234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:33.345 INFO:journalctl@ceph.mds.e.smithi130.stdout:Mar 30 00:55:33 smithi130 systemd[1]: Starting Ceph mds.e for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:55:33.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:33 smithi181 ceph-mon[47622]: Deploying daemon mds.e on smithi130 2024-03-30T00:55:33.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:33 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.116:6836/3445796607,v1:172.21.15.116:6837/3445796607] up:boot 2024-03-30T00:55:33.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:33 smithi181 ceph-mon[47622]: fsmap 4 up:standby 2024-03-30T00:55:33.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:33 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-30T00:55:33.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:33 smithi130 ceph-mon[47626]: Deploying daemon mds.e on smithi130 2024-03-30T00:55:33.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:33 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.116:6836/3445796607,v1:172.21.15.116:6837/3445796607] up:boot 2024-03-30T00:55:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:33 smithi130 ceph-mon[47626]: fsmap 4 up:standby 2024-03-30T00:55:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:33 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-30T00:55:33.845 INFO:journalctl@ceph.mds.e.smithi130.stdout:Mar 30 00:55:33 smithi130 ceph-mds[84244]: -- 172.21.15.130:0/3144099570 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564529d361a0 con 0x564529d0c800 2024-03-30T00:55:33.845 INFO:journalctl@ceph.mds.e.smithi130.stdout:Mar 30 00:55:33 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-e[84221]: starting mds.e at 2024-03-30T00:55:33.845 INFO:journalctl@ceph.mds.e.smithi130.stdout:Mar 30 00:55:33 smithi130 bash[84206]: b900de8a8518275b5d3afd302707f1ebddcf5a3f16cf7d94326bf8b2e86e4ca0 2024-03-30T00:55:33.845 INFO:journalctl@ceph.mds.e.smithi130.stdout:Mar 30 00:55:33 smithi130 systemd[1]: Started Ceph mds.e for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:55:33.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:33 smithi116 ceph-mon[38086]: Deploying daemon mds.e on smithi130 2024-03-30T00:55:33.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:33 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.116:6836/3445796607,v1:172.21.15.116:6837/3445796607] up:boot 2024-03-30T00:55:33.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:33 smithi116 ceph-mon[38086]: fsmap 4 up:standby 2024-03-30T00:55:33.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:33 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-30T00:55:34.450 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:55:34.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.648+0000 7facc6c00700 1 -- 172.21.15.116:0/2276419818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc0100c80 msgr2=0x7facc01010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:34.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.648+0000 7facc6c00700 1 --2- 172.21.15.116:0/2276419818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc0100c80 0x7facc01010a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7facb00099c0 tx=0x7facb0009cd0 comp rx=0 tx=0).stop 2024-03-30T00:55:34.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.648+0000 7facc6c00700 1 -- 172.21.15.116:0/2276419818 shutdown_connections 2024-03-30T00:55:34.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.648+0000 7facc6c00700 1 --2- 172.21.15.116:0/2276419818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facc0102810 0x7facc01090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:34.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.648+0000 7facc6c00700 1 --2- 172.21.15.116:0/2276419818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc0100c80 0x7facc01010a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:34.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.648+0000 7facc6c00700 1 --2- 172.21.15.116:0/2276419818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facc0101e70 0x7facc01022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:34.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.648+0000 7facc6c00700 1 -- 172.21.15.116:0/2276419818 >> 172.21.15.116:0/2276419818 conn(0x7facc00fc1f0 msgr2=0x7facc00fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:34.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.648+0000 7facc6c00700 1 -- 172.21.15.116:0/2276419818 shutdown_connections 2024-03-30T00:55:34.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.649+0000 7facc6c00700 1 -- 172.21.15.116:0/2276419818 wait complete. 2024-03-30T00:55:34.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.649+0000 7facc6c00700 1 Processor -- start 2024-03-30T00:55:34.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.650+0000 7facc6c00700 1 -- start start 2024-03-30T00:55:34.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.650+0000 7facc6c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facc0100c80 0x7facc0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:34.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.650+0000 7facc6c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc0101e70 0x7facc00731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:34.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.650+0000 7facc6c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facc0102810 0x7facc00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:34.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.650+0000 7facc6c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc0073e80 con 0x7facc0101e70 2024-03-30T00:55:34.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.650+0000 7facc6c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc0073ff0 con 0x7facc0102810 2024-03-30T00:55:34.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.650+0000 7facc6c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc01a0c80 con 0x7facc0100c80 2024-03-30T00:55:34.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.650+0000 7facbf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc0101e70 0x7facc00731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:34.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.650+0000 7facbf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc0101e70 0x7facc00731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37344/0 (socket says 172.21.15.116:37344) 2024-03-30T00:55:34.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.650+0000 7facbf400700 1 -- 172.21.15.116:0/1644400047 learned_addr learned my addr 172.21.15.116:0/1644400047 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:34.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.650+0000 7facbfe00700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facc0100c80 0x7facc0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:34.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.650+0000 7facc4c00700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facc0102810 0x7facc00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:34.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.651+0000 7facbf400700 1 -- 172.21.15.116:0/1644400047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facc0100c80 msgr2=0x7facc0074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.651+0000 7facbf400700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facc0100c80 0x7facc0074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.651+0000 7facbf400700 1 -- 172.21.15.116:0/1644400047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facc0102810 msgr2=0x7facc00736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.651+0000 7facbf400700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facc0102810 0x7facc00736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.651+0000 7facbf400700 1 -- 172.21.15.116:0/1644400047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb00095f0 con 0x7facc0101e70 2024-03-30T00:55:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.651+0000 7facbfe00700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facc0100c80 0x7facc0074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.651+0000 7facbf400700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc0101e70 0x7facc00731a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7facac00e9a0 tx=0x7facac00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.651+0000 7facbcc00700 1 -- 172.21.15.116:0/1644400047 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac009790 con 0x7facc0101e70 2024-03-30T00:55:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.651+0000 7facbcc00700 1 -- 172.21.15.116:0/1644400047 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7facac00ce70 con 0x7facc0101e70 2024-03-30T00:55:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.651+0000 7facc6c00700 1 -- 172.21.15.116:0/1644400047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc01a0f10 con 0x7facc0101e70 2024-03-30T00:55:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.651+0000 7facbcc00700 1 -- 172.21.15.116:0/1644400047 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac018420 con 0x7facc0101e70 2024-03-30T00:55:34.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.651+0000 7facc6c00700 1 -- 172.21.15.116:0/1644400047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc01a1390 con 0x7facc0101e70 2024-03-30T00:55:34.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.652+0000 7facc6c00700 1 -- 172.21.15.116:0/1644400047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc01080d0 con 0x7facc0101e70 2024-03-30T00:55:34.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.654+0000 7facbcc00700 1 -- 172.21.15.116:0/1644400047 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7facac0106f0 con 0x7facc0101e70 2024-03-30T00:55:34.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.655+0000 7facbcc00700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7faca40a1120 0x7faca40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:34.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.656+0000 7facbcc00700 1 -- 172.21.15.116:0/1644400047 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7facac014070 con 0x7facc0101e70 2024-03-30T00:55:34.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.656+0000 7facbfe00700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7faca40a1120 0x7faca40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:34.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.656+0000 7facbfe00700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7faca40a1120 0x7faca40a35e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7facb0003730 tx=0x7facb000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:34.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.660+0000 7facbcc00700 1 -- 172.21.15.116:0/1644400047 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7facac0529f0 con 0x7facc0101e70 2024-03-30T00:55:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:34 smithi181 ceph-mon[47622]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:34.798 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:34.798 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:34.798 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:34.798 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:34.798 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:34.799 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:34.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:34 smithi130 ceph-mon[47626]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:34.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:34.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:34.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:34.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:34.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:34.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:34.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.910+0000 7facc6c00700 1 -- 172.21.15.116:0/1644400047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facc0067a10 con 0x7facc0101e70 2024-03-30T00:55:34.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.911+0000 7facbcc00700 1 -- 172.21.15.116:0/1644400047 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4297 (secure 0 0 0) 0x7facac099c70 con 0x7facc0101e70 2024-03-30T00:55:34.918 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:34.918 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24524,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2024-03-30T00:55:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.914+0000 7facc6c00700 1 -- 172.21.15.116:0/1644400047 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7faca40a1120 msgr2=0x7faca40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.914+0000 7facc6c00700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7faca40a1120 0x7faca40a35e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7facb0003730 tx=0x7facb000b540 comp rx=0 tx=0).stop 2024-03-30T00:55:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.914+0000 7facc6c00700 1 -- 172.21.15.116:0/1644400047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc0101e70 msgr2=0x7facc00731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.914+0000 7facc6c00700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc0101e70 0x7facc00731a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7facac00e9a0 tx=0x7facac00ecb0 comp rx=0 tx=0).stop 2024-03-30T00:55:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.914+0000 7facc6c00700 1 -- 172.21.15.116:0/1644400047 shutdown_connections 2024-03-30T00:55:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.914+0000 7facc6c00700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7faca40a1120 0x7faca40a35e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.914+0000 7facc6c00700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facc0100c80 0x7facc0074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.914+0000 7facc6c00700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facc0102810 0x7facc00736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.914+0000 7facc6c00700 1 --2- 172.21.15.116:0/1644400047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc0101e70 0x7facc00731a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.914+0000 7facc6c00700 1 -- 172.21.15.116:0/1644400047 >> 172.21.15.116:0/1644400047 conn(0x7facc00fc1f0 msgr2=0x7facc0105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.914+0000 7facc6c00700 1 -- 172.21.15.116:0/1644400047 shutdown_connections 2024-03-30T00:55:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:34.914+0000 7facc6c00700 1 -- 172.21.15.116:0/1644400047 wait complete. 2024-03-30T00:55:34.922 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 6 2024-03-30T00:55:34.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:34 smithi116 ceph-mon[38086]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:34.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:34.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:34.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:34.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:34.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:34.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:35.252 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 30 00:55:35 smithi181 systemd[1]: Starting Ceph mds.f for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:55:35.752 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 30 00:55:35 smithi181 ceph-mds[85107]: -- 172.21.15.181:0/3359915092 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5640121301a0 con 0x564012106c00 2024-03-30T00:55:35.752 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 30 00:55:35 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-f[85084]: starting mds.f at 2024-03-30T00:55:35.752 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 30 00:55:35 smithi181 bash[85068]: 4f2c42ace9b72d92bc3ab34810364ab891540c7d0be848c5bf79ccf95f54bb06 2024-03-30T00:55:35.752 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 30 00:55:35 smithi181 systemd[1]: Started Ceph mds.f for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:55:35.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:35 smithi181 ceph-mon[47622]: Deploying daemon mds.f on smithi181 2024-03-30T00:55:35.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:35 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.130:6834/3346365474,v1:172.21.15.130:6835/3346365474] up:boot 2024-03-30T00:55:35.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:35 smithi181 ceph-mon[47622]: fsmap 5 up:standby 2024-03-30T00:55:35.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T00:55:35.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:35 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1644400047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:35.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:35.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:35.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:35.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:35.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:35.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:35.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:35 smithi130 ceph-mon[47626]: Deploying daemon mds.f on smithi181 2024-03-30T00:55:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:35 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.130:6834/3346365474,v1:172.21.15.130:6835/3346365474] up:boot 2024-03-30T00:55:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:35 smithi130 ceph-mon[47626]: fsmap 5 up:standby 2024-03-30T00:55:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T00:55:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:35 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1644400047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:35.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:35 smithi116 ceph-mon[38086]: Deploying daemon mds.f on smithi181 2024-03-30T00:55:35.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:35 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.130:6834/3346365474,v1:172.21.15.130:6835/3346365474] up:boot 2024-03-30T00:55:35.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:35 smithi116 ceph-mon[38086]: fsmap 5 up:standby 2024-03-30T00:55:35.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T00:55:35.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:35 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1644400047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:35.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:35.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:35.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:35.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:35.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:35.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:36.703 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 30 00:55:36 smithi181 systemd[1]: Starting Ceph mds.c for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:55:36.703 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:36 smithi181 ceph-mon[47622]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:36.703 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:36 smithi181 ceph-mon[47622]: Deploying daemon mds.c on smithi181 2024-03-30T00:55:36.703 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:36 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.181:6834/702133015,v1:172.21.15.181:6835/702133015] up:boot 2024-03-30T00:55:36.703 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:36 smithi181 ceph-mon[47622]: fsmap 6 up:standby 2024-03-30T00:55:36.703 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:36 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T00:55:36.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:36 smithi130 ceph-mon[47626]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:36.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:36 smithi130 ceph-mon[47626]: Deploying daemon mds.c on smithi181 2024-03-30T00:55:36.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:36 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.181:6834/702133015,v1:172.21.15.181:6835/702133015] up:boot 2024-03-30T00:55:36.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:36 smithi130 ceph-mon[47626]: fsmap 6 up:standby 2024-03-30T00:55:36.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:36 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T00:55:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:36 smithi116 ceph-mon[38086]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:36 smithi116 ceph-mon[38086]: Deploying daemon mds.c on smithi181 2024-03-30T00:55:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:36 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.181:6834/702133015,v1:172.21.15.181:6835/702133015] up:boot 2024-03-30T00:55:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:36 smithi116 ceph-mon[38086]: fsmap 6 up:standby 2024-03-30T00:55:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T00:55:36.941 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:55:37.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.137+0000 7f3f97600700 1 -- 172.21.15.116:0/2006854518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f90074df0 msgr2=0x7f3f90073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:37.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.137+0000 7f3f97600700 1 --2- 172.21.15.116:0/2006854518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f90074df0 0x7f3f90073240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3f84013f60 tx=0x7f3f84013f90 comp rx=0 tx=0).stop 2024-03-30T00:55:37.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.137+0000 7f3f97600700 1 -- 172.21.15.116:0/2006854518 shutdown_connections 2024-03-30T00:55:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.137+0000 7f3f97600700 1 --2- 172.21.15.116:0/2006854518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f90108b40 0x7f3f9010afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.137+0000 7f3f97600700 1 --2- 172.21.15.116:0/2006854518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f90074df0 0x7f3f90073240 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.137+0000 7f3f97600700 1 --2- 172.21.15.116:0/2006854518 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f90073810 0x7f3f90073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.137+0000 7f3f97600700 1 -- 172.21.15.116:0/2006854518 >> 172.21.15.116:0/2006854518 conn(0x7f3f900fc1f0 msgr2=0x7f3f900fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.137+0000 7f3f97600700 1 -- 172.21.15.116:0/2006854518 shutdown_connections 2024-03-30T00:55:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.137+0000 7f3f97600700 1 -- 172.21.15.116:0/2006854518 wait complete. 2024-03-30T00:55:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.138+0000 7f3f97600700 1 Processor -- start 2024-03-30T00:55:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.139+0000 7f3f97600700 1 -- start start 2024-03-30T00:55:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.139+0000 7f3f97600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f90073810 0x7f3f90194a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.139+0000 7f3f97600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f90074df0 0x7f3f90194fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.139+0000 7f3f97600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f90108b40 0x7f3f9019bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.139+0000 7f3f97600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9019c590 con 0x7f3f90073810 2024-03-30T00:55:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.139+0000 7f3f97600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9019c6d0 con 0x7f3f90074df0 2024-03-30T00:55:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.139+0000 7f3f97600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9019c860 con 0x7f3f90108b40 2024-03-30T00:55:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.139+0000 7f3f8fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f90074df0 0x7f3f90194fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.139+0000 7f3f94c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f90073810 0x7f3f90194a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.139+0000 7f3f8fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f90074df0 0x7f3f90194fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:33644/0 (socket says 172.21.15.116:33644) 2024-03-30T00:55:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.139+0000 7f3f8fe00700 1 -- 172.21.15.116:0/3017498305 learned_addr learned my addr 172.21.15.116:0/3017498305 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.139+0000 7f3f95600700 1 --2- 172.21.15.116:0/3017498305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f90108b40 0x7f3f9019bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.140+0000 7f3f94c00700 1 -- 172.21.15.116:0/3017498305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f90108b40 msgr2=0x7f3f9019bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.140+0000 7f3f94c00700 1 --2- 172.21.15.116:0/3017498305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f90108b40 0x7f3f9019bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.140+0000 7f3f94c00700 1 -- 172.21.15.116:0/3017498305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f90074df0 msgr2=0x7f3f90194fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.140+0000 7f3f94c00700 1 --2- 172.21.15.116:0/3017498305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f90074df0 0x7f3f90194fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.140+0000 7f3f94c00700 1 -- 172.21.15.116:0/3017498305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c0095f0 con 0x7f3f90073810 2024-03-30T00:55:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.140+0000 7f3f8fe00700 1 --2- 172.21.15.116:0/3017498305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f90074df0 0x7f3f90194fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.140+0000 7f3f94c00700 1 --2- 172.21.15.116:0/3017498305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f90073810 0x7f3f90194a60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3f84013e80 tx=0x7f3f8401b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.141+0000 7f3f8d600700 1 -- 172.21.15.116:0/3017498305 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f840050c0 con 0x7f3f90073810 2024-03-30T00:55:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.142+0000 7f3f97600700 1 -- 172.21.15.116:0/3017498305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f84013b90 con 0x7f3f90073810 2024-03-30T00:55:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.142+0000 7f3f97600700 1 -- 172.21.15.116:0/3017498305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9019cdd0 con 0x7f3f90073810 2024-03-30T00:55:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.142+0000 7f3f8d600700 1 -- 172.21.15.116:0/3017498305 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f840056e0 con 0x7f3f90073810 2024-03-30T00:55:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.142+0000 7f3f8d600700 1 -- 172.21.15.116:0/3017498305 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f840038e0 con 0x7f3f90073810 2024-03-30T00:55:37.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.144+0000 7f3f82a00700 1 -- 172.21.15.116:0/3017498305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f70005320 con 0x7f3f90073810 2024-03-30T00:55:37.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.145+0000 7f3f8d600700 1 -- 172.21.15.116:0/3017498305 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f3f84022040 con 0x7f3f90073810 2024-03-30T00:55:37.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.147+0000 7f3f8d600700 1 --2- 172.21.15.116:0/3017498305 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3f780a11a0 0x7f3f780a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:37.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.147+0000 7f3f8d600700 1 -- 172.21.15.116:0/3017498305 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3f840d4f90 con 0x7f3f90073810 2024-03-30T00:55:37.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.147+0000 7f3f8fe00700 1 --2- 172.21.15.116:0/3017498305 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3f780a11a0 0x7f3f780a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:37.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.148+0000 7f3f8fe00700 1 --2- 172.21.15.116:0/3017498305 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3f780a11a0 0x7f3f780a3660 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c005950 tx=0x7f3f7c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:37.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.153+0000 7f3f8d600700 1 -- 172.21.15.116:0/3017498305 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3f841089f0 con 0x7f3f90073810 2024-03-30T00:55:37.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.406+0000 7f3f82a00700 1 -- 172.21.15.116:0/3017498305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f70005190 con 0x7f3f90073810 2024-03-30T00:55:37.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.406+0000 7f3f8d600700 1 -- 172.21.15.116:0/3017498305 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5054 (secure 0 0 0) 0x7f3f8409ec50 con 0x7f3f90073810 2024-03-30T00:55:37.414 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:37.414 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24524,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2024-03-30T00:55:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.409+0000 7f3f82a00700 1 -- 172.21.15.116:0/3017498305 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3f780a11a0 msgr2=0x7f3f780a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.409+0000 7f3f82a00700 1 --2- 172.21.15.116:0/3017498305 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3f780a11a0 0x7f3f780a3660 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c005950 tx=0x7f3f7c009310 comp rx=0 tx=0).stop 2024-03-30T00:55:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.409+0000 7f3f82a00700 1 -- 172.21.15.116:0/3017498305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f90073810 msgr2=0x7f3f90194a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.409+0000 7f3f82a00700 1 --2- 172.21.15.116:0/3017498305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f90073810 0x7f3f90194a60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3f84013e80 tx=0x7f3f8401b910 comp rx=0 tx=0).stop 2024-03-30T00:55:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.409+0000 7f3f82a00700 1 -- 172.21.15.116:0/3017498305 shutdown_connections 2024-03-30T00:55:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.409+0000 7f3f82a00700 1 --2- 172.21.15.116:0/3017498305 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3f780a11a0 0x7f3f780a3660 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.409+0000 7f3f82a00700 1 --2- 172.21.15.116:0/3017498305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f90108b40 0x7f3f9019bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.409+0000 7f3f82a00700 1 --2- 172.21.15.116:0/3017498305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f90073810 0x7f3f90194a60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.409+0000 7f3f82a00700 1 --2- 172.21.15.116:0/3017498305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f90074df0 0x7f3f90194fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.409+0000 7f3f82a00700 1 -- 172.21.15.116:0/3017498305 >> 172.21.15.116:0/3017498305 conn(0x7f3f900fc1f0 msgr2=0x7f3f90103a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.409+0000 7f3f82a00700 1 -- 172.21.15.116:0/3017498305 shutdown_connections 2024-03-30T00:55:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:37.409+0000 7f3f82a00700 1 -- 172.21.15.116:0/3017498305 wait complete. 2024-03-30T00:55:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 7 2024-03-30T00:55:37.502 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 30 00:55:37 smithi181 ceph-mds[85453]: -- 172.21.15.181:0/3359128752 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5626b78761a0 con 0x5626b784cc00 2024-03-30T00:55:37.502 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 30 00:55:37 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-c[85430]: starting mds.c at 2024-03-30T00:55:37.502 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 30 00:55:37 smithi181 bash[85414]: 2b3319046e3f31e06a347fa6f4b7d7909e5f4bb70e5bbd04c181301b502f71d4 2024-03-30T00:55:37.502 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 30 00:55:37 smithi181 systemd[1]: Started Ceph mds.c for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:55:38.232 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:38 smithi130 ceph-mon[47626]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:38.232 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:38 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:38.232 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:38 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:38.232 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:38 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:38.232 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:38 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:38.233 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:38 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:38.233 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:38 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:38.233 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:38 smithi130 ceph-mon[47626]: Deploying daemon mds.h on smithi130 2024-03-30T00:55:38.233 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:38 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3017498305' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:38.233 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:38 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.181:6836/941347789,v1:172.21.15.181:6837/941347789] up:boot 2024-03-30T00:55:38.233 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:38 smithi130 ceph-mon[47626]: fsmap 7 up:standby 2024-03-30T00:55:38.233 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:38 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T00:55:38.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:38 smithi116 ceph-mon[38086]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:38.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:38 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:38.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:38 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:38.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:38 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:38.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:38 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:38.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:38 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:38.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:38 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:38.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:38 smithi116 ceph-mon[38086]: Deploying daemon mds.h on smithi130 2024-03-30T00:55:38.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:38 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3017498305' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:38.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:38 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.181:6836/941347789,v1:172.21.15.181:6837/941347789] up:boot 2024-03-30T00:55:38.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:38 smithi116 ceph-mon[38086]: fsmap 7 up:standby 2024-03-30T00:55:38.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:38 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T00:55:38.484 INFO:journalctl@ceph.mds.h.smithi130.stdout:Mar 30 00:55:38 smithi130 systemd[1]: Starting Ceph mds.h for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:55:38.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:38 smithi181 ceph-mon[47622]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:38.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:38 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:38.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:38 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:38.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:38 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:38.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:38 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:38.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:38 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:38.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:38 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:38.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:38 smithi181 ceph-mon[47622]: Deploying daemon mds.h on smithi130 2024-03-30T00:55:38.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:38 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3017498305' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:38.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:38 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.181:6836/941347789,v1:172.21.15.181:6837/941347789] up:boot 2024-03-30T00:55:38.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:38 smithi181 ceph-mon[47622]: fsmap 7 up:standby 2024-03-30T00:55:38.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:38 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T00:55:39.096 INFO:journalctl@ceph.mds.h.smithi130.stdout:Mar 30 00:55:38 smithi130 ceph-mds[84585]: -- 172.21.15.130:0/554883997 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561d86a381a0 con 0x561d86a0e800 2024-03-30T00:55:39.096 INFO:journalctl@ceph.mds.h.smithi130.stdout:Mar 30 00:55:38 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-h[84562]: starting mds.h at 2024-03-30T00:55:39.096 INFO:journalctl@ceph.mds.h.smithi130.stdout:Mar 30 00:55:38 smithi130 bash[84548]: b1282b01fdfa11cf1b268e906d23042c8fa8608a9ca2558db7d304f76c2a6815 2024-03-30T00:55:39.096 INFO:journalctl@ceph.mds.h.smithi130.stdout:Mar 30 00:55:38 smithi130 systemd[1]: Started Ceph mds.h for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:55:39.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:55:39.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.638+0000 7f8fec000700 1 -- 172.21.15.116:0/4123810669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe4101e70 msgr2=0x7f8fe41022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:39.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.638+0000 7f8fec000700 1 --2- 172.21.15.116:0/4123810669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe4101e70 0x7f8fe41022d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00099c0 tx=0x7f8fd0009cd0 comp rx=0 tx=0).stop 2024-03-30T00:55:39.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.638+0000 7f8fec000700 1 -- 172.21.15.116:0/4123810669 shutdown_connections 2024-03-30T00:55:39.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.638+0000 7f8fec000700 1 --2- 172.21.15.116:0/4123810669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fe4102810 0x7f8fe41090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:39.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.638+0000 7f8fec000700 1 --2- 172.21.15.116:0/4123810669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8fe4100c80 0x7f8fe41010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:39.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.638+0000 7f8fec000700 1 --2- 172.21.15.116:0/4123810669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe4101e70 0x7f8fe41022d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:39.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.638+0000 7f8fec000700 1 -- 172.21.15.116:0/4123810669 >> 172.21.15.116:0/4123810669 conn(0x7f8fe40fc1f0 msgr2=0x7f8fe40fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:39.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.638+0000 7f8fec000700 1 -- 172.21.15.116:0/4123810669 shutdown_connections 2024-03-30T00:55:39.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.638+0000 7f8fec000700 1 -- 172.21.15.116:0/4123810669 wait complete. 2024-03-30T00:55:39.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.639+0000 7f8fec000700 1 Processor -- start 2024-03-30T00:55:39.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.639+0000 7f8fec000700 1 -- start start 2024-03-30T00:55:39.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.639+0000 7f8fec000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fe4100c80 0x7f8fe4198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:39.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.639+0000 7f8fec000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8fe4101e70 0x7f8fe4199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:39.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.639+0000 7f8fec000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe4102810 0x7f8fe41a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:39.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.639+0000 7f8fec000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe41a0900 con 0x7f8fe4102810 2024-03-30T00:55:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.639+0000 7f8fec000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe41a0a70 con 0x7f8fe4101e70 2024-03-30T00:55:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.639+0000 7f8fec000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe41a0c30 con 0x7f8fe4100c80 2024-03-30T00:55:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.639+0000 7f8fe9600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fe4100c80 0x7f8fe4198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.639+0000 7f8fe9600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fe4100c80 0x7f8fe4198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:53688/0 (socket says 172.21.15.116:53688) 2024-03-30T00:55:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fe9600700 1 -- 172.21.15.116:0/4196708235 learned_addr learned my addr 172.21.15.116:0/4196708235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fe8c00700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8fe4101e70 0x7f8fe4199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fea000700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe4102810 0x7f8fe41a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fe9600700 1 -- 172.21.15.116:0/4196708235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8fe4101e70 msgr2=0x7f8fe4199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fe9600700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8fe4101e70 0x7f8fe4199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fe9600700 1 -- 172.21.15.116:0/4196708235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe4102810 msgr2=0x7f8fe41a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fe9600700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe4102810 0x7f8fe41a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fe9600700 1 -- 172.21.15.116:0/4196708235 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd00095f0 con 0x7f8fe4100c80 2024-03-30T00:55:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fe8c00700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8fe4101e70 0x7f8fe4199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:39.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fea000700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe4102810 0x7f8fe41a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:39.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fe9600700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fe4100c80 0x7f8fe4198e10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8fd800e970 tx=0x7f8fd800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:39.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fe2000700 1 -- 172.21.15.116:0/4196708235 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8009790 con 0x7f8fe4100c80 2024-03-30T00:55:39.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fe2000700 1 -- 172.21.15.116:0/4196708235 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8fd800c7e0 con 0x7f8fe4100c80 2024-03-30T00:55:39.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fec000700 1 -- 172.21.15.116:0/4196708235 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe41a0f10 con 0x7f8fe4100c80 2024-03-30T00:55:39.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.640+0000 7f8fe2000700 1 -- 172.21.15.116:0/4196708235 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8003c70 con 0x7f8fe4100c80 2024-03-30T00:55:39.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.641+0000 7f8fec000700 1 -- 172.21.15.116:0/4196708235 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe41a1330 con 0x7f8fe4100c80 2024-03-30T00:55:39.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.641+0000 7f8fec000700 1 -- 172.21.15.116:0/4196708235 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe4067a10 con 0x7f8fe4100c80 2024-03-30T00:55:39.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.643+0000 7f8fe2000700 1 -- 172.21.15.116:0/4196708235 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f8fd800cc50 con 0x7f8fe4100c80 2024-03-30T00:55:39.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.644+0000 7f8fe2000700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8fcc0a11a0 0x7f8fcc0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:39.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.645+0000 7f8fe2000700 1 -- 172.21.15.116:0/4196708235 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8fd8014070 con 0x7f8fe4100c80 2024-03-30T00:55:39.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.645+0000 7f8fe8c00700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8fcc0a11a0 0x7f8fcc0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:39.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.645+0000 7f8fe8c00700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8fcc0a11a0 0x7f8fcc0a3660 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009fd0 tx=0x7f8fd00036f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:39.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.647+0000 7f8fe2000700 1 -- 172.21.15.116:0/4196708235 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8fd80968f0 con 0x7f8fe4100c80 2024-03-30T00:55:39.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.895+0000 7f8fec000700 1 -- 172.21.15.116:0/4196708235 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fe410c4e0 con 0x7f8fe4100c80 2024-03-30T00:55:39.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.896+0000 7f8fe2000700 1 -- 172.21.15.116:0/4196708235 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6571 (secure 0 0 0) 0x7f8fd8003dd0 con 0x7f8fe4100c80 2024-03-30T00:55:39.903 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:39.903 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24524,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2024-03-30T00:55:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.898+0000 7f8fec000700 1 -- 172.21.15.116:0/4196708235 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8fcc0a11a0 msgr2=0x7f8fcc0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.898+0000 7f8fec000700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8fcc0a11a0 0x7f8fcc0a3660 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009fd0 tx=0x7f8fd00036f0 comp rx=0 tx=0).stop 2024-03-30T00:55:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.898+0000 7f8fec000700 1 -- 172.21.15.116:0/4196708235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fe4100c80 msgr2=0x7f8fe4198e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.898+0000 7f8fec000700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fe4100c80 0x7f8fe4198e10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8fd800e970 tx=0x7f8fd800ec80 comp rx=0 tx=0).stop 2024-03-30T00:55:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.898+0000 7f8fec000700 1 -- 172.21.15.116:0/4196708235 shutdown_connections 2024-03-30T00:55:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.898+0000 7f8fec000700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8fcc0a11a0 0x7f8fcc0a3660 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.898+0000 7f8fec000700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fe4100c80 0x7f8fe4198e10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.898+0000 7f8fec000700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe4102810 0x7f8fe41a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.898+0000 7f8fec000700 1 --2- 172.21.15.116:0/4196708235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8fe4101e70 0x7f8fe4199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.898+0000 7f8fec000700 1 -- 172.21.15.116:0/4196708235 >> 172.21.15.116:0/4196708235 conn(0x7f8fe40fc1f0 msgr2=0x7f8fe4072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.898+0000 7f8fec000700 1 -- 172.21.15.116:0/4196708235 shutdown_connections 2024-03-30T00:55:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:39.898+0000 7f8fec000700 1 -- 172.21.15.116:0/4196708235 wait complete. 2024-03-30T00:55:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 9 2024-03-30T00:55:40.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:40 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:40.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:40 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:40.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:40 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:40.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:40 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:40.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:40 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:40.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:40 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:40.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:40 smithi116 ceph-mon[38086]: Deploying daemon mds.i on smithi181 2024-03-30T00:55:40.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:40 smithi116 ceph-mon[38086]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:40.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:40 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] up:boot 2024-03-30T00:55:40.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:40 smithi116 ceph-mon[38086]: fsmap 8 up:standby 2024-03-30T00:55:40.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:40 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T00:55:40.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:40 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/4196708235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:40.231 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 00:55:40 smithi181 systemd[1]: Starting Ceph mds.i for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:55:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:40 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:40 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:40 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:40 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:40 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:40 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:40 smithi181 ceph-mon[47622]: Deploying daemon mds.i on smithi181 2024-03-30T00:55:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:40 smithi181 ceph-mon[47622]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:40 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] up:boot 2024-03-30T00:55:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:40 smithi181 ceph-mon[47622]: fsmap 8 up:standby 2024-03-30T00:55:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:40 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T00:55:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:40 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/4196708235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:40.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:40 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:40.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:40 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:40.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:40 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:40.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:40 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:40.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:40 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:40.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:40 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:40.345 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:40 smithi130 ceph-mon[47626]: Deploying daemon mds.i on smithi181 2024-03-30T00:55:40.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:40 smithi130 ceph-mon[47626]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:40.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:40 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] up:boot 2024-03-30T00:55:40.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:40 smithi130 ceph-mon[47626]: fsmap 8 up:standby 2024-03-30T00:55:40.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:40 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T00:55:40.346 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:40 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/4196708235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:41.002 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 00:55:40 smithi181 ceph-mds[85793]: -- 172.21.15.181:0/1356985909 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d55c7101a0 con 0x55d55c6e7000 2024-03-30T00:55:41.002 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 00:55:40 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-i[85770]: starting mds.i at 2024-03-30T00:55:41.002 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 00:55:40 smithi181 bash[85755]: 7560dd3ed75fdde9b5f4b8bfb1281f5f1b50492e4932b93f269219e4b97088b6 2024-03-30T00:55:41.002 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 00:55:40 smithi181 systemd[1]: Started Ceph mds.i for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:55:41.825 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:41.825 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:41.825 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:41.825 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:41.825 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:41.825 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:41.825 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:41 smithi116 ceph-mon[38086]: Deploying daemon mds.g on smithi116 2024-03-30T00:55:41.825 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:41 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] up:boot 2024-03-30T00:55:41.826 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:41 smithi116 ceph-mon[38086]: fsmap 9 up:standby 2024-03-30T00:55:41.826 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:41 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T00:55:41.826 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:41 smithi116 ceph-mon[38086]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:41.924 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:55:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:42.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:42.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:41 smithi181 ceph-mon[47622]: Deploying daemon mds.g on smithi116 2024-03-30T00:55:42.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:41 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] up:boot 2024-03-30T00:55:42.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:41 smithi181 ceph-mon[47622]: fsmap 9 up:standby 2024-03-30T00:55:42.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:41 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T00:55:42.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:41 smithi181 ceph-mon[47622]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:42.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:42.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:42.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:42.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:42.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:42.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:42.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:41 smithi130 ceph-mon[47626]: Deploying daemon mds.g on smithi116 2024-03-30T00:55:42.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:41 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] up:boot 2024-03-30T00:55:42.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:41 smithi130 ceph-mon[47626]: fsmap 9 up:standby 2024-03-30T00:55:42.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:41 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T00:55:42.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:41 smithi130 ceph-mon[47626]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.192+0000 7fec7de00700 1 -- 172.21.15.116:0/3734018939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec78102810 msgr2=0x7fec781090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.192+0000 7fec7de00700 1 --2- 172.21.15.116:0/3734018939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec78102810 0x7fec781090c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fec700099c0 tx=0x7fec70009cd0 comp rx=0 tx=0).stop 2024-03-30T00:55:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.192+0000 7fec7de00700 1 -- 172.21.15.116:0/3734018939 shutdown_connections 2024-03-30T00:55:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.192+0000 7fec7de00700 1 --2- 172.21.15.116:0/3734018939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec78100c80 0x7fec781010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.192+0000 7fec7de00700 1 --2- 172.21.15.116:0/3734018939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec78102810 0x7fec781090c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.192+0000 7fec7de00700 1 --2- 172.21.15.116:0/3734018939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec78101e70 0x7fec781022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.192+0000 7fec7de00700 1 -- 172.21.15.116:0/3734018939 >> 172.21.15.116:0/3734018939 conn(0x7fec780fc210 msgr2=0x7fec780fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.192+0000 7fec7de00700 1 -- 172.21.15.116:0/3734018939 shutdown_connections 2024-03-30T00:55:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.192+0000 7fec7de00700 1 -- 172.21.15.116:0/3734018939 wait complete. 2024-03-30T00:55:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.194+0000 7fec7de00700 1 Processor -- start 2024-03-30T00:55:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.194+0000 7fec7de00700 1 -- start start 2024-03-30T00:55:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.194+0000 7fec7de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec78100c80 0x7fec78198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.194+0000 7fec7de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec78101e70 0x7fec781992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.194+0000 7fec7de00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec78102810 0x7fec781a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.194+0000 7fec7de00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec781a0890 con 0x7fec78100c80 2024-03-30T00:55:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.194+0000 7fec7de00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec781a0a00 con 0x7fec78101e70 2024-03-30T00:55:42.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.194+0000 7fec7de00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec781a0bc0 con 0x7fec78102810 2024-03-30T00:55:42.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.194+0000 7fec76a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec78100c80 0x7fec78198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:42.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.194+0000 7fec76a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec78100c80 0x7fec78198da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43956/0 (socket says 172.21.15.116:43956) 2024-03-30T00:55:42.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.194+0000 7fec76a00700 1 -- 172.21.15.116:0/325067374 learned_addr learned my addr 172.21.15.116:0/325067374 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:42.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.195+0000 7fec77400700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec78102810 0x7fec781a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:42.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.195+0000 7fec76000700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec78101e70 0x7fec781992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:42.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.195+0000 7fec76a00700 1 -- 172.21.15.116:0/325067374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec78102810 msgr2=0x7fec781a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:42.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.195+0000 7fec76a00700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec78102810 0x7fec781a02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:42.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.195+0000 7fec76a00700 1 -- 172.21.15.116:0/325067374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec78101e70 msgr2=0x7fec781992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:42.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.195+0000 7fec76a00700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec78101e70 0x7fec781992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:42.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.195+0000 7fec76a00700 1 -- 172.21.15.116:0/325067374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec700095f0 con 0x7fec78100c80 2024-03-30T00:55:42.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.195+0000 7fec77400700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec78102810 0x7fec781a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:42.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.195+0000 7fec76a00700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec78100c80 0x7fec78198da0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fec6800efc0 tx=0x7fec6800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:42.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.195+0000 7fec7ca00700 1 -- 172.21.15.116:0/325067374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68018070 con 0x7fec78100c80 2024-03-30T00:55:42.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.195+0000 7fec7ca00700 1 -- 172.21.15.116:0/325067374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec6800f040 con 0x7fec78100c80 2024-03-30T00:55:42.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.195+0000 7fec7de00700 1 -- 172.21.15.116:0/325067374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec781a0e40 con 0x7fec78100c80 2024-03-30T00:55:42.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.195+0000 7fec7ca00700 1 -- 172.21.15.116:0/325067374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68004280 con 0x7fec78100c80 2024-03-30T00:55:42.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.196+0000 7fec7de00700 1 -- 172.21.15.116:0/325067374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec781a1260 con 0x7fec78100c80 2024-03-30T00:55:42.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.197+0000 7fec7de00700 1 -- 172.21.15.116:0/325067374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec78067180 con 0x7fec78100c80 2024-03-30T00:55:42.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.199+0000 7fec7ca00700 1 -- 172.21.15.116:0/325067374 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fec68003a00 con 0x7fec78100c80 2024-03-30T00:55:42.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.200+0000 7fec7ca00700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fec5c0a1120 0x7fec5c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.200+0000 7fec7ca00700 1 -- 172.21.15.116:0/325067374 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fec680cc410 con 0x7fec78100c80 2024-03-30T00:55:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.200+0000 7fec76000700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fec5c0a1120 0x7fec5c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.201+0000 7fec76000700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fec5c0a1120 0x7fec5c0a35e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fec64007be0 tx=0x7fec640073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.205+0000 7fec7ca00700 1 -- 172.21.15.116:0/325067374 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fec68096030 con 0x7fec78100c80 2024-03-30T00:55:42.390 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 30 00:55:42 smithi116 systemd[1]: Starting Ceph mds.g for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:55:42.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.471+0000 7fec7de00700 1 -- 172.21.15.116:0/325067374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec7810c720 con 0x7fec78100c80 2024-03-30T00:55:42.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.473+0000 7fec7ca00700 1 -- 172.21.15.116:0/325067374 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7333 (secure 0 0 0) 0x7fec68099650 con 0x7fec78100c80 2024-03-30T00:55:42.479 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:42.479 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24524,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2024-03-30T00:55:42.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.476+0000 7fec7de00700 1 -- 172.21.15.116:0/325067374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fec5c0a1120 msgr2=0x7fec5c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:42.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.476+0000 7fec7de00700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fec5c0a1120 0x7fec5c0a35e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fec64007be0 tx=0x7fec640073d0 comp rx=0 tx=0).stop 2024-03-30T00:55:42.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.476+0000 7fec7de00700 1 -- 172.21.15.116:0/325067374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec78100c80 msgr2=0x7fec78198da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:42.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.476+0000 7fec7de00700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec78100c80 0x7fec78198da0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fec6800efc0 tx=0x7fec6800c3a0 comp rx=0 tx=0).stop 2024-03-30T00:55:42.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.476+0000 7fec7de00700 1 -- 172.21.15.116:0/325067374 shutdown_connections 2024-03-30T00:55:42.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.476+0000 7fec7de00700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fec5c0a1120 0x7fec5c0a35e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:42.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.476+0000 7fec7de00700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec78102810 0x7fec781a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:42.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.476+0000 7fec7de00700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec78100c80 0x7fec78198da0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:42.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.476+0000 7fec7de00700 1 --2- 172.21.15.116:0/325067374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec78101e70 0x7fec781992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:42.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.476+0000 7fec7de00700 1 -- 172.21.15.116:0/325067374 >> 172.21.15.116:0/325067374 conn(0x7fec780fc210 msgr2=0x7fec78072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:42.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.477+0000 7fec7de00700 1 -- 172.21.15.116:0/325067374 shutdown_connections 2024-03-30T00:55:42.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:42.477+0000 7fec7de00700 1 -- 172.21.15.116:0/325067374 wait complete. 2024-03-30T00:55:42.485 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 10 2024-03-30T00:55:43.147 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 30 00:55:42 smithi116 ceph-mds[88230]: -- 172.21.15.116:0/145438060 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b35c09e1a0 con 0x55b35c076800 2024-03-30T00:55:43.147 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 30 00:55:42 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-g[88207]: starting mds.g at 2024-03-30T00:55:43.148 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 30 00:55:42 smithi116 bash[88192]: 1aeabd56bca8a6746a68b5886a2d216c9a5a2f0ef95454443a0ec287d16b2c87 2024-03-30T00:55:43.148 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 30 00:55:42 smithi116 systemd[1]: Started Ceph mds.g for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:55:43.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:43 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/325067374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:43.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:43 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:43.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:43 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:43.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:43 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:43.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:43 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:43.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:43 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:43.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:43 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:43.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:43 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/325067374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:43.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:43 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:43.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:43 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:43.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:43 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:43.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:43 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:43.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:43 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:43.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:43 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:43.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:43 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/325067374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:43.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:43 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:43.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:43 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:43.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:43 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:43.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:43 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:43.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:43 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:43.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:43 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:44.246 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:44 smithi116 ceph-mon[38086]: Deploying daemon mds.a on smithi116 2024-03-30T00:55:44.246 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:44 smithi116 ceph-mon[38086]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:44.246 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:44 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.116:6838/1448499059,v1:172.21.15.116:6839/1448499059] up:boot 2024-03-30T00:55:44.246 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:44 smithi116 ceph-mon[38086]: fsmap 10 up:standby 2024-03-30T00:55:44.246 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:44 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-30T00:55:44.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:44 smithi181 ceph-mon[47622]: Deploying daemon mds.a on smithi116 2024-03-30T00:55:44.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:44 smithi181 ceph-mon[47622]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:44.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:44 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.116:6838/1448499059,v1:172.21.15.116:6839/1448499059] up:boot 2024-03-30T00:55:44.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:44 smithi181 ceph-mon[47622]: fsmap 10 up:standby 2024-03-30T00:55:44.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:44 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-30T00:55:44.504 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:55:44.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:44 smithi130 ceph-mon[47626]: Deploying daemon mds.a on smithi116 2024-03-30T00:55:44.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:44 smithi130 ceph-mon[47626]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:44.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:44 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.116:6838/1448499059,v1:172.21.15.116:6839/1448499059] up:boot 2024-03-30T00:55:44.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:44 smithi130 ceph-mon[47626]: fsmap 10 up:standby 2024-03-30T00:55:44.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:44 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-30T00:55:44.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.901+0000 7fa647a00700 1 -- 172.21.15.116:0/623768221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa640104090 msgr2=0x7fa640106480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:44.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.901+0000 7fa647a00700 1 --2- 172.21.15.116:0/623768221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa640104090 0x7fa640106480 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa62c009990 tx=0x7fa62c009ca0 comp rx=0 tx=0).stop 2024-03-30T00:55:44.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.901+0000 7fa647a00700 1 -- 172.21.15.116:0/623768221 shutdown_connections 2024-03-30T00:55:44.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.901+0000 7fa647a00700 1 --2- 172.21.15.116:0/623768221 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6401069c0 0x7fa640108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:44.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.901+0000 7fa647a00700 1 --2- 172.21.15.116:0/623768221 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa640101760 0x7fa640103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:44.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.901+0000 7fa647a00700 1 --2- 172.21.15.116:0/623768221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa640104090 0x7fa640106480 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:44.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.901+0000 7fa647a00700 1 -- 172.21.15.116:0/623768221 >> 172.21.15.116:0/623768221 conn(0x7fa6400fb0d0 msgr2=0x7fa6400fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:44.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.901+0000 7fa647a00700 1 -- 172.21.15.116:0/623768221 shutdown_connections 2024-03-30T00:55:44.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.901+0000 7fa647a00700 1 -- 172.21.15.116:0/623768221 wait complete. 2024-03-30T00:55:44.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.903+0000 7fa647a00700 1 Processor -- start 2024-03-30T00:55:44.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.903+0000 7fa647a00700 1 -- start start 2024-03-30T00:55:44.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.904+0000 7fa647a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa640101760 0x7fa640196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.904+0000 7fa647a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa640104090 0x7fa6401970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.904+0000 7fa647a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6401069c0 0x7fa64019e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.904+0000 7fa647a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64019e6d0 con 0x7fa640104090 2024-03-30T00:55:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.904+0000 7fa647a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64019e810 con 0x7fa640101760 2024-03-30T00:55:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.904+0000 7fa647a00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64019e9a0 con 0x7fa6401069c0 2024-03-30T00:55:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.904+0000 7fa63fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa640104090 0x7fa6401970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.904+0000 7fa63fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa640104090 0x7fa6401970e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60736/0 (socket says 172.21.15.116:60736) 2024-03-30T00:55:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.904+0000 7fa63fe00700 1 -- 172.21.15.116:0/198678850 learned_addr learned my addr 172.21.15.116:0/198678850 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.904+0000 7fa645000700 1 --2- 172.21.15.116:0/198678850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa640101760 0x7fa640196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.904+0000 7fa645a00700 1 --2- 172.21.15.116:0/198678850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6401069c0 0x7fa64019e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:44.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.905+0000 7fa63fe00700 1 -- 172.21.15.116:0/198678850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6401069c0 msgr2=0x7fa64019e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:44.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.905+0000 7fa63fe00700 1 --2- 172.21.15.116:0/198678850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6401069c0 0x7fa64019e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:44.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.905+0000 7fa63fe00700 1 -- 172.21.15.116:0/198678850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa640101760 msgr2=0x7fa640196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:44.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.905+0000 7fa63fe00700 1 --2- 172.21.15.116:0/198678850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa640101760 0x7fa640196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:44.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.905+0000 7fa63fe00700 1 -- 172.21.15.116:0/198678850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c0095f0 con 0x7fa640104090 2024-03-30T00:55:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.905+0000 7fa63fe00700 1 --2- 172.21.15.116:0/198678850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa640104090 0x7fa6401970e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0098e0 tx=0x7fa62c00f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.905+0000 7fa63d600700 1 -- 172.21.15.116:0/198678850 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c01d070 con 0x7fa640104090 2024-03-30T00:55:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.905+0000 7fa647a00700 1 -- 172.21.15.116:0/198678850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64019ec20 con 0x7fa640104090 2024-03-30T00:55:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.905+0000 7fa63d600700 1 -- 172.21.15.116:0/198678850 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa62c005050 con 0x7fa640104090 2024-03-30T00:55:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.905+0000 7fa63d600700 1 -- 172.21.15.116:0/198678850 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c004ae0 con 0x7fa640104090 2024-03-30T00:55:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.905+0000 7fa647a00700 1 -- 172.21.15.116:0/198678850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64019f050 con 0x7fa640104090 2024-03-30T00:55:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.906+0000 7fa647a00700 1 -- 172.21.15.116:0/198678850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6400fcc50 con 0x7fa640104090 2024-03-30T00:55:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.908+0000 7fa63d600700 1 -- 172.21.15.116:0/198678850 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fa62c004c60 con 0x7fa640104090 2024-03-30T00:55:44.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.909+0000 7fa63d600700 1 --2- 172.21.15.116:0/198678850 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa6280a1120 0x7fa6280a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:44.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.910+0000 7fa63d600700 1 -- 172.21.15.116:0/198678850 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa62c0cd400 con 0x7fa640104090 2024-03-30T00:55:44.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.910+0000 7fa645000700 1 --2- 172.21.15.116:0/198678850 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa6280a1120 0x7fa6280a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.910+0000 7fa645000700 1 --2- 172.21.15.116:0/198678850 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa6280a1120 0x7fa6280a35e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa634005950 tx=0x7fa63400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:44.914+0000 7fa63d600700 1 -- 172.21.15.116:0/198678850 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa62c097020 con 0x7fa640104090 2024-03-30T00:55:44.923 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 30 00:55:44 smithi116 systemd[1]: Starting Ceph mds.a for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:55:45.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:45.184+0000 7fa647a00700 1 -- 172.21.15.116:0/198678850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa640067a10 con 0x7fa640104090 2024-03-30T00:55:45.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:45.185+0000 7fa63d600700 1 -- 172.21.15.116:0/198678850 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8094 (secure 0 0 0) 0x7fa62c09a640 con 0x7fa640104090 2024-03-30T00:55:45.192 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:45.192 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24524,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2024-03-30T00:55:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:45.187+0000 7fa647a00700 1 -- 172.21.15.116:0/198678850 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa6280a1120 msgr2=0x7fa6280a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:45.187+0000 7fa647a00700 1 --2- 172.21.15.116:0/198678850 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa6280a1120 0x7fa6280a35e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa634005950 tx=0x7fa63400b410 comp rx=0 tx=0).stop 2024-03-30T00:55:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:45.187+0000 7fa647a00700 1 -- 172.21.15.116:0/198678850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa640104090 msgr2=0x7fa6401970e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:45.187+0000 7fa647a00700 1 --2- 172.21.15.116:0/198678850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa640104090 0x7fa6401970e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0098e0 tx=0x7fa62c00f670 comp rx=0 tx=0).stop 2024-03-30T00:55:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:45.188+0000 7fa647a00700 1 -- 172.21.15.116:0/198678850 shutdown_connections 2024-03-30T00:55:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:45.188+0000 7fa647a00700 1 --2- 172.21.15.116:0/198678850 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa6280a1120 0x7fa6280a35e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:45.188+0000 7fa647a00700 1 --2- 172.21.15.116:0/198678850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6401069c0 0x7fa64019e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:45.188+0000 7fa647a00700 1 --2- 172.21.15.116:0/198678850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa640101760 0x7fa640196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:45.188+0000 7fa647a00700 1 --2- 172.21.15.116:0/198678850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa640104090 0x7fa6401970e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:45.188+0000 7fa647a00700 1 -- 172.21.15.116:0/198678850 >> 172.21.15.116:0/198678850 conn(0x7fa6400fb0d0 msgr2=0x7fa6401021a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:45.188+0000 7fa647a00700 1 -- 172.21.15.116:0/198678850 shutdown_connections 2024-03-30T00:55:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:45.188+0000 7fa647a00700 1 -- 172.21.15.116:0/198678850 wait complete. 2024-03-30T00:55:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 11 2024-03-30T00:55:45.674 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 30 00:55:45 smithi116 bash[88575]: dc16e43b1112e3163e56b780ead3d6ea08f49ca6cedb6d2e1e52f6268c4ffdb7 2024-03-30T00:55:45.674 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 30 00:55:45 smithi116 systemd[1]: Started Ceph mds.a for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:55:45.674 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 30 00:55:45 smithi116 ceph-mds[88615]: -- 172.21.15.116:0/2091821532 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5586e46721a0 con 0x5586e464a800 2024-03-30T00:55:45.674 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 30 00:55:45 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-a[88590]: starting mds.a at 2024-03-30T00:55:46.410 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:46 smithi130 ceph-mon[47626]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:46.410 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:46 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/198678850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:46.411 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:46.411 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:46.411 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:46.411 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:46.411 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:46.411 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:46 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:46.411 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:46 smithi130 ceph-mon[47626]: Deploying daemon mds.b on smithi130 2024-03-30T00:55:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:46 smithi116 ceph-mon[38086]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:46 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/198678850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:46 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:46.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:46 smithi116 ceph-mon[38086]: Deploying daemon mds.b on smithi130 2024-03-30T00:55:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:46 smithi181 ceph-mon[47622]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:46 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/198678850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T00:55:46.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T00:55:46.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:46 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:46.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:46 smithi181 ceph-mon[47622]: Deploying daemon mds.b on smithi130 2024-03-30T00:55:46.662 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 00:55:46 smithi130 systemd[1]: Starting Ceph mds.b for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T00:55:47.160 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 00:55:47 smithi130 bash[84889]: a0f7f2943bde7b35dd8ff39944bac930eb574154023ee67b32b20a3f26b8765b 2024-03-30T00:55:47.160 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 00:55:47 smithi130 systemd[1]: Started Ceph mds.b for 4909f004-ee2f-11ee-b647-cb9ed24678a4. 2024-03-30T00:55:47.160 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 00:55:47 smithi130 ceph-mds[84927]: -- 172.21.15.130:0/2577352538 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5615a36c81a0 con 0x5615a369e800 2024-03-30T00:55:47.160 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 00:55:47 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-b[84904]: starting mds.b at 2024-03-30T00:55:47.214 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:55:47.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.414+0000 7ff458a00700 1 -- 172.21.15.116:0/3333205564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4541027e0 msgr2=0x7ff454109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:47.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.414+0000 7ff458a00700 1 --2- 172.21.15.116:0/3333205564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4541027e0 0x7ff454109090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00b0d0 tx=0x7ff44c00b3e0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.414+0000 7ff458a00700 1 -- 172.21.15.116:0/3333205564 shutdown_connections 2024-03-30T00:55:47.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.414+0000 7ff458a00700 1 --2- 172.21.15.116:0/3333205564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4541027e0 0x7ff454109090 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.414+0000 7ff458a00700 1 --2- 172.21.15.116:0/3333205564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454100c50 0x7ff454101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.414+0000 7ff458a00700 1 --2- 172.21.15.116:0/3333205564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff454101e40 0x7ff4541022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.414+0000 7ff458a00700 1 -- 172.21.15.116:0/3333205564 >> 172.21.15.116:0/3333205564 conn(0x7ff4540fc1c0 msgr2=0x7ff4540fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:47.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.414+0000 7ff458a00700 1 -- 172.21.15.116:0/3333205564 shutdown_connections 2024-03-30T00:55:47.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.414+0000 7ff458a00700 1 -- 172.21.15.116:0/3333205564 wait complete. 2024-03-30T00:55:47.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.416+0000 7ff458a00700 1 Processor -- start 2024-03-30T00:55:47.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.416+0000 7ff458a00700 1 -- start start 2024-03-30T00:55:47.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.416+0000 7ff458a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff454100c50 0x7ff454074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:47.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff458a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454101e40 0x7ff4540731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:47.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff458a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4541027e0 0x7ff4540736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:47.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff458a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff454073e80 con 0x7ff454101e40 2024-03-30T00:55:47.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff458a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff454073ff0 con 0x7ff454100c50 2024-03-30T00:55:47.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff458a00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4541a0d20 con 0x7ff4541027e0 2024-03-30T00:55:47.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff451e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454101e40 0x7ff4540731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:47.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff452800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff454100c50 0x7ff454074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:47.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff452800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff454100c50 0x7ff454074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:47670/0 (socket says 172.21.15.116:47670) 2024-03-30T00:55:47.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff452800700 1 -- 172.21.15.116:0/3853482491 learned_addr learned my addr 172.21.15.116:0/3853482491 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:47.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff453200700 1 --2- 172.21.15.116:0/3853482491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4541027e0 0x7ff4540736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:47.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:47 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.116:6840/2469944136,v1:172.21.15.116:6841/2469944136] up:boot 2024-03-30T00:55:47.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:47 smithi116 ceph-mon[38086]: fsmap 11 up:standby 2024-03-30T00:55:47.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:47 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-30T00:55:47.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:47 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:47 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:47 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:47 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.426 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:47 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff452800700 1 -- 172.21.15.116:0/3853482491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4541027e0 msgr2=0x7ff4540736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:47.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff452800700 1 --2- 172.21.15.116:0/3853482491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4541027e0 0x7ff4540736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff452800700 1 -- 172.21.15.116:0/3853482491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454101e40 msgr2=0x7ff4540731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:47.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff452800700 1 --2- 172.21.15.116:0/3853482491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454101e40 0x7ff4540731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.417+0000 7ff452800700 1 -- 172.21.15.116:0/3853482491 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44c009d00 con 0x7ff454100c50 2024-03-30T00:55:47.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.418+0000 7ff452800700 1 --2- 172.21.15.116:0/3853482491 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff454100c50 0x7ff454074b10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff44400e970 tx=0x7ff44400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:47.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.418+0000 7ff44b400700 1 -- 172.21.15.116:0/3853482491 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff444009790 con 0x7ff454100c50 2024-03-30T00:55:47.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.418+0000 7ff44b400700 1 -- 172.21.15.116:0/3853482491 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff44400c780 con 0x7ff454100c50 2024-03-30T00:55:47.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.418+0000 7ff458a00700 1 -- 172.21.15.116:0/3853482491 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4541a0fb0 con 0x7ff454100c50 2024-03-30T00:55:47.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.418+0000 7ff44b400700 1 -- 172.21.15.116:0/3853482491 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff444003be0 con 0x7ff454100c50 2024-03-30T00:55:47.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.418+0000 7ff458a00700 1 -- 172.21.15.116:0/3853482491 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4541a13e0 con 0x7ff454100c50 2024-03-30T00:55:47.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.419+0000 7ff458a00700 1 -- 172.21.15.116:0/3853482491 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff454108070 con 0x7ff454100c50 2024-03-30T00:55:47.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.421+0000 7ff44b400700 1 -- 172.21.15.116:0/3853482491 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7ff44400cbf0 con 0x7ff454100c50 2024-03-30T00:55:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.422+0000 7ff44b400700 1 --2- 172.21.15.116:0/3853482491 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff4380a1220 0x7ff4380a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.423+0000 7ff44b400700 1 -- 172.21.15.116:0/3853482491 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff444014070 con 0x7ff454100c50 2024-03-30T00:55:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.423+0000 7ff451e00700 1 --2- 172.21.15.116:0/3853482491 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff4380a1220 0x7ff4380a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:47.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.423+0000 7ff451e00700 1 --2- 172.21.15.116:0/3853482491 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff4380a1220 0x7ff4380a36e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff440009c60 tx=0x7ff440009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:47.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.425+0000 7ff44b400700 1 -- 172.21.15.116:0/3853482491 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff444096860 con 0x7ff454100c50 2024-03-30T00:55:47.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:47 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.116:6840/2469944136,v1:172.21.15.116:6841/2469944136] up:boot 2024-03-30T00:55:47.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:47 smithi181 ceph-mon[47622]: fsmap 11 up:standby 2024-03-30T00:55:47.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:47 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-30T00:55:47.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:47 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:47 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:47 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:47 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:47 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:47 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.116:6840/2469944136,v1:172.21.15.116:6841/2469944136] up:boot 2024-03-30T00:55:47.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:47 smithi130 ceph-mon[47626]: fsmap 11 up:standby 2024-03-30T00:55:47.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:47 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-30T00:55:47.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:47 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:47 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:47 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:47 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:47 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:47.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.692+0000 7ff458a00700 1 -- 172.21.15.116:0/3853482491 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff45410c4c0 con 0x7ff454100c50 2024-03-30T00:55:47.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.693+0000 7ff44b400700 1 -- 172.21.15.116:0/3853482491 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9616 (secure 0 0 0) 0x7ff444099e80 con 0x7ff454100c50 2024-03-30T00:55:47.698 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:47.698 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24524,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2024-03-30T00:55:47.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.694+0000 7ff458a00700 1 -- 172.21.15.116:0/3853482491 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff4380a1220 msgr2=0x7ff4380a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:47.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.694+0000 7ff458a00700 1 --2- 172.21.15.116:0/3853482491 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff4380a1220 0x7ff4380a36e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff440009c60 tx=0x7ff440009260 comp rx=0 tx=0).stop 2024-03-30T00:55:47.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.694+0000 7ff458a00700 1 -- 172.21.15.116:0/3853482491 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff454100c50 msgr2=0x7ff454074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:47.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.694+0000 7ff458a00700 1 --2- 172.21.15.116:0/3853482491 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff454100c50 0x7ff454074b10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff44400e970 tx=0x7ff44400ec80 comp rx=0 tx=0).stop 2024-03-30T00:55:47.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.694+0000 7ff458a00700 1 -- 172.21.15.116:0/3853482491 shutdown_connections 2024-03-30T00:55:47.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.694+0000 7ff458a00700 1 --2- 172.21.15.116:0/3853482491 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff4380a1220 0x7ff4380a36e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.694+0000 7ff458a00700 1 --2- 172.21.15.116:0/3853482491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4541027e0 0x7ff4540736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.694+0000 7ff458a00700 1 --2- 172.21.15.116:0/3853482491 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff454100c50 0x7ff454074b10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.694+0000 7ff458a00700 1 --2- 172.21.15.116:0/3853482491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454101e40 0x7ff4540731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.694+0000 7ff458a00700 1 -- 172.21.15.116:0/3853482491 >> 172.21.15.116:0/3853482491 conn(0x7ff4540fc1c0 msgr2=0x7ff4541051b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:47.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.694+0000 7ff458a00700 1 -- 172.21.15.116:0/3853482491 shutdown_connections 2024-03-30T00:55:47.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.694+0000 7ff458a00700 1 -- 172.21.15.116:0/3853482491 wait complete. 2024-03-30T00:55:47.702 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 13 2024-03-30T00:55:47.722 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:55:47.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.923+0000 7f338e600700 1 -- 172.21.15.116:0/553379712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3388073810 msgr2=0x7f3388073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:47.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.923+0000 7f338e600700 1 --2- 172.21.15.116:0/553379712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3388073810 0x7f3388073c90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3374009990 tx=0x7f3374009ca0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.923+0000 7f338e600700 1 -- 172.21.15.116:0/553379712 shutdown_connections 2024-03-30T00:55:47.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.923+0000 7f338e600700 1 --2- 172.21.15.116:0/553379712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3388108b40 0x7f338810afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.923+0000 7f338e600700 1 --2- 172.21.15.116:0/553379712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3388074df0 0x7f3388073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.923+0000 7f338e600700 1 --2- 172.21.15.116:0/553379712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3388073810 0x7f3388073c90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.923+0000 7f338e600700 1 -- 172.21.15.116:0/553379712 >> 172.21.15.116:0/553379712 conn(0x7f33880fc1f0 msgr2=0x7f33880fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:47.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.923+0000 7f338e600700 1 -- 172.21.15.116:0/553379712 shutdown_connections 2024-03-30T00:55:47.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.923+0000 7f338e600700 1 -- 172.21.15.116:0/553379712 wait complete. 2024-03-30T00:55:47.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.924+0000 7f338e600700 1 Processor -- start 2024-03-30T00:55:47.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.924+0000 7f338e600700 1 -- start start 2024-03-30T00:55:47.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.924+0000 7f338e600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3388073810 0x7f3388198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:47.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.924+0000 7f338e600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3388074df0 0x7f33881993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:47.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.924+0000 7f338e600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3388108b40 0x7f33881a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:47.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.924+0000 7f338e600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33881a09a0 con 0x7f3388073810 2024-03-30T00:55:47.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.924+0000 7f338e600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33881a0ae0 con 0x7f3388108b40 2024-03-30T00:55:47.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.924+0000 7f338e600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33881a0c70 con 0x7f3388074df0 2024-03-30T00:55:47.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.925+0000 7f3387e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3388108b40 0x7f33881a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:47.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.925+0000 7f3387e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3388108b40 0x7f33881a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:47686/0 (socket says 172.21.15.116:47686) 2024-03-30T00:55:47.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.925+0000 7f3387400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3388073810 0x7f3388198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:47.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.925+0000 7f3387e00700 1 -- 172.21.15.116:0/3873906077 learned_addr learned my addr 172.21.15.116:0/3873906077 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:47.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.925+0000 7f3386a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3388074df0 0x7f33881993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:47.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.925+0000 7f3387e00700 1 -- 172.21.15.116:0/3873906077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3388074df0 msgr2=0x7f33881993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:47.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.925+0000 7f3387e00700 1 --2- 172.21.15.116:0/3873906077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3388074df0 0x7f33881993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.925+0000 7f3387e00700 1 -- 172.21.15.116:0/3873906077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3388073810 msgr2=0x7f3388198e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:47.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.925+0000 7f3387e00700 1 --2- 172.21.15.116:0/3873906077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3388073810 0x7f3388198e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:47.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.925+0000 7f3387e00700 1 -- 172.21.15.116:0/3873906077 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740095f0 con 0x7f3388108b40 2024-03-30T00:55:47.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.925+0000 7f3387e00700 1 --2- 172.21.15.116:0/3873906077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3388108b40 0x7f33881a03d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f338000ef30 tx=0x7f338000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:47.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.925+0000 7f337fe00700 1 -- 172.21.15.116:0/3873906077 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3380018070 con 0x7f3388108b40 2024-03-30T00:55:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.925+0000 7f337fe00700 1 -- 172.21.15.116:0/3873906077 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f338000f040 con 0x7f3388108b40 2024-03-30T00:55:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.925+0000 7f338e600700 1 -- 172.21.15.116:0/3873906077 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33881a0ef0 con 0x7f3388108b40 2024-03-30T00:55:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.926+0000 7f337fe00700 1 -- 172.21.15.116:0/3873906077 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338000cd00 con 0x7f3388108b40 2024-03-30T00:55:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.926+0000 7f338e600700 1 -- 172.21.15.116:0/3873906077 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33881a1370 con 0x7f3388108b40 2024-03-30T00:55:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.927+0000 7f338e600700 1 -- 172.21.15.116:0/3873906077 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3388107900 con 0x7f3388108b40 2024-03-30T00:55:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.929+0000 7f337fe00700 1 -- 172.21.15.116:0/3873906077 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f3380003a00 con 0x7f3388108b40 2024-03-30T00:55:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.930+0000 7f337fe00700 1 --2- 172.21.15.116:0/3873906077 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f336c0a1120 0x7f336c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.930+0000 7f337fe00700 1 -- 172.21.15.116:0/3873906077 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f33800cbf50 con 0x7f3388108b40 2024-03-30T00:55:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.930+0000 7f3387400700 1 --2- 172.21.15.116:0/3873906077 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f336c0a1120 0x7f336c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.931+0000 7f3387400700 1 --2- 172.21.15.116:0/3873906077 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f336c0a1120 0x7f336c0a35e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3378009c60 tx=0x7f3378009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:47.934+0000 7f337fe00700 1 -- 172.21.15.116:0/3873906077 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3380095bf0 con 0x7f3388108b40 2024-03-30T00:55:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.193+0000 7f338e600700 1 -- 172.21.15.116:0/3873906077 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f338810e6c0 con 0x7f3388108b40 2024-03-30T00:55:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.194+0000 7f337fe00700 1 -- 172.21.15.116:0/3873906077 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+18490 (secure 0 0 0) 0x7f3380099210 con 0x7f3388108b40 2024-03-30T00:55:48.200 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:48.200 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":73,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:55:17.191932+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:55:48.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.197+0000 7f338e600700 1 -- 172.21.15.116:0/3873906077 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f336c0a1120 msgr2=0x7f336c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:48.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.197+0000 7f338e600700 1 --2- 172.21.15.116:0/3873906077 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f336c0a1120 0x7f336c0a35e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3378009c60 tx=0x7f3378009260 comp rx=0 tx=0).stop 2024-03-30T00:55:48.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.197+0000 7f338e600700 1 -- 172.21.15.116:0/3873906077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3388108b40 msgr2=0x7f33881a03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:48.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.197+0000 7f338e600700 1 --2- 172.21.15.116:0/3873906077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3388108b40 0x7f33881a03d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f338000ef30 tx=0x7f338000c3a0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.197+0000 7f338e600700 1 -- 172.21.15.116:0/3873906077 shutdown_connections 2024-03-30T00:55:48.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.197+0000 7f338e600700 1 --2- 172.21.15.116:0/3873906077 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f336c0a1120 0x7f336c0a35e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.197+0000 7f338e600700 1 --2- 172.21.15.116:0/3873906077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3388108b40 0x7f33881a03d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.197+0000 7f338e600700 1 --2- 172.21.15.116:0/3873906077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3388073810 0x7f3388198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.197+0000 7f338e600700 1 --2- 172.21.15.116:0/3873906077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3388074df0 0x7f33881993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.197+0000 7f338e600700 1 -- 172.21.15.116:0/3873906077 >> 172.21.15.116:0/3873906077 conn(0x7f33880fc1f0 msgr2=0x7f3388104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:48.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.197+0000 7f338e600700 1 -- 172.21.15.116:0/3873906077 shutdown_connections 2024-03-30T00:55:48.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.197+0000 7f338e600700 1 -- 172.21.15.116:0/3873906077 wait complete. 2024-03-30T00:55:48.222 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T00:52:29.602428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-30T00:55:48.223 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T00:55:48.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:48 smithi116 ceph-mon[38086]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:48.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:48 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] up:boot 2024-03-30T00:55:48.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:48 smithi116 ceph-mon[38086]: fsmap 12 up:standby 2024-03-30T00:55:48.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:48 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T00:55:48.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:48 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:48.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:48 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3853482491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.445+0000 7fecd0c00700 1 -- 172.21.15.116:0/2726687368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feccc100d80 msgr2=0x7feccc1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.445+0000 7fecd0c00700 1 --2- 172.21.15.116:0/2726687368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feccc100d80 0x7feccc1011a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fecbc009990 tx=0x7fecbc009ca0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.445+0000 7fecd0c00700 1 -- 172.21.15.116:0/2726687368 shutdown_connections 2024-03-30T00:55:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.445+0000 7fecd0c00700 1 --2- 172.21.15.116:0/2726687368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feccc100d80 0x7feccc1011a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.445+0000 7fecd0c00700 1 --2- 172.21.15.116:0/2726687368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feccc102910 0x7feccc1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.445+0000 7fecd0c00700 1 --2- 172.21.15.116:0/2726687368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feccc101f70 0x7feccc1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.445+0000 7fecd0c00700 1 -- 172.21.15.116:0/2726687368 >> 172.21.15.116:0/2726687368 conn(0x7feccc0fc2f0 msgr2=0x7feccc0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.445+0000 7fecd0c00700 1 -- 172.21.15.116:0/2726687368 shutdown_connections 2024-03-30T00:55:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.446+0000 7fecd0c00700 1 -- 172.21.15.116:0/2726687368 wait complete. 2024-03-30T00:55:48.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.446+0000 7fecd0c00700 1 Processor -- start 2024-03-30T00:55:48.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.447+0000 7fecd0c00700 1 -- start start 2024-03-30T00:55:48.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.447+0000 7fecd0c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feccc100d80 0x7feccc196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:48.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.447+0000 7fecd0c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feccc101f70 0x7feccc1972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:48.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.447+0000 7fecd0c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feccc102910 0x7feccc19e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:48.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.447+0000 7fecd0c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc19e870 con 0x7feccc102910 2024-03-30T00:55:48.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.447+0000 7fecd0c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc19e9e0 con 0x7feccc100d80 2024-03-30T00:55:48.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.447+0000 7fecd0c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc19eba0 con 0x7feccc101f70 2024-03-30T00:55:48.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.447+0000 7fecc9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feccc100d80 0x7feccc196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:48.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.447+0000 7fecc9400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feccc101f70 0x7feccc1972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:48.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.447+0000 7fecc9e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feccc100d80 0x7feccc196d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:47708/0 (socket says 172.21.15.116:47708) 2024-03-30T00:55:48.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.447+0000 7fecc9e00700 1 -- 172.21.15.116:0/2446875951 learned_addr learned my addr 172.21.15.116:0/2446875951 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:48.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.447+0000 7fecca800700 1 --2- 172.21.15.116:0/2446875951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feccc102910 0x7feccc19e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:48.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.448+0000 7fecc9e00700 1 -- 172.21.15.116:0/2446875951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feccc101f70 msgr2=0x7feccc1972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:48.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.448+0000 7fecc9e00700 1 --2- 172.21.15.116:0/2446875951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feccc101f70 0x7feccc1972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.448+0000 7fecc9e00700 1 -- 172.21.15.116:0/2446875951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feccc102910 msgr2=0x7feccc19e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:48.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.448+0000 7fecc9e00700 1 --2- 172.21.15.116:0/2446875951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feccc102910 0x7feccc19e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.448+0000 7fecc9e00700 1 -- 172.21.15.116:0/2446875951 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb8009520 con 0x7feccc100d80 2024-03-30T00:55:48.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.448+0000 7fecca800700 1 --2- 172.21.15.116:0/2446875951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feccc102910 0x7feccc19e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:55:48.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.448+0000 7fecc9e00700 1 --2- 172.21.15.116:0/2446875951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feccc100d80 0x7feccc196d80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fecbc00b680 tx=0x7fecbc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:48.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.448+0000 7fecc2a00700 1 -- 172.21.15.116:0/2446875951 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecbc01d070 con 0x7feccc100d80 2024-03-30T00:55:48.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.448+0000 7fecd0c00700 1 -- 172.21.15.116:0/2446875951 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecbc0095f0 con 0x7feccc100d80 2024-03-30T00:55:48.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.448+0000 7fecc2a00700 1 -- 172.21.15.116:0/2446875951 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fecbc021410 con 0x7feccc100d80 2024-03-30T00:55:48.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.448+0000 7fecc2a00700 1 -- 172.21.15.116:0/2446875951 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecbc017650 con 0x7feccc100d80 2024-03-30T00:55:48.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.448+0000 7fecd0c00700 1 -- 172.21.15.116:0/2446875951 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feccc19f0b0 con 0x7feccc100d80 2024-03-30T00:55:48.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.449+0000 7fecd0c00700 1 -- 172.21.15.116:0/2446875951 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feccc1081d0 con 0x7feccc100d80 2024-03-30T00:55:48.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.451+0000 7fecc2a00700 1 -- 172.21.15.116:0/2446875951 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fecbc021580 con 0x7feccc100d80 2024-03-30T00:55:48.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.452+0000 7fecc2a00700 1 --2- 172.21.15.116:0/2446875951 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fecb00a1120 0x7fecb00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:48.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.452+0000 7fecc2a00700 1 -- 172.21.15.116:0/2446875951 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fecbc0cd8c0 con 0x7feccc100d80 2024-03-30T00:55:48.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.452+0000 7fecc9400700 1 --2- 172.21.15.116:0/2446875951 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fecb00a1120 0x7fecb00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:48.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.453+0000 7fecc9400700 1 --2- 172.21.15.116:0/2446875951 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fecb00a1120 0x7fecb00a35e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fecb8009df0 tx=0x7fecb8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:48.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.456+0000 7fecc2a00700 1 -- 172.21.15.116:0/2446875951 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fecbc097520 con 0x7feccc100d80 2024-03-30T00:55:48.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:48 smithi181 ceph-mon[47622]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:48.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:48 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] up:boot 2024-03-30T00:55:48.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:48 smithi181 ceph-mon[47622]: fsmap 12 up:standby 2024-03-30T00:55:48.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:48 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T00:55:48.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:48 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:48.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:48 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3853482491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:48.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:48 smithi130 ceph-mon[47626]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:48.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:48 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] up:boot 2024-03-30T00:55:48.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:48 smithi130 ceph-mon[47626]: fsmap 12 up:standby 2024-03-30T00:55:48.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:48 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T00:55:48.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:48 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:48.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:48 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3853482491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.710+0000 7fecd0c00700 1 -- 172.21.15.116:0/2446875951 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feccc107bb0 con 0x7feccc100d80 2024-03-30T00:55:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.710+0000 7fecc2a00700 1 -- 172.21.15.116:0/2446875951 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7fecbc09ab40 con 0x7feccc100d80 2024-03-30T00:55:48.716 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-30T00:55:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.713+0000 7fecd0c00700 1 -- 172.21.15.116:0/2446875951 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fecb00a1120 msgr2=0x7fecb00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.713+0000 7fecd0c00700 1 --2- 172.21.15.116:0/2446875951 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fecb00a1120 0x7fecb00a35e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fecb8009df0 tx=0x7fecb8009260 comp rx=0 tx=0).stop 2024-03-30T00:55:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.713+0000 7fecd0c00700 1 -- 172.21.15.116:0/2446875951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feccc100d80 msgr2=0x7feccc196d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.713+0000 7fecd0c00700 1 --2- 172.21.15.116:0/2446875951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feccc100d80 0x7feccc196d80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fecbc00b680 tx=0x7fecbc004730 comp rx=0 tx=0).stop 2024-03-30T00:55:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.713+0000 7fecd0c00700 1 -- 172.21.15.116:0/2446875951 shutdown_connections 2024-03-30T00:55:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.713+0000 7fecd0c00700 1 --2- 172.21.15.116:0/2446875951 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fecb00a1120 0x7fecb00a35e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.713+0000 7fecd0c00700 1 --2- 172.21.15.116:0/2446875951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feccc102910 0x7feccc19e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.713+0000 7fecd0c00700 1 --2- 172.21.15.116:0/2446875951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feccc100d80 0x7feccc196d80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.713+0000 7fecd0c00700 1 --2- 172.21.15.116:0/2446875951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feccc101f70 0x7feccc1972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.713+0000 7fecd0c00700 1 -- 172.21.15.116:0/2446875951 >> 172.21.15.116:0/2446875951 conn(0x7feccc0fc2f0 msgr2=0x7feccc105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.713+0000 7fecd0c00700 1 -- 172.21.15.116:0/2446875951 shutdown_connections 2024-03-30T00:55:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.713+0000 7fecd0c00700 1 -- 172.21.15.116:0/2446875951 wait complete. 2024-03-30T00:55:48.740 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-30T00:55:48.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.944+0000 7fa9a6400700 1 -- 172.21.15.116:0/1199141641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9a0101e50 msgr2=0x7fa9a01022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:48.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.944+0000 7fa9a6400700 1 --2- 172.21.15.116:0/1199141641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9a0101e50 0x7fa9a01022b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0099c0 tx=0x7fa98c009cd0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.944+0000 7fa9a6400700 1 -- 172.21.15.116:0/1199141641 shutdown_connections 2024-03-30T00:55:48.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.944+0000 7fa9a6400700 1 --2- 172.21.15.116:0/1199141641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9a0100c60 0x7fa9a0101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.944+0000 7fa9a6400700 1 --2- 172.21.15.116:0/1199141641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a01027f0 0x7fa9a01090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.944+0000 7fa9a6400700 1 --2- 172.21.15.116:0/1199141641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9a0101e50 0x7fa9a01022b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.944+0000 7fa9a6400700 1 -- 172.21.15.116:0/1199141641 >> 172.21.15.116:0/1199141641 conn(0x7fa9a00fc1f0 msgr2=0x7fa9a00fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.944+0000 7fa9a6400700 1 -- 172.21.15.116:0/1199141641 shutdown_connections 2024-03-30T00:55:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.944+0000 7fa9a6400700 1 -- 172.21.15.116:0/1199141641 wait complete. 2024-03-30T00:55:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.946+0000 7fa9a6400700 1 Processor -- start 2024-03-30T00:55:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.946+0000 7fa9a6400700 1 -- start start 2024-03-30T00:55:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.946+0000 7fa9a6400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9a0100c60 0x7fa9a0198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.946+0000 7fa9a6400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a0101e50 0x7fa9a0199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.946+0000 7fa9a6400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9a01027f0 0x7fa9a01a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.946+0000 7fa9a6400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a01a08e0 con 0x7fa9a0101e50 2024-03-30T00:55:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.946+0000 7fa9a6400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a01a0a50 con 0x7fa9a01027f0 2024-03-30T00:55:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.946+0000 7fa9a6400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a01a0c10 con 0x7fa9a0100c60 2024-03-30T00:55:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.947+0000 7fa9a5000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9a0100c60 0x7fa9a0198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.947+0000 7fa9a5000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9a0100c60 0x7fa9a0198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59508/0 (socket says 172.21.15.116:59508) 2024-03-30T00:55:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.947+0000 7fa9a5000700 1 -- 172.21.15.116:0/3637036994 learned_addr learned my addr 172.21.15.116:0/3637036994 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.947+0000 7fa99fe00700 1 --2- 172.21.15.116:0/3637036994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a0101e50 0x7fa9a0199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.947+0000 7fa9a5a00700 1 --2- 172.21.15.116:0/3637036994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9a01027f0 0x7fa9a01a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.947+0000 7fa9a5000700 1 -- 172.21.15.116:0/3637036994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9a01027f0 msgr2=0x7fa9a01a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.947+0000 7fa9a5000700 1 --2- 172.21.15.116:0/3637036994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9a01027f0 0x7fa9a01a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.947+0000 7fa9a5000700 1 -- 172.21.15.116:0/3637036994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a0101e50 msgr2=0x7fa9a0199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.947+0000 7fa9a5000700 1 --2- 172.21.15.116:0/3637036994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a0101e50 0x7fa9a0199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.947+0000 7fa9a5000700 1 -- 172.21.15.116:0/3637036994 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c0095f0 con 0x7fa9a0100c60 2024-03-30T00:55:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.948+0000 7fa9a5000700 1 --2- 172.21.15.116:0/3637036994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9a0100c60 0x7fa9a0198df0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa99400e970 tx=0x7fa99400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.948+0000 7fa99d600700 1 -- 172.21.15.116:0/3637036994 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994009790 con 0x7fa9a0100c60 2024-03-30T00:55:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.948+0000 7fa9a6400700 1 -- 172.21.15.116:0/3637036994 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a01a0ef0 con 0x7fa9a0100c60 2024-03-30T00:55:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.948+0000 7fa99d600700 1 -- 172.21.15.116:0/3637036994 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa99400c7e0 con 0x7fa9a0100c60 2024-03-30T00:55:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.948+0000 7fa99d600700 1 -- 172.21.15.116:0/3637036994 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994003c70 con 0x7fa9a0100c60 2024-03-30T00:55:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.948+0000 7fa9a6400700 1 -- 172.21.15.116:0/3637036994 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a01a1320 con 0x7fa9a0100c60 2024-03-30T00:55:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.949+0000 7fa9a6400700 1 -- 172.21.15.116:0/3637036994 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a00679a0 con 0x7fa9a0100c60 2024-03-30T00:55:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.951+0000 7fa99d600700 1 -- 172.21.15.116:0/3637036994 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fa99400cc50 con 0x7fa9a0100c60 2024-03-30T00:55:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.952+0000 7fa99d600700 1 --2- 172.21.15.116:0/3637036994 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa9880a1120 0x7fa9880a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.952+0000 7fa99d600700 1 -- 172.21.15.116:0/3637036994 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa994014070 con 0x7fa9a0100c60 2024-03-30T00:55:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.952+0000 7fa99fe00700 1 --2- 172.21.15.116:0/3637036994 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa9880a1120 0x7fa9880a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.953+0000 7fa99fe00700 1 --2- 172.21.15.116:0/3637036994 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa9880a1120 0x7fa9880a35e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa98c003730 tx=0x7fa98c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:48.956+0000 7fa99d600700 1 -- 172.21.15.116:0/3637036994 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa994096230 con 0x7fa9a0100c60 2024-03-30T00:55:49.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.210+0000 7fa9a6400700 1 -- 172.21.15.116:0/3637036994 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa9a010c4c0 con 0x7fa9a0100c60 2024-03-30T00:55:49.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.210+0000 7fa99d600700 1 -- 172.21.15.116:0/3637036994 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v73) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa994003dd0 con 0x7fa9a0100c60 2024-03-30T00:55:49.216 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:49.217 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-03-30T00:55:49.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T00:55:49.217 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 1, 2024-03-30T00:55:49.217 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": ".mgr" 2024-03-30T00:55:49.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T00:55:49.217 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-03-30T00:55:49.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.213+0000 7fa9a6400700 1 -- 172.21.15.116:0/3637036994 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa9880a1120 msgr2=0x7fa9880a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:49.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.213+0000 7fa9a6400700 1 --2- 172.21.15.116:0/3637036994 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa9880a1120 0x7fa9880a35e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa98c003730 tx=0x7fa98c00b540 comp rx=0 tx=0).stop 2024-03-30T00:55:49.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.213+0000 7fa9a6400700 1 -- 172.21.15.116:0/3637036994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9a0100c60 msgr2=0x7fa9a0198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:49.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.213+0000 7fa9a6400700 1 --2- 172.21.15.116:0/3637036994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9a0100c60 0x7fa9a0198df0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa99400e970 tx=0x7fa99400ec80 comp rx=0 tx=0).stop 2024-03-30T00:55:49.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.213+0000 7fa9a6400700 1 -- 172.21.15.116:0/3637036994 shutdown_connections 2024-03-30T00:55:49.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.213+0000 7fa9a6400700 1 --2- 172.21.15.116:0/3637036994 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa9880a1120 0x7fa9880a35e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:49.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.213+0000 7fa9a6400700 1 --2- 172.21.15.116:0/3637036994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9a0100c60 0x7fa9a0198df0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:49.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.213+0000 7fa9a6400700 1 --2- 172.21.15.116:0/3637036994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9a01027f0 0x7fa9a01a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:49.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.213+0000 7fa9a6400700 1 --2- 172.21.15.116:0/3637036994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a0101e50 0x7fa9a0199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:49.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.213+0000 7fa9a6400700 1 -- 172.21.15.116:0/3637036994 >> 172.21.15.116:0/3637036994 conn(0x7fa9a00fc1f0 msgr2=0x7fa9a0072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:49.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.213+0000 7fa9a6400700 1 -- 172.21.15.116:0/3637036994 shutdown_connections 2024-03-30T00:55:49.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.213+0000 7fa9a6400700 1 -- 172.21.15.116:0/3637036994 wait complete. 2024-03-30T00:55:49.238 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-30T00:55:49.238 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-30T00:55:49.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:49 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3873906077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:49.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:49 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2446875951' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:55:49.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:49.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:49.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:49 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.438+0000 7f0bbfa00700 1 -- 172.21.15.116:0/576088229 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bb81017e0 msgr2=0x7f0bb8103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.438+0000 7f0bbfa00700 1 --2- 172.21.15.116:0/576088229 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bb81017e0 0x7f0bb8103bd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0bac009990 tx=0x7f0bac009ca0 comp rx=0 tx=0).stop 2024-03-30T00:55:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.438+0000 7f0bbfa00700 1 -- 172.21.15.116:0/576088229 shutdown_connections 2024-03-30T00:55:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.438+0000 7f0bbfa00700 1 --2- 172.21.15.116:0/576088229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bb8106a40 0x7f0bb8108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.438+0000 7f0bbfa00700 1 --2- 172.21.15.116:0/576088229 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bb81017e0 0x7f0bb8103bd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.438+0000 7f0bbfa00700 1 --2- 172.21.15.116:0/576088229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bb8104110 0x7f0bb8106500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.438+0000 7f0bbfa00700 1 -- 172.21.15.116:0/576088229 >> 172.21.15.116:0/576088229 conn(0x7f0bb80fb1d0 msgr2=0x7f0bb80fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.438+0000 7f0bbfa00700 1 -- 172.21.15.116:0/576088229 shutdown_connections 2024-03-30T00:55:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.439+0000 7f0bbfa00700 1 -- 172.21.15.116:0/576088229 wait complete. 2024-03-30T00:55:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.439+0000 7f0bbfa00700 1 Processor -- start 2024-03-30T00:55:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.440+0000 7f0bbfa00700 1 -- start start 2024-03-30T00:55:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.440+0000 7f0bbfa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bb81017e0 0x7f0bb8101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.440+0000 7f0bbfa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bb8104110 0x7f0bb80ff790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.440+0000 7f0bbfa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bb8106a40 0x7f0bb80ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.440+0000 7f0bbfa00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb8100470 con 0x7f0bb8106a40 2024-03-30T00:55:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.440+0000 7f0bbfa00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb81005e0 con 0x7f0bb8104110 2024-03-30T00:55:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.440+0000 7f0bbfa00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb81a0d60 con 0x7f0bb81017e0 2024-03-30T00:55:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.440+0000 7f0bb7e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bb8104110 0x7f0bb80ff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.440+0000 7f0bbd000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bb81017e0 0x7f0bb8101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.440+0000 7f0bb7e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bb8104110 0x7f0bb80ff790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:47742/0 (socket says 172.21.15.116:47742) 2024-03-30T00:55:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.440+0000 7f0bb7e00700 1 -- 172.21.15.116:0/136240444 learned_addr learned my addr 172.21.15.116:0/136240444 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.440+0000 7f0bbda00700 1 --2- 172.21.15.116:0/136240444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bb8106a40 0x7f0bb80ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.441+0000 7f0bb7e00700 1 -- 172.21.15.116:0/136240444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bb81017e0 msgr2=0x7f0bb8101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.441+0000 7f0bb7e00700 1 --2- 172.21.15.116:0/136240444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bb81017e0 0x7f0bb8101100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.441+0000 7f0bb7e00700 1 -- 172.21.15.116:0/136240444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bb8106a40 msgr2=0x7f0bb80ffcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.441+0000 7f0bb7e00700 1 --2- 172.21.15.116:0/136240444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bb8106a40 0x7f0bb80ffcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.441+0000 7f0bb7e00700 1 -- 172.21.15.116:0/136240444 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac0095f0 con 0x7f0bb8104110 2024-03-30T00:55:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.441+0000 7f0bbd000700 1 --2- 172.21.15.116:0/136240444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bb81017e0 0x7f0bb8101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.441+0000 7f0bb7e00700 1 --2- 172.21.15.116:0/136240444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bb8104110 0x7f0bb80ff790 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0ba400e9a0 tx=0x7f0ba400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.441+0000 7f0bb5600700 1 -- 172.21.15.116:0/136240444 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4009790 con 0x7f0bb8104110 2024-03-30T00:55:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.441+0000 7f0bb5600700 1 -- 172.21.15.116:0/136240444 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ba400ce70 con 0x7f0bb8104110 2024-03-30T00:55:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.441+0000 7f0bb5600700 1 -- 172.21.15.116:0/136240444 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4018420 con 0x7f0bb8104110 2024-03-30T00:55:49.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.441+0000 7f0bbfa00700 1 -- 172.21.15.116:0/136240444 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb81a0ff0 con 0x7f0bb8104110 2024-03-30T00:55:49.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.441+0000 7f0bbfa00700 1 -- 172.21.15.116:0/136240444 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb81a1420 con 0x7f0bb8104110 2024-03-30T00:55:49.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.442+0000 7f0bbfa00700 1 -- 172.21.15.116:0/136240444 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb80fcd50 con 0x7f0bb8104110 2024-03-30T00:55:49.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.444+0000 7f0bb5600700 1 -- 172.21.15.116:0/136240444 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f0ba40106f0 con 0x7f0bb8104110 2024-03-30T00:55:49.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.445+0000 7f0bb5600700 1 --2- 172.21.15.116:0/136240444 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0ba00a11a0 0x7f0ba00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:49.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.445+0000 7f0bb5600700 1 -- 172.21.15.116:0/136240444 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0ba4014070 con 0x7f0bb8104110 2024-03-30T00:55:49.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.446+0000 7f0bbd000700 1 --2- 172.21.15.116:0/136240444 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0ba00a11a0 0x7f0ba00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:49.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.446+0000 7f0bbd000700 1 --2- 172.21.15.116:0/136240444 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0ba00a11a0 0x7f0ba00a3660 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00b2e0 tx=0x7f0bac005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:49.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.448+0000 7f0bb5600700 1 -- 172.21.15.116:0/136240444 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0ba40960e0 con 0x7f0bb8104110 2024-03-30T00:55:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:49 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3873906077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:49 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2446875951' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:55:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:49.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:49.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:49 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:49 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3873906077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:49.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:49 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2446875951' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:55:49.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:49.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:49.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:49 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:49.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:49.664+0000 7f0bbfa00700 1 -- 172.21.15.116:0/136240444 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0bb810c590 con 0x7f0bb8104110 2024-03-30T00:55:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.279+0000 7f0bb5600700 1 -- 172.21.15.116:0/136240444 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v74) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0ba4099700 con 0x7f0bb8104110 2024-03-30T00:55:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.283+0000 7f0bbfa00700 1 -- 172.21.15.116:0/136240444 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0ba00a11a0 msgr2=0x7f0ba00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.283+0000 7f0bbfa00700 1 --2- 172.21.15.116:0/136240444 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0ba00a11a0 0x7f0ba00a3660 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00b2e0 tx=0x7f0bac005fb0 comp rx=0 tx=0).stop 2024-03-30T00:55:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.283+0000 7f0bbfa00700 1 -- 172.21.15.116:0/136240444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bb8104110 msgr2=0x7f0bb80ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.283+0000 7f0bbfa00700 1 --2- 172.21.15.116:0/136240444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bb8104110 0x7f0bb80ff790 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0ba400e9a0 tx=0x7f0ba400ecb0 comp rx=0 tx=0).stop 2024-03-30T00:55:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.283+0000 7f0bbfa00700 1 -- 172.21.15.116:0/136240444 shutdown_connections 2024-03-30T00:55:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.283+0000 7f0bbfa00700 1 --2- 172.21.15.116:0/136240444 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0ba00a11a0 0x7f0ba00a3660 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.283+0000 7f0bbfa00700 1 --2- 172.21.15.116:0/136240444 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0bb81017e0 0x7f0bb8101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.283+0000 7f0bbfa00700 1 --2- 172.21.15.116:0/136240444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bb8106a40 0x7f0bb80ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.283+0000 7f0bbfa00700 1 --2- 172.21.15.116:0/136240444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bb8104110 0x7f0bb80ff790 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.283+0000 7f0bbfa00700 1 -- 172.21.15.116:0/136240444 >> 172.21.15.116:0/136240444 conn(0x7f0bb80fb1d0 msgr2=0x7f0bb81022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.283+0000 7f0bbfa00700 1 -- 172.21.15.116:0/136240444 shutdown_connections 2024-03-30T00:55:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.283+0000 7f0bbfa00700 1 -- 172.21.15.116:0/136240444 wait complete. 2024-03-30T00:55:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' created 2024-03-30T00:55:50.307 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-30T00:55:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:50 smithi181 ceph-mon[47622]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:50 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3637036994' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-30T00:55:50.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:50 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.518+0000 7f3ff2600700 1 -- 172.21.15.116:0/1396771545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3fec101810 msgr2=0x7f3fec103c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.518+0000 7f3ff2600700 1 --2- 172.21.15.116:0/1396771545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3fec101810 0x7f3fec103c00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009990 tx=0x7f3fdc009ca0 comp rx=0 tx=0).stop 2024-03-30T00:55:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.518+0000 7f3ff2600700 1 -- 172.21.15.116:0/1396771545 shutdown_connections 2024-03-30T00:55:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.518+0000 7f3ff2600700 1 --2- 172.21.15.116:0/1396771545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec106a70 0x7f3fec108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.518+0000 7f3ff2600700 1 --2- 172.21.15.116:0/1396771545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3fec101810 0x7f3fec103c00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.518+0000 7f3ff2600700 1 --2- 172.21.15.116:0/1396771545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fec104140 0x7f3fec106530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.518+0000 7f3ff2600700 1 -- 172.21.15.116:0/1396771545 >> 172.21.15.116:0/1396771545 conn(0x7f3fec0fb200 msgr2=0x7f3fec0fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.518+0000 7f3ff2600700 1 -- 172.21.15.116:0/1396771545 shutdown_connections 2024-03-30T00:55:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.518+0000 7f3ff2600700 1 -- 172.21.15.116:0/1396771545 wait complete. 2024-03-30T00:55:50.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.519+0000 7f3ff2600700 1 Processor -- start 2024-03-30T00:55:50.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.520+0000 7f3ff2600700 1 -- start start 2024-03-30T00:55:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.520+0000 7f3ff2600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3fec101810 0x7f3fec101130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.520+0000 7f3ff2600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec104140 0x7f3fec0ff7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.520+0000 7f3ff2600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fec106a70 0x7f3fec0ffd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.520+0000 7f3ff2600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec1004a0 con 0x7f3fec104140 2024-03-30T00:55:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.520+0000 7f3ff2600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec100610 con 0x7f3fec101810 2024-03-30T00:55:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.520+0000 7f3ff2600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec1a0d50 con 0x7f3fec106a70 2024-03-30T00:55:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.520+0000 7f3feaa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec104140 0x7f3fec0ff7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.520+0000 7f3feb400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3fec101810 0x7f3fec101130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.520+0000 7f3feaa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec104140 0x7f3fec0ff7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60872/0 (socket says 172.21.15.116:60872) 2024-03-30T00:55:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.520+0000 7f3feb400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3fec101810 0x7f3fec101130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:47764/0 (socket says 172.21.15.116:47764) 2024-03-30T00:55:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.520+0000 7f3feaa00700 1 -- 172.21.15.116:0/1677976537 learned_addr learned my addr 172.21.15.116:0/1677976537 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.520+0000 7f3febe00700 1 --2- 172.21.15.116:0/1677976537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fec106a70 0x7f3fec0ffd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.521+0000 7f3feb400700 1 -- 172.21.15.116:0/1677976537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fec106a70 msgr2=0x7f3fec0ffd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.521+0000 7f3feb400700 1 --2- 172.21.15.116:0/1677976537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fec106a70 0x7f3fec0ffd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.521+0000 7f3feb400700 1 -- 172.21.15.116:0/1677976537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec104140 msgr2=0x7f3fec0ff7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.521+0000 7f3feb400700 1 --2- 172.21.15.116:0/1677976537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec104140 0x7f3fec0ff7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.521+0000 7f3feb400700 1 -- 172.21.15.116:0/1677976537 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc0095f0 con 0x7f3fec101810 2024-03-30T00:55:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.521+0000 7f3feaa00700 1 --2- 172.21.15.116:0/1677976537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec104140 0x7f3fec0ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.521+0000 7f3feb400700 1 --2- 172.21.15.116:0/1677976537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3fec101810 0x7f3fec101130 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc000c00 tx=0x7f3fdc009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:50.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.522+0000 7f3fe3e00700 1 -- 172.21.15.116:0/1677976537 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc01d070 con 0x7f3fec101810 2024-03-30T00:55:50.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.522+0000 7f3ff2600700 1 -- 172.21.15.116:0/1677976537 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fec1a0f80 con 0x7f3fec101810 2024-03-30T00:55:50.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.522+0000 7f3fe3e00700 1 -- 172.21.15.116:0/1677976537 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3fdc021410 con 0x7f3fec101810 2024-03-30T00:55:50.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.522+0000 7f3fe3e00700 1 -- 172.21.15.116:0/1677976537 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc017650 con 0x7f3fec101810 2024-03-30T00:55:50.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.522+0000 7f3ff2600700 1 -- 172.21.15.116:0/1677976537 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fec1a12f0 con 0x7f3fec101810 2024-03-30T00:55:50.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.523+0000 7f3ff2600700 1 -- 172.21.15.116:0/1677976537 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fec0fcd80 con 0x7f3fec101810 2024-03-30T00:55:50.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.524+0000 7f3fe3e00700 1 -- 172.21.15.116:0/1677976537 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f3fdc021580 con 0x7f3fec101810 2024-03-30T00:55:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.527+0000 7f3fe3e00700 1 --2- 172.21.15.116:0/1677976537 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3fd00a11a0 0x7f3fd00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.527+0000 7f3fe3e00700 1 -- 172.21.15.116:0/1677976537 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f3fdc0cd900 con 0x7f3fec101810 2024-03-30T00:55:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.527+0000 7f3feaa00700 1 --2- 172.21.15.116:0/1677976537 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3fd00a11a0 0x7f3fd00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.528+0000 7f3feaa00700 1 --2- 172.21.15.116:0/1677976537 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3fd00a11a0 0x7f3fd00a3660 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8011550 tx=0x7f3fd8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.530+0000 7f3fe3e00700 1 -- 172.21.15.116:0/1677976537 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3fdc0973b0 con 0x7f3fec101810 2024-03-30T00:55:50.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:50 smithi130 ceph-mon[47626]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:50.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:50 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3637036994' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-30T00:55:50.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:50 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:50.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:50 smithi116 ceph-mon[38086]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:50.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:50 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3637036994' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-30T00:55:50.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:50 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:55:50.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:50.752+0000 7f3ff2600700 1 -- 172.21.15.116:0/1677976537 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f3fec067260 con 0x7f3fec101810 2024-03-30T00:55:51.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.287+0000 7f3fe3e00700 1 -- 172.21.15.116:0/1677976537 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v75) v1 ==== 171+0+0 (secure 0 0 0) 0x7f3fdc005740 con 0x7f3fec101810 2024-03-30T00:55:51.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.295+0000 7f3ff2600700 1 -- 172.21.15.116:0/1677976537 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3fd00a11a0 msgr2=0x7f3fd00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:51.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.295+0000 7f3ff2600700 1 --2- 172.21.15.116:0/1677976537 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3fd00a11a0 0x7f3fd00a3660 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8011550 tx=0x7f3fd8009260 comp rx=0 tx=0).stop 2024-03-30T00:55:51.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.295+0000 7f3ff2600700 1 -- 172.21.15.116:0/1677976537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3fec101810 msgr2=0x7f3fec101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:51.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.295+0000 7f3ff2600700 1 --2- 172.21.15.116:0/1677976537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3fec101810 0x7f3fec101130 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc000c00 tx=0x7f3fdc009580 comp rx=0 tx=0).stop 2024-03-30T00:55:51.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.296+0000 7f3ff2600700 1 -- 172.21.15.116:0/1677976537 shutdown_connections 2024-03-30T00:55:51.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.296+0000 7f3ff2600700 1 --2- 172.21.15.116:0/1677976537 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3fd00a11a0 0x7f3fd00a3660 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.296+0000 7f3ff2600700 1 --2- 172.21.15.116:0/1677976537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fec106a70 0x7f3fec0ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.296+0000 7f3ff2600700 1 --2- 172.21.15.116:0/1677976537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3fec101810 0x7f3fec101130 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.296+0000 7f3ff2600700 1 --2- 172.21.15.116:0/1677976537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec104140 0x7f3fec0ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.296+0000 7f3ff2600700 1 -- 172.21.15.116:0/1677976537 >> 172.21.15.116:0/1677976537 conn(0x7f3fec0fb200 msgr2=0x7f3fec1036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:51.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.296+0000 7f3ff2600700 1 -- 172.21.15.116:0/1677976537 shutdown_connections 2024-03-30T00:55:51.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.297+0000 7f3ff2600700 1 -- 172.21.15.116:0/1677976537 wait complete. 2024-03-30T00:55:51.305 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' created 2024-03-30T00:55:51.334 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-30T00:55:51.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.526+0000 7fc856000700 1 -- 172.21.15.116:0/3962846554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc850101f70 msgr2=0x7fc8501023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:51.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.526+0000 7fc856000700 1 --2- 172.21.15.116:0/3962846554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc850101f70 0x7fc8501023d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009990 tx=0x7fc83c009ca0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.526+0000 7fc856000700 1 -- 172.21.15.116:0/3962846554 shutdown_connections 2024-03-30T00:55:51.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.526+0000 7fc856000700 1 --2- 172.21.15.116:0/3962846554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc850102910 0x7fc8501091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.526+0000 7fc856000700 1 --2- 172.21.15.116:0/3962846554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc850100d80 0x7fc8501011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.526+0000 7fc856000700 1 --2- 172.21.15.116:0/3962846554 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc850101f70 0x7fc8501023d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.526+0000 7fc856000700 1 -- 172.21.15.116:0/3962846554 >> 172.21.15.116:0/3962846554 conn(0x7fc8500fc2f0 msgr2=0x7fc8500fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:51.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.526+0000 7fc856000700 1 -- 172.21.15.116:0/3962846554 shutdown_connections 2024-03-30T00:55:51.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.526+0000 7fc856000700 1 -- 172.21.15.116:0/3962846554 wait complete. 2024-03-30T00:55:51.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.527+0000 7fc856000700 1 Processor -- start 2024-03-30T00:55:51.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.527+0000 7fc856000700 1 -- start start 2024-03-30T00:55:51.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.527+0000 7fc856000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc850100d80 0x7fc850198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:51.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.527+0000 7fc856000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc850101f70 0x7fc850199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:51.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.527+0000 7fc856000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc850102910 0x7fc8501a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:51.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.527+0000 7fc856000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8501a0a50 con 0x7fc850102910 2024-03-30T00:55:51.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.527+0000 7fc856000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8501a0b90 con 0x7fc850100d80 2024-03-30T00:55:51.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.527+0000 7fc856000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8501a0d20 con 0x7fc850101f70 2024-03-30T00:55:51.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.527+0000 7fc84fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc850102910 0x7fc8501a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:51.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc84f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc850100d80 0x7fc850198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:51.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc84fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc850102910 0x7fc8501a0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60890/0 (socket says 172.21.15.116:60890) 2024-03-30T00:55:51.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc84fe00700 1 -- 172.21.15.116:0/1955428405 learned_addr learned my addr 172.21.15.116:0/1955428405 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:51.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc84ea00700 1 --2- 172.21.15.116:0/1955428405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc850101f70 0x7fc850199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:51.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc84f400700 1 -- 172.21.15.116:0/1955428405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc850101f70 msgr2=0x7fc850199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:51.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc84f400700 1 --2- 172.21.15.116:0/1955428405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc850101f70 0x7fc850199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc84f400700 1 -- 172.21.15.116:0/1955428405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc850102910 msgr2=0x7fc8501a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:51.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc84f400700 1 --2- 172.21.15.116:0/1955428405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc850102910 0x7fc8501a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc84f400700 1 -- 172.21.15.116:0/1955428405 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc840009520 con 0x7fc850100d80 2024-03-30T00:55:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc84fe00700 1 --2- 172.21.15.116:0/1955428405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc850102910 0x7fc8501a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc84f400700 1 --2- 172.21.15.116:0/1955428405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc850100d80 0x7fc850198f20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc84000e9c0 tx=0x7fc84000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc847e00700 1 -- 172.21.15.116:0/1955428405 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84000cb70 con 0x7fc850100d80 2024-03-30T00:55:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc856000700 1 -- 172.21.15.116:0/1955428405 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83c0095f0 con 0x7fc850100d80 2024-03-30T00:55:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc847e00700 1 -- 172.21.15.116:0/1955428405 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc84000ee60 con 0x7fc850100d80 2024-03-30T00:55:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.528+0000 7fc847e00700 1 -- 172.21.15.116:0/1955428405 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc840018680 con 0x7fc850100d80 2024-03-30T00:55:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.529+0000 7fc856000700 1 -- 172.21.15.116:0/1955428405 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8501a1290 con 0x7fc850100d80 2024-03-30T00:55:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.529+0000 7fc856000700 1 -- 172.21.15.116:0/1955428405 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc850067af0 con 0x7fc850100d80 2024-03-30T00:55:51.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.531+0000 7fc847e00700 1 -- 172.21.15.116:0/1955428405 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fc840010430 con 0x7fc850100d80 2024-03-30T00:55:51.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.532+0000 7fc847e00700 1 --2- 172.21.15.116:0/1955428405 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc8340a1120 0x7fc8340a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:51.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.533+0000 7fc847e00700 1 -- 172.21.15.116:0/1955428405 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fc840014070 con 0x7fc850100d80 2024-03-30T00:55:51.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.533+0000 7fc84ea00700 1 --2- 172.21.15.116:0/1955428405 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc8340a1120 0x7fc8340a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:51.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.533+0000 7fc84ea00700 1 --2- 172.21.15.116:0/1955428405 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc8340a1120 0x7fc8340a35e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00b2e0 tx=0x7fc83c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:51.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.535+0000 7fc847e00700 1 -- 172.21.15.116:0/1955428405 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc840096800 con 0x7fc850100d80 2024-03-30T00:55:51.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:51 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/136240444' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-30T00:55:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:51 smithi130 ceph-mon[47626]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-30T00:55:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:51 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:51 smithi130 ceph-mon[47626]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-30T00:55:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:51 smithi130 ceph-mon[47626]: osdmap e74: 12 total, 12 up, 12 in 2024-03-30T00:55:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:51 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:51 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:51 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:51 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:51 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/136240444' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-30T00:55:51.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:51 smithi116 ceph-mon[38086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-30T00:55:51.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:51 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:51 smithi116 ceph-mon[38086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-30T00:55:51.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:51 smithi116 ceph-mon[38086]: osdmap e74: 12 total, 12 up, 12 in 2024-03-30T00:55:51.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:51 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:51 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:51 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:51 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:51 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/136240444' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-30T00:55:51.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:51 smithi181 ceph-mon[47622]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-30T00:55:51.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:51 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:51 smithi181 ceph-mon[47622]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-30T00:55:51.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:51 smithi181 ceph-mon[47622]: osdmap e74: 12 total, 12 up, 12 in 2024-03-30T00:55:51.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:51 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:51 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:51 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:51 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.788+0000 7fc856000700 1 -- 172.21.15.116:0/1955428405 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc85010c5e0 con 0x7fc850100d80 2024-03-30T00:55:51.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.799+0000 7fc847e00700 1 -- 172.21.15.116:0/1955428405 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fc840099e20 con 0x7fc850100d80 2024-03-30T00:55:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.804+0000 7fc856000700 1 -- 172.21.15.116:0/1955428405 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc8340a1120 msgr2=0x7fc8340a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.804+0000 7fc856000700 1 --2- 172.21.15.116:0/1955428405 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc8340a1120 0x7fc8340a35e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00b2e0 tx=0x7fc83c005fb0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.804+0000 7fc856000700 1 -- 172.21.15.116:0/1955428405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc850100d80 msgr2=0x7fc850198f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.804+0000 7fc856000700 1 --2- 172.21.15.116:0/1955428405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc850100d80 0x7fc850198f20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc84000e9c0 tx=0x7fc84000ecd0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.804+0000 7fc856000700 1 -- 172.21.15.116:0/1955428405 shutdown_connections 2024-03-30T00:55:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.804+0000 7fc856000700 1 --2- 172.21.15.116:0/1955428405 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc8340a1120 0x7fc8340a35e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.804+0000 7fc856000700 1 --2- 172.21.15.116:0/1955428405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc850102910 0x7fc8501a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.804+0000 7fc856000700 1 --2- 172.21.15.116:0/1955428405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc850100d80 0x7fc850198f20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.804+0000 7fc856000700 1 --2- 172.21.15.116:0/1955428405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc850101f70 0x7fc850199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.804+0000 7fc856000700 1 -- 172.21.15.116:0/1955428405 >> 172.21.15.116:0/1955428405 conn(0x7fc8500fc2f0 msgr2=0x7fc850072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.805+0000 7fc856000700 1 -- 172.21.15.116:0/1955428405 shutdown_connections 2024-03-30T00:55:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:51.805+0000 7fc856000700 1 -- 172.21.15.116:0/1955428405 wait complete. 2024-03-30T00:55:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:new fs with metadata pool 2 and data pool 3 2024-03-30T00:55:51.841 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-30T00:55:51.841 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-30T00:55:52.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.033+0000 7f17cc600700 1 -- 172.21.15.116:0/3087338198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c4100d80 msgr2=0x7f17c41011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:52.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.033+0000 7f17cc600700 1 --2- 172.21.15.116:0/3087338198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c4100d80 0x7f17c41011a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f17b80099c0 tx=0x7f17b8009cd0 comp rx=0 tx=0).stop 2024-03-30T00:55:52.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.033+0000 7f17cc600700 1 -- 172.21.15.116:0/3087338198 shutdown_connections 2024-03-30T00:55:52.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.033+0000 7f17cc600700 1 --2- 172.21.15.116:0/3087338198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17c4102910 0x7f17c41091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:52.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.033+0000 7f17cc600700 1 --2- 172.21.15.116:0/3087338198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c4100d80 0x7f17c41011a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:52.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.033+0000 7f17cc600700 1 --2- 172.21.15.116:0/3087338198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17c4101f70 0x7f17c41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:52.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.033+0000 7f17cc600700 1 -- 172.21.15.116:0/3087338198 >> 172.21.15.116:0/3087338198 conn(0x7f17c40fc2f0 msgr2=0x7f17c40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:52.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.033+0000 7f17cc600700 1 -- 172.21.15.116:0/3087338198 shutdown_connections 2024-03-30T00:55:52.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.033+0000 7f17cc600700 1 -- 172.21.15.116:0/3087338198 wait complete. 2024-03-30T00:55:52.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.034+0000 7f17cc600700 1 Processor -- start 2024-03-30T00:55:52.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.034+0000 7f17cc600700 1 -- start start 2024-03-30T00:55:52.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.034+0000 7f17cc600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17c4100d80 0x7f17c4074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:52.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.034+0000 7f17cc600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17c4101f70 0x7f17c40731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:52.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.034+0000 7f17cc600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c4102910 0x7f17c40736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:52.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.034+0000 7f17cc600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c4073e80 con 0x7f17c4102910 2024-03-30T00:55:52.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.034+0000 7f17cc600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c4073ff0 con 0x7f17c4101f70 2024-03-30T00:55:52.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.034+0000 7f17cc600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c41a0dc0 con 0x7f17c4100d80 2024-03-30T00:55:52.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17ca600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c4102910 0x7f17c40736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:52.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17c9c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17c4100d80 0x7f17c4074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:52.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17c9200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17c4101f70 0x7f17c40731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:52.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17ca600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c4102910 0x7f17c40736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60904/0 (socket says 172.21.15.116:60904) 2024-03-30T00:55:52.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17ca600700 1 -- 172.21.15.116:0/1198311613 learned_addr learned my addr 172.21.15.116:0/1198311613 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:52.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17ca600700 1 -- 172.21.15.116:0/1198311613 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17c4100d80 msgr2=0x7f17c4074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:52.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17ca600700 1 --2- 172.21.15.116:0/1198311613 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17c4100d80 0x7f17c4074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:52.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17ca600700 1 -- 172.21.15.116:0/1198311613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17c4101f70 msgr2=0x7f17c40731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:52.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17ca600700 1 --2- 172.21.15.116:0/1198311613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17c4101f70 0x7f17c40731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:52.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17ca600700 1 -- 172.21.15.116:0/1198311613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b80095f0 con 0x7f17c4102910 2024-03-30T00:55:52.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17c9c00700 1 --2- 172.21.15.116:0/1198311613 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17c4100d80 0x7f17c4074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:52.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17ca600700 1 --2- 172.21.15.116:0/1198311613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c4102910 0x7f17c40736e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00ef30 tx=0x7f17bc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:52.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17c2000700 1 -- 172.21.15.116:0/1198311613 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc018070 con 0x7f17c4102910 2024-03-30T00:55:52.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17c2000700 1 -- 172.21.15.116:0/1198311613 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17bc00f040 con 0x7f17c4102910 2024-03-30T00:55:52.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.035+0000 7f17cc600700 1 -- 172.21.15.116:0/1198311613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c41a0ff0 con 0x7f17c4102910 2024-03-30T00:55:52.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.036+0000 7f17c2000700 1 -- 172.21.15.116:0/1198311613 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc00cca0 con 0x7f17c4102910 2024-03-30T00:55:52.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.036+0000 7f17cc600700 1 -- 172.21.15.116:0/1198311613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c41a1470 con 0x7f17c4102910 2024-03-30T00:55:52.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.037+0000 7f17cc600700 1 -- 172.21.15.116:0/1198311613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c4108220 con 0x7f17c4102910 2024-03-30T00:55:52.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.039+0000 7f17c2000700 1 -- 172.21.15.116:0/1198311613 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f17bc003a00 con 0x7f17c4102910 2024-03-30T00:55:52.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.040+0000 7f17c2000700 1 --2- 172.21.15.116:0/1198311613 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f17ac0a11a0 0x7f17ac0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:52.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.040+0000 7f17c2000700 1 -- 172.21.15.116:0/1198311613 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f17bc0ccbc0 con 0x7f17c4102910 2024-03-30T00:55:52.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.040+0000 7f17c9c00700 1 --2- 172.21.15.116:0/1198311613 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f17ac0a11a0 0x7f17ac0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:52.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.041+0000 7f17c9c00700 1 --2- 172.21.15.116:0/1198311613 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f17ac0a11a0 0x7f17ac0a3660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f17b8003730 tx=0x7f17b800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:52.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.043+0000 7f17c2000700 1 -- 172.21.15.116:0/1198311613 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f17bc096490 con 0x7f17c4102910 2024-03-30T00:55:52.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.264+0000 7f17cc600700 1 -- 172.21.15.116:0/1198311613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f17c410c5e0 con 0x7f17c4102910 2024-03-30T00:55:52.595 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-b[84904]: 2024-03-30T00:55:52.166+0000 7ffbd37aa700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-30T00:55:52.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1677976537' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-30T00:55:52.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-30T00:55:52.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: pgmap v222: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:52.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-30T00:55:52.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: osdmap e75: 12 total, 12 up, 12 in 2024-03-30T00:55:52.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:52.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:52.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:52.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:52.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:52.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1955428405' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-30T00:55:52.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-30T00:55:52.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-30T00:55:52.597 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: osdmap e76: 12 total, 12 up, 12 in 2024-03-30T00:55:52.597 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-30T00:55:52.597 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-30T00:55:52.597 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-30T00:55:52.597 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] up:active 2024-03-30T00:55:52.597 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T00:55:52.597 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:52 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1198311613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T00:55:52.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1677976537' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-30T00:55:52.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-30T00:55:52.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: pgmap v222: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:52.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-30T00:55:52.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: osdmap e75: 12 total, 12 up, 12 in 2024-03-30T00:55:52.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:52.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:52.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:52.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:52.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:52.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1955428405' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-30T00:55:52.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-30T00:55:52.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-30T00:55:52.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: osdmap e76: 12 total, 12 up, 12 in 2024-03-30T00:55:52.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-30T00:55:52.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-30T00:55:52.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-30T00:55:52.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] up:active 2024-03-30T00:55:52.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T00:55:52.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:52 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1198311613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T00:55:52.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1677976537' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-30T00:55:52.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-30T00:55:52.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: pgmap v222: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:52.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-30T00:55:52.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: osdmap e75: 12 total, 12 up, 12 in 2024-03-30T00:55:52.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:52.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:52.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:55:52.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:55:52.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:55:52.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1955428405' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-30T00:55:52.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-30T00:55:52.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-30T00:55:52.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: osdmap e76: 12 total, 12 up, 12 in 2024-03-30T00:55:52.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-30T00:55:52.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-30T00:55:52.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-30T00:55:52.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] up:active 2024-03-30T00:55:52.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T00:55:52.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:52 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1198311613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T00:55:52.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.797+0000 7f17c2000700 1 -- 172.21.15.116:0/1198311613 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v77) v1 ==== 157+0+0 (secure 0 0 0) 0x7f17bc099ab0 con 0x7f17c4102910 2024-03-30T00:55:52.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.799+0000 7f17cc600700 1 -- 172.21.15.116:0/1198311613 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f17ac0a11a0 msgr2=0x7f17ac0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:52.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.799+0000 7f17cc600700 1 --2- 172.21.15.116:0/1198311613 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f17ac0a11a0 0x7f17ac0a3660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f17b8003730 tx=0x7f17b800b540 comp rx=0 tx=0).stop 2024-03-30T00:55:52.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.799+0000 7f17cc600700 1 -- 172.21.15.116:0/1198311613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c4102910 msgr2=0x7f17c40736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:52.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.799+0000 7f17cc600700 1 --2- 172.21.15.116:0/1198311613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c4102910 0x7f17c40736e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00ef30 tx=0x7f17bc00c3a0 comp rx=0 tx=0).stop 2024-03-30T00:55:52.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.799+0000 7f17cc600700 1 -- 172.21.15.116:0/1198311613 shutdown_connections 2024-03-30T00:55:52.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.799+0000 7f17cc600700 1 --2- 172.21.15.116:0/1198311613 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f17ac0a11a0 0x7f17ac0a3660 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:52.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.799+0000 7f17cc600700 1 --2- 172.21.15.116:0/1198311613 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17c4100d80 0x7f17c4074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:52.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.799+0000 7f17cc600700 1 --2- 172.21.15.116:0/1198311613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c4102910 0x7f17c40736e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:52.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.799+0000 7f17cc600700 1 --2- 172.21.15.116:0/1198311613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17c4101f70 0x7f17c40731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:52.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.800+0000 7f17cc600700 1 -- 172.21.15.116:0/1198311613 >> 172.21.15.116:0/1198311613 conn(0x7f17c40fc2f0 msgr2=0x7f17c4105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:52.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.800+0000 7f17cc600700 1 -- 172.21.15.116:0/1198311613 shutdown_connections 2024-03-30T00:55:52.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:52.800+0000 7f17cc600700 1 -- 172.21.15.116:0/1198311613 wait complete. 2024-03-30T00:55:52.824 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-30T00:55:53.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.021+0000 7f1d1d600700 1 -- 172.21.15.116:0/1152728841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d18101e80 msgr2=0x7f1d181022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:53.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.021+0000 7f1d1d600700 1 --2- 172.21.15.116:0/1152728841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d18101e80 0x7f1d181022e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1d040099c0 tx=0x7f1d04009cd0 comp rx=0 tx=0).stop 2024-03-30T00:55:53.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.021+0000 7f1d1d600700 1 -- 172.21.15.116:0/1152728841 shutdown_connections 2024-03-30T00:55:53.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.021+0000 7f1d1d600700 1 --2- 172.21.15.116:0/1152728841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d18100c90 0x7f1d181010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:53.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.021+0000 7f1d1d600700 1 --2- 172.21.15.116:0/1152728841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d18102820 0x7f1d181090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:53.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.021+0000 7f1d1d600700 1 --2- 172.21.15.116:0/1152728841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d18101e80 0x7f1d181022e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:53.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.021+0000 7f1d1d600700 1 -- 172.21.15.116:0/1152728841 >> 172.21.15.116:0/1152728841 conn(0x7f1d180fc220 msgr2=0x7f1d180fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:53.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.021+0000 7f1d1d600700 1 -- 172.21.15.116:0/1152728841 shutdown_connections 2024-03-30T00:55:53.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.021+0000 7f1d1d600700 1 -- 172.21.15.116:0/1152728841 wait complete. 2024-03-30T00:55:53.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.022+0000 7f1d1d600700 1 Processor -- start 2024-03-30T00:55:53.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.022+0000 7f1d1d600700 1 -- start start 2024-03-30T00:55:53.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.022+0000 7f1d1d600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d18100c90 0x7f1d18198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.022+0000 7f1d1d600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d18101e80 0x7f1d18199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.022+0000 7f1d1d600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d18102820 0x7f1d181a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.022+0000 7f1d1d600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d181a0940 con 0x7f1d18102820 2024-03-30T00:55:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.022+0000 7f1d1d600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d181a0a80 con 0x7f1d18100c90 2024-03-30T00:55:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.022+0000 7f1d1d600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d181a0c10 con 0x7f1d18101e80 2024-03-30T00:55:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.023+0000 7f1d17400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d18102820 0x7f1d181a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.023+0000 7f1d16a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d18100c90 0x7f1d18198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.023+0000 7f1d16000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d18101e80 0x7f1d18199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.023+0000 7f1d17400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d18102820 0x7f1d181a0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60910/0 (socket says 172.21.15.116:60910) 2024-03-30T00:55:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.023+0000 7f1d17400700 1 -- 172.21.15.116:0/3712650729 learned_addr learned my addr 172.21.15.116:0/3712650729 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.023+0000 7f1d16a00700 1 -- 172.21.15.116:0/3712650729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d18101e80 msgr2=0x7f1d18199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.023+0000 7f1d16a00700 1 --2- 172.21.15.116:0/3712650729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d18101e80 0x7f1d18199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.023+0000 7f1d16a00700 1 -- 172.21.15.116:0/3712650729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d18102820 msgr2=0x7f1d181a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.023+0000 7f1d16a00700 1 --2- 172.21.15.116:0/3712650729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d18102820 0x7f1d181a0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.023+0000 7f1d16a00700 1 -- 172.21.15.116:0/3712650729 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d040095f0 con 0x7f1d18100c90 2024-03-30T00:55:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.023+0000 7f1d17400700 1 --2- 172.21.15.116:0/3712650729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d18102820 0x7f1d181a0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.023+0000 7f1d16a00700 1 --2- 172.21.15.116:0/3712650729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d18100c90 0x7f1d18198e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800e9c0 tx=0x7f1d0800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.024+0000 7f1d0f400700 1 -- 172.21.15.116:0/3712650729 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d08004220 con 0x7f1d18100c90 2024-03-30T00:55:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.024+0000 7f1d0f400700 1 -- 172.21.15.116:0/3712650729 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d08004380 con 0x7f1d18100c90 2024-03-30T00:55:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.024+0000 7f1d0f400700 1 -- 172.21.15.116:0/3712650729 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d08003be0 con 0x7f1d18100c90 2024-03-30T00:55:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.024+0000 7f1d1d600700 1 -- 172.21.15.116:0/3712650729 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d181a0ef0 con 0x7f1d18100c90 2024-03-30T00:55:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.024+0000 7f1d1d600700 1 -- 172.21.15.116:0/3712650729 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d181a12b0 con 0x7f1d18100c90 2024-03-30T00:55:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.025+0000 7f1d1d600700 1 -- 172.21.15.116:0/3712650729 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d18067a10 con 0x7f1d18100c90 2024-03-30T00:55:53.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.026+0000 7f1d0f400700 1 -- 172.21.15.116:0/3712650729 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f1d08003d40 con 0x7f1d18100c90 2024-03-30T00:55:53.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.028+0000 7f1d0f400700 1 --2- 172.21.15.116:0/3712650729 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1cfc0a1110 0x7f1cfc0a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:53.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.028+0000 7f1d0f400700 1 -- 172.21.15.116:0/3712650729 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8205+0+0 (secure 0 0 0) 0x7f1d08014070 con 0x7f1d18100c90 2024-03-30T00:55:53.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.028+0000 7f1d16000700 1 --2- 172.21.15.116:0/3712650729 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1cfc0a1110 0x7f1cfc0a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.028+0000 7f1d16000700 1 --2- 172.21.15.116:0/3712650729 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1cfc0a1110 0x7f1cfc0a35d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1d04003730 tx=0x7f1d0400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.030+0000 7f1d0f400700 1 -- 172.21.15.116:0/3712650729 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1d08096a00 con 0x7f1d18100c90 2024-03-30T00:55:53.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:53.238+0000 7f1d1d600700 1 -- 172.21.15.116:0/3712650729 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1d181a1500 con 0x7f1d18100c90 2024-03-30T00:55:53.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:53 smithi130 ceph-mon[47626]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T00:55:53.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:53 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1198311613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-30T00:55:53.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:53 smithi130 ceph-mon[47626]: osdmap e77: 12 total, 12 up, 12 in 2024-03-30T00:55:53.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:53 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3712650729' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-30T00:55:53.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:53 smithi130 ceph-mon[47626]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-30T00:55:53.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:53 smithi116 ceph-mon[38086]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T00:55:53.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:53 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1198311613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-30T00:55:53.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:53 smithi116 ceph-mon[38086]: osdmap e77: 12 total, 12 up, 12 in 2024-03-30T00:55:53.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:53 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3712650729' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-30T00:55:53.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:53 smithi116 ceph-mon[38086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-30T00:55:53.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:53 smithi181 ceph-mon[47622]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T00:55:53.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:53 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1198311613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-30T00:55:53.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:53 smithi181 ceph-mon[47622]: osdmap e77: 12 total, 12 up, 12 in 2024-03-30T00:55:53.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:53 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3712650729' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-30T00:55:53.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:53 smithi181 ceph-mon[47622]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-30T00:55:54.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:54 smithi130 ceph-mon[47626]: pgmap v226: 97 pgs: 13 creating+peering, 11 active+clean, 73 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:54.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:54 smithi130 ceph-mon[47626]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T00:55:54.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:54 smithi130 ceph-mon[47626]: Cluster is now healthy 2024-03-30T00:55:54.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:54 smithi116 ceph-mon[38086]: pgmap v226: 97 pgs: 13 creating+peering, 11 active+clean, 73 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:54.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:54 smithi116 ceph-mon[38086]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T00:55:54.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:54 smithi116 ceph-mon[38086]: Cluster is now healthy 2024-03-30T00:55:54.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:54 smithi181 ceph-mon[47622]: pgmap v226: 97 pgs: 13 creating+peering, 11 active+clean, 73 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T00:55:54.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:54 smithi181 ceph-mon[47622]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T00:55:54.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:54 smithi181 ceph-mon[47622]: Cluster is now healthy 2024-03-30T00:55:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:54.807+0000 7f1d0f400700 1 -- 172.21.15.116:0/3712650729 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v79) v1 ==== 228+0+0 (secure 0 0 0) 0x7f1d0809a020 con 0x7f1d18100c90 2024-03-30T00:55:54.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:54.814+0000 7f1d1d600700 1 -- 172.21.15.116:0/3712650729 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1cfc0a1110 msgr2=0x7f1cfc0a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:54.814+0000 7f1d1d600700 1 --2- 172.21.15.116:0/3712650729 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1cfc0a1110 0x7f1cfc0a35d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1d04003730 tx=0x7f1d0400b540 comp rx=0 tx=0).stop 2024-03-30T00:55:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:54.814+0000 7f1d1d600700 1 -- 172.21.15.116:0/3712650729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d18100c90 msgr2=0x7f1d18198e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:54.814+0000 7f1d1d600700 1 --2- 172.21.15.116:0/3712650729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d18100c90 0x7f1d18198e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800e9c0 tx=0x7f1d0800ecd0 comp rx=0 tx=0).stop 2024-03-30T00:55:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:54.814+0000 7f1d1d600700 1 -- 172.21.15.116:0/3712650729 shutdown_connections 2024-03-30T00:55:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:54.814+0000 7f1d1d600700 1 --2- 172.21.15.116:0/3712650729 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1cfc0a1110 0x7f1cfc0a35d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:54.814+0000 7f1d1d600700 1 --2- 172.21.15.116:0/3712650729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d18102820 0x7f1d181a0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:54.814+0000 7f1d1d600700 1 --2- 172.21.15.116:0/3712650729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d18100c90 0x7f1d18198e10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:54.814+0000 7f1d1d600700 1 --2- 172.21.15.116:0/3712650729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d18101e80 0x7f1d18199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:54.814+0000 7f1d1d600700 1 -- 172.21.15.116:0/3712650729 >> 172.21.15.116:0/3712650729 conn(0x7f1d180fc220 msgr2=0x7f1d18072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:54.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:54.815+0000 7f1d1d600700 1 -- 172.21.15.116:0/3712650729 shutdown_connections 2024-03-30T00:55:54.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:54.815+0000 7f1d1d600700 1 -- 172.21.15.116:0/3712650729 wait complete. 2024-03-30T00:55:54.823 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data_ec' created 2024-03-30T00:55:54.847 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-30T00:55:55.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.040+0000 7f6a4d000700 1 -- 172.21.15.116:0/3647185623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a48100d80 msgr2=0x7f6a481011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:55.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.040+0000 7f6a4d000700 1 --2- 172.21.15.116:0/3647185623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a48100d80 0x7f6a481011a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6a380099c0 tx=0x7f6a38009cd0 comp rx=0 tx=0).stop 2024-03-30T00:55:55.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.040+0000 7f6a4d000700 1 -- 172.21.15.116:0/3647185623 shutdown_connections 2024-03-30T00:55:55.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.040+0000 7f6a4d000700 1 --2- 172.21.15.116:0/3647185623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a48102910 0x7f6a481091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:55.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.040+0000 7f6a4d000700 1 --2- 172.21.15.116:0/3647185623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a48100d80 0x7f6a481011a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:55.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.040+0000 7f6a4d000700 1 --2- 172.21.15.116:0/3647185623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a48101f70 0x7f6a481023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:55.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.040+0000 7f6a4d000700 1 -- 172.21.15.116:0/3647185623 >> 172.21.15.116:0/3647185623 conn(0x7f6a480fc2f0 msgr2=0x7f6a480fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:55.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.040+0000 7f6a4d000700 1 -- 172.21.15.116:0/3647185623 shutdown_connections 2024-03-30T00:55:55.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.040+0000 7f6a4d000700 1 -- 172.21.15.116:0/3647185623 wait complete. 2024-03-30T00:55:55.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.041+0000 7f6a4d000700 1 Processor -- start 2024-03-30T00:55:55.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.041+0000 7f6a4d000700 1 -- start start 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.041+0000 7f6a4d000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a48100d80 0x7f6a48198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.041+0000 7f6a4d000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a48101f70 0x7f6a481994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a4d000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a48102910 0x7f6a481a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a4d000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a481a0ab0 con 0x7f6a48101f70 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a4d000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a481a0bf0 con 0x7f6a48100d80 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a4d000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a481a0d80 con 0x7f6a48102910 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a45400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a48101f70 0x7f6a481994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a45e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a48100d80 0x7f6a48198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a45400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a48101f70 0x7f6a481994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45728/0 (socket says 172.21.15.116:45728) 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a45400700 1 -- 172.21.15.116:0/4221352950 learned_addr learned my addr 172.21.15.116:0/4221352950 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a46800700 1 --2- 172.21.15.116:0/4221352950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a48102910 0x7f6a481a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a45400700 1 -- 172.21.15.116:0/4221352950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a48102910 msgr2=0x7f6a481a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a45400700 1 --2- 172.21.15.116:0/4221352950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a48102910 0x7f6a481a04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a45400700 1 -- 172.21.15.116:0/4221352950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a48100d80 msgr2=0x7f6a48198f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:55.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a45400700 1 --2- 172.21.15.116:0/4221352950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a48100d80 0x7f6a48198f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:55.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a45400700 1 -- 172.21.15.116:0/4221352950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a380095f0 con 0x7f6a48101f70 2024-03-30T00:55:55.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a45e00700 1 --2- 172.21.15.116:0/4221352950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a48100d80 0x7f6a48198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:55.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.042+0000 7f6a45400700 1 --2- 172.21.15.116:0/4221352950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a48101f70 0x7f6a481994c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400e9a0 tx=0x7f6a3400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:55.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.043+0000 7f6a3ea00700 1 -- 172.21.15.116:0/4221352950 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a34009790 con 0x7f6a48101f70 2024-03-30T00:55:55.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.043+0000 7f6a4d000700 1 -- 172.21.15.116:0/4221352950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a481a1060 con 0x7f6a48101f70 2024-03-30T00:55:55.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.043+0000 7f6a3ea00700 1 -- 172.21.15.116:0/4221352950 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a34010430 con 0x7f6a48101f70 2024-03-30T00:55:55.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.043+0000 7f6a3ea00700 1 -- 172.21.15.116:0/4221352950 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a34018680 con 0x7f6a48101f70 2024-03-30T00:55:55.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.043+0000 7f6a4d000700 1 -- 172.21.15.116:0/4221352950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a481a15c0 con 0x7f6a48101f70 2024-03-30T00:55:55.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.044+0000 7f6a4d000700 1 -- 172.21.15.116:0/4221352950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a48067af0 con 0x7f6a48101f70 2024-03-30T00:55:55.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.046+0000 7f6a3ea00700 1 -- 172.21.15.116:0/4221352950 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f6a340105a0 con 0x7f6a48101f70 2024-03-30T00:55:55.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.047+0000 7f6a3ea00700 1 --2- 172.21.15.116:0/4221352950 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6a2c0a11a0 0x7f6a2c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:55.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.047+0000 7f6a3ea00700 1 -- 172.21.15.116:0/4221352950 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f6a34014070 con 0x7f6a48101f70 2024-03-30T00:55:55.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.047+0000 7f6a45e00700 1 --2- 172.21.15.116:0/4221352950 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6a2c0a11a0 0x7f6a2c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:55.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.048+0000 7f6a45e00700 1 --2- 172.21.15.116:0/4221352950 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6a2c0a11a0 0x7f6a2c0a3660 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6a38003730 tx=0x7f6a3800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:55.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.050+0000 7f6a3ea00700 1 -- 172.21.15.116:0/4221352950 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6a34096420 con 0x7f6a48101f70 2024-03-30T00:55:55.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.255+0000 7f6a4d000700 1 -- 172.21.15.116:0/4221352950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f6a4810c5e0 con 0x7f6a48101f70 2024-03-30T00:55:55.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:55 smithi130 ceph-mon[47626]: osdmap e78: 12 total, 12 up, 12 in 2024-03-30T00:55:55.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:55 smithi130 ceph-mon[47626]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-30T00:55:55.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:55 smithi130 ceph-mon[47626]: osdmap e79: 12 total, 12 up, 12 in 2024-03-30T00:55:55.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:55 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/4221352950' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-30T00:55:55.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:55 smithi116 ceph-mon[38086]: osdmap e78: 12 total, 12 up, 12 in 2024-03-30T00:55:55.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:55 smithi116 ceph-mon[38086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-30T00:55:55.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:55 smithi116 ceph-mon[38086]: osdmap e79: 12 total, 12 up, 12 in 2024-03-30T00:55:55.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:55 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/4221352950' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-30T00:55:55.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:55 smithi181 ceph-mon[47622]: osdmap e78: 12 total, 12 up, 12 in 2024-03-30T00:55:55.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:55 smithi181 ceph-mon[47622]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-30T00:55:55.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:55 smithi181 ceph-mon[47622]: osdmap e79: 12 total, 12 up, 12 in 2024-03-30T00:55:55.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:55 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/4221352950' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-30T00:55:55.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.813+0000 7f6a3ea00700 1 -- 172.21.15.116:0/4221352950 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v80) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6a34026020 con 0x7f6a48101f70 2024-03-30T00:55:55.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.819+0000 7f6a4d000700 1 -- 172.21.15.116:0/4221352950 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6a2c0a11a0 msgr2=0x7f6a2c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:55.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.819+0000 7f6a4d000700 1 --2- 172.21.15.116:0/4221352950 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6a2c0a11a0 0x7f6a2c0a3660 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6a38003730 tx=0x7f6a3800b540 comp rx=0 tx=0).stop 2024-03-30T00:55:55.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.819+0000 7f6a4d000700 1 -- 172.21.15.116:0/4221352950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a48101f70 msgr2=0x7f6a481994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:55.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.819+0000 7f6a4d000700 1 --2- 172.21.15.116:0/4221352950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a48101f70 0x7f6a481994c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400e9a0 tx=0x7f6a3400ecb0 comp rx=0 tx=0).stop 2024-03-30T00:55:55.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.820+0000 7f6a4d000700 1 -- 172.21.15.116:0/4221352950 shutdown_connections 2024-03-30T00:55:55.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.820+0000 7f6a4d000700 1 --2- 172.21.15.116:0/4221352950 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6a2c0a11a0 0x7f6a2c0a3660 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:55.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.820+0000 7f6a4d000700 1 --2- 172.21.15.116:0/4221352950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a48102910 0x7f6a481a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:55.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.820+0000 7f6a4d000700 1 --2- 172.21.15.116:0/4221352950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a48100d80 0x7f6a48198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:55.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.820+0000 7f6a4d000700 1 --2- 172.21.15.116:0/4221352950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a48101f70 0x7f6a481994c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:55.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.820+0000 7f6a4d000700 1 -- 172.21.15.116:0/4221352950 >> 172.21.15.116:0/4221352950 conn(0x7f6a480fc2f0 msgr2=0x7f6a48072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:55.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.821+0000 7f6a4d000700 1 -- 172.21.15.116:0/4221352950 shutdown_connections 2024-03-30T00:55:55.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:55.821+0000 7f6a4d000700 1 -- 172.21.15.116:0/4221352950 wait complete. 2024-03-30T00:55:55.829 INFO:teuthology.orchestra.run.smithi116.stderr:set pool 4 allow_ec_overwrites to true 2024-03-30T00:55:55.858 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-30T00:55:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.053+0000 7f8f82000700 1 -- 172.21.15.116:0/3253527948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f7c102910 msgr2=0x7f8f7c1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.053+0000 7f8f82000700 1 --2- 172.21.15.116:0/3253527948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f7c102910 0x7f8f7c1091c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400b0d0 tx=0x7f8f7400b3e0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.053+0000 7f8f82000700 1 -- 172.21.15.116:0/3253527948 shutdown_connections 2024-03-30T00:55:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.053+0000 7f8f82000700 1 --2- 172.21.15.116:0/3253527948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f7c102910 0x7f8f7c1091c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.053+0000 7f8f82000700 1 --2- 172.21.15.116:0/3253527948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f7c100d80 0x7f8f7c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.053+0000 7f8f82000700 1 --2- 172.21.15.116:0/3253527948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f7c101f70 0x7f8f7c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.053+0000 7f8f82000700 1 -- 172.21.15.116:0/3253527948 >> 172.21.15.116:0/3253527948 conn(0x7f8f7c0fc2f0 msgr2=0x7f8f7c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.053+0000 7f8f82000700 1 -- 172.21.15.116:0/3253527948 shutdown_connections 2024-03-30T00:55:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.053+0000 7f8f82000700 1 -- 172.21.15.116:0/3253527948 wait complete. 2024-03-30T00:55:56.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.054+0000 7f8f82000700 1 Processor -- start 2024-03-30T00:55:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.054+0000 7f8f82000700 1 -- start start 2024-03-30T00:55:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.054+0000 7f8f82000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f7c100d80 0x7f8f7c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.054+0000 7f8f82000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f7c101f70 0x7f8f7c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.054+0000 7f8f82000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f7c102910 0x7f8f7c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.054+0000 7f8f82000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c073e80 con 0x7f8f7c102910 2024-03-30T00:55:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.054+0000 7f8f82000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c073ff0 con 0x7f8f7c101f70 2024-03-30T00:55:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.054+0000 7f8f82000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c1a0e60 con 0x7f8f7c100d80 2024-03-30T00:55:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.055+0000 7f8f7be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f7c102910 0x7f8f7c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.055+0000 7f8f7aa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f7c101f70 0x7f8f7c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.055+0000 7f8f7be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f7c102910 0x7f8f7c0736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45756/0 (socket says 172.21.15.116:45756) 2024-03-30T00:55:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.055+0000 7f8f7be00700 1 -- 172.21.15.116:0/2330031533 learned_addr learned my addr 172.21.15.116:0/2330031533 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.055+0000 7f8f7b400700 1 --2- 172.21.15.116:0/2330031533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f7c100d80 0x7f8f7c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.055+0000 7f8f7be00700 1 -- 172.21.15.116:0/2330031533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f7c100d80 msgr2=0x7f8f7c074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.055+0000 7f8f7be00700 1 --2- 172.21.15.116:0/2330031533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f7c100d80 0x7f8f7c074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.055+0000 7f8f7be00700 1 -- 172.21.15.116:0/2330031533 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f7c101f70 msgr2=0x7f8f7c0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.055+0000 7f8f7be00700 1 --2- 172.21.15.116:0/2330031533 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f7c101f70 0x7f8f7c0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.055+0000 7f8f7be00700 1 -- 172.21.15.116:0/2330031533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f74009d00 con 0x7f8f7c102910 2024-03-30T00:55:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.055+0000 7f8f7be00700 1 --2- 172.21.15.116:0/2330031533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f7c102910 0x7f8f7c0736e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8f74007b60 tx=0x7f8f74008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.055+0000 7f8f73e00700 1 -- 172.21.15.116:0/2330031533 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7400e040 con 0x7f8f7c102910 2024-03-30T00:55:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.056+0000 7f8f73e00700 1 -- 172.21.15.116:0/2330031533 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f7401d930 con 0x7f8f7c102910 2024-03-30T00:55:56.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.056+0000 7f8f82000700 1 -- 172.21.15.116:0/2330031533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7c1a1090 con 0x7f8f7c102910 2024-03-30T00:55:56.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.056+0000 7f8f73e00700 1 -- 172.21.15.116:0/2330031533 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7401b9f0 con 0x7f8f7c102910 2024-03-30T00:55:56.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.056+0000 7f8f82000700 1 -- 172.21.15.116:0/2330031533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7c1a14b0 con 0x7f8f7c102910 2024-03-30T00:55:56.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.057+0000 7f8f82000700 1 -- 172.21.15.116:0/2330031533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7c108130 con 0x7f8f7c102910 2024-03-30T00:55:56.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.058+0000 7f8f73e00700 1 -- 172.21.15.116:0/2330031533 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f8f7401daa0 con 0x7f8f7c102910 2024-03-30T00:55:56.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.060+0000 7f8f73e00700 1 --2- 172.21.15.116:0/2330031533 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8f600a11a0 0x7f8f600a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:56.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.060+0000 7f8f73e00700 1 -- 172.21.15.116:0/2330031533 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f8f740ce5c0 con 0x7f8f7c102910 2024-03-30T00:55:56.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.060+0000 7f8f7b400700 1 --2- 172.21.15.116:0/2330031533 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8f600a11a0 0x7f8f600a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:56.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.060+0000 7f8f7b400700 1 --2- 172.21.15.116:0/2330031533 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8f600a11a0 0x7f8f600a3660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c101dd0 tx=0x7f8f6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:56.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.062+0000 7f8f73e00700 1 -- 172.21.15.116:0/2330031533 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8f74097bb0 con 0x7f8f7c102910 2024-03-30T00:55:56.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.323+0000 7f8f82000700 1 -- 172.21.15.116:0/2330031533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f8f7c10c5f0 con 0x7f8f7c102910 2024-03-30T00:55:56.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.331+0000 7f8f73e00700 1 -- 172.21.15.116:0/2330031533 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) v1 ==== 137+0+0 (secure 0 0 0) 0x7f8f7409b1d0 con 0x7f8f7c102910 2024-03-30T00:55:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.334+0000 7f8f82000700 1 -- 172.21.15.116:0/2330031533 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8f600a11a0 msgr2=0x7f8f600a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.334+0000 7f8f82000700 1 --2- 172.21.15.116:0/2330031533 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8f600a11a0 0x7f8f600a3660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c101dd0 tx=0x7f8f6c009260 comp rx=0 tx=0).stop 2024-03-30T00:55:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.334+0000 7f8f82000700 1 -- 172.21.15.116:0/2330031533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f7c102910 msgr2=0x7f8f7c0736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.334+0000 7f8f82000700 1 --2- 172.21.15.116:0/2330031533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f7c102910 0x7f8f7c0736e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8f74007b60 tx=0x7f8f74008e40 comp rx=0 tx=0).stop 2024-03-30T00:55:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.334+0000 7f8f82000700 1 -- 172.21.15.116:0/2330031533 shutdown_connections 2024-03-30T00:55:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.334+0000 7f8f82000700 1 --2- 172.21.15.116:0/2330031533 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8f600a11a0 0x7f8f600a3660 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.334+0000 7f8f82000700 1 --2- 172.21.15.116:0/2330031533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f7c100d80 0x7f8f7c074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.334+0000 7f8f82000700 1 --2- 172.21.15.116:0/2330031533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f7c102910 0x7f8f7c0736e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.334+0000 7f8f82000700 1 --2- 172.21.15.116:0/2330031533 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f7c101f70 0x7f8f7c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.334+0000 7f8f82000700 1 -- 172.21.15.116:0/2330031533 >> 172.21.15.116:0/2330031533 conn(0x7f8f7c0fc2f0 msgr2=0x7f8f7c105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:56.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.334+0000 7f8f82000700 1 -- 172.21.15.116:0/2330031533 shutdown_connections 2024-03-30T00:55:56.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.334+0000 7f8f82000700 1 -- 172.21.15.116:0/2330031533 wait complete. 2024-03-30T00:55:56.341 INFO:teuthology.orchestra.run.smithi116.stderr:added data pool 4 to fsmap 2024-03-30T00:55:56.365 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:55:56.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.552+0000 7fe7cd400700 1 -- 172.21.15.116:0/2362468759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c8100c80 msgr2=0x7fe7c81010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:56.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.552+0000 7fe7cd400700 1 --2- 172.21.15.116:0/2362468759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c8100c80 0x7fe7c81010a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80099c0 tx=0x7fe7b8009cd0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.552+0000 7fe7cd400700 1 -- 172.21.15.116:0/2362468759 shutdown_connections 2024-03-30T00:55:56.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.552+0000 7fe7cd400700 1 --2- 172.21.15.116:0/2362468759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7c8102810 0x7fe7c81090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.552+0000 7fe7cd400700 1 --2- 172.21.15.116:0/2362468759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c8100c80 0x7fe7c81010a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.552+0000 7fe7cd400700 1 --2- 172.21.15.116:0/2362468759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7c8101e70 0x7fe7c81022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.552+0000 7fe7cd400700 1 -- 172.21.15.116:0/2362468759 >> 172.21.15.116:0/2362468759 conn(0x7fe7c80fc1f0 msgr2=0x7fe7c80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:56.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.553+0000 7fe7cd400700 1 -- 172.21.15.116:0/2362468759 shutdown_connections 2024-03-30T00:55:56.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.553+0000 7fe7cd400700 1 -- 172.21.15.116:0/2362468759 wait complete. 2024-03-30T00:55:56.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.553+0000 7fe7cd400700 1 Processor -- start 2024-03-30T00:55:56.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.554+0000 7fe7cd400700 1 -- start start 2024-03-30T00:55:56.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.554+0000 7fe7cd400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7c8100c80 0x7fe7c8198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:56.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.554+0000 7fe7cd400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c8101e70 0x7fe7c8199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:56.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.554+0000 7fe7cd400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7c8102810 0x7fe7c81a0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:56.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.554+0000 7fe7cd400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c81a0950 con 0x7fe7c8101e70 2024-03-30T00:55:56.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.554+0000 7fe7cd400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c81a0a90 con 0x7fe7c8100c80 2024-03-30T00:55:56.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.554+0000 7fe7cd400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c81a0c20 con 0x7fe7c8102810 2024-03-30T00:55:56.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.554+0000 7fe7c5e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c8101e70 0x7fe7c8199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:56.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.554+0000 7fe7c5e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c8101e70 0x7fe7c8199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45770/0 (socket says 172.21.15.116:45770) 2024-03-30T00:55:56.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.554+0000 7fe7c5e00700 1 -- 172.21.15.116:0/3190764437 learned_addr learned my addr 172.21.15.116:0/3190764437 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:56.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.554+0000 7fe7c6800700 1 --2- 172.21.15.116:0/3190764437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7c8100c80 0x7fe7c8198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:56.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.554+0000 7fe7c7200700 1 --2- 172.21.15.116:0/3190764437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7c8102810 0x7fe7c81a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:56.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.555+0000 7fe7c5e00700 1 -- 172.21.15.116:0/3190764437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7c8102810 msgr2=0x7fe7c81a0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:56.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.555+0000 7fe7c5e00700 1 --2- 172.21.15.116:0/3190764437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7c8102810 0x7fe7c81a0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.555+0000 7fe7c5e00700 1 -- 172.21.15.116:0/3190764437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7c8100c80 msgr2=0x7fe7c8198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:56.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.555+0000 7fe7c5e00700 1 --2- 172.21.15.116:0/3190764437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7c8100c80 0x7fe7c8198e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.555+0000 7fe7c5e00700 1 -- 172.21.15.116:0/3190764437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b80095f0 con 0x7fe7c8101e70 2024-03-30T00:55:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.555+0000 7fe7c5e00700 1 --2- 172.21.15.116:0/3190764437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c8101e70 0x7fe7c8199360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400e9a0 tx=0x7fe7b400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.555+0000 7fe7bf400700 1 -- 172.21.15.116:0/3190764437 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4009790 con 0x7fe7c8101e70 2024-03-30T00:55:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.555+0000 7fe7bf400700 1 -- 172.21.15.116:0/3190764437 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7b4010430 con 0x7fe7c8101e70 2024-03-30T00:55:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.555+0000 7fe7bf400700 1 -- 172.21.15.116:0/3190764437 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4018680 con 0x7fe7c8101e70 2024-03-30T00:55:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.555+0000 7fe7cd400700 1 -- 172.21.15.116:0/3190764437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c81a0f00 con 0x7fe7c8101e70 2024-03-30T00:55:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.555+0000 7fe7cd400700 1 -- 172.21.15.116:0/3190764437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c81a1380 con 0x7fe7c8101e70 2024-03-30T00:55:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.556+0000 7fe7cd400700 1 -- 172.21.15.116:0/3190764437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c8067a10 con 0x7fe7c8101e70 2024-03-30T00:55:56.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.558+0000 7fe7bf400700 1 -- 172.21.15.116:0/3190764437 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fe7b40105a0 con 0x7fe7c8101e70 2024-03-30T00:55:56.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.559+0000 7fe7bf400700 1 --2- 172.21.15.116:0/3190764437 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe7ac0a11a0 0x7fe7ac0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:56.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.559+0000 7fe7bf400700 1 -- 172.21.15.116:0/3190764437 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe7b4014070 con 0x7fe7c8101e70 2024-03-30T00:55:56.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.560+0000 7fe7c6800700 1 --2- 172.21.15.116:0/3190764437 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe7ac0a11a0 0x7fe7ac0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:56.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.561+0000 7fe7c6800700 1 --2- 172.21.15.116:0/3190764437 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe7ac0a11a0 0x7fe7ac0a3660 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8003730 tx=0x7fe7b8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:56.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.564+0000 7fe7bf400700 1 -- 172.21.15.116:0/3190764437 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe7b40529f0 con 0x7fe7c8101e70 2024-03-30T00:55:56.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:56 smithi130 ceph-mon[47626]: pgmap v229: 129 pgs: 13 creating+peering, 48 active+clean, 68 unknown; 578 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 3 op/s 2024-03-30T00:55:56.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:56 smithi130 ceph-mon[47626]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T00:55:56.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:56 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/4221352950' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-30T00:55:56.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:56 smithi130 ceph-mon[47626]: osdmap e80: 12 total, 12 up, 12 in 2024-03-30T00:55:56.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:56 smithi116 ceph-mon[38086]: pgmap v229: 129 pgs: 13 creating+peering, 48 active+clean, 68 unknown; 578 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 3 op/s 2024-03-30T00:55:56.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:56 smithi116 ceph-mon[38086]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T00:55:56.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:56 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/4221352950' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-30T00:55:56.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:56 smithi116 ceph-mon[38086]: osdmap e80: 12 total, 12 up, 12 in 2024-03-30T00:55:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:56 smithi181 ceph-mon[47622]: pgmap v229: 129 pgs: 13 creating+peering, 48 active+clean, 68 unknown; 578 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 3 op/s 2024-03-30T00:55:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:56 smithi181 ceph-mon[47622]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T00:55:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:56 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/4221352950' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-30T00:55:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:56 smithi181 ceph-mon[47622]: osdmap e80: 12 total, 12 up, 12 in 2024-03-30T00:55:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.815+0000 7fe7cd400700 1 -- 172.21.15.116:0/3190764437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7c810c4e0 con 0x7fe7c8101e70 2024-03-30T00:55:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.816+0000 7fe7bf400700 1 -- 172.21.15.116:0/3190764437 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+10825 (secure 0 0 0) 0x7fe7b40961a0 con 0x7fe7c8101e70 2024-03-30T00:55:56.822 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:56.822 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24524,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:55:56.324997+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T00:55:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.818+0000 7fe7cd400700 1 -- 172.21.15.116:0/3190764437 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe7ac0a11a0 msgr2=0x7fe7ac0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.818+0000 7fe7cd400700 1 --2- 172.21.15.116:0/3190764437 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe7ac0a11a0 0x7fe7ac0a3660 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8003730 tx=0x7fe7b8000bc0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.818+0000 7fe7cd400700 1 -- 172.21.15.116:0/3190764437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c8101e70 msgr2=0x7fe7c8199360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.818+0000 7fe7cd400700 1 --2- 172.21.15.116:0/3190764437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c8101e70 0x7fe7c8199360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400e9a0 tx=0x7fe7b400ecb0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.818+0000 7fe7cd400700 1 -- 172.21.15.116:0/3190764437 shutdown_connections 2024-03-30T00:55:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.818+0000 7fe7cd400700 1 --2- 172.21.15.116:0/3190764437 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe7ac0a11a0 0x7fe7ac0a3660 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.818+0000 7fe7cd400700 1 --2- 172.21.15.116:0/3190764437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7c8102810 0x7fe7c81a0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.818+0000 7fe7cd400700 1 --2- 172.21.15.116:0/3190764437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7c8100c80 0x7fe7c8198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.818+0000 7fe7cd400700 1 --2- 172.21.15.116:0/3190764437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c8101e70 0x7fe7c8199360 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.818+0000 7fe7cd400700 1 -- 172.21.15.116:0/3190764437 >> 172.21.15.116:0/3190764437 conn(0x7fe7c80fc1f0 msgr2=0x7fe7c8072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.819+0000 7fe7cd400700 1 -- 172.21.15.116:0/3190764437 shutdown_connections 2024-03-30T00:55:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:56.819+0000 7fe7cd400700 1 -- 172.21.15.116:0/3190764437 wait complete. 2024-03-30T00:55:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 16 2024-03-30T00:55:56.844 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:55:57.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.078+0000 7f85b1400700 1 -- 172.21.15.116:0/2076022921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac101e70 msgr2=0x7f85ac1022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:57.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.078+0000 7f85b1400700 1 --2- 172.21.15.116:0/2076022921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac101e70 0x7f85ac1022d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f85980099c0 tx=0x7f8598009cd0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.078+0000 7f85b1400700 1 -- 172.21.15.116:0/2076022921 shutdown_connections 2024-03-30T00:55:57.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.078+0000 7f85b1400700 1 --2- 172.21.15.116:0/2076022921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85ac102810 0x7f85ac1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.078+0000 7f85b1400700 1 --2- 172.21.15.116:0/2076022921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85ac100c80 0x7f85ac1010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.078+0000 7f85b1400700 1 --2- 172.21.15.116:0/2076022921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac101e70 0x7f85ac1022d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.078+0000 7f85b1400700 1 -- 172.21.15.116:0/2076022921 >> 172.21.15.116:0/2076022921 conn(0x7f85ac0fc1f0 msgr2=0x7f85ac0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:57.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.078+0000 7f85b1400700 1 -- 172.21.15.116:0/2076022921 shutdown_connections 2024-03-30T00:55:57.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.078+0000 7f85b1400700 1 -- 172.21.15.116:0/2076022921 wait complete. 2024-03-30T00:55:57.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.079+0000 7f85b1400700 1 Processor -- start 2024-03-30T00:55:57.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.079+0000 7f85b1400700 1 -- start start 2024-03-30T00:55:57.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85b1400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85ac100c80 0x7f85ac196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:57.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85b1400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac101e70 0x7f85ac197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:57.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85b1400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85ac102810 0x7f85ac19e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:57.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85b1400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac19e6f0 con 0x7f85ac101e70 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85b1400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac19e830 con 0x7f85ac102810 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85b1400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac19e9c0 con 0x7f85ac100c80 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85a9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac101e70 0x7f85ac197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85a9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac101e70 0x7f85ac197100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45790/0 (socket says 172.21.15.116:45790) 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85a9e00700 1 -- 172.21.15.116:0/3377663397 learned_addr learned my addr 172.21.15.116:0/3377663397 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85ab200700 1 --2- 172.21.15.116:0/3377663397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85ac102810 0x7f85ac19e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85aa800700 1 --2- 172.21.15.116:0/3377663397 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85ac100c80 0x7f85ac196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85a9e00700 1 -- 172.21.15.116:0/3377663397 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85ac100c80 msgr2=0x7f85ac196bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85a9e00700 1 --2- 172.21.15.116:0/3377663397 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85ac100c80 0x7f85ac196bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85a9e00700 1 -- 172.21.15.116:0/3377663397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85ac102810 msgr2=0x7f85ac19e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85a9e00700 1 --2- 172.21.15.116:0/3377663397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85ac102810 0x7f85ac19e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.080+0000 7f85a9e00700 1 -- 172.21.15.116:0/3377663397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85980095f0 con 0x7f85ac101e70 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.081+0000 7f85a9e00700 1 --2- 172.21.15.116:0/3377663397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac101e70 0x7f85ac197100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8598006010 tx=0x7f859800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.081+0000 7f85a3400700 1 -- 172.21.15.116:0/3377663397 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859801d070 con 0x7f85ac101e70 2024-03-30T00:55:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.081+0000 7f85a3400700 1 -- 172.21.15.116:0/3377663397 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8598005030 con 0x7f85ac101e70 2024-03-30T00:55:57.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.081+0000 7f85b1400700 1 -- 172.21.15.116:0/3377663397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ac19ec40 con 0x7f85ac101e70 2024-03-30T00:55:57.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.081+0000 7f85a3400700 1 -- 172.21.15.116:0/3377663397 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8598004b10 con 0x7f85ac101e70 2024-03-30T00:55:57.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.081+0000 7f85b1400700 1 -- 172.21.15.116:0/3377663397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ac19f0c0 con 0x7f85ac101e70 2024-03-30T00:55:57.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.082+0000 7f85b1400700 1 -- 172.21.15.116:0/3377663397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ac108030 con 0x7f85ac101e70 2024-03-30T00:55:57.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.084+0000 7f85a3400700 1 -- 172.21.15.116:0/3377663397 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f85980051a0 con 0x7f85ac101e70 2024-03-30T00:55:57.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.086+0000 7f85a3400700 1 --2- 172.21.15.116:0/3377663397 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f85900a11a0 0x7f85900a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:57.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.086+0000 7f85a3400700 1 -- 172.21.15.116:0/3377663397 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f85980cd6e0 con 0x7f85ac101e70 2024-03-30T00:55:57.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.086+0000 7f85aa800700 1 --2- 172.21.15.116:0/3377663397 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f85900a11a0 0x7f85900a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:57.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.087+0000 7f85aa800700 1 --2- 172.21.15.116:0/3377663397 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f85900a11a0 0x7f85900a3660 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f85ac101cd0 tx=0x7f859c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:57.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.090+0000 7f85a3400700 1 -- 172.21.15.116:0/3377663397 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8598096d10 con 0x7f85ac101e70 2024-03-30T00:55:57.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.333+0000 7f85b1400700 1 -- 172.21.15.116:0/3377663397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85ac067a10 con 0x7f85ac101e70 2024-03-30T00:55:57.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.334+0000 7f85a3400700 1 -- 172.21.15.116:0/3377663397 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+24028 (secure 0 0 0) 0x7f859809a330 con 0x7f85ac101e70 2024-03-30T00:55:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:57.341 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:55:56.324776+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:55:57.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.337+0000 7f85b1400700 1 -- 172.21.15.116:0/3377663397 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f85900a11a0 msgr2=0x7f85900a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:57.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.337+0000 7f85b1400700 1 --2- 172.21.15.116:0/3377663397 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f85900a11a0 0x7f85900a3660 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f85ac101cd0 tx=0x7f859c009260 comp rx=0 tx=0).stop 2024-03-30T00:55:57.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.337+0000 7f85b1400700 1 -- 172.21.15.116:0/3377663397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac101e70 msgr2=0x7f85ac197100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:57.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.337+0000 7f85b1400700 1 --2- 172.21.15.116:0/3377663397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac101e70 0x7f85ac197100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8598006010 tx=0x7f859800f690 comp rx=0 tx=0).stop 2024-03-30T00:55:57.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.337+0000 7f85b1400700 1 -- 172.21.15.116:0/3377663397 shutdown_connections 2024-03-30T00:55:57.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.337+0000 7f85b1400700 1 --2- 172.21.15.116:0/3377663397 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f85900a11a0 0x7f85900a3660 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.337+0000 7f85b1400700 1 --2- 172.21.15.116:0/3377663397 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85ac100c80 0x7f85ac196bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.337+0000 7f85b1400700 1 --2- 172.21.15.116:0/3377663397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85ac102810 0x7f85ac19e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.337+0000 7f85b1400700 1 --2- 172.21.15.116:0/3377663397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac101e70 0x7f85ac197100 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.337+0000 7f85b1400700 1 -- 172.21.15.116:0/3377663397 >> 172.21.15.116:0/3377663397 conn(0x7f85ac0fc1f0 msgr2=0x7f85ac105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:57.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.337+0000 7f85b1400700 1 -- 172.21.15.116:0/3377663397 shutdown_connections 2024-03-30T00:55:57.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.337+0000 7f85b1400700 1 -- 172.21.15.116:0/3377663397 wait complete. 2024-03-30T00:55:57.362 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:55:57.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.547+0000 7fb1f6c00700 1 -- 172.21.15.116:0/2528753733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0102810 msgr2=0x7fb1f01090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:57.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.547+0000 7fb1f6c00700 1 --2- 172.21.15.116:0/2528753733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0102810 0x7fb1f01090c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800b0d0 tx=0x7fb1e800b3e0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.547+0000 7fb1f6c00700 1 -- 172.21.15.116:0/2528753733 shutdown_connections 2024-03-30T00:55:57.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.547+0000 7fb1f6c00700 1 --2- 172.21.15.116:0/2528753733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f0100c80 0x7fb1f01010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.547+0000 7fb1f6c00700 1 --2- 172.21.15.116:0/2528753733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0102810 0x7fb1f01090c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.547+0000 7fb1f6c00700 1 --2- 172.21.15.116:0/2528753733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1f0101e70 0x7fb1f01022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.547+0000 7fb1f6c00700 1 -- 172.21.15.116:0/2528753733 >> 172.21.15.116:0/2528753733 conn(0x7fb1f00fc1f0 msgr2=0x7fb1f00fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:57.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.548+0000 7fb1f6c00700 1 -- 172.21.15.116:0/2528753733 shutdown_connections 2024-03-30T00:55:57.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.548+0000 7fb1f6c00700 1 -- 172.21.15.116:0/2528753733 wait complete. 2024-03-30T00:55:57.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.549+0000 7fb1f6c00700 1 Processor -- start 2024-03-30T00:55:57.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.549+0000 7fb1f6c00700 1 -- start start 2024-03-30T00:55:57.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.549+0000 7fb1f6c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0100c80 0x7fb1f0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:57.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.549+0000 7fb1f6c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f0101e70 0x7fb1f00731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:57.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.549+0000 7fb1f6c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1f0102810 0x7fb1f00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:57.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.549+0000 7fb1f6c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f0073e80 con 0x7fb1f0100c80 2024-03-30T00:55:57.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.549+0000 7fb1f6c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f0073ff0 con 0x7fb1f0102810 2024-03-30T00:55:57.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.549+0000 7fb1f6c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f01a0d50 con 0x7fb1f0101e70 2024-03-30T00:55:57.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.549+0000 7fb1efe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0100c80 0x7fb1f0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:57.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.549+0000 7fb1efe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0100c80 0x7fb1f0074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45804/0 (socket says 172.21.15.116:45804) 2024-03-30T00:55:57.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.549+0000 7fb1efe00700 1 -- 172.21.15.116:0/3080804991 learned_addr learned my addr 172.21.15.116:0/3080804991 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:57.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.549+0000 7fb1f4c00700 1 --2- 172.21.15.116:0/3080804991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1f0102810 0x7fb1f00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:57.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.549+0000 7fb1ef400700 1 --2- 172.21.15.116:0/3080804991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f0101e70 0x7fb1f00731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:57.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.550+0000 7fb1efe00700 1 -- 172.21.15.116:0/3080804991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f0101e70 msgr2=0x7fb1f00731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:57.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.550+0000 7fb1efe00700 1 --2- 172.21.15.116:0/3080804991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f0101e70 0x7fb1f00731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.550+0000 7fb1efe00700 1 -- 172.21.15.116:0/3080804991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1f0102810 msgr2=0x7fb1f00736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:57.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.550+0000 7fb1efe00700 1 --2- 172.21.15.116:0/3080804991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1f0102810 0x7fb1f00736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.550+0000 7fb1efe00700 1 -- 172.21.15.116:0/3080804991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e8009d00 con 0x7fb1f0100c80 2024-03-30T00:55:57.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.550+0000 7fb1efe00700 1 --2- 172.21.15.116:0/3080804991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0100c80 0x7fb1f0074b10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb1e000d980 tx=0x7fb1e000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:57.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.550+0000 7fb1ecc00700 1 -- 172.21.15.116:0/3080804991 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0004020 con 0x7fb1f0100c80 2024-03-30T00:55:57.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.550+0000 7fb1ecc00700 1 -- 172.21.15.116:0/3080804991 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb1e000f690 con 0x7fb1f0100c80 2024-03-30T00:55:57.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.550+0000 7fb1f6c00700 1 -- 172.21.15.116:0/3080804991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f01a0fe0 con 0x7fb1f0100c80 2024-03-30T00:55:57.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.550+0000 7fb1ecc00700 1 -- 172.21.15.116:0/3080804991 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0003a60 con 0x7fb1f0100c80 2024-03-30T00:55:57.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.550+0000 7fb1f6c00700 1 -- 172.21.15.116:0/3080804991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f01a1410 con 0x7fb1f0100c80 2024-03-30T00:55:57.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.551+0000 7fb1f6c00700 1 -- 172.21.15.116:0/3080804991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f0108030 con 0x7fb1f0100c80 2024-03-30T00:55:57.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.553+0000 7fb1ecc00700 1 -- 172.21.15.116:0/3080804991 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fb1e000fb00 con 0x7fb1f0100c80 2024-03-30T00:55:57.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.555+0000 7fb1ecc00700 1 --2- 172.21.15.116:0/3080804991 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb1d40a11a0 0x7fb1d40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:57.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.555+0000 7fb1ecc00700 1 -- 172.21.15.116:0/3080804991 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb1e00cc5d0 con 0x7fb1f0100c80 2024-03-30T00:55:57.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.555+0000 7fb1ef400700 1 --2- 172.21.15.116:0/3080804991 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb1d40a11a0 0x7fb1d40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:57.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.556+0000 7fb1ef400700 1 --2- 172.21.15.116:0/3080804991 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb1d40a11a0 0x7fb1d40a3660 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc005950 tx=0x7fb1dc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:57.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.560+0000 7fb1ecc00700 1 -- 172.21.15.116:0/3080804991 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb1e0095b10 con 0x7fb1f0100c80 2024-03-30T00:55:57.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:57 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2330031533' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-30T00:55:57.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:57 smithi130 ceph-mon[47626]: osdmap e81: 12 total, 12 up, 12 in 2024-03-30T00:55:57.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:57 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2330031533' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-30T00:55:57.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:57 smithi130 ceph-mon[47626]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T00:55:57.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:57 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3190764437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:57.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:57 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2330031533' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-30T00:55:57.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:57 smithi116 ceph-mon[38086]: osdmap e81: 12 total, 12 up, 12 in 2024-03-30T00:55:57.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:57 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2330031533' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-30T00:55:57.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:57 smithi116 ceph-mon[38086]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T00:55:57.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:57 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3190764437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:57.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:57 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2330031533' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-30T00:55:57.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:57 smithi181 ceph-mon[47622]: osdmap e81: 12 total, 12 up, 12 in 2024-03-30T00:55:57.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:57 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2330031533' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-30T00:55:57.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:57 smithi181 ceph-mon[47622]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T00:55:57.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:57 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3190764437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:57.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.795+0000 7fb1f6c00700 1 -- 172.21.15.116:0/3080804991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1f0067a10 con 0x7fb1f0100c80 2024-03-30T00:55:57.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.797+0000 7fb1ecc00700 1 -- 172.21.15.116:0/3080804991 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+24028 (secure 0 0 0) 0x7fb1e0099130 con 0x7fb1f0100c80 2024-03-30T00:55:57.803 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:57.803 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:55:56.324776+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:55:57.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.799+0000 7fb1f6c00700 1 -- 172.21.15.116:0/3080804991 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb1d40a11a0 msgr2=0x7fb1d40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:57.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.799+0000 7fb1f6c00700 1 --2- 172.21.15.116:0/3080804991 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb1d40a11a0 0x7fb1d40a3660 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc005950 tx=0x7fb1dc00b410 comp rx=0 tx=0).stop 2024-03-30T00:55:57.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.799+0000 7fb1f6c00700 1 -- 172.21.15.116:0/3080804991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0100c80 msgr2=0x7fb1f0074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:57.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.799+0000 7fb1f6c00700 1 --2- 172.21.15.116:0/3080804991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0100c80 0x7fb1f0074b10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb1e000d980 tx=0x7fb1e000dc90 comp rx=0 tx=0).stop 2024-03-30T00:55:57.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.799+0000 7fb1f6c00700 1 -- 172.21.15.116:0/3080804991 shutdown_connections 2024-03-30T00:55:57.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.799+0000 7fb1f6c00700 1 --2- 172.21.15.116:0/3080804991 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb1d40a11a0 0x7fb1d40a3660 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.799+0000 7fb1f6c00700 1 --2- 172.21.15.116:0/3080804991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1f0102810 0x7fb1f00736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.799+0000 7fb1f6c00700 1 --2- 172.21.15.116:0/3080804991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0100c80 0x7fb1f0074b10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.799+0000 7fb1f6c00700 1 --2- 172.21.15.116:0/3080804991 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1f0101e70 0x7fb1f00731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:57.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.799+0000 7fb1f6c00700 1 -- 172.21.15.116:0/3080804991 >> 172.21.15.116:0/3080804991 conn(0x7fb1f00fc1f0 msgr2=0x7fb1f0105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:57.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.800+0000 7fb1f6c00700 1 -- 172.21.15.116:0/3080804991 shutdown_connections 2024-03-30T00:55:57.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:57.800+0000 7fb1f6c00700 1 -- 172.21.15.116:0/3080804991 wait complete. 2024-03-30T00:55:57.839 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:55:58.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.027+0000 7ff5ea800700 1 -- 172.21.15.116:0/2925518587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e4101e70 msgr2=0x7ff5e41022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:58.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.027+0000 7ff5ea800700 1 --2- 172.21.15.116:0/2925518587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e4101e70 0x7ff5e41022d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00099c0 tx=0x7ff5d0009cd0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.027+0000 7ff5ea800700 1 -- 172.21.15.116:0/2925518587 shutdown_connections 2024-03-30T00:55:58.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.027+0000 7ff5ea800700 1 --2- 172.21.15.116:0/2925518587 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5e4100c80 0x7ff5e41010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.027+0000 7ff5ea800700 1 --2- 172.21.15.116:0/2925518587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5e4102810 0x7ff5e41090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.027+0000 7ff5ea800700 1 --2- 172.21.15.116:0/2925518587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e4101e70 0x7ff5e41022d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.027+0000 7ff5ea800700 1 -- 172.21.15.116:0/2925518587 >> 172.21.15.116:0/2925518587 conn(0x7ff5e40fc1f0 msgr2=0x7ff5e40fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.027+0000 7ff5ea800700 1 -- 172.21.15.116:0/2925518587 shutdown_connections 2024-03-30T00:55:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.027+0000 7ff5ea800700 1 -- 172.21.15.116:0/2925518587 wait complete. 2024-03-30T00:55:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.028+0000 7ff5ea800700 1 Processor -- start 2024-03-30T00:55:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.028+0000 7ff5ea800700 1 -- start start 2024-03-30T00:55:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5ea800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e4100c80 0x7ff5e4198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5ea800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5e4101e70 0x7ff5e4199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5ea800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5e4102810 0x7ff5e41a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5ea800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e41a0900 con 0x7ff5e4100c80 2024-03-30T00:55:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5ea800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e41a0a70 con 0x7ff5e4102810 2024-03-30T00:55:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5ea800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e41a0c30 con 0x7ff5e4101e70 2024-03-30T00:55:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5e3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e4100c80 0x7ff5e4198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5e3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e4100c80 0x7ff5e4198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45818/0 (socket says 172.21.15.116:45818) 2024-03-30T00:55:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5e3400700 1 -- 172.21.15.116:0/3211072300 learned_addr learned my addr 172.21.15.116:0/3211072300 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5e2a00700 1 --2- 172.21.15.116:0/3211072300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5e4101e70 0x7ff5e4199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5e3e00700 1 --2- 172.21.15.116:0/3211072300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5e4102810 0x7ff5e41a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5e3400700 1 -- 172.21.15.116:0/3211072300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5e4101e70 msgr2=0x7ff5e4199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5e3400700 1 --2- 172.21.15.116:0/3211072300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5e4101e70 0x7ff5e4199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5e3400700 1 -- 172.21.15.116:0/3211072300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5e4102810 msgr2=0x7ff5e41a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5e3400700 1 --2- 172.21.15.116:0/3211072300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5e4102810 0x7ff5e41a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.029+0000 7ff5e3400700 1 -- 172.21.15.116:0/3211072300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d00095f0 con 0x7ff5e4100c80 2024-03-30T00:55:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.030+0000 7ff5e3400700 1 --2- 172.21.15.116:0/3211072300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e4100c80 0x7ff5e4198e10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400e970 tx=0x7ff5d400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.030+0000 7ff5dbe00700 1 -- 172.21.15.116:0/3211072300 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4009790 con 0x7ff5e4100c80 2024-03-30T00:55:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.030+0000 7ff5dbe00700 1 -- 172.21.15.116:0/3211072300 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff5d400eec0 con 0x7ff5e4100c80 2024-03-30T00:55:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.030+0000 7ff5ea800700 1 -- 172.21.15.116:0/3211072300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e41a0f10 con 0x7ff5e4100c80 2024-03-30T00:55:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.030+0000 7ff5dbe00700 1 -- 172.21.15.116:0/3211072300 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4003cf0 con 0x7ff5e4100c80 2024-03-30T00:55:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.030+0000 7ff5ea800700 1 -- 172.21.15.116:0/3211072300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e41a1330 con 0x7ff5e4100c80 2024-03-30T00:55:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.031+0000 7ff5ea800700 1 -- 172.21.15.116:0/3211072300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e4067a10 con 0x7ff5e4100c80 2024-03-30T00:55:58.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.033+0000 7ff5dbe00700 1 -- 172.21.15.116:0/3211072300 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7ff5d400ccd0 con 0x7ff5e4100c80 2024-03-30T00:55:58.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.035+0000 7ff5dbe00700 1 --2- 172.21.15.116:0/3211072300 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff5c80a1120 0x7ff5c80a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.035+0000 7ff5dbe00700 1 -- 172.21.15.116:0/3211072300 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff5d4014070 con 0x7ff5e4100c80 2024-03-30T00:55:58.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.035+0000 7ff5e2a00700 1 --2- 172.21.15.116:0/3211072300 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff5c80a1120 0x7ff5c80a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.036+0000 7ff5e2a00700 1 --2- 172.21.15.116:0/3211072300 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff5c80a1120 0x7ff5c80a35e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0003730 tx=0x7ff5d000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:58.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.039+0000 7ff5dbe00700 1 -- 172.21.15.116:0/3211072300 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff5d40962b0 con 0x7ff5e4100c80 2024-03-30T00:55:58.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.288+0000 7ff5ea800700 1 -- 172.21.15.116:0/3211072300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5e41a1580 con 0x7ff5e4100c80 2024-03-30T00:55:58.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.289+0000 7ff5dbe00700 1 -- 172.21.15.116:0/3211072300 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+10825 (secure 0 0 0) 0x7ff5d40998d0 con 0x7ff5e4100c80 2024-03-30T00:55:58.294 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:58.295 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24524,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:55:56.324997+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T00:55:58.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.291+0000 7ff5ea800700 1 -- 172.21.15.116:0/3211072300 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff5c80a1120 msgr2=0x7ff5c80a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:58.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.291+0000 7ff5ea800700 1 --2- 172.21.15.116:0/3211072300 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff5c80a1120 0x7ff5c80a35e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0003730 tx=0x7ff5d000b540 comp rx=0 tx=0).stop 2024-03-30T00:55:58.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.291+0000 7ff5ea800700 1 -- 172.21.15.116:0/3211072300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e4100c80 msgr2=0x7ff5e4198e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:58.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.291+0000 7ff5ea800700 1 --2- 172.21.15.116:0/3211072300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e4100c80 0x7ff5e4198e10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400e970 tx=0x7ff5d400ec80 comp rx=0 tx=0).stop 2024-03-30T00:55:58.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.291+0000 7ff5ea800700 1 -- 172.21.15.116:0/3211072300 shutdown_connections 2024-03-30T00:55:58.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.291+0000 7ff5ea800700 1 --2- 172.21.15.116:0/3211072300 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff5c80a1120 0x7ff5c80a35e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.291+0000 7ff5ea800700 1 --2- 172.21.15.116:0/3211072300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5e4102810 0x7ff5e41a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.291+0000 7ff5ea800700 1 --2- 172.21.15.116:0/3211072300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e4100c80 0x7ff5e4198e10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.291+0000 7ff5ea800700 1 --2- 172.21.15.116:0/3211072300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5e4101e70 0x7ff5e4199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.291+0000 7ff5ea800700 1 -- 172.21.15.116:0/3211072300 >> 172.21.15.116:0/3211072300 conn(0x7ff5e40fc1f0 msgr2=0x7ff5e4072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:58.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.291+0000 7ff5ea800700 1 -- 172.21.15.116:0/3211072300 shutdown_connections 2024-03-30T00:55:58.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.291+0000 7ff5ea800700 1 -- 172.21.15.116:0/3211072300 wait complete. 2024-03-30T00:55:58.298 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 16 2024-03-30T00:55:58.315 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:55:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.502+0000 7f0b97000700 1 -- 172.21.15.116:0/2119556393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b90100c80 msgr2=0x7f0b901010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.502+0000 7f0b97000700 1 --2- 172.21.15.116:0/2119556393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b90100c80 0x7f0b901010a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0b800099c0 tx=0x7f0b80009cd0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.502+0000 7f0b97000700 1 -- 172.21.15.116:0/2119556393 shutdown_connections 2024-03-30T00:55:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.502+0000 7f0b97000700 1 --2- 172.21.15.116:0/2119556393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b90102810 0x7f0b901090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.502+0000 7f0b97000700 1 --2- 172.21.15.116:0/2119556393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b90100c80 0x7f0b901010a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.502+0000 7f0b97000700 1 --2- 172.21.15.116:0/2119556393 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b90101e70 0x7f0b901022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.502+0000 7f0b97000700 1 -- 172.21.15.116:0/2119556393 >> 172.21.15.116:0/2119556393 conn(0x7f0b900fc1f0 msgr2=0x7f0b900fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.502+0000 7f0b97000700 1 -- 172.21.15.116:0/2119556393 shutdown_connections 2024-03-30T00:55:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.502+0000 7f0b97000700 1 -- 172.21.15.116:0/2119556393 wait complete. 2024-03-30T00:55:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.503+0000 7f0b97000700 1 Processor -- start 2024-03-30T00:55:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.503+0000 7f0b97000700 1 -- start start 2024-03-30T00:55:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b97000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b90100c80 0x7f0b90074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b97000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b90101e70 0x7f0b900731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b97000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b90102810 0x7f0b900736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b97000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b90073e80 con 0x7f0b90101e70 2024-03-30T00:55:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b97000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b90073ff0 con 0x7f0b90102810 2024-03-30T00:55:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b97000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b901a0c00 con 0x7f0b90100c80 2024-03-30T00:55:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b8f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b90101e70 0x7f0b900731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b8f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b90101e70 0x7f0b900731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45840/0 (socket says 172.21.15.116:45840) 2024-03-30T00:55:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b8f400700 1 -- 172.21.15.116:0/1427717921 learned_addr learned my addr 172.21.15.116:0/1427717921 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b8fe00700 1 --2- 172.21.15.116:0/1427717921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b90100c80 0x7f0b90074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b95000700 1 --2- 172.21.15.116:0/1427717921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b90102810 0x7f0b900736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b8f400700 1 -- 172.21.15.116:0/1427717921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b90100c80 msgr2=0x7f0b90074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b8f400700 1 --2- 172.21.15.116:0/1427717921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b90100c80 0x7f0b90074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b8f400700 1 -- 172.21.15.116:0/1427717921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b90102810 msgr2=0x7f0b900736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b8f400700 1 --2- 172.21.15.116:0/1427717921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b90102810 0x7f0b900736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.504+0000 7f0b8f400700 1 -- 172.21.15.116:0/1427717921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b800095f0 con 0x7f0b90101e70 2024-03-30T00:55:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.505+0000 7f0b8f400700 1 --2- 172.21.15.116:0/1427717921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b90101e70 0x7f0b900731a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c00e9a0 tx=0x7f0b7c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.505+0000 7f0b8cc00700 1 -- 172.21.15.116:0/1427717921 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c009790 con 0x7f0b90101e70 2024-03-30T00:55:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.505+0000 7f0b8cc00700 1 -- 172.21.15.116:0/1427717921 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b7c00cb70 con 0x7f0b90101e70 2024-03-30T00:55:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.505+0000 7f0b97000700 1 -- 172.21.15.116:0/1427717921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b901a0e90 con 0x7f0b90101e70 2024-03-30T00:55:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.505+0000 7f0b8cc00700 1 -- 172.21.15.116:0/1427717921 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c010ea0 con 0x7f0b90101e70 2024-03-30T00:55:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.505+0000 7f0b97000700 1 -- 172.21.15.116:0/1427717921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b901a1310 con 0x7f0b90101e70 2024-03-30T00:55:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.506+0000 7f0b97000700 1 -- 172.21.15.116:0/1427717921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b901080b0 con 0x7f0b90101e70 2024-03-30T00:55:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.508+0000 7f0b8cc00700 1 -- 172.21.15.116:0/1427717921 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f0b7c00cce0 con 0x7f0b90101e70 2024-03-30T00:55:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.509+0000 7f0b8cc00700 1 --2- 172.21.15.116:0/1427717921 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0b740a1120 0x7f0b740a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.509+0000 7f0b8cc00700 1 -- 172.21.15.116:0/1427717921 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0b7c014070 con 0x7f0b90101e70 2024-03-30T00:55:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.510+0000 7f0b8fe00700 1 --2- 172.21.15.116:0/1427717921 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0b740a1120 0x7f0b740a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.510+0000 7f0b8fe00700 1 --2- 172.21.15.116:0/1427717921 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0b740a1120 0x7f0b740a35e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0b80009990 tx=0x7f0b80000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.513+0000 7f0b8cc00700 1 -- 172.21.15.116:0/1427717921 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0b7c0529f0 con 0x7f0b90101e70 2024-03-30T00:55:58.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:58 smithi130 ceph-mon[47626]: pgmap v232: 129 pgs: 7 creating+peering, 106 active+clean, 16 unknown; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-03-30T00:55:58.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:58 smithi130 ceph-mon[47626]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T00:55:58.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:58 smithi130 ceph-mon[47626]: Cluster is now healthy 2024-03-30T00:55:58.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:58 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3377663397' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:58.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:58 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3080804991' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:58.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:58 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3211072300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:58.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:58 smithi116 ceph-mon[38086]: pgmap v232: 129 pgs: 7 creating+peering, 106 active+clean, 16 unknown; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-03-30T00:55:58.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:58 smithi116 ceph-mon[38086]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T00:55:58.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:58 smithi116 ceph-mon[38086]: Cluster is now healthy 2024-03-30T00:55:58.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:58 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3377663397' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:58.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:58 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3080804991' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:58.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:58 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3211072300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:58.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:58 smithi181 ceph-mon[47622]: pgmap v232: 129 pgs: 7 creating+peering, 106 active+clean, 16 unknown; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-03-30T00:55:58.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:58 smithi181 ceph-mon[47622]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T00:55:58.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:58 smithi181 ceph-mon[47622]: Cluster is now healthy 2024-03-30T00:55:58.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:58 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3377663397' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:58.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:58 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3080804991' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:58.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:58 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3211072300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:55:58.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.761+0000 7f0b97000700 1 -- 172.21.15.116:0/1427717921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b90067a10 con 0x7f0b90101e70 2024-03-30T00:55:58.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.763+0000 7f0b8cc00700 1 -- 172.21.15.116:0/1427717921 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+24028 (secure 0 0 0) 0x7f0b7c0962a0 con 0x7f0b90101e70 2024-03-30T00:55:58.769 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:55:58.769 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:55:56.324776+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:55:58.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.765+0000 7f0b97000700 1 -- 172.21.15.116:0/1427717921 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0b740a1120 msgr2=0x7f0b740a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:58.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.765+0000 7f0b97000700 1 --2- 172.21.15.116:0/1427717921 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0b740a1120 0x7f0b740a35e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0b80009990 tx=0x7f0b80000bc0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.765+0000 7f0b97000700 1 -- 172.21.15.116:0/1427717921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b90101e70 msgr2=0x7f0b900731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:58.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.765+0000 7f0b97000700 1 --2- 172.21.15.116:0/1427717921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b90101e70 0x7f0b900731a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c00e9a0 tx=0x7f0b7c00ecb0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.765+0000 7f0b97000700 1 -- 172.21.15.116:0/1427717921 shutdown_connections 2024-03-30T00:55:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.765+0000 7f0b97000700 1 --2- 172.21.15.116:0/1427717921 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0b740a1120 0x7f0b740a35e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.765+0000 7f0b97000700 1 --2- 172.21.15.116:0/1427717921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b90100c80 0x7f0b90074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.765+0000 7f0b97000700 1 --2- 172.21.15.116:0/1427717921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b90102810 0x7f0b900736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.765+0000 7f0b97000700 1 --2- 172.21.15.116:0/1427717921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b90101e70 0x7f0b900731a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.765+0000 7f0b97000700 1 -- 172.21.15.116:0/1427717921 >> 172.21.15.116:0/1427717921 conn(0x7f0b900fc1f0 msgr2=0x7f0b901051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.765+0000 7f0b97000700 1 -- 172.21.15.116:0/1427717921 shutdown_connections 2024-03-30T00:55:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.765+0000 7f0b97000700 1 -- 172.21.15.116:0/1427717921 wait complete. 2024-03-30T00:55:58.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-30T00:55:51.790326+0000', 'modified': '2024-03-30T00:55:56.324997+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24587}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.130:6839/1184545085', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 1184545085}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 1184545085}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-03-30T00:55:58.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-30T00:55:58.791 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-30T00:55:58.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.978+0000 7f1862e00700 1 -- 172.21.15.116:0/2230110788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185c100d80 msgr2=0x7f185c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:58.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.978+0000 7f1862e00700 1 --2- 172.21.15.116:0/2230110788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185c100d80 0x7f185c1011a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f184c0098f0 tx=0x7f184c009c00 comp rx=0 tx=0).stop 2024-03-30T00:55:58.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.978+0000 7f1862e00700 1 -- 172.21.15.116:0/2230110788 shutdown_connections 2024-03-30T00:55:58.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.978+0000 7f1862e00700 1 --2- 172.21.15.116:0/2230110788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185c100d80 0x7f185c1011a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.978+0000 7f1862e00700 1 --2- 172.21.15.116:0/2230110788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c102910 0x7f185c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.978+0000 7f1862e00700 1 --2- 172.21.15.116:0/2230110788 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f185c101f70 0x7f185c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.978+0000 7f1862e00700 1 -- 172.21.15.116:0/2230110788 >> 172.21.15.116:0/2230110788 conn(0x7f185c0fc2f0 msgr2=0x7f185c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:58.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.978+0000 7f1862e00700 1 -- 172.21.15.116:0/2230110788 shutdown_connections 2024-03-30T00:55:58.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.978+0000 7f1862e00700 1 -- 172.21.15.116:0/2230110788 wait complete. 2024-03-30T00:55:58.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.979+0000 7f1862e00700 1 Processor -- start 2024-03-30T00:55:58.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.979+0000 7f1862e00700 1 -- start start 2024-03-30T00:55:58.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.979+0000 7f1862e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185c100d80 0x7f185c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.979+0000 7f1862e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f185c101f70 0x7f185c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.979+0000 7f1862e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c102910 0x7f185c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.979+0000 7f1862e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c073e80 con 0x7f185c102910 2024-03-30T00:55:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.979+0000 7f1862e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c073ff0 con 0x7f185c101f70 2024-03-30T00:55:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.979+0000 7f1862e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c1a0e40 con 0x7f185c100d80 2024-03-30T00:55:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.980+0000 7f1860e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c102910 0x7f185c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.980+0000 7f1860e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c102910 0x7f185c0736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45862/0 (socket says 172.21.15.116:45862) 2024-03-30T00:55:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.980+0000 7f1860e00700 1 -- 172.21.15.116:0/964719142 learned_addr learned my addr 172.21.15.116:0/964719142 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.980+0000 7f185be00700 1 --2- 172.21.15.116:0/964719142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185c100d80 0x7f185c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.980+0000 7f185b400700 1 --2- 172.21.15.116:0/964719142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f185c101f70 0x7f185c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.980+0000 7f1860e00700 1 -- 172.21.15.116:0/964719142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185c100d80 msgr2=0x7f185c074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.980+0000 7f1860e00700 1 --2- 172.21.15.116:0/964719142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185c100d80 0x7f185c074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.980+0000 7f1860e00700 1 -- 172.21.15.116:0/964719142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f185c101f70 msgr2=0x7f185c0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.980+0000 7f1860e00700 1 --2- 172.21.15.116:0/964719142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f185c101f70 0x7f185c0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.980+0000 7f1860e00700 1 -- 172.21.15.116:0/964719142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1854009d00 con 0x7f185c102910 2024-03-30T00:55:58.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.980+0000 7f1860e00700 1 --2- 172.21.15.116:0/964719142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c102910 0x7f185c0736e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f185400e3c0 tx=0x7f185400e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:58.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.980+0000 7f1858c00700 1 -- 172.21.15.116:0/964719142 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18540090d0 con 0x7f185c102910 2024-03-30T00:55:58.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.980+0000 7f1858c00700 1 -- 172.21.15.116:0/964719142 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f185400f040 con 0x7f185c102910 2024-03-30T00:55:58.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.980+0000 7f1862e00700 1 -- 172.21.15.116:0/964719142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184c009520 con 0x7f185c102910 2024-03-30T00:55:58.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.981+0000 7f1858c00700 1 -- 172.21.15.116:0/964719142 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854007bb0 con 0x7f185c102910 2024-03-30T00:55:58.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.981+0000 7f1862e00700 1 -- 172.21.15.116:0/964719142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185c1a14a0 con 0x7f185c102910 2024-03-30T00:55:58.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.984+0000 7f1858c00700 1 -- 172.21.15.116:0/964719142 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f1854007d10 con 0x7f185c102910 2024-03-30T00:55:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.986+0000 7f1876a0ab80 1 Processor -- start 2024-03-30T00:55:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.986+0000 7f1858c00700 1 --2- 172.21.15.116:0/964719142 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f18400a11a0 0x7f18400a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.986+0000 7f1876a0ab80 1 -- start start 2024-03-30T00:55:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.986+0000 7f1858c00700 1 -- 172.21.15.116:0/964719142 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f18540cc9f0 con 0x7f185c102910 2024-03-30T00:55:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.986+0000 7f185b400700 1 --2- 172.21.15.116:0/964719142 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f18400a11a0 0x7f18400a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.986+0000 7f1876a0ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b90535f780 0x55b90535fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.986+0000 7f1876a0ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9053600e0 0x55b9053609a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.986+0000 7f1876a0ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 0x55b905361400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.986+0000 7f1876a0ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9052b7940 con 0x55b9053600e0 2024-03-30T00:55:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.986+0000 7f1876a0ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9052b5300 con 0x55b90535f780 2024-03-30T00:55:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.986+0000 7f1876a0ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9052b0bf0 con 0x55b905360f80 2024-03-30T00:55:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.987+0000 7f185b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9053600e0 0x55b9053609a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.987+0000 7f185b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9053600e0 0x55b9053609a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45872/0 (socket says 172.21.15.116:45872) 2024-03-30T00:55:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.987+0000 7f185b400700 1 -- 172.21.15.116:0/3727268548 learned_addr learned my addr 172.21.15.116:0/3727268548 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.987+0000 7f185be00700 1 --2- 172.21.15.116:0/3727268548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b90535f780 0x55b90535fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.987+0000 7f1860e00700 1 --2- 172.21.15.116:0/3727268548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 0x55b905361400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.987+0000 7f185b400700 1 --2- 172.21.15.116:0/964719142 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f18400a11a0 0x7f18400a3660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f18480125e0 tx=0x7f18480129a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.987+0000 7f185b400700 1 -- 172.21.15.116:0/3727268548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 msgr2=0x55b905361400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:55:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.987+0000 7f185b400700 1 --2- 172.21.15.116:0/3727268548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 0x55b905361400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.987+0000 7f185b400700 1 -- 172.21.15.116:0/3727268548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b90535f780 msgr2=0x55b90535fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:55:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.987+0000 7f185b400700 1 --2- 172.21.15.116:0/3727268548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b90535f780 0x55b90535fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.987+0000 7f185b400700 1 -- 172.21.15.116:0/3727268548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b905132360 con 0x55b9053600e0 2024-03-30T00:55:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.987+0000 7f185b400700 1 --2- 172.21.15.116:0/3727268548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9053600e0 0x55b9053609a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f184801e610 tx=0x7f184801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b3d7ea5b796a01d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.988+0000 7f1851600700 1 -- 172.21.15.116:0/3727268548 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184801ecf0 con 0x55b9053600e0 2024-03-30T00:55:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.988+0000 7f1851600700 1 -- 172.21.15.116:0/3727268548 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f184801ee50 con 0x55b9053600e0 2024-03-30T00:55:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.988+0000 7f1851600700 1 -- 172.21.15.116:0/3727268548 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184800f740 con 0x55b9053600e0 2024-03-30T00:55:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.988+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3727268548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9053600e0 msgr2=0x55b9053609a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.988+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3727268548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9053600e0 0x55b9053609a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f184801e610 tx=0x7f184801e000 comp rx=0 tx=0).stop 2024-03-30T00:55:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.988+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3727268548 shutdown_connections 2024-03-30T00:55:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.988+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3727268548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 0x55b905361400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.988+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3727268548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b90535f780 0x55b90535fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.988+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3727268548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9053600e0 0x55b9053609a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.988+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3727268548 >> 172.21.15.116:0/3727268548 conn(0x55b9052be650 msgr2=0x55b90535a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.989+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3727268548 shutdown_connections 2024-03-30T00:55:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.989+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3727268548 wait complete. 2024-03-30T00:55:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.989+0000 7f1876a0ab80 1 -- start start 2024-03-30T00:55:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.989+0000 7f1876a0ab80 1 Processor -- start 2024-03-30T00:55:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.990+0000 7f1876a0ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b90535f780 0x55b9053695a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.990+0000 7f1876a0ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9053600e0 0x55b905369ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.990+0000 7f1876a0ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 0x55b905370b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.990+0000 7f1876a0ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b90524cd30 con 0x55b9053600e0 2024-03-30T00:55:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.990+0000 7f1876a0ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b90524c9d0 con 0x55b90535f780 2024-03-30T00:55:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.990+0000 7f1876a0ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b905134650 con 0x55b905360f80 2024-03-30T00:55:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.990+0000 7f1876a0ab80 10 client.0 _ll_register_callbacks cb 0x55b905256420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:55:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.990+0000 7f185b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9053600e0 0x55b905369ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.990+0000 7f185b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9053600e0 0x55b905369ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45882/0 (socket says 172.21.15.116:45882) 2024-03-30T00:55:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.990+0000 7f185b400700 1 -- 172.21.15.116:0/3929286082 learned_addr learned my addr 172.21.15.116:0/3929286082 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.990+0000 7f1860e00700 1 --2- 172.21.15.116:0/3929286082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 0x55b905370b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.990+0000 7f185be00700 1 --2- 172.21.15.116:0/3929286082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b90535f780 0x55b9053695a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f1860e00700 1 -- 172.21.15.116:0/3929286082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b90535f780 msgr2=0x55b9053695a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:55:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f1860e00700 1 --2- 172.21.15.116:0/3929286082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b90535f780 0x55b9053695a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f1860e00700 1 -- 172.21.15.116:0/3929286082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9053600e0 msgr2=0x55b905369ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:55:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f1860e00700 1 --2- 172.21.15.116:0/3929286082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9053600e0 0x55b905369ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f1860e00700 1 -- 172.21.15.116:0/3929286082 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184801e3b0 con 0x55b905360f80 2024-03-30T00:55:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f185b400700 1 --2- 172.21.15.116:0/3929286082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9053600e0 0x55b905369ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:55:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f1860e00700 1 --2- 172.21.15.116:0/3929286082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 0x55b905370b20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f18540ca2a0 tx=0x7f1854099660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=245e86b0e23fe91f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f1846000700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T00:55:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f1846000700 1 -- 172.21.15.116:0/3929286082 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854099c80 con 0x55b905360f80 2024-03-30T00:55:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f1846000700 1 -- 172.21.15.116:0/3929286082 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f185409dab0 con 0x55b905360f80 2024-03-30T00:55:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3929286082 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b905271330 con 0x55b905360f80 2024-03-30T00:55:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f1846000700 1 -- 172.21.15.116:0/3929286082 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18540a2070 con 0x55b905360f80 2024-03-30T00:55:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f1876a0ab80 10 client.24676 fetch_fsmap learned FSMap version 16 2024-03-30T00:55:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3929286082 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x55b9050deb10 con 0x55b905360f80 2024-03-30T00:55:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.991+0000 7f1846000700 1 -- 172.21.15.116:0/3929286082 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b905271330 con 0x55b905360f80 2024-03-30T00:55:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.992+0000 7f1846000700 1 -- 172.21.15.116:0/3929286082 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 16) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f1854099290 con 0x55b905360f80 2024-03-30T00:55:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.992+0000 7f1876a0ab80 10 client.24676 fetch_fsmap finished waiting for FSMap version 16 2024-03-30T00:55:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.992+0000 7f1876a0ab80 10 client.24676 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-30T00:55:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.992+0000 7f1876a0ab80 20 client.24676 populate_metadata read hostname 'smithi116' 2024-03-30T00:55:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.992+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3929286082 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b905374670 0x55b905379970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.992+0000 7f1876a0ab80 4 client.24676 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:55:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.992+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3929286082 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9052a96b0 con 0x55b905374670 2024-03-30T00:55:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.992+0000 7f185be00700 1 --2- 172.21.15.116:0/3929286082 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b905374670 0x55b905379970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.993+0000 7f1846000700 10 client.24676 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.993+0000 7f185be00700 1 --2- 172.21.15.116:0/3929286082 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b905374670 0x55b905379970 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c201aa4aeea919c server_cookie=c869983b776fd1c3 in_seq=0 out_seq=0 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.993+0000 7f1846000700 1 -- 172.21.15.116:0/3929286082 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b9052a96b0 con 0x55b905374670 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.993+0000 7f1846000700 10 client.24676 handle_command_reply: tid=0 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.994+0000 7f1876a0ab80 1 client.24676 shutdown 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.994+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3929286082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 msgr2=0x55b905370b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.994+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3929286082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 0x55b905370b20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f18540ca2a0 tx=0x7f1854099660 comp rx=0 tx=0).stop 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.994+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3929286082 shutdown_connections 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.994+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3929286082 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b905374670 0x55b905379970 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.994+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3929286082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 0x55b905370b20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.994+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3929286082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b90535f780 0x55b9053695a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.994+0000 7f1846000700 0 client.24676 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.994+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3929286082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9053600e0 0x55b905369ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.994+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3929286082 >> 172.21.15.116:0/3929286082 conn(0x55b9052be650 msgr2=0x55b90535e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.994+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3929286082 shutdown_connections 2024-03-30T00:55:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.994+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3929286082 wait complete. 2024-03-30T00:55:59.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:58.994+0000 7f1876a0ab80 20 client.24676 trim_cache size 0 max 16384 2024-03-30T00:55:59.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.012+0000 7f1876a0ab80 1 Processor -- start 2024-03-30T00:55:59.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.012+0000 7f1876a0ab80 1 -- start start 2024-03-30T00:55:59.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.012+0000 7f1876a0ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b90535f780 0x55b90535fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:59.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.012+0000 7f1876a0ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9053600e0 0x55b905370ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:59.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.012+0000 7f1876a0ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 0x55b905361360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:59.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.012+0000 7f1876a0ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9052a96e0 con 0x55b90535f780 2024-03-30T00:55:59.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.012+0000 7f1876a0ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b905271360 con 0x55b9053600e0 2024-03-30T00:55:59.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.012+0000 7f1876a0ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9052a9990 con 0x55b905360f80 2024-03-30T00:55:59.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.013+0000 7f185be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b90535f780 0x55b90535fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.013+0000 7f185be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b90535f780 0x55b90535fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45898/0 (socket says 172.21.15.116:45898) 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.013+0000 7f185be00700 1 -- 172.21.15.116:0/3818563666 learned_addr learned my addr 172.21.15.116:0/3818563666 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.013+0000 7f1860e00700 1 --2- 172.21.15.116:0/3818563666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 0x55b905361360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.013+0000 7f185b400700 1 --2- 172.21.15.116:0/3818563666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9053600e0 0x55b905370ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.013+0000 7f185be00700 1 -- 172.21.15.116:0/3818563666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 msgr2=0x55b905361360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.013+0000 7f185be00700 1 --2- 172.21.15.116:0/3818563666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 0x55b905361360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.013+0000 7f185be00700 1 -- 172.21.15.116:0/3818563666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9053600e0 msgr2=0x55b905370ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.013+0000 7f185be00700 1 --2- 172.21.15.116:0/3818563666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9053600e0 0x55b905370ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.013+0000 7f185be00700 1 -- 172.21.15.116:0/3818563666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b905276e50 con 0x55b90535f780 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.013+0000 7f185be00700 1 --2- 172.21.15.116:0/3818563666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b90535f780 0x55b90535fb60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f184c0098f0 tx=0x7f184c009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ca48768f916402 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.013+0000 7f1852000700 1 -- 172.21.15.116:0/3818563666 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c027a80 con 0x55b90535f780 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.013+0000 7f1852000700 1 -- 172.21.15.116:0/3818563666 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f184c01d040 con 0x55b90535f780 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.014+0000 7f1852000700 1 -- 172.21.15.116:0/3818563666 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c00b880 con 0x55b90535f780 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.014+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3818563666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b90535f780 msgr2=0x55b90535fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:59.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.014+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3818563666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b90535f780 0x55b90535fb60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f184c0098f0 tx=0x7f184c009f50 comp rx=0 tx=0).stop 2024-03-30T00:55:59.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.014+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3818563666 shutdown_connections 2024-03-30T00:55:59.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.014+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3818563666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b905360f80 0x55b905361360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.014+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3818563666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b90535f780 0x55b90535fb60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.014+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3818563666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9053600e0 0x55b905370ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.014+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3818563666 >> 172.21.15.116:0/3818563666 conn(0x55b9052be650 msgr2=0x55b905353550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:59.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.014+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3818563666 shutdown_connections 2024-03-30T00:55:59.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.014+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3818563666 wait complete. 2024-03-30T00:55:59.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.015+0000 7f1876a0ab80 1 -- start start 2024-03-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.015+0000 7f1876a0ab80 1 Processor -- start 2024-03-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.015+0000 7f1876a0ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b90535f780 0x55b905365e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.015+0000 7f1876a0ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9053600e0 0x55b905366360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.015+0000 7f1876a0ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b905360f80 0x55b9053668a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.015+0000 7f1876a0ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b905369de0 con 0x55b90535f780 2024-03-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.015+0000 7f1876a0ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b905369f20 con 0x55b905360f80 2024-03-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.016+0000 7f1876a0ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9053c8f60 con 0x55b9053600e0 2024-03-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.016+0000 7f1876a0ab80 10 client.0 _ll_register_callbacks cb 0x55b905256420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.016+0000 7f185be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b90535f780 0x55b905365e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.016+0000 7f185be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b90535f780 0x55b905365e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45910/0 (socket says 172.21.15.116:45910) 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.016+0000 7f185be00700 1 -- 172.21.15.116:0/3942989183 learned_addr learned my addr 172.21.15.116:0/3942989183 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.016+0000 7f185b400700 1 --2- 172.21.15.116:0/3942989183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9053600e0 0x55b905366360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.016+0000 7f1860e00700 1 --2- 172.21.15.116:0/3942989183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b905360f80 0x55b9053668a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.016+0000 7f185be00700 1 -- 172.21.15.116:0/3942989183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9053600e0 msgr2=0x55b905366360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.016+0000 7f185be00700 1 --2- 172.21.15.116:0/3942989183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9053600e0 0x55b905366360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.016+0000 7f185be00700 1 -- 172.21.15.116:0/3942989183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b905360f80 msgr2=0x55b9053668a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.016+0000 7f185be00700 1 --2- 172.21.15.116:0/3942989183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b905360f80 0x55b9053668a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.016+0000 7f185be00700 1 -- 172.21.15.116:0/3942989183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185c109b10 con 0x55b90535f780 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.016+0000 7f185be00700 1 --2- 172.21.15.116:0/3942989183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b90535f780 0x55b905365e20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f184c009230 tx=0x7f184c00f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5026c01d7f83f9d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.016+0000 7f1846000700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.017+0000 7f1846000700 1 -- 172.21.15.116:0/3942989183 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c027500 con 0x55b90535f780 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.017+0000 7f1846000700 1 -- 172.21.15.116:0/3942989183 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f184c027d80 con 0x55b90535f780 2024-03-30T00:55:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.017+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3942989183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f184801ee50 con 0x55b90535f780 2024-03-30T00:55:59.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.017+0000 7f1846000700 1 -- 172.21.15.116:0/3942989183 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f184c00b660 con 0x55b90535f780 2024-03-30T00:55:59.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.017+0000 7f1876a0ab80 10 client.14634 fetch_fsmap learned FSMap version 16 2024-03-30T00:55:59.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.017+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3942989183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x55b905136c80 con 0x55b90535f780 2024-03-30T00:55:59.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.017+0000 7f1846000700 1 -- 172.21.15.116:0/3942989183 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f184801ee50 con 0x55b90535f780 2024-03-30T00:55:59.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.017+0000 7f1846000700 1 -- 172.21.15.116:0/3942989183 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 16) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f184c009920 con 0x55b90535f780 2024-03-30T00:55:59.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.017+0000 7f1876a0ab80 10 client.14634 fetch_fsmap finished waiting for FSMap version 16 2024-03-30T00:55:59.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.017+0000 7f1876a0ab80 10 client.14634 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-30T00:55:59.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.017+0000 7f1876a0ab80 20 client.14634 populate_metadata read hostname 'smithi116' 2024-03-30T00:55:59.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.017+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3942989183 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b9053cd7d0 0x55b9053ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:55:59.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.017+0000 7f1876a0ab80 4 client.14634 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T00:55:59.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.017+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3942989183 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b9050ffca0 con 0x55b9053cd7d0 2024-03-30T00:55:59.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.018+0000 7f185b400700 1 --2- 172.21.15.116:0/3942989183 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b9053cd7d0 0x55b9053ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:55:59.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.018+0000 7f1846000700 10 client.14634 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:55:59.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.018+0000 7f185b400700 1 --2- 172.21.15.116:0/3942989183 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b9053cd7d0 0x55b9053ca1d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c73a9f28967ef16 server_cookie=3157ac762098f205 in_seq=0 out_seq=0 2024-03-30T00:55:59.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.018+0000 7f1846000700 1 -- 172.21.15.116:0/3942989183 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x55b9050ffca0 con 0x55b9053cd7d0 2024-03-30T00:55:59.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.018+0000 7f1846000700 10 client.14634 handle_command_reply: tid=0 2024-03-30T00:55:59.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.019+0000 7f1876a0ab80 1 client.14634 shutdown 2024-03-30T00:55:59.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.019+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3942989183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b90535f780 msgr2=0x55b905365e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:59.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.019+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3942989183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b90535f780 0x55b905365e20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f184c009230 tx=0x7f184c00f670 comp rx=0 tx=0).stop 2024-03-30T00:55:59.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.019+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3942989183 shutdown_connections 2024-03-30T00:55:59.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.019+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3942989183 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b9053cd7d0 0x55b9053ca1d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.019+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3942989183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b905360f80 0x55b9053668a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.019+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3942989183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b90535f780 0x55b905365e20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.019+0000 7f1876a0ab80 1 --2- 172.21.15.116:0/3942989183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9053600e0 0x55b905366360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.019+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3942989183 >> 172.21.15.116:0/3942989183 conn(0x55b9052be650 msgr2=0x55b905368a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:59.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.019+0000 7f1846000700 0 client.14634 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:55:59.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.019+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3942989183 shutdown_connections 2024-03-30T00:55:59.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.019+0000 7f1876a0ab80 1 -- 172.21.15.116:0/3942989183 wait complete. 2024-03-30T00:55:59.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.019+0000 7f1876a0ab80 20 client.14634 trim_cache size 0 max 16384 2024-03-30T00:55:59.032 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:55:59.032 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "4909f004-ee2f-11ee-b647-cb9ed24678a4", 2024-03-30T00:55:59.032 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-03-30T00:55:59.032 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24587, 2024-03-30T00:55:59.032 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-30T00:55:59.032 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-30T00:55:59.032 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-30T00:55:59.032 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 7.225625763, 2024-03-30T00:55:59.032 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 16, 2024-03-30T00:55:59.033 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 81, 2024-03-30T00:55:59.033 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 80, 2024-03-30T00:55:59.033 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 11.987439188 2024-03-30T00:55:59.033 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:55:59.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.020+0000 7f1862e00700 1 -- 172.21.15.116:0/964719142 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f18400a11a0 msgr2=0x7f18400a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:59.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.020+0000 7f1862e00700 1 --2- 172.21.15.116:0/964719142 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f18400a11a0 0x7f18400a3660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f18480125e0 tx=0x7f18480129a0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.020+0000 7f1862e00700 1 -- 172.21.15.116:0/964719142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c102910 msgr2=0x7f185c0736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:55:59.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.020+0000 7f1862e00700 1 --2- 172.21.15.116:0/964719142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c102910 0x7f185c0736e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f185400e3c0 tx=0x7f185400e6d0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.020+0000 7f1862e00700 1 -- 172.21.15.116:0/964719142 shutdown_connections 2024-03-30T00:55:59.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.020+0000 7f1862e00700 1 --2- 172.21.15.116:0/964719142 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f18400a11a0 0x7f18400a3660 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.020+0000 7f1862e00700 1 --2- 172.21.15.116:0/964719142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f185c100d80 0x7f185c074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.020+0000 7f1862e00700 1 --2- 172.21.15.116:0/964719142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c102910 0x7f185c0736e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.020+0000 7f1862e00700 1 --2- 172.21.15.116:0/964719142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f185c101f70 0x7f185c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:55:59.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.020+0000 7f1862e00700 1 -- 172.21.15.116:0/964719142 >> 172.21.15.116:0/964719142 conn(0x7f185c0fc2f0 msgr2=0x7f185c1053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:55:59.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.020+0000 7f1862e00700 1 -- 172.21.15.116:0/964719142 shutdown_connections 2024-03-30T00:55:59.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:55:59.020+0000 7f1862e00700 1 -- 172.21.15.116:0/964719142 wait complete. 2024-03-30T00:55:59.044 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-30T00:55:59.044 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-30T00:55:59.044 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph fs ls 2024-03-30T00:55:59.399 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:55:59.581 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:55:59 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1427717921' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:59.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:55:59 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1427717921' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:55:59.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:55:59 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1427717921' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:00.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.538+0000 7f18cb600700 1 -- 172.21.15.116:0/1738074686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c4100e60 msgr2=0x7f18c41012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:00.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.538+0000 7f18cb600700 1 --2- 172.21.15.116:0/1738074686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c4100e60 0x7f18c41012e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f18b00099c0 tx=0x7f18b0009cd0 comp rx=0 tx=0).stop 2024-03-30T00:56:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.538+0000 7f18cb600700 1 -- 172.21.15.116:0/1738074686 shutdown_connections 2024-03-30T00:56:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.538+0000 7f18cb600700 1 --2- 172.21.15.116:0/1738074686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18c4106ef0 0x7f18c41072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.538+0000 7f18cb600700 1 --2- 172.21.15.116:0/1738074686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f18c4101820 0x7f18c410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.538+0000 7f18cb600700 1 --2- 172.21.15.116:0/1738074686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c4100e60 0x7f18c41012e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.538+0000 7f18cb600700 1 -- 172.21.15.116:0/1738074686 >> 172.21.15.116:0/1738074686 conn(0x7f18c4077fa0 msgr2=0x7f18c40783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.538+0000 7f18cb600700 1 -- 172.21.15.116:0/1738074686 shutdown_connections 2024-03-30T00:56:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.538+0000 7f18cb600700 1 -- 172.21.15.116:0/1738074686 wait complete. 2024-03-30T00:56:00.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.539+0000 7f18cb600700 1 Processor -- start 2024-03-30T00:56:00.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.540+0000 7f18cb600700 1 -- start start 2024-03-30T00:56:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.540+0000 7f18cb600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18c4100e60 0x7f18c4197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.540+0000 7f18cb600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c4101820 0x7f18c4197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.540+0000 7f18cb600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f18c4106ef0 0x7f18c419bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.540+0000 7f18cb600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c419c0d0 con 0x7f18c4101820 2024-03-30T00:56:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.540+0000 7f18cb600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c419c240 con 0x7f18c4106ef0 2024-03-30T00:56:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.540+0000 7f18cb600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c419c400 con 0x7f18c4100e60 2024-03-30T00:56:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.540+0000 7f18c3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c4101820 0x7f18c4197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.540+0000 7f18c3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c4101820 0x7f18c4197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45916/0 (socket says 172.21.15.116:45916) 2024-03-30T00:56:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.540+0000 7f18c3e00700 1 -- 172.21.15.116:0/673479459 learned_addr learned my addr 172.21.15.116:0/673479459 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.540+0000 7f18c8c00700 1 --2- 172.21.15.116:0/673479459 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18c4100e60 0x7f18c4197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.540+0000 7f18c9600700 1 --2- 172.21.15.116:0/673479459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f18c4106ef0 0x7f18c419bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.541+0000 7f18c3e00700 1 -- 172.21.15.116:0/673479459 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18c4100e60 msgr2=0x7f18c4197220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.541+0000 7f18c3e00700 1 --2- 172.21.15.116:0/673479459 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18c4100e60 0x7f18c4197220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.541+0000 7f18c3e00700 1 -- 172.21.15.116:0/673479459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f18c4106ef0 msgr2=0x7f18c419bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.541+0000 7f18c3e00700 1 --2- 172.21.15.116:0/673479459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f18c4106ef0 0x7f18c419bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.541+0000 7f18c3e00700 1 -- 172.21.15.116:0/673479459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b00095f0 con 0x7f18c4101820 2024-03-30T00:56:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.541+0000 7f18c3e00700 1 --2- 172.21.15.116:0/673479459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c4101820 0x7f18c4197760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f18b0005b40 tx=0x7f18b000ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.541+0000 7f18c1600700 1 -- 172.21.15.116:0/673479459 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b001d070 con 0x7f18c4101820 2024-03-30T00:56:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.541+0000 7f18cb600700 1 -- 172.21.15.116:0/673479459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c419c680 con 0x7f18c4101820 2024-03-30T00:56:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.541+0000 7f18c1600700 1 -- 172.21.15.116:0/673479459 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18b000be70 con 0x7f18c4101820 2024-03-30T00:56:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.541+0000 7f18c1600700 1 -- 172.21.15.116:0/673479459 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b000fb70 con 0x7f18c4101820 2024-03-30T00:56:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.541+0000 7f18cb600700 1 -- 172.21.15.116:0/673479459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c41a3f40 con 0x7f18c4101820 2024-03-30T00:56:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.542+0000 7f18cb600700 1 -- 172.21.15.116:0/673479459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c41092f0 con 0x7f18c4101820 2024-03-30T00:56:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.544+0000 7f18c1600700 1 -- 172.21.15.116:0/673479459 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f18b002b440 con 0x7f18c4101820 2024-03-30T00:56:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.545+0000 7f18c1600700 1 --2- 172.21.15.116:0/673479459 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f18ac0a1120 0x7f18ac0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.546+0000 7f18c1600700 1 -- 172.21.15.116:0/673479459 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f18b00cd7d0 con 0x7f18c4101820 2024-03-30T00:56:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.546+0000 7f18c8c00700 1 --2- 172.21.15.116:0/673479459 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f18ac0a1120 0x7f18ac0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.546+0000 7f18c8c00700 1 --2- 172.21.15.116:0/673479459 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f18ac0a1120 0x7f18ac0a35e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f18b8007be0 tx=0x7f18b80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:00.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.549+0000 7f18c1600700 1 -- 172.21.15.116:0/673479459 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f18b0096da0 con 0x7f18c4101820 2024-03-30T00:56:00.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:00 smithi130 ceph-mon[47626]: pgmap v233: 129 pgs: 7 creating+peering, 117 active+clean, 5 unknown; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-03-30T00:56:00.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:00 smithi116 ceph-mon[38086]: pgmap v233: 129 pgs: 7 creating+peering, 117 active+clean, 5 unknown; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-03-30T00:56:00.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:00 smithi181 ceph-mon[47622]: pgmap v233: 129 pgs: 7 creating+peering, 117 active+clean, 5 unknown; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-03-30T00:56:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.795+0000 7f18cb600700 1 -- 172.21.15.116:0/673479459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f18c404ea80 con 0x7f18c4101820 2024-03-30T00:56:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.795+0000 7f18c1600700 1 -- 172.21.15.116:0/673479459 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+88 (secure 0 0 0) 0x7f18b009a3c0 con 0x7f18c4101820 2024-03-30T00:56:00.801 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-30T00:56:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.798+0000 7f18cb600700 1 -- 172.21.15.116:0/673479459 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f18ac0a1120 msgr2=0x7f18ac0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.798+0000 7f18cb600700 1 --2- 172.21.15.116:0/673479459 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f18ac0a1120 0x7f18ac0a35e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f18b8007be0 tx=0x7f18b80073d0 comp rx=0 tx=0).stop 2024-03-30T00:56:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.798+0000 7f18cb600700 1 -- 172.21.15.116:0/673479459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c4101820 msgr2=0x7f18c4197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.798+0000 7f18cb600700 1 --2- 172.21.15.116:0/673479459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c4101820 0x7f18c4197760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f18b0005b40 tx=0x7f18b000ba30 comp rx=0 tx=0).stop 2024-03-30T00:56:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.798+0000 7f18cb600700 1 -- 172.21.15.116:0/673479459 shutdown_connections 2024-03-30T00:56:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.798+0000 7f18cb600700 1 --2- 172.21.15.116:0/673479459 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f18ac0a1120 0x7f18ac0a35e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.798+0000 7f18cb600700 1 --2- 172.21.15.116:0/673479459 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18c4100e60 0x7f18c4197220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.798+0000 7f18cb600700 1 --2- 172.21.15.116:0/673479459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f18c4106ef0 0x7f18c419bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.798+0000 7f18cb600700 1 --2- 172.21.15.116:0/673479459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c4101820 0x7f18c4197760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.798+0000 7f18cb600700 1 -- 172.21.15.116:0/673479459 >> 172.21.15.116:0/673479459 conn(0x7f18c4077fa0 msgr2=0x7f18c40ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.798+0000 7f18cb600700 1 -- 172.21.15.116:0/673479459 shutdown_connections 2024-03-30T00:56:00.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:00.798+0000 7f18cb600700 1 -- 172.21.15.116:0/673479459 wait complete. 2024-03-30T00:56:01.167 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-30T00:56:01.167 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-30T00:56:01.167 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2024-03-30T00:56:01.167 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-30T00:56:01.167 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-30T00:56:01.167 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-30T00:56:01.167 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-30T00:56:01.168 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-30T00:56:01.168 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-30T00:56:01.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:01.168 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2024-03-30T00:56:01.211 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-30T00:56:01.212 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-30T00:56:01.212 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2024-03-30T00:56:01.212 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-30T00:56:01.212 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2024-03-30T00:56:01.212 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-30T00:56:01.212 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T00:56:01.212 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2024-03-30T00:56:01.212 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2024-03-30T00:56:01.212 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T00:56:01.212 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2024-03-30T00:56:01.212 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2024-03-30T00:56:01.212 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T00:56:01.212 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2024-03-30T00:56:01.212 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-30T00:56:01.213 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2024-03-30T00:56:01.213 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-30T00:56:01.213 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 41895sec preferred_lft 41895sec 2024-03-30T00:56:01.213 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-03-30T00:56:01.213 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-30T00:56:01.215 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-30T00:56:01.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:01.215 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-30T00:56:01.215 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-30T00:56:01.216 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add name ceph-brx type bridge 2024-03-30T00:56:01.216 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr flush dev ceph-brx 2024-03-30T00:56:01.216 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set ceph-brx up 2024-03-30T00:56:01.216 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-30T00:56:01.216 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-30T00:56:01.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:01 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/673479459' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T00:56:01.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:01 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/673479459' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T00:56:01.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:01.684 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-30T00:56:01.751 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-03-30T00:56:01.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:01 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/673479459' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T00:56:01.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:01.754 DEBUG:teuthology.orchestra.run.smithi116:> route 2024-03-30T00:56:01.825 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2024-03-30T00:56:01.825 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-30T00:56:01.825 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-30T00:56:01.825 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-30T00:56:01.825 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-30T00:56:01.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:01.826 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-30T00:56:01.826 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-30T00:56:01.826 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-30T00:56:01.826 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-30T00:56:01.826 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-30T00:56:01.826 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-30T00:56:02.514 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:02 smithi116 ceph-mon[38086]: pgmap v234: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 6 op/s 2024-03-30T00:56:02.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:02.517 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-30T00:56:02.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:02.560 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2024-03-30T00:56:02.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:02 smithi130 ceph-mon[47626]: pgmap v234: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 6 op/s 2024-03-30T00:56:02.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:02.642 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-30T00:56:02.642 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-30T00:56:02.642 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-30T00:56:02.642 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-30T00:56:02.642 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-30T00:56:02.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:02 smithi181 ceph-mon[47622]: pgmap v234: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 6 op/s 2024-03-30T00:56:02.820 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-30T00:56:02.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:02.820 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-30T00:56:02.820 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-30T00:56:02.820 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-30T00:56:02.820 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-30T00:56:02.820 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-30T00:56:02.821 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-30T00:56:02.821 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-30T00:56:02.821 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-30T00:56:03.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:03.089 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-30T00:56:03.089 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-30T00:56:03.089 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2024-03-30T00:56:03.089 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.0 master ceph-brx 2024-03-30T00:56:03.089 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-30T00:56:03.255 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-30T00:56:03.255 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-30T00:56:03.256 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-30T00:56:03.297 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-30T00:56:03.298 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-30T00:56:03.298 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-30T00:56:03.379 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-30T00:56:03.486 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-30T00:56:03.487 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-30T00:56:03.555 INFO:teuthology.orchestra.run.smithi116.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-30T00:56:03.558 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-30T00:56:03.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:03.559 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-30T00:56:03.601 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-30T00:56:03.601 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-30T00:56:03.652 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-30T00:56:03.717 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-30T00:56:03.717 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-30T00:56:03.783 INFO:teuthology.orchestra.run.smithi116.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-30T00:56:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-30T00:56:03.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:03.786 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-30T00:56:03.826 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi116.stderr:2024-03-30T00:56:03.821+0000 7fd2f0ea05c0 -1 init, newargv = 0x55e1a6365270 newargc=15 2024-03-30T00:56:03.827 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi116.stderr:ceph-fuse[90980]: starting ceph client 2024-03-30T00:56:03.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi116.stderr:ceph-fuse[90980]: starting fuse 2024-03-30T00:56:04.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:04 smithi130 ceph-mon[47626]: pgmap v235: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 4 op/s 2024-03-30T00:56:04.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:04 smithi116 ceph-mon[38086]: pgmap v235: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 4 op/s 2024-03-30T00:56:04.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:04 smithi181 ceph-mon[47622]: pgmap v235: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 4 op/s 2024-03-30T00:56:04.930 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-30T00:56:05.005 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-30T00:56:05.005 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-30T00:56:05.071 INFO:teuthology.orchestra.run.smithi116.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-30T00:56:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-30T00:56:05.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:05.074 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-30T00:56:05.116 INFO:teuthology.orchestra.run.smithi116.stdout:122 2024-03-30T00:56:05.116 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [122] 2024-03-30T00:56:05.116 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> import re 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> import os 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-30T00:56:05.117 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2024-03-30T00:56:05.118 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-30T00:56:05.118 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-30T00:56:05.118 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-30T00:56:05.118 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2024-03-30T00:56:05.118 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2024-03-30T00:56:05.118 DEBUG:teuthology.orchestra.run.smithi116:> return f 2024-03-30T00:56:05.118 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-30T00:56:05.118 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-30T00:56:05.118 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.admin")) 2024-03-30T00:56:05.118 DEBUG:teuthology.orchestra.run.smithi116:> ' 2024-03-30T00:56:05.295 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.admin.90980.asok 2024-03-30T00:56:05.473 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.90980.asok 2024-03-30T00:56:05.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:05.473 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.90980.asok status 2024-03-30T00:56:05.693 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:56:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2024-03-30T00:56:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "470100a4ccbcf7769052bf09a5413881ccf1b97f", 2024-03-30T00:56:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)", 2024-03-30T00:56:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "admin", 2024-03-30T00:56:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2024-03-30T00:56:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-30T00:56:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "90980", 2024-03-30T00:56:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2024-03-30T00:56:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T00:56:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2024-03-30T00:56:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2024-03-30T00:56:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24694, 2024-03-30T00:56:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2024-03-30T00:56:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2024-03-30T00:56:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2024-03-30T00:56:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 24694 2024-03-30T00:56:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T00:56:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2024-03-30T00:56:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-30T00:56:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-30T00:56:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 249181635 2024-03-30T00:56:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T00:56:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T00:56:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2024-03-30T00:56:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-30T00:56:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-30T00:56:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 249181635 2024-03-30T00:56:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T00:56:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.24694 172.21.15.116:0/249181635", 2024-03-30T00:56:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "172.21.15.116:0/249181635", 2024-03-30T00:56:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2024-03-30T00:56:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 16, 2024-03-30T00:56:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 81, 2024-03-30T00:56:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2024-03-30T00:56:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2024-03-30T00:56:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2024-03-30T00:56:05.696 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:56:05.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:05.709 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2024-03-30T00:56:05.709 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-30T00:56:05.751 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2024-03-30T00:56:05.752 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-30T00:56:05.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:05.752 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-30T00:56:05.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:05.858 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-30T00:56:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:# file: . 2024-03-30T00:56:06.001 INFO:teuthology.orchestra.run.smithi116.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-30T00:56:06.002 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:56:06.003 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:56:06.003 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-30T00:56:06.045 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:56:06.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.270+0000 7efe1b400700 1 -- 172.21.15.116:0/1601576312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe14074df0 msgr2=0x7efe14073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:06.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.270+0000 7efe1b400700 1 --2- 172.21.15.116:0/1601576312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe14074df0 0x7efe14073240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efe08009990 tx=0x7efe08009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.270+0000 7efe1b400700 1 -- 172.21.15.116:0/1601576312 shutdown_connections 2024-03-30T00:56:06.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.270+0000 7efe1b400700 1 --2- 172.21.15.116:0/1601576312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efe14108b40 0x7efe1410afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.270+0000 7efe1b400700 1 --2- 172.21.15.116:0/1601576312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe14074df0 0x7efe14073240 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.270+0000 7efe1b400700 1 --2- 172.21.15.116:0/1601576312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe14073810 0x7efe14073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.270+0000 7efe1b400700 1 -- 172.21.15.116:0/1601576312 >> 172.21.15.116:0/1601576312 conn(0x7efe140fc1f0 msgr2=0x7efe140fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:06.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.270+0000 7efe1b400700 1 -- 172.21.15.116:0/1601576312 shutdown_connections 2024-03-30T00:56:06.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.270+0000 7efe1b400700 1 -- 172.21.15.116:0/1601576312 wait complete. 2024-03-30T00:56:06.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.271+0000 7efe1b400700 1 Processor -- start 2024-03-30T00:56:06.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.271+0000 7efe1b400700 1 -- start start 2024-03-30T00:56:06.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.271+0000 7efe1b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efe14073810 0x7efe14198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:06.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.271+0000 7efe1b400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe14074df0 0x7efe14199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:06.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.271+0000 7efe1b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe14108b40 0x7efe141a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:06.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.271+0000 7efe1b400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe141a0940 con 0x7efe14108b40 2024-03-30T00:56:06.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.271+0000 7efe1b400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe141a0a80 con 0x7efe14073810 2024-03-30T00:56:06.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.271+0000 7efe1b400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe141a0c10 con 0x7efe14074df0 2024-03-30T00:56:06.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.272+0000 7efe18a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efe14073810 0x7efe14198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:06.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.272+0000 7efe19400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe14108b40 0x7efe141a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:06.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.272+0000 7efe18a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efe14073810 0x7efe14198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:36380/0 (socket says 172.21.15.116:36380) 2024-03-30T00:56:06.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.272+0000 7efe18a00700 1 -- 172.21.15.116:0/3586767025 learned_addr learned my addr 172.21.15.116:0/3586767025 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:06.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.272+0000 7efe13e00700 1 --2- 172.21.15.116:0/3586767025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe14074df0 0x7efe14199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:06.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.272+0000 7efe19400700 1 -- 172.21.15.116:0/3586767025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe14074df0 msgr2=0x7efe14199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:06.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.272+0000 7efe19400700 1 --2- 172.21.15.116:0/3586767025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe14074df0 0x7efe14199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.272+0000 7efe19400700 1 -- 172.21.15.116:0/3586767025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efe14073810 msgr2=0x7efe14198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:06.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.272+0000 7efe19400700 1 --2- 172.21.15.116:0/3586767025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efe14073810 0x7efe14198e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.272+0000 7efe19400700 1 -- 172.21.15.116:0/3586767025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe080095f0 con 0x7efe14108b40 2024-03-30T00:56:06.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.272+0000 7efe18a00700 1 --2- 172.21.15.116:0/3586767025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efe14073810 0x7efe14198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:06.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.272+0000 7efe19400700 1 --2- 172.21.15.116:0/3586767025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe14108b40 0x7efe141a0370 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efe0c007f10 tx=0x7efe0c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:06.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.273+0000 7efe11600700 1 -- 172.21.15.116:0/3586767025 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c004db0 con 0x7efe14108b40 2024-03-30T00:56:06.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.273+0000 7efe11600700 1 -- 172.21.15.116:0/3586767025 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe0c00f040 con 0x7efe14108b40 2024-03-30T00:56:06.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.273+0000 7efe1b400700 1 -- 172.21.15.116:0/3586767025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe141a0ef0 con 0x7efe14108b40 2024-03-30T00:56:06.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.273+0000 7efe11600700 1 -- 172.21.15.116:0/3586767025 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c005990 con 0x7efe14108b40 2024-03-30T00:56:06.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.273+0000 7efe1b400700 1 -- 172.21.15.116:0/3586767025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe141a13c0 con 0x7efe14108b40 2024-03-30T00:56:06.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.274+0000 7efe1b400700 1 -- 172.21.15.116:0/3586767025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe14107900 con 0x7efe14108b40 2024-03-30T00:56:06.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.276+0000 7efe11600700 1 -- 172.21.15.116:0/3586767025 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7efe0c00db20 con 0x7efe14108b40 2024-03-30T00:56:06.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.278+0000 7efe11600700 1 --2- 172.21.15.116:0/3586767025 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efdfc0a11a0 0x7efdfc0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:06.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.278+0000 7efe11600700 1 -- 172.21.15.116:0/3586767025 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7efe0c0ccae0 con 0x7efe14108b40 2024-03-30T00:56:06.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.278+0000 7efe18a00700 1 --2- 172.21.15.116:0/3586767025 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efdfc0a11a0 0x7efdfc0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:06.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.278+0000 7efe18a00700 1 --2- 172.21.15.116:0/3586767025 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efdfc0a11a0 0x7efdfc0a3660 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efe0800b2e0 tx=0x7efe0800b750 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.281+0000 7efe11600700 1 -- 172.21.15.116:0/3586767025 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efe0c096020 con 0x7efe14108b40 2024-03-30T00:56:06.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:06 smithi116 ceph-mon[38086]: pgmap v236: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 220 B/s wr, 0 op/s 2024-03-30T00:56:06.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.525+0000 7efe1b400700 1 -- 172.21.15.116:0/3586767025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe14067a10 con 0x7efe14108b40 2024-03-30T00:56:06.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.527+0000 7efe11600700 1 -- 172.21.15.116:0/3586767025 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+24028 (secure 0 0 0) 0x7efe0c023070 con 0x7efe14108b40 2024-03-30T00:56:06.532 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:56:06.533 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:55:56.324776+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:56:06.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.529+0000 7efe1b400700 1 -- 172.21.15.116:0/3586767025 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efdfc0a11a0 msgr2=0x7efdfc0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:06.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.529+0000 7efe1b400700 1 --2- 172.21.15.116:0/3586767025 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efdfc0a11a0 0x7efdfc0a3660 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efe0800b2e0 tx=0x7efe0800b750 comp rx=0 tx=0).stop 2024-03-30T00:56:06.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.529+0000 7efe1b400700 1 -- 172.21.15.116:0/3586767025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe14108b40 msgr2=0x7efe141a0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:06.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.529+0000 7efe1b400700 1 --2- 172.21.15.116:0/3586767025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe14108b40 0x7efe141a0370 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efe0c007f10 tx=0x7efe0c00d3b0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.529+0000 7efe1b400700 1 -- 172.21.15.116:0/3586767025 shutdown_connections 2024-03-30T00:56:06.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.529+0000 7efe1b400700 1 --2- 172.21.15.116:0/3586767025 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efdfc0a11a0 0x7efdfc0a3660 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.529+0000 7efe1b400700 1 --2- 172.21.15.116:0/3586767025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe14108b40 0x7efe141a0370 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.529+0000 7efe1b400700 1 --2- 172.21.15.116:0/3586767025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efe14073810 0x7efe14198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.529+0000 7efe1b400700 1 --2- 172.21.15.116:0/3586767025 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe14074df0 0x7efe14199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.529+0000 7efe1b400700 1 -- 172.21.15.116:0/3586767025 >> 172.21.15.116:0/3586767025 conn(0x7efe140fc1f0 msgr2=0x7efe14104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:06.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.529+0000 7efe1b400700 1 -- 172.21.15.116:0/3586767025 shutdown_connections 2024-03-30T00:56:06.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.529+0000 7efe1b400700 1 -- 172.21.15.116:0/3586767025 wait complete. 2024-03-30T00:56:06.553 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T00:52:29.602428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T00:55:50.220543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T00:55:51.274498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T00:55:54.301591+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T00:56:06.554 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T00:56:06.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:06 smithi130 ceph-mon[47626]: pgmap v236: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 220 B/s wr, 0 op/s 2024-03-30T00:56:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.742+0000 7f6ffbe00700 1 -- 172.21.15.116:0/132345275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff4074df0 msgr2=0x7f6ff4073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.742+0000 7f6ffbe00700 1 --2- 172.21.15.116:0/132345275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff4074df0 0x7f6ff4073240 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009990 tx=0x7f6fe8009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.742+0000 7f6ffbe00700 1 -- 172.21.15.116:0/132345275 shutdown_connections 2024-03-30T00:56:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.742+0000 7f6ffbe00700 1 --2- 172.21.15.116:0/132345275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ff4108c40 0x7f6ff410b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.742+0000 7f6ffbe00700 1 --2- 172.21.15.116:0/132345275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff4074df0 0x7f6ff4073240 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.742+0000 7f6ffbe00700 1 --2- 172.21.15.116:0/132345275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6ff4073810 0x7f6ff4073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.742+0000 7f6ffbe00700 1 -- 172.21.15.116:0/132345275 >> 172.21.15.116:0/132345275 conn(0x7f6ff40fc2f0 msgr2=0x7f6ff40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.742+0000 7f6ffbe00700 1 -- 172.21.15.116:0/132345275 shutdown_connections 2024-03-30T00:56:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.742+0000 7f6ffbe00700 1 -- 172.21.15.116:0/132345275 wait complete. 2024-03-30T00:56:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.743+0000 7f6ffbe00700 1 Processor -- start 2024-03-30T00:56:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.743+0000 7f6ffbe00700 1 -- start start 2024-03-30T00:56:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ffbe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6ff4073810 0x7f6ff4198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ffbe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ff4074df0 0x7f6ff41994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ffbe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff4108c40 0x7f6ff41a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ffbe00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41a0a50 con 0x7f6ff4108c40 2024-03-30T00:56:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ffbe00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41a0b90 con 0x7f6ff4073810 2024-03-30T00:56:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ffbe00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41a0d20 con 0x7f6ff4074df0 2024-03-30T00:56:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ff9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff4108c40 0x7f6ff41a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ff9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff4108c40 0x7f6ff41a0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37794/0 (socket says 172.21.15.116:37794) 2024-03-30T00:56:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ff9e00700 1 -- 172.21.15.116:0/2865808535 learned_addr learned my addr 172.21.15.116:0/2865808535 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ff9400700 1 --2- 172.21.15.116:0/2865808535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6ff4073810 0x7f6ff4198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ff8a00700 1 --2- 172.21.15.116:0/2865808535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ff4074df0 0x7f6ff41994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ff9e00700 1 -- 172.21.15.116:0/2865808535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ff4074df0 msgr2=0x7f6ff41994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ff9e00700 1 --2- 172.21.15.116:0/2865808535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ff4074df0 0x7f6ff41994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ff9e00700 1 -- 172.21.15.116:0/2865808535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6ff4073810 msgr2=0x7f6ff4198f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ff9e00700 1 --2- 172.21.15.116:0/2865808535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6ff4073810 0x7f6ff4198f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ff9e00700 1 -- 172.21.15.116:0/2865808535 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe80095f0 con 0x7f6ff4108c40 2024-03-30T00:56:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.744+0000 7f6ff9e00700 1 --2- 172.21.15.116:0/2865808535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff4108c40 0x7f6ff41a0480 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00ef30 tx=0x7f6fec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.745+0000 7f6ff2000700 1 -- 172.21.15.116:0/2865808535 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec018070 con 0x7f6ff4108c40 2024-03-30T00:56:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.745+0000 7f6ff2000700 1 -- 172.21.15.116:0/2865808535 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fec00f040 con 0x7f6ff4108c40 2024-03-30T00:56:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.745+0000 7f6ffbe00700 1 -- 172.21.15.116:0/2865808535 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff41a0fa0 con 0x7f6ff4108c40 2024-03-30T00:56:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.745+0000 7f6ff2000700 1 -- 172.21.15.116:0/2865808535 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec00cd00 con 0x7f6ff4108c40 2024-03-30T00:56:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.745+0000 7f6ffbe00700 1 -- 172.21.15.116:0/2865808535 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff41a1420 con 0x7f6ff4108c40 2024-03-30T00:56:06.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:06 smithi181 ceph-mon[47622]: pgmap v236: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 220 B/s wr, 0 op/s 2024-03-30T00:56:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.746+0000 7f6ffbe00700 1 -- 172.21.15.116:0/2865808535 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff4107a00 con 0x7f6ff4108c40 2024-03-30T00:56:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.748+0000 7f6ff2000700 1 -- 172.21.15.116:0/2865808535 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f6fec003a00 con 0x7f6ff4108c40 2024-03-30T00:56:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.749+0000 7f6ff2000700 1 --2- 172.21.15.116:0/2865808535 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6fdc0a1120 0x7f6fdc0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.749+0000 7f6ff2000700 1 -- 172.21.15.116:0/2865808535 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6fec0cc620 con 0x7f6ff4108c40 2024-03-30T00:56:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.749+0000 7f6ff9400700 1 --2- 172.21.15.116:0/2865808535 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6fdc0a1120 0x7f6fdc0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.750+0000 7f6ff9400700 1 --2- 172.21.15.116:0/2865808535 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6fdc0a1120 0x7f6fdc0a35e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8006010 tx=0x7f6fe80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.753+0000 7f6ff2000700 1 -- 172.21.15.116:0/2865808535 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6fec095bf0 con 0x7f6ff4108c40 2024-03-30T00:56:06.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.961+0000 7f6ffbe00700 1 -- 172.21.15.116:0/2865808535 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6ff402ccb0 con 0x7f6ff4108c40 2024-03-30T00:56:06.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.961+0000 7f6ff2000700 1 -- 172.21.15.116:0/2865808535 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6fec099210 con 0x7f6ff4108c40 2024-03-30T00:56:06.967 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-30T00:56:06.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.964+0000 7f6ffbe00700 1 -- 172.21.15.116:0/2865808535 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6fdc0a1120 msgr2=0x7f6fdc0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:06.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.964+0000 7f6ffbe00700 1 --2- 172.21.15.116:0/2865808535 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6fdc0a1120 0x7f6fdc0a35e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8006010 tx=0x7f6fe80058e0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.964+0000 7f6ffbe00700 1 -- 172.21.15.116:0/2865808535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff4108c40 msgr2=0x7f6ff41a0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:06.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.964+0000 7f6ffbe00700 1 --2- 172.21.15.116:0/2865808535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff4108c40 0x7f6ff41a0480 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00ef30 tx=0x7f6fec00c3a0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.964+0000 7f6ffbe00700 1 -- 172.21.15.116:0/2865808535 shutdown_connections 2024-03-30T00:56:06.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.964+0000 7f6ffbe00700 1 --2- 172.21.15.116:0/2865808535 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6fdc0a1120 0x7f6fdc0a35e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.964+0000 7f6ffbe00700 1 --2- 172.21.15.116:0/2865808535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ff4108c40 0x7f6ff41a0480 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.964+0000 7f6ffbe00700 1 --2- 172.21.15.116:0/2865808535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6ff4073810 0x7f6ff4198f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.964+0000 7f6ffbe00700 1 --2- 172.21.15.116:0/2865808535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ff4074df0 0x7f6ff41994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:06.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.964+0000 7f6ffbe00700 1 -- 172.21.15.116:0/2865808535 >> 172.21.15.116:0/2865808535 conn(0x7f6ff40fc2f0 msgr2=0x7f6ff4104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:06.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.964+0000 7f6ffbe00700 1 -- 172.21.15.116:0/2865808535 shutdown_connections 2024-03-30T00:56:06.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:06.964+0000 7f6ffbe00700 1 -- 172.21.15.116:0/2865808535 wait complete. 2024-03-30T00:56:06.987 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T00:56:07.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.180+0000 7fb3aae00700 1 -- 172.21.15.116:0/1668871564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4102910 msgr2=0x7fb3a41091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:07.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.180+0000 7fb3aae00700 1 --2- 172.21.15.116:0/1668871564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4102910 0x7fb3a41091c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00b0d0 tx=0x7fb39c00b3e0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.180+0000 7fb3aae00700 1 -- 172.21.15.116:0/1668871564 shutdown_connections 2024-03-30T00:56:07.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.180+0000 7fb3aae00700 1 --2- 172.21.15.116:0/1668871564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3a4100d80 0x7fb3a41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.180+0000 7fb3aae00700 1 --2- 172.21.15.116:0/1668871564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4102910 0x7fb3a41091c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.180+0000 7fb3aae00700 1 --2- 172.21.15.116:0/1668871564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3a4101f70 0x7fb3a41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.180+0000 7fb3aae00700 1 -- 172.21.15.116:0/1668871564 >> 172.21.15.116:0/1668871564 conn(0x7fb3a40fc2f0 msgr2=0x7fb3a40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:07.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.180+0000 7fb3aae00700 1 -- 172.21.15.116:0/1668871564 shutdown_connections 2024-03-30T00:56:07.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.180+0000 7fb3aae00700 1 -- 172.21.15.116:0/1668871564 wait complete. 2024-03-30T00:56:07.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.181+0000 7fb3aae00700 1 Processor -- start 2024-03-30T00:56:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.181+0000 7fb3aae00700 1 -- start start 2024-03-30T00:56:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.181+0000 7fb3aae00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3a4100d80 0x7fb3a4074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.181+0000 7fb3aae00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4101f70 0x7fb3a40731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.181+0000 7fb3aae00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3a4102910 0x7fb3a40736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.181+0000 7fb3aae00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a4073e80 con 0x7fb3a4101f70 2024-03-30T00:56:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.181+0000 7fb3aae00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a4073ff0 con 0x7fb3a4100d80 2024-03-30T00:56:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.181+0000 7fb3aae00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a41a0eb0 con 0x7fb3a4102910 2024-03-30T00:56:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.182+0000 7fb3a3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4101f70 0x7fb3a40731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.182+0000 7fb3a3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4101f70 0x7fb3a40731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37806/0 (socket says 172.21.15.116:37806) 2024-03-30T00:56:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.182+0000 7fb3a3400700 1 -- 172.21.15.116:0/563275937 learned_addr learned my addr 172.21.15.116:0/563275937 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.182+0000 7fb3a3e00700 1 --2- 172.21.15.116:0/563275937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3a4100d80 0x7fb3a4074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.182+0000 7fb3a8e00700 1 --2- 172.21.15.116:0/563275937 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3a4102910 0x7fb3a40736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.182+0000 7fb3a3400700 1 -- 172.21.15.116:0/563275937 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3a4102910 msgr2=0x7fb3a40736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.182+0000 7fb3a3400700 1 --2- 172.21.15.116:0/563275937 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3a4102910 0x7fb3a40736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.182+0000 7fb3a3400700 1 -- 172.21.15.116:0/563275937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3a4100d80 msgr2=0x7fb3a4074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.182+0000 7fb3a3400700 1 --2- 172.21.15.116:0/563275937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3a4100d80 0x7fb3a4074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.182+0000 7fb3a3400700 1 -- 172.21.15.116:0/563275937 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c009d00 con 0x7fb3a4101f70 2024-03-30T00:56:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.182+0000 7fb3a3400700 1 --2- 172.21.15.116:0/563275937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4101f70 0x7fb3a40731a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb39000d950 tx=0x7fb39000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.183+0000 7fb3a0c00700 1 -- 172.21.15.116:0/563275937 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39000f890 con 0x7fb3a4101f70 2024-03-30T00:56:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.183+0000 7fb3aae00700 1 -- 172.21.15.116:0/563275937 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a41a1140 con 0x7fb3a4101f70 2024-03-30T00:56:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.183+0000 7fb3a0c00700 1 -- 172.21.15.116:0/563275937 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb390010460 con 0x7fb3a4101f70 2024-03-30T00:56:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.183+0000 7fb3a0c00700 1 -- 172.21.15.116:0/563275937 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3900195c0 con 0x7fb3a4101f70 2024-03-30T00:56:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.183+0000 7fb3aae00700 1 -- 172.21.15.116:0/563275937 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a41a1650 con 0x7fb3a4101f70 2024-03-30T00:56:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.184+0000 7fb3aae00700 1 -- 172.21.15.116:0/563275937 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a4108180 con 0x7fb3a4101f70 2024-03-30T00:56:07.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.186+0000 7fb3a0c00700 1 -- 172.21.15.116:0/563275937 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fb3900108d0 con 0x7fb3a4101f70 2024-03-30T00:56:07.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.187+0000 7fb3a0c00700 1 --2- 172.21.15.116:0/563275937 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb3880a1120 0x7fb3880a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:07.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.187+0000 7fb3a0c00700 1 -- 172.21.15.116:0/563275937 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb3900cc8b0 con 0x7fb3a4101f70 2024-03-30T00:56:07.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.188+0000 7fb3a3e00700 1 --2- 172.21.15.116:0/563275937 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb3880a1120 0x7fb3880a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:07.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.188+0000 7fb3a3e00700 1 --2- 172.21.15.116:0/563275937 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb3880a1120 0x7fb3880a35e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4101dd0 tx=0x7fb394009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:07.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.192+0000 7fb3a0c00700 1 -- 172.21.15.116:0/563275937 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb390095df0 con 0x7fb3a4101f70 2024-03-30T00:56:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.402+0000 7fb3aae00700 1 -- 172.21.15.116:0/563275937 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb3a410c5f0 con 0x7fb3a4101f70 2024-03-30T00:56:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.403+0000 7fb3a0c00700 1 -- 172.21.15.116:0/563275937 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb390016020 con 0x7fb3a4101f70 2024-03-30T00:56:07.409 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:56:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.405+0000 7fb3aae00700 1 -- 172.21.15.116:0/563275937 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb3880a1120 msgr2=0x7fb3880a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.405+0000 7fb3aae00700 1 --2- 172.21.15.116:0/563275937 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb3880a1120 0x7fb3880a35e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4101dd0 tx=0x7fb394009260 comp rx=0 tx=0).stop 2024-03-30T00:56:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.405+0000 7fb3aae00700 1 -- 172.21.15.116:0/563275937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4101f70 msgr2=0x7fb3a40731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.405+0000 7fb3aae00700 1 --2- 172.21.15.116:0/563275937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4101f70 0x7fb3a40731a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb39000d950 tx=0x7fb39000dc60 comp rx=0 tx=0).stop 2024-03-30T00:56:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.405+0000 7fb3aae00700 1 -- 172.21.15.116:0/563275937 shutdown_connections 2024-03-30T00:56:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.405+0000 7fb3aae00700 1 --2- 172.21.15.116:0/563275937 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb3880a1120 0x7fb3880a35e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.405+0000 7fb3aae00700 1 --2- 172.21.15.116:0/563275937 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3a4102910 0x7fb3a40736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.405+0000 7fb3aae00700 1 --2- 172.21.15.116:0/563275937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3a4100d80 0x7fb3a4074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.405+0000 7fb3aae00700 1 --2- 172.21.15.116:0/563275937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4101f70 0x7fb3a40731a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.405+0000 7fb3aae00700 1 -- 172.21.15.116:0/563275937 >> 172.21.15.116:0/563275937 conn(0x7fb3a40fc2f0 msgr2=0x7fb3a41052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.405+0000 7fb3aae00700 1 -- 172.21.15.116:0/563275937 shutdown_connections 2024-03-30T00:56:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.405+0000 7fb3aae00700 1 -- 172.21.15.116:0/563275937 wait complete. 2024-03-30T00:56:07.429 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T00:56:07.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:07 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3586767025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:07.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:07 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2865808535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:56:07.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.619+0000 7fdc36200700 1 -- 172.21.15.116:0/3781581962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc30102910 msgr2=0x7fdc301091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.619+0000 7fdc36200700 1 --2- 172.21.15.116:0/3781581962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc30102910 0x7fdc301091c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800b0d0 tx=0x7fdc2800b3e0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.619+0000 7fdc36200700 1 -- 172.21.15.116:0/3781581962 shutdown_connections 2024-03-30T00:56:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.619+0000 7fdc36200700 1 --2- 172.21.15.116:0/3781581962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc30100d80 0x7fdc301011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.619+0000 7fdc36200700 1 --2- 172.21.15.116:0/3781581962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc30102910 0x7fdc301091c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.619+0000 7fdc36200700 1 --2- 172.21.15.116:0/3781581962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc30101f70 0x7fdc301023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.619+0000 7fdc36200700 1 -- 172.21.15.116:0/3781581962 >> 172.21.15.116:0/3781581962 conn(0x7fdc300fc2f0 msgr2=0x7fdc300fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.619+0000 7fdc36200700 1 -- 172.21.15.116:0/3781581962 shutdown_connections 2024-03-30T00:56:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.619+0000 7fdc36200700 1 -- 172.21.15.116:0/3781581962 wait complete. 2024-03-30T00:56:07.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.620+0000 7fdc36200700 1 Processor -- start 2024-03-30T00:56:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.620+0000 7fdc36200700 1 -- start start 2024-03-30T00:56:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.620+0000 7fdc36200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc30100d80 0x7fdc30074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.620+0000 7fdc36200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc30101f70 0x7fdc300731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.620+0000 7fdc36200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc30102910 0x7fdc300736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.620+0000 7fdc36200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc30073e80 con 0x7fdc30101f70 2024-03-30T00:56:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.620+0000 7fdc36200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc30073ff0 con 0x7fdc30102910 2024-03-30T00:56:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.620+0000 7fdc36200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc301a0eb0 con 0x7fdc30100d80 2024-03-30T00:56:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.620+0000 7fdc2ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc30101f70 0x7fdc300731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.621+0000 7fdc2f400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc30100d80 0x7fdc30074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.621+0000 7fdc2f400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc30100d80 0x7fdc30074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:56270/0 (socket says 172.21.15.116:56270) 2024-03-30T00:56:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.621+0000 7fdc2f400700 1 -- 172.21.15.116:0/3825773501 learned_addr learned my addr 172.21.15.116:0/3825773501 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.621+0000 7fdc2fe00700 1 --2- 172.21.15.116:0/3825773501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc30102910 0x7fdc300736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:07.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.621+0000 7fdc2fe00700 1 -- 172.21.15.116:0/3825773501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc30100d80 msgr2=0x7fdc30074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:07.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.621+0000 7fdc2fe00700 1 --2- 172.21.15.116:0/3825773501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc30100d80 0x7fdc30074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.621+0000 7fdc2fe00700 1 -- 172.21.15.116:0/3825773501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc30101f70 msgr2=0x7fdc300731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:07.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.621+0000 7fdc2fe00700 1 --2- 172.21.15.116:0/3825773501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc30101f70 0x7fdc300731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.621+0000 7fdc2fe00700 1 -- 172.21.15.116:0/3825773501 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc28009d00 con 0x7fdc30102910 2024-03-30T00:56:07.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.621+0000 7fdc2ea00700 1 --2- 172.21.15.116:0/3825773501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc30101f70 0x7fdc300731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:07.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.621+0000 7fdc2fe00700 1 --2- 172.21.15.116:0/3825773501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc30102910 0x7fdc300736e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdc28007b60 tx=0x7fdc28008e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.622+0000 7fdc27e00700 1 -- 172.21.15.116:0/3825773501 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2800e040 con 0x7fdc30102910 2024-03-30T00:56:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.622+0000 7fdc27e00700 1 -- 172.21.15.116:0/3825773501 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc28005070 con 0x7fdc30102910 2024-03-30T00:56:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.622+0000 7fdc36200700 1 -- 172.21.15.116:0/3825773501 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc301a10e0 con 0x7fdc30102910 2024-03-30T00:56:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.622+0000 7fdc27e00700 1 -- 172.21.15.116:0/3825773501 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2801d6d0 con 0x7fdc30102910 2024-03-30T00:56:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.622+0000 7fdc36200700 1 -- 172.21.15.116:0/3825773501 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc301a14b0 con 0x7fdc30102910 2024-03-30T00:56:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.623+0000 7fdc36200700 1 -- 172.21.15.116:0/3825773501 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc30108180 con 0x7fdc30102910 2024-03-30T00:56:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.624+0000 7fdc27e00700 1 -- 172.21.15.116:0/3825773501 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fdc280051e0 con 0x7fdc30102910 2024-03-30T00:56:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.626+0000 7fdc27e00700 1 --2- 172.21.15.116:0/3825773501 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdc140a1120 0x7fdc140a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.626+0000 7fdc27e00700 1 -- 172.21.15.116:0/3825773501 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdc280cda80 con 0x7fdc30102910 2024-03-30T00:56:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.626+0000 7fdc2f400700 1 --2- 172.21.15.116:0/3825773501 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdc140a1120 0x7fdc140a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:07.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.627+0000 7fdc2f400700 1 --2- 172.21.15.116:0/3825773501 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdc140a1120 0x7fdc140a35e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdc20005950 tx=0x7fdc20009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.630+0000 7fdc27e00700 1 -- 172.21.15.116:0/3825773501 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdc28097050 con 0x7fdc30102910 2024-03-30T00:56:07.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:07 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3586767025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:07.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:07 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2865808535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:56:07.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:07 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3586767025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:07.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:07 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2865808535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:56:07.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.843+0000 7fdc36200700 1 -- 172.21.15.116:0/3825773501 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdc3010c7f0 con 0x7fdc30102910 2024-03-30T00:56:07.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.843+0000 7fdc27e00700 1 -- 172.21.15.116:0/3825773501 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdc2809a670 con 0x7fdc30102910 2024-03-30T00:56:07.849 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-30T00:56:07.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.846+0000 7fdc36200700 1 -- 172.21.15.116:0/3825773501 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdc140a1120 msgr2=0x7fdc140a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:07.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.846+0000 7fdc36200700 1 --2- 172.21.15.116:0/3825773501 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdc140a1120 0x7fdc140a35e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdc20005950 tx=0x7fdc20009260 comp rx=0 tx=0).stop 2024-03-30T00:56:07.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.846+0000 7fdc36200700 1 -- 172.21.15.116:0/3825773501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc30102910 msgr2=0x7fdc300736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:07.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.846+0000 7fdc36200700 1 --2- 172.21.15.116:0/3825773501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc30102910 0x7fdc300736e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdc28007b60 tx=0x7fdc28008e90 comp rx=0 tx=0).stop 2024-03-30T00:56:07.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.846+0000 7fdc36200700 1 -- 172.21.15.116:0/3825773501 shutdown_connections 2024-03-30T00:56:07.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.846+0000 7fdc36200700 1 --2- 172.21.15.116:0/3825773501 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdc140a1120 0x7fdc140a35e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.846+0000 7fdc36200700 1 --2- 172.21.15.116:0/3825773501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc30100d80 0x7fdc30074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.846+0000 7fdc36200700 1 --2- 172.21.15.116:0/3825773501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc30102910 0x7fdc300736e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.846+0000 7fdc36200700 1 --2- 172.21.15.116:0/3825773501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc30101f70 0x7fdc300731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:07.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.846+0000 7fdc36200700 1 -- 172.21.15.116:0/3825773501 >> 172.21.15.116:0/3825773501 conn(0x7fdc300fc2f0 msgr2=0x7fdc301052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:07.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.846+0000 7fdc36200700 1 -- 172.21.15.116:0/3825773501 shutdown_connections 2024-03-30T00:56:07.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:07.846+0000 7fdc36200700 1 -- 172.21.15.116:0/3825773501 wait complete. 2024-03-30T00:56:07.870 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T00:56:08.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.058+0000 7f61b8e00700 1 -- 172.21.15.116:0/1352432819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b4100d80 msgr2=0x7f61b41011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:08.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.058+0000 7f61b8e00700 1 --2- 172.21.15.116:0/1352432819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b4100d80 0x7f61b41011a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f61a40099c0 tx=0x7f61a4009cd0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.058+0000 7f61b8e00700 1 -- 172.21.15.116:0/1352432819 shutdown_connections 2024-03-30T00:56:08.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.058+0000 7f61b8e00700 1 --2- 172.21.15.116:0/1352432819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61b4102910 0x7f61b41091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.058+0000 7f61b8e00700 1 --2- 172.21.15.116:0/1352432819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b4100d80 0x7f61b41011a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.058+0000 7f61b8e00700 1 --2- 172.21.15.116:0/1352432819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61b4101f70 0x7f61b41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.058+0000 7f61b8e00700 1 -- 172.21.15.116:0/1352432819 >> 172.21.15.116:0/1352432819 conn(0x7f61b40fc2f0 msgr2=0x7f61b40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:08.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.058+0000 7f61b8e00700 1 -- 172.21.15.116:0/1352432819 shutdown_connections 2024-03-30T00:56:08.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.058+0000 7f61b8e00700 1 -- 172.21.15.116:0/1352432819 wait complete. 2024-03-30T00:56:08.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.059+0000 7f61b8e00700 1 Processor -- start 2024-03-30T00:56:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.059+0000 7f61b8e00700 1 -- start start 2024-03-30T00:56:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.059+0000 7f61b8e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b4100d80 0x7f61b4198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.059+0000 7f61b8e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61b4101f70 0x7f61b41994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.059+0000 7f61b8e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61b4102910 0x7f61b41a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.059+0000 7f61b8e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b41a0a20 con 0x7f61b4100d80 2024-03-30T00:56:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.059+0000 7f61b8e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b41a0b60 con 0x7f61b4101f70 2024-03-30T00:56:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.059+0000 7f61b8e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b41a0cf0 con 0x7f61b4102910 2024-03-30T00:56:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.060+0000 7f61b1e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b4100d80 0x7f61b4198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.060+0000 7f61b1400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61b4101f70 0x7f61b41994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.060+0000 7f61b1e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b4100d80 0x7f61b4198f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37836/0 (socket says 172.21.15.116:37836) 2024-03-30T00:56:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.060+0000 7f61b1e00700 1 -- 172.21.15.116:0/792759801 learned_addr learned my addr 172.21.15.116:0/792759801 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.060+0000 7f61b2800700 1 --2- 172.21.15.116:0/792759801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61b4102910 0x7f61b41a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.060+0000 7f61b1e00700 1 -- 172.21.15.116:0/792759801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61b4102910 msgr2=0x7f61b41a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.060+0000 7f61b1e00700 1 --2- 172.21.15.116:0/792759801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61b4102910 0x7f61b41a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.060+0000 7f61b1e00700 1 -- 172.21.15.116:0/792759801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61b4101f70 msgr2=0x7f61b41994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.060+0000 7f61b1e00700 1 --2- 172.21.15.116:0/792759801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61b4101f70 0x7f61b41994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.060+0000 7f61b1e00700 1 -- 172.21.15.116:0/792759801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a40095f0 con 0x7f61b4100d80 2024-03-30T00:56:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.060+0000 7f61b2800700 1 --2- 172.21.15.116:0/792759801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61b4102910 0x7f61b41a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:56:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.060+0000 7f61b1e00700 1 --2- 172.21.15.116:0/792759801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b4100d80 0x7f61b4198f90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f61a4006010 tx=0x7f61a4005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.060+0000 7f61aaa00700 1 -- 172.21.15.116:0/792759801 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a401d070 con 0x7f61b4100d80 2024-03-30T00:56:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.061+0000 7f61aaa00700 1 -- 172.21.15.116:0/792759801 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61a4004050 con 0x7f61b4100d80 2024-03-30T00:56:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.061+0000 7f61b8e00700 1 -- 172.21.15.116:0/792759801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b41a0f70 con 0x7f61b4100d80 2024-03-30T00:56:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.061+0000 7f61aaa00700 1 -- 172.21.15.116:0/792759801 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4022e00 con 0x7f61b4100d80 2024-03-30T00:56:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.061+0000 7f61b8e00700 1 -- 172.21.15.116:0/792759801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b41a12e0 con 0x7f61b4100d80 2024-03-30T00:56:08.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.062+0000 7f61b8e00700 1 -- 172.21.15.116:0/792759801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b4067af0 con 0x7f61b4100d80 2024-03-30T00:56:08.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.063+0000 7f61aaa00700 1 -- 172.21.15.116:0/792759801 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f61a4022500 con 0x7f61b4100d80 2024-03-30T00:56:08.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.065+0000 7f61aaa00700 1 --2- 172.21.15.116:0/792759801 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f61980a1120 0x7f61980a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:08.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.065+0000 7f61aaa00700 1 -- 172.21.15.116:0/792759801 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f61a40cdef0 con 0x7f61b4100d80 2024-03-30T00:56:08.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.065+0000 7f61b1400700 1 --2- 172.21.15.116:0/792759801 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f61980a1120 0x7f61980a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:08.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.065+0000 7f61b1400700 1 --2- 172.21.15.116:0/792759801 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f61980a1120 0x7f61980a35e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f61a0007c60 tx=0x7f61a00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:08.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.068+0000 7f61aaa00700 1 -- 172.21.15.116:0/792759801 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f61a40974c0 con 0x7f61b4100d80 2024-03-30T00:56:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.273+0000 7f61b8e00700 1 -- 172.21.15.116:0/792759801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f61b410c5e0 con 0x7f61b4100d80 2024-03-30T00:56:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.274+0000 7f61aaa00700 1 -- 172.21.15.116:0/792759801 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v81) v1 ==== 103+0+11 (secure 0 0 0) 0x7f61b410c5e0 con 0x7f61b4100d80 2024-03-30T00:56:08.280 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:56:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.276+0000 7f61b8e00700 1 -- 172.21.15.116:0/792759801 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f61980a1120 msgr2=0x7f61980a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.276+0000 7f61b8e00700 1 --2- 172.21.15.116:0/792759801 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f61980a1120 0x7f61980a35e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f61a0007c60 tx=0x7f61a00073d0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.276+0000 7f61b8e00700 1 -- 172.21.15.116:0/792759801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b4100d80 msgr2=0x7f61b4198f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.276+0000 7f61b8e00700 1 --2- 172.21.15.116:0/792759801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b4100d80 0x7f61b4198f90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f61a4006010 tx=0x7f61a4005740 comp rx=0 tx=0).stop 2024-03-30T00:56:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.276+0000 7f61b8e00700 1 -- 172.21.15.116:0/792759801 shutdown_connections 2024-03-30T00:56:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.276+0000 7f61b8e00700 1 --2- 172.21.15.116:0/792759801 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f61980a1120 0x7f61980a35e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.276+0000 7f61b8e00700 1 --2- 172.21.15.116:0/792759801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61b4102910 0x7f61b41a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.276+0000 7f61b8e00700 1 --2- 172.21.15.116:0/792759801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b4100d80 0x7f61b4198f90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.276+0000 7f61b8e00700 1 --2- 172.21.15.116:0/792759801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61b4101f70 0x7f61b41994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.276+0000 7f61b8e00700 1 -- 172.21.15.116:0/792759801 >> 172.21.15.116:0/792759801 conn(0x7f61b40fc2f0 msgr2=0x7f61b4072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.276+0000 7f61b8e00700 1 -- 172.21.15.116:0/792759801 shutdown_connections 2024-03-30T00:56:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.277+0000 7f61b8e00700 1 -- 172.21.15.116:0/792759801 wait complete. 2024-03-30T00:56:08.300 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-30T00:56:08.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.487+0000 7ff216a00700 1 -- 172.21.15.116:0/3662789427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff210102910 msgr2=0x7ff2101091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:08.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.487+0000 7ff216a00700 1 --2- 172.21.15.116:0/3662789427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff210102910 0x7ff2101091c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff20800b0d0 tx=0x7ff20800b3e0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.487+0000 7ff216a00700 1 -- 172.21.15.116:0/3662789427 shutdown_connections 2024-03-30T00:56:08.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.487+0000 7ff216a00700 1 --2- 172.21.15.116:0/3662789427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff210100d80 0x7ff2101011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.487+0000 7ff216a00700 1 --2- 172.21.15.116:0/3662789427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff210102910 0x7ff2101091c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.487+0000 7ff216a00700 1 --2- 172.21.15.116:0/3662789427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210101f70 0x7ff2101023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.487+0000 7ff216a00700 1 -- 172.21.15.116:0/3662789427 >> 172.21.15.116:0/3662789427 conn(0x7ff2100fc2f0 msgr2=0x7ff2100fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.487+0000 7ff216a00700 1 -- 172.21.15.116:0/3662789427 shutdown_connections 2024-03-30T00:56:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.487+0000 7ff216a00700 1 -- 172.21.15.116:0/3662789427 wait complete. 2024-03-30T00:56:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.488+0000 7ff216a00700 1 Processor -- start 2024-03-30T00:56:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.489+0000 7ff216a00700 1 -- start start 2024-03-30T00:56:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.489+0000 7ff216a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff210100d80 0x7ff210199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.489+0000 7ff216a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210101f70 0x7ff210199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.489+0000 7ff216a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff210102910 0x7ff2101a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.489+0000 7ff216a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2101a0b60 con 0x7ff210101f70 2024-03-30T00:56:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.489+0000 7ff216a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2101a0ca0 con 0x7ff210100d80 2024-03-30T00:56:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.489+0000 7ff216a00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2101a0e30 con 0x7ff210102910 2024-03-30T00:56:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.489+0000 7ff20fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff210100d80 0x7ff210199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.489+0000 7ff20f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210101f70 0x7ff210199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.489+0000 7ff20fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff210100d80 0x7ff210199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:36494/0 (socket says 172.21.15.116:36494) 2024-03-30T00:56:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.489+0000 7ff20fe00700 1 -- 172.21.15.116:0/3837158794 learned_addr learned my addr 172.21.15.116:0/3837158794 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.489+0000 7ff214a00700 1 --2- 172.21.15.116:0/3837158794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff210102910 0x7ff2101a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.490+0000 7ff20fe00700 1 -- 172.21.15.116:0/3837158794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff210102910 msgr2=0x7ff2101a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.490+0000 7ff20fe00700 1 --2- 172.21.15.116:0/3837158794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff210102910 0x7ff2101a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.490+0000 7ff20fe00700 1 -- 172.21.15.116:0/3837158794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210101f70 msgr2=0x7ff210199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.490+0000 7ff20fe00700 1 --2- 172.21.15.116:0/3837158794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210101f70 0x7ff210199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.490+0000 7ff20fe00700 1 -- 172.21.15.116:0/3837158794 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff208009d00 con 0x7ff210100d80 2024-03-30T00:56:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.490+0000 7ff20f400700 1 --2- 172.21.15.116:0/3837158794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210101f70 0x7ff210199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.490+0000 7ff20fe00700 1 --2- 172.21.15.116:0/3837158794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff210100d80 0x7ff210199030 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff20000e970 tx=0x7ff20000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.490+0000 7ff20cc00700 1 -- 172.21.15.116:0/3837158794 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200009790 con 0x7ff210100d80 2024-03-30T00:56:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.490+0000 7ff216a00700 1 -- 172.21.15.116:0/3837158794 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2101a1110 con 0x7ff210100d80 2024-03-30T00:56:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.490+0000 7ff20cc00700 1 -- 172.21.15.116:0/3837158794 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff20000c7e0 con 0x7ff210100d80 2024-03-30T00:56:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.490+0000 7ff20cc00700 1 -- 172.21.15.116:0/3837158794 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200003c70 con 0x7ff210100d80 2024-03-30T00:56:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.490+0000 7ff216a00700 1 -- 172.21.15.116:0/3837158794 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2101a1670 con 0x7ff210100d80 2024-03-30T00:56:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.491+0000 7ff216a00700 1 -- 172.21.15.116:0/3837158794 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff210067af0 con 0x7ff210100d80 2024-03-30T00:56:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.493+0000 7ff20cc00700 1 -- 172.21.15.116:0/3837158794 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7ff20000cc50 con 0x7ff210100d80 2024-03-30T00:56:08.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.494+0000 7ff20cc00700 1 --2- 172.21.15.116:0/3837158794 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1f40a11a0 0x7ff1f40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:08.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.494+0000 7ff20cc00700 1 -- 172.21.15.116:0/3837158794 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff200014070 con 0x7ff210100d80 2024-03-30T00:56:08.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.495+0000 7ff20f400700 1 --2- 172.21.15.116:0/3837158794 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1f40a11a0 0x7ff1f40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:08.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.495+0000 7ff20f400700 1 --2- 172.21.15.116:0/3837158794 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1f40a11a0 0x7ff1f40a3660 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc011490 tx=0x7ff1fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:08.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.497+0000 7ff20cc00700 1 -- 172.21.15.116:0/3837158794 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff200096350 con 0x7ff210100d80 2024-03-30T00:56:08.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:08 smithi116 ceph-mon[38086]: pgmap v237: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 190 B/s wr, 0 op/s 2024-03-30T00:56:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:08 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/563275937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:56:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:08 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3825773501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:56:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:08 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/792759801' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:56:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:08 smithi181 ceph-mon[47622]: pgmap v237: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 190 B/s wr, 0 op/s 2024-03-30T00:56:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:08 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/563275937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:56:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:08 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3825773501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:56:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:08 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/792759801' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:56:08.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.762+0000 7ff216a00700 1 -- 172.21.15.116:0/3837158794 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff21010c5f0 con 0x7ff210100d80 2024-03-30T00:56:08.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.763+0000 7ff20cc00700 1 -- 172.21.15.116:0/3837158794 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v81) v1 ==== 81+0+598 (secure 0 0 0) 0x7ff200099970 con 0x7ff210100d80 2024-03-30T00:56:08.768 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/309656254 2024-03-31T00:50:35.029431+0000 2024-03-30T00:56:08.769 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/131150228 2024-03-31T00:50:35.029431+0000 2024-03-30T00:56:08.769 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/755518827 2024-03-31T00:50:35.029431+0000 2024-03-30T00:56:08.769 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/3183509903 2024-03-31T00:50:11.610313+0000 2024-03-30T00:56:08.769 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/4184421214 2024-03-31T00:50:11.610313+0000 2024-03-30T00:56:08.769 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/3183509903 2024-03-31T00:50:11.610313+0000 2024-03-30T00:56:08.769 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/131150228 2024-03-31T00:50:35.029431+0000 2024-03-30T00:56:08.769 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2567186791 2024-03-31T00:50:35.029431+0000 2024-03-30T00:56:08.769 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2997035221 2024-03-31T00:50:11.610313+0000 2024-03-30T00:56:08.769 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2367387359 2024-03-31T00:50:11.610313+0000 2024-03-30T00:56:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.765+0000 7ff216a00700 1 -- 172.21.15.116:0/3837158794 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1f40a11a0 msgr2=0x7ff1f40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.765+0000 7ff216a00700 1 --2- 172.21.15.116:0/3837158794 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1f40a11a0 0x7ff1f40a3660 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc011490 tx=0x7ff1fc009260 comp rx=0 tx=0).stop 2024-03-30T00:56:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.765+0000 7ff216a00700 1 -- 172.21.15.116:0/3837158794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff210100d80 msgr2=0x7ff210199030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.765+0000 7ff216a00700 1 --2- 172.21.15.116:0/3837158794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff210100d80 0x7ff210199030 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff20000e970 tx=0x7ff20000ec80 comp rx=0 tx=0).stop 2024-03-30T00:56:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.765+0000 7ff216a00700 1 -- 172.21.15.116:0/3837158794 shutdown_connections 2024-03-30T00:56:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.765+0000 7ff216a00700 1 --2- 172.21.15.116:0/3837158794 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1f40a11a0 0x7ff1f40a3660 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.765+0000 7ff216a00700 1 --2- 172.21.15.116:0/3837158794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff210102910 0x7ff2101a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.765+0000 7ff216a00700 1 --2- 172.21.15.116:0/3837158794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff210100d80 0x7ff210199030 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.765+0000 7ff216a00700 1 --2- 172.21.15.116:0/3837158794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210101f70 0x7ff210199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.765+0000 7ff216a00700 1 -- 172.21.15.116:0/3837158794 >> 172.21.15.116:0/3837158794 conn(0x7ff2100fc2f0 msgr2=0x7ff210072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.765+0000 7ff216a00700 1 -- 172.21.15.116:0/3837158794 shutdown_connections 2024-03-30T00:56:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:08.765+0000 7ff216a00700 1 -- 172.21.15.116:0/3837158794 wait complete. 2024-03-30T00:56:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:listed 10 entries 2024-03-30T00:56:08.789 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:56:08.789 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-30T00:56:08.834 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:56:08.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:08 smithi130 ceph-mon[47626]: pgmap v237: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 190 B/s wr, 0 op/s 2024-03-30T00:56:08.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:08 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/563275937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:56:08.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:08 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3825773501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:56:08.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:08 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/792759801' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:56:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.058+0000 7fb468000700 1 -- 172.21.15.116:0/913082995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb460102810 msgr2=0x7fb4601090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.058+0000 7fb468000700 1 --2- 172.21.15.116:0/913082995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb460102810 0x7fb4601090c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb45800b0d0 tx=0x7fb45800b3e0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.058+0000 7fb468000700 1 -- 172.21.15.116:0/913082995 shutdown_connections 2024-03-30T00:56:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.058+0000 7fb468000700 1 --2- 172.21.15.116:0/913082995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb460100c80 0x7fb4601010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.058+0000 7fb468000700 1 --2- 172.21.15.116:0/913082995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb460102810 0x7fb4601090c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.058+0000 7fb468000700 1 --2- 172.21.15.116:0/913082995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb460101e70 0x7fb4601022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.058+0000 7fb468000700 1 -- 172.21.15.116:0/913082995 >> 172.21.15.116:0/913082995 conn(0x7fb4600fc1f0 msgr2=0x7fb4600fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.058+0000 7fb468000700 1 -- 172.21.15.116:0/913082995 shutdown_connections 2024-03-30T00:56:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.058+0000 7fb468000700 1 -- 172.21.15.116:0/913082995 wait complete. 2024-03-30T00:56:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.059+0000 7fb468000700 1 Processor -- start 2024-03-30T00:56:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.059+0000 7fb468000700 1 -- start start 2024-03-30T00:56:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.059+0000 7fb468000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb460100c80 0x7fb460074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.059+0000 7fb468000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb460101e70 0x7fb4600731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.059+0000 7fb468000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb460102810 0x7fb4600736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.059+0000 7fb468000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb460073e80 con 0x7fb460101e70 2024-03-30T00:56:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.059+0000 7fb468000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb460073ff0 con 0x7fb460102810 2024-03-30T00:56:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.059+0000 7fb468000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4601a0d50 con 0x7fb460100c80 2024-03-30T00:56:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.060+0000 7fb464c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb460101e70 0x7fb4600731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.060+0000 7fb464c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb460101e70 0x7fb4600731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37872/0 (socket says 172.21.15.116:37872) 2024-03-30T00:56:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.060+0000 7fb464c00700 1 -- 172.21.15.116:0/3924167772 learned_addr learned my addr 172.21.15.116:0/3924167772 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.060+0000 7fb466000700 1 --2- 172.21.15.116:0/3924167772 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb460102810 0x7fb4600736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.060+0000 7fb465600700 1 --2- 172.21.15.116:0/3924167772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb460100c80 0x7fb460074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.060+0000 7fb464c00700 1 -- 172.21.15.116:0/3924167772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb460100c80 msgr2=0x7fb460074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.060+0000 7fb464c00700 1 --2- 172.21.15.116:0/3924167772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb460100c80 0x7fb460074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.060+0000 7fb464c00700 1 -- 172.21.15.116:0/3924167772 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb460102810 msgr2=0x7fb4600736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.060+0000 7fb464c00700 1 --2- 172.21.15.116:0/3924167772 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb460102810 0x7fb4600736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.060+0000 7fb464c00700 1 -- 172.21.15.116:0/3924167772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb458009d00 con 0x7fb460101e70 2024-03-30T00:56:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.060+0000 7fb464c00700 1 --2- 172.21.15.116:0/3924167772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb460101e70 0x7fb4600731a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00d980 tx=0x7fb44c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.061+0000 7fb45e000700 1 -- 172.21.15.116:0/3924167772 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c004220 con 0x7fb460101e70 2024-03-30T00:56:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.061+0000 7fb45e000700 1 -- 172.21.15.116:0/3924167772 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb44c007640 con 0x7fb460101e70 2024-03-30T00:56:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.061+0000 7fb45e000700 1 -- 172.21.15.116:0/3924167772 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c005690 con 0x7fb460101e70 2024-03-30T00:56:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.061+0000 7fb468000700 1 -- 172.21.15.116:0/3924167772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4601a0fe0 con 0x7fb460101e70 2024-03-30T00:56:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.061+0000 7fb468000700 1 -- 172.21.15.116:0/3924167772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4601a1410 con 0x7fb460101e70 2024-03-30T00:56:09.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.062+0000 7fb468000700 1 -- 172.21.15.116:0/3924167772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb460108030 con 0x7fb460101e70 2024-03-30T00:56:09.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.064+0000 7fb45e000700 1 -- 172.21.15.116:0/3924167772 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fb44c004750 con 0x7fb460101e70 2024-03-30T00:56:09.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.065+0000 7fb45e000700 1 --2- 172.21.15.116:0/3924167772 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4480a1120 0x7fb4480a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:09.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.065+0000 7fb45e000700 1 -- 172.21.15.116:0/3924167772 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb44c0cc870 con 0x7fb460101e70 2024-03-30T00:56:09.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.065+0000 7fb465600700 1 --2- 172.21.15.116:0/3924167772 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4480a1120 0x7fb4480a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:09.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.066+0000 7fb465600700 1 --2- 172.21.15.116:0/3924167772 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4480a1120 0x7fb4480a35e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb460101cd0 tx=0x7fb454009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.069+0000 7fb45e000700 1 -- 172.21.15.116:0/3924167772 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb44c095db0 con 0x7fb460101e70 2024-03-30T00:56:09.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.306+0000 7fb468000700 1 -- 172.21.15.116:0/3924167772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb460067a10 con 0x7fb460101e70 2024-03-30T00:56:09.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.307+0000 7fb45e000700 1 -- 172.21.15.116:0/3924167772 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+24028 (secure 0 0 0) 0x7fb44c016090 con 0x7fb460101e70 2024-03-30T00:56:09.315 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:56:09.315 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:55:56.324776+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:56:09.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.310+0000 7fb468000700 1 -- 172.21.15.116:0/3924167772 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4480a1120 msgr2=0x7fb4480a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:09.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.310+0000 7fb468000700 1 --2- 172.21.15.116:0/3924167772 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4480a1120 0x7fb4480a35e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb460101cd0 tx=0x7fb454009260 comp rx=0 tx=0).stop 2024-03-30T00:56:09.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.310+0000 7fb468000700 1 -- 172.21.15.116:0/3924167772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb460101e70 msgr2=0x7fb4600731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:09.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.310+0000 7fb468000700 1 --2- 172.21.15.116:0/3924167772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb460101e70 0x7fb4600731a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00d980 tx=0x7fb44c00dc90 comp rx=0 tx=0).stop 2024-03-30T00:56:09.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.310+0000 7fb468000700 1 -- 172.21.15.116:0/3924167772 shutdown_connections 2024-03-30T00:56:09.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.310+0000 7fb468000700 1 --2- 172.21.15.116:0/3924167772 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4480a1120 0x7fb4480a35e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.310+0000 7fb468000700 1 --2- 172.21.15.116:0/3924167772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb460100c80 0x7fb460074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.310+0000 7fb468000700 1 --2- 172.21.15.116:0/3924167772 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb460102810 0x7fb4600736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.310+0000 7fb468000700 1 --2- 172.21.15.116:0/3924167772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb460101e70 0x7fb4600731a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.310+0000 7fb468000700 1 -- 172.21.15.116:0/3924167772 >> 172.21.15.116:0/3924167772 conn(0x7fb4600fc1f0 msgr2=0x7fb460105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:09.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.310+0000 7fb468000700 1 -- 172.21.15.116:0/3924167772 shutdown_connections 2024-03-30T00:56:09.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.310+0000 7fb468000700 1 -- 172.21.15.116:0/3924167772 wait complete. 2024-03-30T00:56:09.335 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T00:52:29.602428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T00:55:50.220543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T00:55:51.274498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T00:55:54.301591+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T00:56:09.335 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T00:56:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.525+0000 7f1ef6200700 1 -- 172.21.15.116:0/2925800484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0100d80 msgr2=0x7f1ef01011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.525+0000 7f1ef6200700 1 --2- 172.21.15.116:0/2925800484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0100d80 0x7f1ef01011a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1ee00099c0 tx=0x7f1ee0009cd0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.525+0000 7f1ef6200700 1 -- 172.21.15.116:0/2925800484 shutdown_connections 2024-03-30T00:56:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.525+0000 7f1ef6200700 1 --2- 172.21.15.116:0/2925800484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ef0102910 0x7f1ef01091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.525+0000 7f1ef6200700 1 --2- 172.21.15.116:0/2925800484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0100d80 0x7f1ef01011a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.525+0000 7f1ef6200700 1 --2- 172.21.15.116:0/2925800484 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ef0101f70 0x7f1ef01023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.525+0000 7f1ef6200700 1 -- 172.21.15.116:0/2925800484 >> 172.21.15.116:0/2925800484 conn(0x7f1ef00fc2f0 msgr2=0x7f1ef00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.525+0000 7f1ef6200700 1 -- 172.21.15.116:0/2925800484 shutdown_connections 2024-03-30T00:56:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.525+0000 7f1ef6200700 1 -- 172.21.15.116:0/2925800484 wait complete. 2024-03-30T00:56:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.526+0000 7f1ef6200700 1 Processor -- start 2024-03-30T00:56:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.526+0000 7f1ef6200700 1 -- start start 2024-03-30T00:56:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.526+0000 7f1ef6200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ef0100d80 0x7f1ef0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.526+0000 7f1ef6200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ef0101f70 0x7f1ef00731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.526+0000 7f1ef6200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0102910 0x7f1ef00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.526+0000 7f1ef6200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef0073e80 con 0x7f1ef0102910 2024-03-30T00:56:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.526+0000 7f1ef6200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef0073ff0 con 0x7f1ef0100d80 2024-03-30T00:56:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.526+0000 7f1ef6200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef01a0db0 con 0x7f1ef0101f70 2024-03-30T00:56:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.527+0000 7f1eefe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0102910 0x7f1ef00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.527+0000 7f1eefe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0102910 0x7f1ef00736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37894/0 (socket says 172.21.15.116:37894) 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.527+0000 7f1eefe00700 1 -- 172.21.15.116:0/2939438832 learned_addr learned my addr 172.21.15.116:0/2939438832 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.527+0000 7f1eef400700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ef0100d80 0x7f1ef0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.527+0000 7f1eeea00700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ef0101f70 0x7f1ef00731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.527+0000 7f1eefe00700 1 -- 172.21.15.116:0/2939438832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ef0101f70 msgr2=0x7f1ef00731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.527+0000 7f1eefe00700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ef0101f70 0x7f1ef00731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.527+0000 7f1eefe00700 1 -- 172.21.15.116:0/2939438832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ef0100d80 msgr2=0x7f1ef0074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.527+0000 7f1eefe00700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ef0100d80 0x7f1ef0074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.527+0000 7f1eefe00700 1 -- 172.21.15.116:0/2939438832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee00095f0 con 0x7f1ef0102910 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.527+0000 7f1eef400700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ef0100d80 0x7f1ef0074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.527+0000 7f1eefe00700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0102910 0x7f1ef00736e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8007ee0 tx=0x7f1ee800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.527+0000 7f1ee7e00700 1 -- 172.21.15.116:0/2939438832 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee800dcc0 con 0x7f1ef0102910 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.528+0000 7f1ee7e00700 1 -- 172.21.15.116:0/2939438832 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ee800f040 con 0x7f1ef0102910 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.528+0000 7f1ef6200700 1 -- 172.21.15.116:0/2939438832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef01a0fe0 con 0x7f1ef0102910 2024-03-30T00:56:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.528+0000 7f1ee7e00700 1 -- 172.21.15.116:0/2939438832 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8004830 con 0x7f1ef0102910 2024-03-30T00:56:09.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.528+0000 7f1ef6200700 1 -- 172.21.15.116:0/2939438832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef01a14c0 con 0x7f1ef0102910 2024-03-30T00:56:09.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.529+0000 7f1ef6200700 1 -- 172.21.15.116:0/2939438832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef0108200 con 0x7f1ef0102910 2024-03-30T00:56:09.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.531+0000 7f1ee7e00700 1 -- 172.21.15.116:0/2939438832 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f1ee8003a00 con 0x7f1ef0102910 2024-03-30T00:56:09.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.532+0000 7f1ee7e00700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1ed40a11a0 0x7f1ed40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:09.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.532+0000 7f1ee7e00700 1 -- 172.21.15.116:0/2939438832 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1ee80cc280 con 0x7f1ef0102910 2024-03-30T00:56:09.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.532+0000 7f1eef400700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1ed40a11a0 0x7f1ed40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:09.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.532+0000 7f1eef400700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1ed40a11a0 0x7f1ed40a3660 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0003730 tx=0x7f1ee000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:09.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.536+0000 7f1ee7e00700 1 -- 172.21.15.116:0/2939438832 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1ee80957c0 con 0x7f1ef0102910 2024-03-30T00:56:09.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:09 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3837158794' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T00:56:09.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:09 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3924167772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.744+0000 7f1ef6200700 1 -- 172.21.15.116:0/2939438832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1ef002ccb0 con 0x7f1ef0102910 2024-03-30T00:56:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.745+0000 7f1ee7e00700 1 -- 172.21.15.116:0/2939438832 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1ee8004000 con 0x7f1ef0102910 2024-03-30T00:56:09.751 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-30T00:56:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:09 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3837158794' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T00:56:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:09 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3924167772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.747+0000 7f1ef6200700 1 -- 172.21.15.116:0/2939438832 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1ed40a11a0 msgr2=0x7f1ed40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.747+0000 7f1ef6200700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1ed40a11a0 0x7f1ed40a3660 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0003730 tx=0x7f1ee000b540 comp rx=0 tx=0).stop 2024-03-30T00:56:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.747+0000 7f1ef6200700 1 -- 172.21.15.116:0/2939438832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0102910 msgr2=0x7f1ef00736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.747+0000 7f1ef6200700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0102910 0x7f1ef00736e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8007ee0 tx=0x7f1ee800d3b0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.747+0000 7f1ef6200700 1 -- 172.21.15.116:0/2939438832 shutdown_connections 2024-03-30T00:56:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.747+0000 7f1ef6200700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1ed40a11a0 0x7f1ed40a3660 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.747+0000 7f1ef6200700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ef0102910 0x7f1ef00736e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.747+0000 7f1ef6200700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ef0100d80 0x7f1ef0074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.747+0000 7f1ef6200700 1 --2- 172.21.15.116:0/2939438832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ef0101f70 0x7f1ef00731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.747+0000 7f1ef6200700 1 -- 172.21.15.116:0/2939438832 >> 172.21.15.116:0/2939438832 conn(0x7f1ef00fc2f0 msgr2=0x7f1ef0105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.747+0000 7f1ef6200700 1 -- 172.21.15.116:0/2939438832 shutdown_connections 2024-03-30T00:56:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.747+0000 7f1ef6200700 1 -- 172.21.15.116:0/2939438832 wait complete. 2024-03-30T00:56:09.771 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T00:56:09.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:09 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3837158794' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T00:56:09.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:09 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3924167772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:09.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.960+0000 7f2e03800700 1 -- 172.21.15.116:0/885135561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dfc102910 msgr2=0x7f2dfc1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:09.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.960+0000 7f2e03800700 1 --2- 172.21.15.116:0/885135561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dfc102910 0x7f2dfc1091c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2df400b0d0 tx=0x7f2df400b3e0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.960+0000 7f2e03800700 1 -- 172.21.15.116:0/885135561 shutdown_connections 2024-03-30T00:56:09.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.960+0000 7f2e03800700 1 --2- 172.21.15.116:0/885135561 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dfc100d80 0x7f2dfc1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.960+0000 7f2e03800700 1 --2- 172.21.15.116:0/885135561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dfc102910 0x7f2dfc1091c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.960+0000 7f2e03800700 1 --2- 172.21.15.116:0/885135561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dfc101f70 0x7f2dfc1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.960+0000 7f2e03800700 1 -- 172.21.15.116:0/885135561 >> 172.21.15.116:0/885135561 conn(0x7f2dfc0fc2f0 msgr2=0x7f2dfc0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.960+0000 7f2e03800700 1 -- 172.21.15.116:0/885135561 shutdown_connections 2024-03-30T00:56:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.960+0000 7f2e03800700 1 -- 172.21.15.116:0/885135561 wait complete. 2024-03-30T00:56:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.961+0000 7f2e03800700 1 Processor -- start 2024-03-30T00:56:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.961+0000 7f2e03800700 1 -- start start 2024-03-30T00:56:09.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.961+0000 7f2e03800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dfc100d80 0x7f2dfc106a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:09.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.961+0000 7f2e03800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dfc101f70 0x7f2dfc106fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:09.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.961+0000 7f2e03800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dfc102910 0x7f2dfc1035f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:09.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.961+0000 7f2e03800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc103d90 con 0x7f2dfc101f70 2024-03-30T00:56:09.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.961+0000 7f2e03800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc103f00 con 0x7f2dfc100d80 2024-03-30T00:56:09.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.961+0000 7f2e03800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc1040c0 con 0x7f2dfc102910 2024-03-30T00:56:09.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.962+0000 7f2dfbe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dfc101f70 0x7f2dfc106fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:09.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.962+0000 7f2dfbe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dfc101f70 0x7f2dfc106fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37924/0 (socket says 172.21.15.116:37924) 2024-03-30T00:56:09.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.962+0000 7f2dfbe00700 1 -- 172.21.15.116:0/1001692111 learned_addr learned my addr 172.21.15.116:0/1001692111 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:09.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.962+0000 7f2e01800700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dfc102910 0x7f2dfc1035f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:09.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.962+0000 7f2e00e00700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dfc100d80 0x7f2dfc106a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:09.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.962+0000 7f2dfbe00700 1 -- 172.21.15.116:0/1001692111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dfc102910 msgr2=0x7f2dfc1035f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:09.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.962+0000 7f2dfbe00700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dfc102910 0x7f2dfc1035f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.962+0000 7f2dfbe00700 1 -- 172.21.15.116:0/1001692111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dfc100d80 msgr2=0x7f2dfc106a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.962+0000 7f2dfbe00700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dfc100d80 0x7f2dfc106a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.962+0000 7f2dfbe00700 1 -- 172.21.15.116:0/1001692111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df4009d00 con 0x7f2dfc101f70 2024-03-30T00:56:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.962+0000 7f2e00e00700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dfc100d80 0x7f2dfc106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:56:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.962+0000 7f2dfbe00700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dfc101f70 0x7f2dfc106fa0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2de800e9a0 tx=0x7f2de800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.962+0000 7f2df9600700 1 -- 172.21.15.116:0/1001692111 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8009790 con 0x7f2dfc101f70 2024-03-30T00:56:09.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.963+0000 7f2df9600700 1 -- 172.21.15.116:0/1001692111 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2de800cbd0 con 0x7f2dfc101f70 2024-03-30T00:56:09.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.963+0000 7f2e03800700 1 -- 172.21.15.116:0/1001692111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dfc1a5590 con 0x7f2dfc101f70 2024-03-30T00:56:09.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.963+0000 7f2df9600700 1 -- 172.21.15.116:0/1001692111 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8018680 con 0x7f2dfc101f70 2024-03-30T00:56:09.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.963+0000 7f2e03800700 1 -- 172.21.15.116:0/1001692111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dfc1a5aa0 con 0x7f2dfc101f70 2024-03-30T00:56:09.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.964+0000 7f2e03800700 1 -- 172.21.15.116:0/1001692111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dfc067af0 con 0x7f2dfc101f70 2024-03-30T00:56:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.966+0000 7f2df9600700 1 -- 172.21.15.116:0/1001692111 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f2de800cd40 con 0x7f2dfc101f70 2024-03-30T00:56:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.968+0000 7f2df9600700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2de40a1120 0x7f2de40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.968+0000 7f2df9600700 1 -- 172.21.15.116:0/1001692111 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2de8014070 con 0x7f2dfc101f70 2024-03-30T00:56:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.968+0000 7f2e00e00700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2de40a1120 0x7f2de40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:09.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.968+0000 7f2e00e00700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2de40a1120 0x7f2de40a35e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc101dd0 tx=0x7f2df0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:09.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:09.972+0000 7f2df9600700 1 -- 172.21.15.116:0/1001692111 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2de80529f0 con 0x7f2dfc101f70 2024-03-30T00:56:10.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.176+0000 7f2e03800700 1 -- 172.21.15.116:0/1001692111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2dfc1a5d90 con 0x7f2dfc101f70 2024-03-30T00:56:10.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.176+0000 7f2df9600700 1 -- 172.21.15.116:0/1001692111 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2de80962d0 con 0x7f2dfc101f70 2024-03-30T00:56:10.182 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:56:10.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.178+0000 7f2e03800700 1 -- 172.21.15.116:0/1001692111 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2de40a1120 msgr2=0x7f2de40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:10.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.178+0000 7f2e03800700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2de40a1120 0x7f2de40a35e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc101dd0 tx=0x7f2df0009260 comp rx=0 tx=0).stop 2024-03-30T00:56:10.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.178+0000 7f2e03800700 1 -- 172.21.15.116:0/1001692111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dfc101f70 msgr2=0x7f2dfc106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:10.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.178+0000 7f2e03800700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dfc101f70 0x7f2dfc106fa0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2de800e9a0 tx=0x7f2de800ecb0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.179+0000 7f2e03800700 1 -- 172.21.15.116:0/1001692111 shutdown_connections 2024-03-30T00:56:10.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.179+0000 7f2e03800700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2de40a1120 0x7f2de40a35e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.179+0000 7f2e03800700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2dfc102910 0x7f2dfc1035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.179+0000 7f2e03800700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dfc100d80 0x7f2dfc106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.179+0000 7f2e03800700 1 --2- 172.21.15.116:0/1001692111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dfc101f70 0x7f2dfc106fa0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.179+0000 7f2e03800700 1 -- 172.21.15.116:0/1001692111 >> 172.21.15.116:0/1001692111 conn(0x7f2dfc0fc2f0 msgr2=0x7f2dfc072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:10.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.179+0000 7f2e03800700 1 -- 172.21.15.116:0/1001692111 shutdown_connections 2024-03-30T00:56:10.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.179+0000 7f2e03800700 1 -- 172.21.15.116:0/1001692111 wait complete. 2024-03-30T00:56:10.203 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T00:56:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.395+0000 7f1baf200700 1 -- 172.21.15.116:0/2785357946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba8100d80 msgr2=0x7f1ba81011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.395+0000 7f1baf200700 1 --2- 172.21.15.116:0/2785357946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba8100d80 0x7f1ba81011a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1b98009990 tx=0x7f1b98009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.395+0000 7f1baf200700 1 -- 172.21.15.116:0/2785357946 shutdown_connections 2024-03-30T00:56:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.395+0000 7f1baf200700 1 --2- 172.21.15.116:0/2785357946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba8100d80 0x7f1ba81011a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.395+0000 7f1baf200700 1 --2- 172.21.15.116:0/2785357946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8102910 0x7f1ba81091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.395+0000 7f1baf200700 1 --2- 172.21.15.116:0/2785357946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ba8101f70 0x7f1ba81023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.395+0000 7f1baf200700 1 -- 172.21.15.116:0/2785357946 >> 172.21.15.116:0/2785357946 conn(0x7f1ba80fc2f0 msgr2=0x7f1ba80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.395+0000 7f1baf200700 1 -- 172.21.15.116:0/2785357946 shutdown_connections 2024-03-30T00:56:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.395+0000 7f1baf200700 1 -- 172.21.15.116:0/2785357946 wait complete. 2024-03-30T00:56:10.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.396+0000 7f1baf200700 1 Processor -- start 2024-03-30T00:56:10.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.396+0000 7f1baf200700 1 -- start start 2024-03-30T00:56:10.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.396+0000 7f1baf200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ba8100d80 0x7f1ba8196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:10.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.396+0000 7f1baf200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8101f70 0x7f1ba8197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:10.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.396+0000 7f1baf200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba8102910 0x7f1ba819e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:10.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.396+0000 7f1baf200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba819e880 con 0x7f1ba8101f70 2024-03-30T00:56:10.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.396+0000 7f1baf200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba819e9f0 con 0x7f1ba8100d80 2024-03-30T00:56:10.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.396+0000 7f1baf200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba819ebb0 con 0x7f1ba8102910 2024-03-30T00:56:10.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.397+0000 7f1ba7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8101f70 0x7f1ba8197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.397+0000 7f1ba7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8101f70 0x7f1ba8197320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37950/0 (socket says 172.21.15.116:37950) 2024-03-30T00:56:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.397+0000 7f1ba7400700 1 -- 172.21.15.116:0/2037983446 learned_addr learned my addr 172.21.15.116:0/2037983446 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.397+0000 7f1ba7e00700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ba8100d80 0x7f1ba8196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.397+0000 7f1bad200700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba8102910 0x7f1ba819e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.397+0000 7f1ba7400700 1 -- 172.21.15.116:0/2037983446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba8102910 msgr2=0x7f1ba819e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.397+0000 7f1ba7400700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba8102910 0x7f1ba819e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.397+0000 7f1ba7400700 1 -- 172.21.15.116:0/2037983446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ba8100d80 msgr2=0x7f1ba8196de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.397+0000 7f1ba7400700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ba8100d80 0x7f1ba8196de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.397+0000 7f1ba7400700 1 -- 172.21.15.116:0/2037983446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b980095f0 con 0x7f1ba8101f70 2024-03-30T00:56:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.397+0000 7f1ba7e00700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ba8100d80 0x7f1ba8196de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.397+0000 7f1ba7400700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8101f70 0x7f1ba8197320 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1b9400e9a0 tx=0x7f1b9400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.398+0000 7f1ba4c00700 1 -- 172.21.15.116:0/2037983446 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b94009790 con 0x7f1ba8101f70 2024-03-30T00:56:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.398+0000 7f1ba4c00700 1 -- 172.21.15.116:0/2037983446 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b94010430 con 0x7f1ba8101f70 2024-03-30T00:56:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.398+0000 7f1ba4c00700 1 -- 172.21.15.116:0/2037983446 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b94018680 con 0x7f1ba8101f70 2024-03-30T00:56:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.398+0000 7f1baf200700 1 -- 172.21.15.116:0/2037983446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba819ee90 con 0x7f1ba8101f70 2024-03-30T00:56:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.398+0000 7f1baf200700 1 -- 172.21.15.116:0/2037983446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba819f260 con 0x7f1ba8101f70 2024-03-30T00:56:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.399+0000 7f1baf200700 1 -- 172.21.15.116:0/2037983446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba8108270 con 0x7f1ba8101f70 2024-03-30T00:56:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.401+0000 7f1ba4c00700 1 -- 172.21.15.116:0/2037983446 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f1b940105a0 con 0x7f1ba8101f70 2024-03-30T00:56:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.403+0000 7f1ba4c00700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1b8c0a1120 0x7f1b8c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:10.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.403+0000 7f1ba4c00700 1 -- 172.21.15.116:0/2037983446 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1b94014070 con 0x7f1ba8101f70 2024-03-30T00:56:10.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.403+0000 7f1ba7e00700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1b8c0a1120 0x7f1b8c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:10.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.404+0000 7f1ba7e00700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1b8c0a1120 0x7f1b8c0a35e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8198210 tx=0x7f1b98005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:10.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.405+0000 7f1ba4c00700 1 -- 172.21.15.116:0/2037983446 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1b940963f0 con 0x7f1ba8101f70 2024-03-30T00:56:10.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.608+0000 7f1baf200700 1 -- 172.21.15.116:0/2037983446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1ba810c5e0 con 0x7f1ba8101f70 2024-03-30T00:56:10.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.609+0000 7f1ba4c00700 1 -- 172.21.15.116:0/2037983446 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1b940260a0 con 0x7f1ba8101f70 2024-03-30T00:56:10.615 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-30T00:56:10.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.611+0000 7f1baf200700 1 -- 172.21.15.116:0/2037983446 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1b8c0a1120 msgr2=0x7f1b8c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:10.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.611+0000 7f1baf200700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1b8c0a1120 0x7f1b8c0a35e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8198210 tx=0x7f1b98005fd0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.611+0000 7f1baf200700 1 -- 172.21.15.116:0/2037983446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8101f70 msgr2=0x7f1ba8197320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:10.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.611+0000 7f1baf200700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8101f70 0x7f1ba8197320 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1b9400e9a0 tx=0x7f1b9400ecb0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.612+0000 7f1baf200700 1 -- 172.21.15.116:0/2037983446 shutdown_connections 2024-03-30T00:56:10.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.612+0000 7f1baf200700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1b8c0a1120 0x7f1b8c0a35e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.612+0000 7f1baf200700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba8102910 0x7f1ba819e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.612+0000 7f1baf200700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ba8100d80 0x7f1ba8196de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.612+0000 7f1baf200700 1 --2- 172.21.15.116:0/2037983446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8101f70 0x7f1ba8197320 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.612+0000 7f1baf200700 1 -- 172.21.15.116:0/2037983446 >> 172.21.15.116:0/2037983446 conn(0x7f1ba80fc2f0 msgr2=0x7f1ba81053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:10.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.612+0000 7f1baf200700 1 -- 172.21.15.116:0/2037983446 shutdown_connections 2024-03-30T00:56:10.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.612+0000 7f1baf200700 1 -- 172.21.15.116:0/2037983446 wait complete. 2024-03-30T00:56:10.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:10 smithi116 ceph-mon[38086]: pgmap v238: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-30T00:56:10.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:10 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2939438832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:56:10.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:10 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1001692111' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:56:10.635 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T00:56:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:10 smithi181 ceph-mon[47622]: pgmap v238: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-30T00:56:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:10 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2939438832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:56:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:10 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1001692111' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:56:10.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.824+0000 7f1d0fa00700 1 -- 172.21.15.116:0/4005872477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d08100d80 msgr2=0x7f1d081011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.824+0000 7f1d0fa00700 1 --2- 172.21.15.116:0/4005872477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d08100d80 0x7f1d081011a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009920 tx=0x7f1cfc009c30 comp rx=0 tx=0).stop 2024-03-30T00:56:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.824+0000 7f1d0fa00700 1 -- 172.21.15.116:0/4005872477 shutdown_connections 2024-03-30T00:56:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.824+0000 7f1d0fa00700 1 --2- 172.21.15.116:0/4005872477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d08102910 0x7f1d081091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.824+0000 7f1d0fa00700 1 --2- 172.21.15.116:0/4005872477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d08100d80 0x7f1d081011a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.824+0000 7f1d0fa00700 1 --2- 172.21.15.116:0/4005872477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d08101f70 0x7f1d081023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.824+0000 7f1d0fa00700 1 -- 172.21.15.116:0/4005872477 >> 172.21.15.116:0/4005872477 conn(0x7f1d080fc2f0 msgr2=0x7f1d080fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.824+0000 7f1d0fa00700 1 -- 172.21.15.116:0/4005872477 shutdown_connections 2024-03-30T00:56:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.824+0000 7f1d0fa00700 1 -- 172.21.15.116:0/4005872477 wait complete. 2024-03-30T00:56:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.825+0000 7f1d0fa00700 1 Processor -- start 2024-03-30T00:56:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.825+0000 7f1d0fa00700 1 -- start start 2024-03-30T00:56:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.825+0000 7f1d0fa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d08100d80 0x7f1d08074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.825+0000 7f1d0fa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d08101f70 0x7f1d080731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.825+0000 7f1d0fa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d08102910 0x7f1d080736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.825+0000 7f1d0fa00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d08073de0 con 0x7f1d08102910 2024-03-30T00:56:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.825+0000 7f1d0fa00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d08073f50 con 0x7f1d08101f70 2024-03-30T00:56:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.825+0000 7f1d0fa00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d081a0e90 con 0x7f1d08100d80 2024-03-30T00:56:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.826+0000 7f1d0da00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d08102910 0x7f1d080736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.826+0000 7f1d0da00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d08102910 0x7f1d080736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37968/0 (socket says 172.21.15.116:37968) 2024-03-30T00:56:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.826+0000 7f1d0da00700 1 -- 172.21.15.116:0/787189286 learned_addr learned my addr 172.21.15.116:0/787189286 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.826+0000 7f1d0d000700 1 --2- 172.21.15.116:0/787189286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d08100d80 0x7f1d08074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.826+0000 7f1d07e00700 1 --2- 172.21.15.116:0/787189286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d08101f70 0x7f1d080731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.826+0000 7f1d0da00700 1 -- 172.21.15.116:0/787189286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d08100d80 msgr2=0x7f1d08074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.826+0000 7f1d0da00700 1 --2- 172.21.15.116:0/787189286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d08100d80 0x7f1d08074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.826+0000 7f1d0da00700 1 -- 172.21.15.116:0/787189286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d08101f70 msgr2=0x7f1d080731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.826+0000 7f1d0da00700 1 --2- 172.21.15.116:0/787189286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d08101f70 0x7f1d080731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.826+0000 7f1d0da00700 1 -- 172.21.15.116:0/787189286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cfc009520 con 0x7f1d08102910 2024-03-30T00:56:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.826+0000 7f1d0da00700 1 --2- 172.21.15.116:0/787189286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d08102910 0x7f1d080736e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000c110 tx=0x7f1d0000c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.827+0000 7f1d05600700 1 -- 172.21.15.116:0/787189286 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d00017070 con 0x7f1d08102910 2024-03-30T00:56:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.827+0000 7f1d05600700 1 -- 172.21.15.116:0/787189286 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d0000e050 con 0x7f1d08102910 2024-03-30T00:56:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.827+0000 7f1d0fa00700 1 -- 172.21.15.116:0/787189286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d081a1090 con 0x7f1d08102910 2024-03-30T00:56:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.827+0000 7f1d05600700 1 -- 172.21.15.116:0/787189286 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d000078f0 con 0x7f1d08102910 2024-03-30T00:56:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.827+0000 7f1d0fa00700 1 -- 172.21.15.116:0/787189286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d081a15a0 con 0x7f1d08102910 2024-03-30T00:56:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.828+0000 7f1d0fa00700 1 -- 172.21.15.116:0/787189286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d081082c0 con 0x7f1d08102910 2024-03-30T00:56:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.829+0000 7f1d05600700 1 -- 172.21.15.116:0/787189286 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f1d00007a50 con 0x7f1d08102910 2024-03-30T00:56:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.831+0000 7f1d05600700 1 --2- 172.21.15.116:0/787189286 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1cf00a1120 0x7f1cf00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.831+0000 7f1d05600700 1 -- 172.21.15.116:0/787189286 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1d000cc510 con 0x7f1d08102910 2024-03-30T00:56:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.831+0000 7f1d0d000700 1 --2- 172.21.15.116:0/787189286 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1cf00a1120 0x7f1cf00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:10.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.831+0000 7f1d0d000700 1 --2- 172.21.15.116:0/787189286 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1cf00a1120 0x7f1cf00a35e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc0098f0 tx=0x7f1cfc019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:10.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:10.835+0000 7f1d05600700 1 -- 172.21.15.116:0/787189286 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1d00095a50 con 0x7f1d08102910 2024-03-30T00:56:10.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:10 smithi130 ceph-mon[47626]: pgmap v238: 129 pgs: 129 active+clean; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-30T00:56:10.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:10 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2939438832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:56:10.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:10 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1001692111' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:56:11.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.042+0000 7f1d0fa00700 1 -- 172.21.15.116:0/787189286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1d08107ca0 con 0x7f1d08102910 2024-03-30T00:56:11.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.043+0000 7f1d05600700 1 -- 172.21.15.116:0/787189286 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v81) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1d00099070 con 0x7f1d08102910 2024-03-30T00:56:11.056 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:56:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.045+0000 7f1d0fa00700 1 -- 172.21.15.116:0/787189286 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1cf00a1120 msgr2=0x7f1cf00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:11.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.045+0000 7f1d0fa00700 1 --2- 172.21.15.116:0/787189286 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1cf00a1120 0x7f1cf00a35e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc0098f0 tx=0x7f1cfc019040 comp rx=0 tx=0).stop 2024-03-30T00:56:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.045+0000 7f1d0fa00700 1 -- 172.21.15.116:0/787189286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d08102910 msgr2=0x7f1d080736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.045+0000 7f1d0fa00700 1 --2- 172.21.15.116:0/787189286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d08102910 0x7f1d080736e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000c110 tx=0x7f1d0000c420 comp rx=0 tx=0).stop 2024-03-30T00:56:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.045+0000 7f1d0fa00700 1 -- 172.21.15.116:0/787189286 shutdown_connections 2024-03-30T00:56:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.045+0000 7f1d0fa00700 1 --2- 172.21.15.116:0/787189286 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1cf00a1120 0x7f1cf00a35e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.045+0000 7f1d0fa00700 1 --2- 172.21.15.116:0/787189286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d08100d80 0x7f1d08074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.045+0000 7f1d0fa00700 1 --2- 172.21.15.116:0/787189286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d08102910 0x7f1d080736e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.045+0000 7f1d0fa00700 1 --2- 172.21.15.116:0/787189286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d08101f70 0x7f1d080731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.045+0000 7f1d0fa00700 1 -- 172.21.15.116:0/787189286 >> 172.21.15.116:0/787189286 conn(0x7f1d080fc2f0 msgr2=0x7f1d08105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.045+0000 7f1d0fa00700 1 -- 172.21.15.116:0/787189286 shutdown_connections 2024-03-30T00:56:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.045+0000 7f1d0fa00700 1 -- 172.21.15.116:0/787189286 wait complete. 2024-03-30T00:56:11.071 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-30T00:56:11.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.258+0000 7f2a19200700 1 -- 172.21.15.116:0/1925279219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a14100d80 msgr2=0x7f2a141011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:11.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.258+0000 7f2a19200700 1 --2- 172.21.15.116:0/1925279219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a14100d80 0x7f2a141011a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2a040099c0 tx=0x7f2a04009cd0 comp rx=0 tx=0).stop 2024-03-30T00:56:11.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.258+0000 7f2a19200700 1 -- 172.21.15.116:0/1925279219 shutdown_connections 2024-03-30T00:56:11.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.258+0000 7f2a19200700 1 --2- 172.21.15.116:0/1925279219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2a14102910 0x7f2a141091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:11.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.258+0000 7f2a19200700 1 --2- 172.21.15.116:0/1925279219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a14100d80 0x7f2a141011a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.258+0000 7f2a19200700 1 --2- 172.21.15.116:0/1925279219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a14101f70 0x7f2a141023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.258+0000 7f2a19200700 1 -- 172.21.15.116:0/1925279219 >> 172.21.15.116:0/1925279219 conn(0x7f2a140fc2f0 msgr2=0x7f2a140fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.258+0000 7f2a19200700 1 -- 172.21.15.116:0/1925279219 shutdown_connections 2024-03-30T00:56:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.258+0000 7f2a19200700 1 -- 172.21.15.116:0/1925279219 wait complete. 2024-03-30T00:56:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.259+0000 7f2a19200700 1 Processor -- start 2024-03-30T00:56:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.259+0000 7f2a19200700 1 -- start start 2024-03-30T00:56:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.259+0000 7f2a19200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a14100d80 0x7f2a14074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.259+0000 7f2a19200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a14101f70 0x7f2a140731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a19200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2a14102910 0x7f2a140736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a19200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a14073e80 con 0x7f2a14100d80 2024-03-30T00:56:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a19200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a14073ff0 con 0x7f2a14102910 2024-03-30T00:56:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a19200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a141a0dc0 con 0x7f2a14101f70 2024-03-30T00:56:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a11e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a14100d80 0x7f2a14074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a11e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a14100d80 0x7f2a14074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37980/0 (socket says 172.21.15.116:37980) 2024-03-30T00:56:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a11e00700 1 -- 172.21.15.116:0/641366784 learned_addr learned my addr 172.21.15.116:0/641366784 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a0be00700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a14101f70 0x7f2a140731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a12800700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2a14102910 0x7f2a140736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a11e00700 1 -- 172.21.15.116:0/641366784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a14101f70 msgr2=0x7f2a140731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a11e00700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a14101f70 0x7f2a140731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a11e00700 1 -- 172.21.15.116:0/641366784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2a14102910 msgr2=0x7f2a140736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a11e00700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2a14102910 0x7f2a140736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a11e00700 1 -- 172.21.15.116:0/641366784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a040095f0 con 0x7f2a14100d80 2024-03-30T00:56:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a12800700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2a14102910 0x7f2a140736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:56:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.260+0000 7f2a11e00700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a14100d80 0x7f2a14074b10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2a04003730 tx=0x7f2a0400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.261+0000 7f2a0aa00700 1 -- 172.21.15.116:0/641366784 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0401d070 con 0x7f2a14100d80 2024-03-30T00:56:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.261+0000 7f2a0aa00700 1 -- 172.21.15.116:0/641366784 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a04005050 con 0x7f2a14100d80 2024-03-30T00:56:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.261+0000 7f2a0aa00700 1 -- 172.21.15.116:0/641366784 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a04004af0 con 0x7f2a14100d80 2024-03-30T00:56:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.261+0000 7f2a19200700 1 -- 172.21.15.116:0/641366784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a141a0ff0 con 0x7f2a14100d80 2024-03-30T00:56:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.261+0000 7f2a19200700 1 -- 172.21.15.116:0/641366784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a141a1360 con 0x7f2a14100d80 2024-03-30T00:56:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.262+0000 7f2a19200700 1 -- 172.21.15.116:0/641366784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a14108220 con 0x7f2a14100d80 2024-03-30T00:56:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.264+0000 7f2a0aa00700 1 -- 172.21.15.116:0/641366784 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f2a040051c0 con 0x7f2a14100d80 2024-03-30T00:56:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.265+0000 7f2a0aa00700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f29f80a11a0 0x7f29f80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:11.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.265+0000 7f2a0aa00700 1 -- 172.21.15.116:0/641366784 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2a040cdca0 con 0x7f2a14100d80 2024-03-30T00:56:11.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.265+0000 7f2a0be00700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f29f80a11a0 0x7f29f80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:11.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.266+0000 7f2a0be00700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f29f80a11a0 0x7f29f80a3660 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009c60 tx=0x7f29fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:11.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.267+0000 7f2a0aa00700 1 -- 172.21.15.116:0/641366784 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2a04097270 con 0x7f2a14100d80 2024-03-30T00:56:11.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.529+0000 7f2a19200700 1 -- 172.21.15.116:0/641366784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2a1410c5e0 con 0x7f2a14100d80 2024-03-30T00:56:11.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.530+0000 7f2a0aa00700 1 -- 172.21.15.116:0/641366784 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v81) v1 ==== 81+0+598 (secure 0 0 0) 0x7f2a0409a890 con 0x7f2a14100d80 2024-03-30T00:56:11.536 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/309656254 2024-03-31T00:50:35.029431+0000 2024-03-30T00:56:11.536 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/131150228 2024-03-31T00:50:35.029431+0000 2024-03-30T00:56:11.536 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/755518827 2024-03-31T00:50:35.029431+0000 2024-03-30T00:56:11.536 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/3183509903 2024-03-31T00:50:11.610313+0000 2024-03-30T00:56:11.536 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/4184421214 2024-03-31T00:50:11.610313+0000 2024-03-30T00:56:11.536 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/3183509903 2024-03-31T00:50:11.610313+0000 2024-03-30T00:56:11.536 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/131150228 2024-03-31T00:50:35.029431+0000 2024-03-30T00:56:11.536 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2567186791 2024-03-31T00:50:35.029431+0000 2024-03-30T00:56:11.536 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2997035221 2024-03-31T00:50:11.610313+0000 2024-03-30T00:56:11.536 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2367387359 2024-03-31T00:50:11.610313+0000 2024-03-30T00:56:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.532+0000 7f2a19200700 1 -- 172.21.15.116:0/641366784 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f29f80a11a0 msgr2=0x7f29f80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.532+0000 7f2a19200700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f29f80a11a0 0x7f29f80a3660 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009c60 tx=0x7f29fc009260 comp rx=0 tx=0).stop 2024-03-30T00:56:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.532+0000 7f2a19200700 1 -- 172.21.15.116:0/641366784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a14100d80 msgr2=0x7f2a14074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.532+0000 7f2a19200700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a14100d80 0x7f2a14074b10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2a04003730 tx=0x7f2a0400f690 comp rx=0 tx=0).stop 2024-03-30T00:56:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.532+0000 7f2a19200700 1 -- 172.21.15.116:0/641366784 shutdown_connections 2024-03-30T00:56:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.532+0000 7f2a19200700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f29f80a11a0 0x7f29f80a3660 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.532+0000 7f2a19200700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2a14102910 0x7f2a140736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.532+0000 7f2a19200700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a14100d80 0x7f2a14074b10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.532+0000 7f2a19200700 1 --2- 172.21.15.116:0/641366784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a14101f70 0x7f2a140731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.532+0000 7f2a19200700 1 -- 172.21.15.116:0/641366784 >> 172.21.15.116:0/641366784 conn(0x7f2a140fc2f0 msgr2=0x7f2a14105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.533+0000 7f2a19200700 1 -- 172.21.15.116:0/641366784 shutdown_connections 2024-03-30T00:56:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:11.533+0000 7f2a19200700 1 -- 172.21.15.116:0/641366784 wait complete. 2024-03-30T00:56:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:listed 10 entries 2024-03-30T00:56:11.557 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2024-03-30T00:56:11.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:11.558 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-30T00:56:11.615 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:11 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2037983446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:56:11.615 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:11 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/787189286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:56:11.620 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi116.stderr:ceph-fuse[90980]: fuse finished with error 0 and tester_r 0 2024-03-30T00:56:11.624 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T00:56:11.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:11 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2037983446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:56:11.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:11 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/787189286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:56:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:11 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2037983446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:56:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:11 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/787189286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:56:12.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:12 smithi116 ceph-mon[38086]: pgmap v239: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T00:56:12.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:12 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/641366784' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T00:56:12.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:12 smithi181 ceph-mon[47622]: pgmap v239: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T00:56:12.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:12 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/641366784' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T00:56:12.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:12 smithi130 ceph-mon[47626]: pgmap v239: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T00:56:12.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:12 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/641366784' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T00:56:14.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:14 smithi116 ceph-mon[38086]: pgmap v240: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T00:56:14.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:14 smithi181 ceph-mon[47622]: pgmap v240: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T00:56:14.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:14 smithi130 ceph-mon[47626]: pgmap v240: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T00:56:16.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:16 smithi116 ceph-mon[38086]: pgmap v241: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-30T00:56:16.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:16 smithi181 ceph-mon[47622]: pgmap v241: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-30T00:56:16.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:16 smithi130 ceph-mon[47626]: pgmap v241: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-30T00:56:17.727 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2024-03-30T00:56:17.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:56:17.727 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-30T00:56:17.792 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:56:17.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.983+0000 7f7071000700 1 -- 172.21.15.116:0/2377764513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f706c1016e0 msgr2=0x7f706c103ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:17.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.983+0000 7f7071000700 1 --2- 172.21.15.116:0/2377764513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f706c1016e0 0x7f706c103ad0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f705c0098f0 tx=0x7f705c009c00 comp rx=0 tx=0).stop 2024-03-30T00:56:17.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.984+0000 7f7071000700 1 -- 172.21.15.116:0/2377764513 shutdown_connections 2024-03-30T00:56:17.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.984+0000 7f7071000700 1 --2- 172.21.15.116:0/2377764513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f706c106940 0x7f706c108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:17.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.984+0000 7f7071000700 1 --2- 172.21.15.116:0/2377764513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f706c1016e0 0x7f706c103ad0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:17.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.984+0000 7f7071000700 1 --2- 172.21.15.116:0/2377764513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f706c104010 0x7f706c106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:17.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.984+0000 7f7071000700 1 -- 172.21.15.116:0/2377764513 >> 172.21.15.116:0/2377764513 conn(0x7f706c0fb0d0 msgr2=0x7f706c0fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:17.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.984+0000 7f7071000700 1 -- 172.21.15.116:0/2377764513 shutdown_connections 2024-03-30T00:56:17.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.984+0000 7f7071000700 1 -- 172.21.15.116:0/2377764513 wait complete. 2024-03-30T00:56:17.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.985+0000 7f7071000700 1 Processor -- start 2024-03-30T00:56:17.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.985+0000 7f7071000700 1 -- start start 2024-03-30T00:56:17.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.985+0000 7f7071000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f706c1016e0 0x7f706c198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:17.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.985+0000 7f7071000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f706c104010 0x7f706c199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:17.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.985+0000 7f7071000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f706c106940 0x7f706c1a03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:17.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.985+0000 7f7071000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c1a0980 con 0x7f706c104010 2024-03-30T00:56:17.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.985+0000 7f7071000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c1a0ac0 con 0x7f706c106940 2024-03-30T00:56:17.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.985+0000 7f7071000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c1a0c50 con 0x7f706c1016e0 2024-03-30T00:56:17.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.985+0000 7f7069400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f706c104010 0x7f706c199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:17.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.985+0000 7f7069400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f706c104010 0x7f706c199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57500/0 (socket says 172.21.15.116:57500) 2024-03-30T00:56:17.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.985+0000 7f7069400700 1 -- 172.21.15.116:0/940749523 learned_addr learned my addr 172.21.15.116:0/940749523 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:17.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.985+0000 7f706a800700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f706c106940 0x7f706c1a03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:17.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.985+0000 7f7069e00700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f706c1016e0 0x7f706c198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:17.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.986+0000 7f706a800700 1 -- 172.21.15.116:0/940749523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f706c1016e0 msgr2=0x7f706c198e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:17.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.986+0000 7f706a800700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f706c1016e0 0x7f706c198e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:17.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.986+0000 7f706a800700 1 -- 172.21.15.116:0/940749523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f706c104010 msgr2=0x7f706c199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:17.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.986+0000 7f706a800700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f706c104010 0x7f706c199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:17.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.986+0000 7f706a800700 1 -- 172.21.15.116:0/940749523 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7064009d00 con 0x7f706c106940 2024-03-30T00:56:17.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.986+0000 7f7069400700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f706c104010 0x7f706c199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:17.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.986+0000 7f706a800700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f706c106940 0x7f706c1a03b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7064009cd0 tx=0x7f706400c620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:17.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.986+0000 7f7062a00700 1 -- 172.21.15.116:0/940749523 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064017070 con 0x7f706c106940 2024-03-30T00:56:17.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.986+0000 7f7071000700 1 -- 172.21.15.116:0/940749523 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f705c009520 con 0x7f706c106940 2024-03-30T00:56:17.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.986+0000 7f7062a00700 1 -- 172.21.15.116:0/940749523 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f706400f040 con 0x7f706c106940 2024-03-30T00:56:17.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.986+0000 7f7062a00700 1 -- 172.21.15.116:0/940749523 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706400e050 con 0x7f706c106940 2024-03-30T00:56:17.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.986+0000 7f7071000700 1 -- 172.21.15.116:0/940749523 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706c1a1240 con 0x7f706c106940 2024-03-30T00:56:17.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.987+0000 7f7071000700 1 -- 172.21.15.116:0/940749523 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f706c108510 con 0x7f706c106940 2024-03-30T00:56:17.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.989+0000 7f7062a00700 1 -- 172.21.15.116:0/940749523 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f7064004ad0 con 0x7f706c106940 2024-03-30T00:56:17.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.991+0000 7f7062a00700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f70500a1120 0x7f70500a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:17.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.991+0000 7f7062a00700 1 -- 172.21.15.116:0/940749523 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f70640cc750 con 0x7f706c106940 2024-03-30T00:56:17.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.991+0000 7f7069e00700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f70500a1120 0x7f70500a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:17.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.992+0000 7f7069e00700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f70500a1120 0x7f70500a35e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f705c00fa40 tx=0x7f705c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:17.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:17.993+0000 7f7062a00700 1 -- 172.21.15.116:0/940749523 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7064095d20 con 0x7f706c106940 2024-03-30T00:56:18.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.226+0000 7f7071000700 1 -- 172.21.15.116:0/940749523 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f706c10c490 con 0x7f706c106940 2024-03-30T00:56:18.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.228+0000 7f7062a00700 1 -- 172.21.15.116:0/940749523 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+24028 (secure 0 0 0) 0x7f7064099340 con 0x7f706c106940 2024-03-30T00:56:18.235 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:56:18.235 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:55:56.324776+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:56:18.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.230+0000 7f7071000700 1 -- 172.21.15.116:0/940749523 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f70500a1120 msgr2=0x7f70500a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:18.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.230+0000 7f7071000700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f70500a1120 0x7f70500a35e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f705c00fa40 tx=0x7f705c019040 comp rx=0 tx=0).stop 2024-03-30T00:56:18.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.230+0000 7f7071000700 1 -- 172.21.15.116:0/940749523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f706c106940 msgr2=0x7f706c1a03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:18.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.230+0000 7f7071000700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f706c106940 0x7f706c1a03b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7064009cd0 tx=0x7f706400c620 comp rx=0 tx=0).stop 2024-03-30T00:56:18.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.230+0000 7f7071000700 1 -- 172.21.15.116:0/940749523 shutdown_connections 2024-03-30T00:56:18.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.230+0000 7f7071000700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f70500a1120 0x7f70500a35e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.230+0000 7f7071000700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f706c1016e0 0x7f706c198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.230+0000 7f7071000700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f706c106940 0x7f706c1a03b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.230+0000 7f7071000700 1 --2- 172.21.15.116:0/940749523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f706c104010 0x7f706c199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.230+0000 7f7071000700 1 -- 172.21.15.116:0/940749523 >> 172.21.15.116:0/940749523 conn(0x7f706c0fb0d0 msgr2=0x7f706c100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:18.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.231+0000 7f7071000700 1 -- 172.21.15.116:0/940749523 shutdown_connections 2024-03-30T00:56:18.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.231+0000 7f7071000700 1 -- 172.21.15.116:0/940749523 wait complete. 2024-03-30T00:56:18.256 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:56:18.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.442+0000 7fb7bd800700 1 -- 172.21.15.116:0/658351956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8100c80 msgr2=0x7fb7b81010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:18.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.442+0000 7fb7bd800700 1 --2- 172.21.15.116:0/658351956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8100c80 0x7fb7b81010a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb7a80099c0 tx=0x7fb7a8009cd0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.442+0000 7fb7bd800700 1 -- 172.21.15.116:0/658351956 shutdown_connections 2024-03-30T00:56:18.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.442+0000 7fb7bd800700 1 --2- 172.21.15.116:0/658351956 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7b8102810 0x7fb7b81090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.442+0000 7fb7bd800700 1 --2- 172.21.15.116:0/658351956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8100c80 0x7fb7b81010a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.442+0000 7fb7bd800700 1 --2- 172.21.15.116:0/658351956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7b8101e70 0x7fb7b81022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.442+0000 7fb7bd800700 1 -- 172.21.15.116:0/658351956 >> 172.21.15.116:0/658351956 conn(0x7fb7b80fc1f0 msgr2=0x7fb7b80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:18.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.442+0000 7fb7bd800700 1 -- 172.21.15.116:0/658351956 shutdown_connections 2024-03-30T00:56:18.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.442+0000 7fb7bd800700 1 -- 172.21.15.116:0/658351956 wait complete. 2024-03-30T00:56:18.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.443+0000 7fb7bd800700 1 Processor -- start 2024-03-30T00:56:18.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.443+0000 7fb7bd800700 1 -- start start 2024-03-30T00:56:18.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7bd800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7b8100c80 0x7fb7b8198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:18.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7bd800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8101e70 0x7fb7b81992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:18.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7bd800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7b8102810 0x7fb7b81a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:18.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7bd800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b81a08d0 con 0x7fb7b8101e70 2024-03-30T00:56:18.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7bd800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b81a0a10 con 0x7fb7b8100c80 2024-03-30T00:56:18.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7bd800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b81a0ba0 con 0x7fb7b8102810 2024-03-30T00:56:18.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7b6000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8101e70 0x7fb7b81992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:18.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7b6a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7b8100c80 0x7fb7b8198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:18.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7b6000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8101e70 0x7fb7b81992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57526/0 (socket says 172.21.15.116:57526) 2024-03-30T00:56:18.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7b6000700 1 -- 172.21.15.116:0/2309629442 learned_addr learned my addr 172.21.15.116:0/2309629442 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:18.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7b7400700 1 --2- 172.21.15.116:0/2309629442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7b8102810 0x7fb7b81a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:18.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7b6a00700 1 -- 172.21.15.116:0/2309629442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7b8102810 msgr2=0x7fb7b81a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:18.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7b6a00700 1 --2- 172.21.15.116:0/2309629442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7b8102810 0x7fb7b81a0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7b6a00700 1 -- 172.21.15.116:0/2309629442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8101e70 msgr2=0x7fb7b81992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:18.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7b6a00700 1 --2- 172.21.15.116:0/2309629442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8101e70 0x7fb7b81992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7b6a00700 1 -- 172.21.15.116:0/2309629442 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a80095f0 con 0x7fb7b8100c80 2024-03-30T00:56:18.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.444+0000 7fb7b7400700 1 --2- 172.21.15.116:0/2309629442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7b8102810 0x7fb7b81a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:56:18.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.445+0000 7fb7b6a00700 1 --2- 172.21.15.116:0/2309629442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7b8100c80 0x7fb7b8198da0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb7a80098e0 tx=0x7fb7a800f710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:18.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.445+0000 7fb7af400700 1 -- 172.21.15.116:0/2309629442 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a801c070 con 0x7fb7b8100c80 2024-03-30T00:56:18.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.445+0000 7fb7af400700 1 -- 172.21.15.116:0/2309629442 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7a80058e0 con 0x7fb7b8100c80 2024-03-30T00:56:18.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.445+0000 7fb7bd800700 1 -- 172.21.15.116:0/2309629442 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7b81a0e20 con 0x7fb7b8100c80 2024-03-30T00:56:18.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.445+0000 7fb7af400700 1 -- 172.21.15.116:0/2309629442 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8022c30 con 0x7fb7b8100c80 2024-03-30T00:56:18.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.445+0000 7fb7bd800700 1 -- 172.21.15.116:0/2309629442 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7b81a1210 con 0x7fb7b8100c80 2024-03-30T00:56:18.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.446+0000 7fb7bd800700 1 -- 172.21.15.116:0/2309629442 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7b8067a10 con 0x7fb7b8100c80 2024-03-30T00:56:18.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.448+0000 7fb7af400700 1 -- 172.21.15.116:0/2309629442 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fb7a8020070 con 0x7fb7b8100c80 2024-03-30T00:56:18.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.449+0000 7fb7af400700 1 --2- 172.21.15.116:0/2309629442 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb79c0a1120 0x7fb79c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:18.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.449+0000 7fb7af400700 1 -- 172.21.15.116:0/2309629442 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb7a80cdba0 con 0x7fb7b8100c80 2024-03-30T00:56:18.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.449+0000 7fb7b6000700 1 --2- 172.21.15.116:0/2309629442 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb79c0a1120 0x7fb79c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:18.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.450+0000 7fb7b6000700 1 --2- 172.21.15.116:0/2309629442 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb79c0a1120 0x7fb79c0a35e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4009d20 tx=0x7fb7a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:18.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.452+0000 7fb7af400700 1 -- 172.21.15.116:0/2309629442 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb7a8097170 con 0x7fb7b8100c80 2024-03-30T00:56:18.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:18 smithi116 ceph-mon[38086]: pgmap v242: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-30T00:56:18.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:18 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/940749523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.690+0000 7fb7bd800700 1 -- 172.21.15.116:0/2309629442 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7b810c490 con 0x7fb7b8100c80 2024-03-30T00:56:18.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.691+0000 7fb7af400700 1 -- 172.21.15.116:0/2309629442 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+24028 (secure 0 0 0) 0x7fb7a809a790 con 0x7fb7b8100c80 2024-03-30T00:56:18.699 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:56:18.699 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:55:56.324776+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:56:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.694+0000 7fb7bd800700 1 -- 172.21.15.116:0/2309629442 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb79c0a1120 msgr2=0x7fb79c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.694+0000 7fb7bd800700 1 --2- 172.21.15.116:0/2309629442 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb79c0a1120 0x7fb79c0a35e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4009d20 tx=0x7fb7a4009260 comp rx=0 tx=0).stop 2024-03-30T00:56:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.694+0000 7fb7bd800700 1 -- 172.21.15.116:0/2309629442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7b8100c80 msgr2=0x7fb7b8198da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.694+0000 7fb7bd800700 1 --2- 172.21.15.116:0/2309629442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7b8100c80 0x7fb7b8198da0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb7a80098e0 tx=0x7fb7a800f710 comp rx=0 tx=0).stop 2024-03-30T00:56:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.694+0000 7fb7bd800700 1 -- 172.21.15.116:0/2309629442 shutdown_connections 2024-03-30T00:56:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.694+0000 7fb7bd800700 1 --2- 172.21.15.116:0/2309629442 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb79c0a1120 0x7fb79c0a35e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.694+0000 7fb7bd800700 1 --2- 172.21.15.116:0/2309629442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7b8102810 0x7fb7b81a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.694+0000 7fb7bd800700 1 --2- 172.21.15.116:0/2309629442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7b8100c80 0x7fb7b8198da0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.694+0000 7fb7bd800700 1 --2- 172.21.15.116:0/2309629442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8101e70 0x7fb7b81992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.694+0000 7fb7bd800700 1 -- 172.21.15.116:0/2309629442 >> 172.21.15.116:0/2309629442 conn(0x7fb7b80fc1f0 msgr2=0x7fb7b8107640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.694+0000 7fb7bd800700 1 -- 172.21.15.116:0/2309629442 shutdown_connections 2024-03-30T00:56:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.694+0000 7fb7bd800700 1 -- 172.21.15.116:0/2309629442 wait complete. 2024-03-30T00:56:18.720 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-30T00:56:18.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:18 smithi181 ceph-mon[47622]: pgmap v242: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-30T00:56:18.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:18 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/940749523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:18.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:18 smithi130 ceph-mon[47626]: pgmap v242: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-30T00:56:18.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:18 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/940749523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.908+0000 7f27bb000700 1 -- 172.21.15.116:0/1335765703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b4100d80 msgr2=0x7f27b41011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.908+0000 7f27bb000700 1 --2- 172.21.15.116:0/1335765703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b4100d80 0x7f27b41011a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f27a40099c0 tx=0x7f27a4009cd0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.908+0000 7f27bb000700 1 -- 172.21.15.116:0/1335765703 shutdown_connections 2024-03-30T00:56:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.908+0000 7f27bb000700 1 --2- 172.21.15.116:0/1335765703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27b4102910 0x7f27b41091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.908+0000 7f27bb000700 1 --2- 172.21.15.116:0/1335765703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b4100d80 0x7f27b41011a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.908+0000 7f27bb000700 1 --2- 172.21.15.116:0/1335765703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27b4101f70 0x7f27b41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.908+0000 7f27bb000700 1 -- 172.21.15.116:0/1335765703 >> 172.21.15.116:0/1335765703 conn(0x7f27b40fc2f0 msgr2=0x7f27b40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.908+0000 7f27bb000700 1 -- 172.21.15.116:0/1335765703 shutdown_connections 2024-03-30T00:56:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.908+0000 7f27bb000700 1 -- 172.21.15.116:0/1335765703 wait complete. 2024-03-30T00:56:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.909+0000 7f27bb000700 1 Processor -- start 2024-03-30T00:56:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.909+0000 7f27bb000700 1 -- start start 2024-03-30T00:56:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27bb000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b4100d80 0x7f27b4196d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27bb000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27b4101f70 0x7f27b4197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27bb000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27b4102910 0x7f27b419e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27bb000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b419e840 con 0x7f27b4100d80 2024-03-30T00:56:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27bb000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b419e9b0 con 0x7f27b4101f70 2024-03-30T00:56:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27bb000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b419eb70 con 0x7f27b4102910 2024-03-30T00:56:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27b3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b4100d80 0x7f27b4196d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27b3400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27b4101f70 0x7f27b4197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27b3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b4100d80 0x7f27b4196d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57544/0 (socket says 172.21.15.116:57544) 2024-03-30T00:56:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27b3e00700 1 -- 172.21.15.116:0/125220276 learned_addr learned my addr 172.21.15.116:0/125220276 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27b9000700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27b4102910 0x7f27b419e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27b3e00700 1 -- 172.21.15.116:0/125220276 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27b4102910 msgr2=0x7f27b419e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27b3e00700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27b4102910 0x7f27b419e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27b3e00700 1 -- 172.21.15.116:0/125220276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27b4101f70 msgr2=0x7f27b4197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27b3e00700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27b4101f70 0x7f27b4197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27b3e00700 1 -- 172.21.15.116:0/125220276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a40095f0 con 0x7f27b4100d80 2024-03-30T00:56:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27b9000700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27b4102910 0x7f27b419e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:56:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.910+0000 7f27b3400700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27b4101f70 0x7f27b4197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.911+0000 7f27b3e00700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b4100d80 0x7f27b4196d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f27a4006010 tx=0x7f27a400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.911+0000 7f27b0c00700 1 -- 172.21.15.116:0/125220276 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a401d070 con 0x7f27b4100d80 2024-03-30T00:56:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.911+0000 7f27b0c00700 1 -- 172.21.15.116:0/125220276 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27a4005020 con 0x7f27b4100d80 2024-03-30T00:56:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.911+0000 7f27bb000700 1 -- 172.21.15.116:0/125220276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b419edf0 con 0x7f27b4100d80 2024-03-30T00:56:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.911+0000 7f27b0c00700 1 -- 172.21.15.116:0/125220276 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a4004bb0 con 0x7f27b4100d80 2024-03-30T00:56:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.911+0000 7f27bb000700 1 -- 172.21.15.116:0/125220276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b419f1c0 con 0x7f27b4100d80 2024-03-30T00:56:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.912+0000 7f27bb000700 1 -- 172.21.15.116:0/125220276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b4108270 con 0x7f27b4100d80 2024-03-30T00:56:18.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.914+0000 7f27b0c00700 1 -- 172.21.15.116:0/125220276 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f27a4005190 con 0x7f27b4100d80 2024-03-30T00:56:18.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.915+0000 7f27b0c00700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f27980a11a0 0x7f27980a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:18.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.915+0000 7f27b0c00700 1 -- 172.21.15.116:0/125220276 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f27a40cd510 con 0x7f27b4100d80 2024-03-30T00:56:18.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.915+0000 7f27b3400700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f27980a11a0 0x7f27980a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:18.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.916+0000 7f27b3400700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f27980a11a0 0x7f27980a3660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f27a0011490 tx=0x7f27a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:18.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:18.919+0000 7f27b0c00700 1 -- 172.21.15.116:0/125220276 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f27a4096ae0 con 0x7f27b4100d80 2024-03-30T00:56:19.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.173+0000 7f27bb000700 1 -- 172.21.15.116:0/125220276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f27b4067af0 con 0x7f27b4100d80 2024-03-30T00:56:19.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.369+0000 7f27b0c00700 1 -- 172.21.15.116:0/125220276 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f27a409a100 con 0x7f27b4100d80 2024-03-30T00:56:19.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.372+0000 7f27bb000700 1 -- 172.21.15.116:0/125220276 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f27980a11a0 msgr2=0x7f27980a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:19.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.372+0000 7f27bb000700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f27980a11a0 0x7f27980a3660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f27a0011490 tx=0x7f27a0009260 comp rx=0 tx=0).stop 2024-03-30T00:56:19.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.372+0000 7f27bb000700 1 -- 172.21.15.116:0/125220276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b4100d80 msgr2=0x7f27b4196d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:19.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.372+0000 7f27bb000700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b4100d80 0x7f27b4196d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f27a4006010 tx=0x7f27a400f690 comp rx=0 tx=0).stop 2024-03-30T00:56:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.372+0000 7f27bb000700 1 -- 172.21.15.116:0/125220276 shutdown_connections 2024-03-30T00:56:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.372+0000 7f27bb000700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f27980a11a0 0x7f27980a3660 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.372+0000 7f27bb000700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27b4102910 0x7f27b419e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.372+0000 7f27bb000700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b4100d80 0x7f27b4196d50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.372+0000 7f27bb000700 1 --2- 172.21.15.116:0/125220276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27b4101f70 0x7f27b4197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.372+0000 7f27bb000700 1 -- 172.21.15.116:0/125220276 >> 172.21.15.116:0/125220276 conn(0x7f27b40fc2f0 msgr2=0x7f27b41053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.372+0000 7f27bb000700 1 -- 172.21.15.116:0/125220276 shutdown_connections 2024-03-30T00:56:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.372+0000 7f27bb000700 1 -- 172.21.15.116:0/125220276 wait complete. 2024-03-30T00:56:19.398 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-30T00:56:19.398 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-30T00:56:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.588+0000 7fee1aa00700 1 -- 172.21.15.116:0/3290846517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14100d80 msgr2=0x7fee141011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.588+0000 7fee1aa00700 1 --2- 172.21.15.116:0/3290846517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14100d80 0x7fee141011a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fee040099c0 tx=0x7fee04009cd0 comp rx=0 tx=0).stop 2024-03-30T00:56:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.588+0000 7fee1aa00700 1 -- 172.21.15.116:0/3290846517 shutdown_connections 2024-03-30T00:56:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.588+0000 7fee1aa00700 1 --2- 172.21.15.116:0/3290846517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee14102910 0x7fee141091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:19.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.588+0000 7fee1aa00700 1 --2- 172.21.15.116:0/3290846517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14100d80 0x7fee141011a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:19.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.588+0000 7fee1aa00700 1 --2- 172.21.15.116:0/3290846517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee14101f70 0x7fee141023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:19.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.588+0000 7fee1aa00700 1 -- 172.21.15.116:0/3290846517 >> 172.21.15.116:0/3290846517 conn(0x7fee140fc2f0 msgr2=0x7fee140fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:19.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.588+0000 7fee1aa00700 1 -- 172.21.15.116:0/3290846517 shutdown_connections 2024-03-30T00:56:19.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.588+0000 7fee1aa00700 1 -- 172.21.15.116:0/3290846517 wait complete. 2024-03-30T00:56:19.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.589+0000 7fee1aa00700 1 Processor -- start 2024-03-30T00:56:19.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.589+0000 7fee1aa00700 1 -- start start 2024-03-30T00:56:19.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee1aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14100d80 0x7fee14198fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:19.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee1aa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee14101f70 0x7fee14199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:19.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee1aa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee14102910 0x7fee141a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:19.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee1aa00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee141a0aa0 con 0x7fee14100d80 2024-03-30T00:56:19.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee1aa00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee141a0be0 con 0x7fee14101f70 2024-03-30T00:56:19.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee1aa00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee141a0d70 con 0x7fee14102910 2024-03-30T00:56:19.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee13e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14100d80 0x7fee14198fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:19.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee13e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14100d80 0x7fee14198fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57562/0 (socket says 172.21.15.116:57562) 2024-03-30T00:56:19.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee13e00700 1 -- 172.21.15.116:0/3661296464 learned_addr learned my addr 172.21.15.116:0/3661296464 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:19.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee18a00700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee14102910 0x7fee141a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:19.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee13400700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee14101f70 0x7fee14199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:19.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee13e00700 1 -- 172.21.15.116:0/3661296464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee14102910 msgr2=0x7fee141a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:19.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee13e00700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee14102910 0x7fee141a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:19.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee13e00700 1 -- 172.21.15.116:0/3661296464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee14101f70 msgr2=0x7fee14199500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:19.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee13e00700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee14101f70 0x7fee14199500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:19.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee13e00700 1 -- 172.21.15.116:0/3661296464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee040095f0 con 0x7fee14100d80 2024-03-30T00:56:19.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.590+0000 7fee13400700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee14101f70 0x7fee14199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:56:19.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.591+0000 7fee13e00700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14100d80 0x7fee14198fc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fee04006010 tx=0x7fee04005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:19.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.591+0000 7fee10c00700 1 -- 172.21.15.116:0/3661296464 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0401d070 con 0x7fee14100d80 2024-03-30T00:56:19.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.591+0000 7fee10c00700 1 -- 172.21.15.116:0/3661296464 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee04004050 con 0x7fee14100d80 2024-03-30T00:56:19.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.591+0000 7fee10c00700 1 -- 172.21.15.116:0/3661296464 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04022e00 con 0x7fee14100d80 2024-03-30T00:56:19.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.591+0000 7fee1aa00700 1 -- 172.21.15.116:0/3661296464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee141a0ff0 con 0x7fee14100d80 2024-03-30T00:56:19.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.591+0000 7fee1aa00700 1 -- 172.21.15.116:0/3661296464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee141a1360 con 0x7fee14100d80 2024-03-30T00:56:19.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.592+0000 7fee1aa00700 1 -- 172.21.15.116:0/3661296464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee14067af0 con 0x7fee14100d80 2024-03-30T00:56:19.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.594+0000 7fee10c00700 1 -- 172.21.15.116:0/3661296464 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fee04022500 con 0x7fee14100d80 2024-03-30T00:56:19.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.595+0000 7fee10c00700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fedf80a11a0 0x7fedf80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:19.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.595+0000 7fee10c00700 1 -- 172.21.15.116:0/3661296464 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fee040ce550 con 0x7fee14100d80 2024-03-30T00:56:19.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.595+0000 7fee13400700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fedf80a11a0 0x7fedf80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:19.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.596+0000 7fee13400700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fedf80a11a0 0x7fedf80a3660 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fee00005fd0 tx=0x7fee00005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:19.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.597+0000 7fee10c00700 1 -- 172.21.15.116:0/3661296464 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fee04097b20 con 0x7fee14100d80 2024-03-30T00:56:19.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:19 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2309629442' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:19.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:19 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/125220276' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-30T00:56:19.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:19 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2309629442' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:19.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:19 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/125220276' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-30T00:56:19.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:19 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2309629442' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:19.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:19 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/125220276' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-30T00:56:19.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:19.850+0000 7fee1aa00700 1 -- 172.21.15.116:0/3661296464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7fee1410c830 con 0x7fee14100d80 2024-03-30T00:56:20.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.379+0000 7fee10c00700 1 -- 172.21.15.116:0/3661296464 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7fee040270b0 con 0x7fee14100d80 2024-03-30T00:56:20.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.382+0000 7fee1aa00700 1 -- 172.21.15.116:0/3661296464 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fedf80a11a0 msgr2=0x7fedf80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:20.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.382+0000 7fee1aa00700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fedf80a11a0 0x7fedf80a3660 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fee00005fd0 tx=0x7fee00005f60 comp rx=0 tx=0).stop 2024-03-30T00:56:20.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.382+0000 7fee1aa00700 1 -- 172.21.15.116:0/3661296464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14100d80 msgr2=0x7fee14198fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:20.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.382+0000 7fee1aa00700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14100d80 0x7fee14198fc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fee04006010 tx=0x7fee04005740 comp rx=0 tx=0).stop 2024-03-30T00:56:20.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.382+0000 7fee1aa00700 1 -- 172.21.15.116:0/3661296464 shutdown_connections 2024-03-30T00:56:20.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.382+0000 7fee1aa00700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fedf80a11a0 0x7fedf80a3660 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:20.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.382+0000 7fee1aa00700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee14102910 0x7fee141a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:20.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.382+0000 7fee1aa00700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14100d80 0x7fee14198fc0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:20.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.382+0000 7fee1aa00700 1 --2- 172.21.15.116:0/3661296464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee14101f70 0x7fee14199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:20.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.382+0000 7fee1aa00700 1 -- 172.21.15.116:0/3661296464 >> 172.21.15.116:0/3661296464 conn(0x7fee140fc2f0 msgr2=0x7fee14072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:20.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.382+0000 7fee1aa00700 1 -- 172.21.15.116:0/3661296464 shutdown_connections 2024-03-30T00:56:20.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.382+0000 7fee1aa00700 1 -- 172.21.15.116:0/3661296464 wait complete. 2024-03-30T00:56:20.414 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-30T00:56:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.603+0000 7f53faa00700 1 -- 172.21.15.116:0/3239508035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f4101f70 msgr2=0x7f53f41023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.603+0000 7f53faa00700 1 --2- 172.21.15.116:0/3239508035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f4101f70 0x7f53f41023d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f53e00099c0 tx=0x7f53e0009cd0 comp rx=0 tx=0).stop 2024-03-30T00:56:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.603+0000 7f53faa00700 1 -- 172.21.15.116:0/3239508035 shutdown_connections 2024-03-30T00:56:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.603+0000 7f53faa00700 1 --2- 172.21.15.116:0/3239508035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53f4102910 0x7f53f41091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.603+0000 7f53faa00700 1 --2- 172.21.15.116:0/3239508035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53f4100d80 0x7f53f41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.603+0000 7f53faa00700 1 --2- 172.21.15.116:0/3239508035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f4101f70 0x7f53f41023d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.603+0000 7f53faa00700 1 -- 172.21.15.116:0/3239508035 >> 172.21.15.116:0/3239508035 conn(0x7f53f40fc2f0 msgr2=0x7f53f40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.603+0000 7f53faa00700 1 -- 172.21.15.116:0/3239508035 shutdown_connections 2024-03-30T00:56:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.604+0000 7f53faa00700 1 -- 172.21.15.116:0/3239508035 wait complete. 2024-03-30T00:56:20.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.604+0000 7f53faa00700 1 Processor -- start 2024-03-30T00:56:20.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.605+0000 7f53faa00700 1 -- start start 2024-03-30T00:56:20.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.605+0000 7f53faa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53f4100d80 0x7f53f4198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:20.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.605+0000 7f53faa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f4101f70 0x7f53f4199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:20.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.605+0000 7f53faa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53f4102910 0x7f53f41a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:20.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.605+0000 7f53faa00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f41a0a30 con 0x7f53f4101f70 2024-03-30T00:56:20.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.605+0000 7f53faa00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f41a0b70 con 0x7f53f4102910 2024-03-30T00:56:20.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.605+0000 7f53faa00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f41a0d00 con 0x7f53f4100d80 2024-03-30T00:56:20.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.605+0000 7f53f3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f4101f70 0x7f53f4199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:20.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.605+0000 7f53f8a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53f4102910 0x7f53f41a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:20.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.605+0000 7f53f3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f4101f70 0x7f53f4199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57574/0 (socket says 172.21.15.116:57574) 2024-03-30T00:56:20.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.605+0000 7f53f3400700 1 -- 172.21.15.116:0/1400382968 learned_addr learned my addr 172.21.15.116:0/1400382968 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:20.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.605+0000 7f53f3e00700 1 --2- 172.21.15.116:0/1400382968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53f4100d80 0x7f53f4198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:20.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.606+0000 7f53f3400700 1 -- 172.21.15.116:0/1400382968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53f4100d80 msgr2=0x7f53f4198f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:20.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.606+0000 7f53f3400700 1 --2- 172.21.15.116:0/1400382968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53f4100d80 0x7f53f4198f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:20.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.606+0000 7f53f3400700 1 -- 172.21.15.116:0/1400382968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53f4102910 msgr2=0x7f53f41a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:20.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.606+0000 7f53f3400700 1 --2- 172.21.15.116:0/1400382968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53f4102910 0x7f53f41a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:20.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.606+0000 7f53f3400700 1 -- 172.21.15.116:0/1400382968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e00095f0 con 0x7f53f4101f70 2024-03-30T00:56:20.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.606+0000 7f53f8a00700 1 --2- 172.21.15.116:0/1400382968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53f4102910 0x7f53f41a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:20.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.606+0000 7f53f3400700 1 --2- 172.21.15.116:0/1400382968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f4101f70 0x7f53f4199440 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f53e0006010 tx=0x7f53e000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:20.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.606+0000 7f53f0c00700 1 -- 172.21.15.116:0/1400382968 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e001d070 con 0x7f53f4101f70 2024-03-30T00:56:20.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.606+0000 7f53f0c00700 1 -- 172.21.15.116:0/1400382968 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53e0005030 con 0x7f53f4101f70 2024-03-30T00:56:20.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.606+0000 7f53faa00700 1 -- 172.21.15.116:0/1400382968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f41a0f80 con 0x7f53f4101f70 2024-03-30T00:56:20.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.606+0000 7f53f0c00700 1 -- 172.21.15.116:0/1400382968 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e0004b10 con 0x7f53f4101f70 2024-03-30T00:56:20.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.606+0000 7f53faa00700 1 -- 172.21.15.116:0/1400382968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f41a13a0 con 0x7f53f4101f70 2024-03-30T00:56:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.607+0000 7f53faa00700 1 -- 172.21.15.116:0/1400382968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f4067af0 con 0x7f53f4101f70 2024-03-30T00:56:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.609+0000 7f53f0c00700 1 -- 172.21.15.116:0/1400382968 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f53e00051a0 con 0x7f53f4101f70 2024-03-30T00:56:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.610+0000 7f53f0c00700 1 --2- 172.21.15.116:0/1400382968 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f53d80a11a0 0x7f53d80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.611+0000 7f53f0c00700 1 -- 172.21.15.116:0/1400382968 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f53e00cd740 con 0x7f53f4101f70 2024-03-30T00:56:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.611+0000 7f53f3e00700 1 --2- 172.21.15.116:0/1400382968 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f53d80a11a0 0x7f53d80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.611+0000 7f53f3e00700 1 --2- 172.21.15.116:0/1400382968 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f53d80a11a0 0x7f53d80a3660 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f53f4101dd0 tx=0x7f53e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:20.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.614+0000 7f53f0c00700 1 -- 172.21.15.116:0/1400382968 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f53e0096d10 con 0x7f53f4101f70 2024-03-30T00:56:20.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:20 smithi116 ceph-mon[38086]: pgmap v243: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-30T00:56:20.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:20 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/125220276' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-30T00:56:20.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:20 smithi116 ceph-mon[38086]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T00:56:20.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:20 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3661296464' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-30T00:56:20.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:20 smithi181 ceph-mon[47622]: pgmap v243: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-30T00:56:20.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:20 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/125220276' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-30T00:56:20.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:20 smithi181 ceph-mon[47622]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T00:56:20.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:20 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3661296464' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-30T00:56:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:20 smithi130 ceph-mon[47626]: pgmap v243: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-30T00:56:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:20 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/125220276' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-30T00:56:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:20 smithi130 ceph-mon[47626]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T00:56:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:20 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3661296464' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-30T00:56:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:20.867+0000 7f53faa00700 1 -- 172.21.15.116:0/1400382968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f53f410c5e0 con 0x7f53f4101f70 2024-03-30T00:56:21.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.394+0000 7f53f0c00700 1 -- 172.21.15.116:0/1400382968 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f53e009a330 con 0x7f53f4101f70 2024-03-30T00:56:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.396+0000 7f53faa00700 1 -- 172.21.15.116:0/1400382968 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f53d80a11a0 msgr2=0x7f53d80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.396+0000 7f53faa00700 1 --2- 172.21.15.116:0/1400382968 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f53d80a11a0 0x7f53d80a3660 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f53f4101dd0 tx=0x7f53e4009260 comp rx=0 tx=0).stop 2024-03-30T00:56:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.396+0000 7f53faa00700 1 -- 172.21.15.116:0/1400382968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f4101f70 msgr2=0x7f53f4199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.396+0000 7f53faa00700 1 --2- 172.21.15.116:0/1400382968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f4101f70 0x7f53f4199440 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f53e0006010 tx=0x7f53e000f690 comp rx=0 tx=0).stop 2024-03-30T00:56:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.396+0000 7f53faa00700 1 -- 172.21.15.116:0/1400382968 shutdown_connections 2024-03-30T00:56:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.396+0000 7f53faa00700 1 --2- 172.21.15.116:0/1400382968 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f53d80a11a0 0x7f53d80a3660 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:21.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.396+0000 7f53faa00700 1 --2- 172.21.15.116:0/1400382968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53f4100d80 0x7f53f4198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:21.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.396+0000 7f53faa00700 1 --2- 172.21.15.116:0/1400382968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53f4102910 0x7f53f41a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:21.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.396+0000 7f53faa00700 1 --2- 172.21.15.116:0/1400382968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53f4101f70 0x7f53f4199440 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:21.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.396+0000 7f53faa00700 1 -- 172.21.15.116:0/1400382968 >> 172.21.15.116:0/1400382968 conn(0x7f53f40fc2f0 msgr2=0x7f53f4107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:21.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.397+0000 7f53faa00700 1 -- 172.21.15.116:0/1400382968 shutdown_connections 2024-03-30T00:56:21.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.397+0000 7f53faa00700 1 -- 172.21.15.116:0/1400382968 wait complete. 2024-03-30T00:56:21.426 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-30T00:56:21.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.614+0000 7ff7d7c00700 1 -- 172.21.15.116:0/2326094688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7d0074df0 msgr2=0x7ff7d0073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:21.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.614+0000 7ff7d7c00700 1 --2- 172.21.15.116:0/2326094688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7d0074df0 0x7ff7d0073240 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4009990 tx=0x7ff7c4009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:21.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.614+0000 7ff7d7c00700 1 -- 172.21.15.116:0/2326094688 shutdown_connections 2024-03-30T00:56:21.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.614+0000 7ff7d7c00700 1 --2- 172.21.15.116:0/2326094688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7d0108c40 0x7ff7d010b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:21.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.614+0000 7ff7d7c00700 1 --2- 172.21.15.116:0/2326094688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7d0074df0 0x7ff7d0073240 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:21.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.614+0000 7ff7d7c00700 1 --2- 172.21.15.116:0/2326094688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7d0073810 0x7ff7d0073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:21.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.614+0000 7ff7d7c00700 1 -- 172.21.15.116:0/2326094688 >> 172.21.15.116:0/2326094688 conn(0x7ff7d00fc2f0 msgr2=0x7ff7d00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:21.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.614+0000 7ff7d7c00700 1 -- 172.21.15.116:0/2326094688 shutdown_connections 2024-03-30T00:56:21.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.614+0000 7ff7d7c00700 1 -- 172.21.15.116:0/2326094688 wait complete. 2024-03-30T00:56:21.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.615+0000 7ff7d7c00700 1 Processor -- start 2024-03-30T00:56:21.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.615+0000 7ff7d7c00700 1 -- start start 2024-03-30T00:56:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.615+0000 7ff7d7c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7d0073810 0x7ff7d0198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.615+0000 7ff7d7c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7d0074df0 0x7ff7d0199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.615+0000 7ff7d7c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7d0108c40 0x7ff7d01a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.615+0000 7ff7d7c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d01a0a50 con 0x7ff7d0073810 2024-03-30T00:56:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.615+0000 7ff7d7c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d01a0b90 con 0x7ff7d0108c40 2024-03-30T00:56:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.615+0000 7ff7d7c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d01a0d20 con 0x7ff7d0074df0 2024-03-30T00:56:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.615+0000 7ff7d5200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7d0073810 0x7ff7d0198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.615+0000 7ff7cfe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7d0074df0 0x7ff7d0199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.615+0000 7ff7d5200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7d0073810 0x7ff7d0198f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57582/0 (socket says 172.21.15.116:57582) 2024-03-30T00:56:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.615+0000 7ff7d5200700 1 -- 172.21.15.116:0/1345059362 learned_addr learned my addr 172.21.15.116:0/1345059362 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.616+0000 7ff7d5c00700 1 --2- 172.21.15.116:0/1345059362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7d0108c40 0x7ff7d01a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.616+0000 7ff7d5200700 1 -- 172.21.15.116:0/1345059362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7d0074df0 msgr2=0x7ff7d0199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.616+0000 7ff7d5200700 1 --2- 172.21.15.116:0/1345059362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7d0074df0 0x7ff7d0199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.616+0000 7ff7d5200700 1 -- 172.21.15.116:0/1345059362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7d0108c40 msgr2=0x7ff7d01a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.616+0000 7ff7d5200700 1 --2- 172.21.15.116:0/1345059362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7d0108c40 0x7ff7d01a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.616+0000 7ff7d5200700 1 -- 172.21.15.116:0/1345059362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c40095f0 con 0x7ff7d0073810 2024-03-30T00:56:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.616+0000 7ff7cfe00700 1 --2- 172.21.15.116:0/1345059362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7d0074df0 0x7ff7d0199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.616+0000 7ff7d5200700 1 --2- 172.21.15.116:0/1345059362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7d0073810 0x7ff7d0198f20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400b680 tx=0x7ff7c4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.616+0000 7ff7cd600700 1 -- 172.21.15.116:0/1345059362 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c401d070 con 0x7ff7d0073810 2024-03-30T00:56:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.616+0000 7ff7cd600700 1 -- 172.21.15.116:0/1345059362 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff7c4022470 con 0x7ff7d0073810 2024-03-30T00:56:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.616+0000 7ff7cd600700 1 -- 172.21.15.116:0/1345059362 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4017650 con 0x7ff7d0073810 2024-03-30T00:56:21.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.616+0000 7ff7d7c00700 1 -- 172.21.15.116:0/1345059362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d01a0fa0 con 0x7ff7d0073810 2024-03-30T00:56:21.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.617+0000 7ff7d7c00700 1 -- 172.21.15.116:0/1345059362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d01a1310 con 0x7ff7d0073810 2024-03-30T00:56:21.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.617+0000 7ff7d7c00700 1 -- 172.21.15.116:0/1345059362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d0107a00 con 0x7ff7d0073810 2024-03-30T00:56:21.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.619+0000 7ff7cd600700 1 -- 172.21.15.116:0/1345059362 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7ff7c40225e0 con 0x7ff7d0073810 2024-03-30T00:56:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.621+0000 7ff7cd600700 1 --2- 172.21.15.116:0/1345059362 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff7b80a11a0 0x7ff7b80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.621+0000 7ff7cd600700 1 -- 172.21.15.116:0/1345059362 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff7c40cdd70 con 0x7ff7d0073810 2024-03-30T00:56:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.621+0000 7ff7cfe00700 1 --2- 172.21.15.116:0/1345059362 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff7b80a11a0 0x7ff7b80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.621+0000 7ff7cfe00700 1 --2- 172.21.15.116:0/1345059362 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff7b80a11a0 0x7ff7b80a3660 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0074b10 tx=0x7ff7bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:21.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.624+0000 7ff7cd600700 1 -- 172.21.15.116:0/1345059362 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff7c4097340 con 0x7ff7d0073810 2024-03-30T00:56:21.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:21 smithi116 ceph-mon[38086]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-30T00:56:21.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:21 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3661296464' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-30T00:56:21.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:21 smithi116 ceph-mon[38086]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-30T00:56:21.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:21 smithi116 ceph-mon[38086]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T00:56:21.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:21 smithi116 ceph-mon[38086]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-30T00:56:21.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:21 smithi116 ceph-mon[38086]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-30T00:56:21.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:21 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1400382968' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-30T00:56:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:21 smithi181 ceph-mon[47622]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-30T00:56:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:21 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3661296464' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-30T00:56:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:21 smithi181 ceph-mon[47622]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-30T00:56:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:21 smithi181 ceph-mon[47622]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T00:56:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:21 smithi181 ceph-mon[47622]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-30T00:56:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:21 smithi181 ceph-mon[47622]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-30T00:56:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:21 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1400382968' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-30T00:56:21.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:21 smithi130 ceph-mon[47626]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-30T00:56:21.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:21 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3661296464' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-30T00:56:21.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:21 smithi130 ceph-mon[47626]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-30T00:56:21.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:21 smithi130 ceph-mon[47626]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T00:56:21.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:21 smithi130 ceph-mon[47626]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-30T00:56:21.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:21 smithi130 ceph-mon[47626]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-30T00:56:21.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:21 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1400382968' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-30T00:56:21.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:21.877+0000 7ff7d7c00700 1 -- 172.21.15.116:0/1345059362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7ff7d010e7c0 con 0x7ff7d0073810 2024-03-30T00:56:22.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.147+0000 7ff7cd600700 1 -- 172.21.15.116:0/1345059362 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7ff7c409a960 con 0x7ff7d0073810 2024-03-30T00:56:22.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.150+0000 7ff7d7c00700 1 -- 172.21.15.116:0/1345059362 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff7b80a11a0 msgr2=0x7ff7b80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:22.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.150+0000 7ff7d7c00700 1 --2- 172.21.15.116:0/1345059362 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff7b80a11a0 0x7ff7b80a3660 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0074b10 tx=0x7ff7bc009260 comp rx=0 tx=0).stop 2024-03-30T00:56:22.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.150+0000 7ff7d7c00700 1 -- 172.21.15.116:0/1345059362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7d0073810 msgr2=0x7ff7d0198f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:22.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.150+0000 7ff7d7c00700 1 --2- 172.21.15.116:0/1345059362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7d0073810 0x7ff7d0198f20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400b680 tx=0x7ff7c4009580 comp rx=0 tx=0).stop 2024-03-30T00:56:22.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.150+0000 7ff7d7c00700 1 -- 172.21.15.116:0/1345059362 shutdown_connections 2024-03-30T00:56:22.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.150+0000 7ff7d7c00700 1 --2- 172.21.15.116:0/1345059362 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff7b80a11a0 0x7ff7b80a3660 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.150+0000 7ff7d7c00700 1 --2- 172.21.15.116:0/1345059362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7d0108c40 0x7ff7d01a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.150+0000 7ff7d7c00700 1 --2- 172.21.15.116:0/1345059362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7d0073810 0x7ff7d0198f20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.150+0000 7ff7d7c00700 1 --2- 172.21.15.116:0/1345059362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7d0074df0 0x7ff7d0199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.150+0000 7ff7d7c00700 1 -- 172.21.15.116:0/1345059362 >> 172.21.15.116:0/1345059362 conn(0x7ff7d00fc2f0 msgr2=0x7ff7d0104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:22.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.150+0000 7ff7d7c00700 1 -- 172.21.15.116:0/1345059362 shutdown_connections 2024-03-30T00:56:22.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.150+0000 7ff7d7c00700 1 -- 172.21.15.116:0/1345059362 wait complete. 2024-03-30T00:56:22.176 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-30T00:56:22.177 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-30T00:56:22.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.366+0000 7f07d8600700 1 -- 172.21.15.116:0/941249055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d0108c40 msgr2=0x7f07d010b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:22.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.366+0000 7f07d8600700 1 --2- 172.21.15.116:0/941249055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d0108c40 0x7f07d010b0d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f07c800b0a0 tx=0x7f07c800b3b0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.366+0000 7f07d8600700 1 -- 172.21.15.116:0/941249055 shutdown_connections 2024-03-30T00:56:22.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.366+0000 7f07d8600700 1 --2- 172.21.15.116:0/941249055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07d0074df0 0x7f07d0073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.366+0000 7f07d8600700 1 --2- 172.21.15.116:0/941249055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d0108c40 0x7f07d010b0d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.366+0000 7f07d8600700 1 --2- 172.21.15.116:0/941249055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07d0073810 0x7f07d0073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.366+0000 7f07d8600700 1 -- 172.21.15.116:0/941249055 >> 172.21.15.116:0/941249055 conn(0x7f07d00fc2f0 msgr2=0x7f07d00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:22.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.366+0000 7f07d8600700 1 -- 172.21.15.116:0/941249055 shutdown_connections 2024-03-30T00:56:22.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.366+0000 7f07d8600700 1 -- 172.21.15.116:0/941249055 wait complete. 2024-03-30T00:56:22.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.367+0000 7f07d8600700 1 Processor -- start 2024-03-30T00:56:22.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.367+0000 7f07d8600700 1 -- start start 2024-03-30T00:56:22.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d8600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07d0073810 0x7f07d0199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:22.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d8600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07d0074df0 0x7f07d0199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:22.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d8600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d0108c40 0x7f07d01a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:22.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d8600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d01a0b80 con 0x7f07d0108c40 2024-03-30T00:56:22.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d8600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d01a0cc0 con 0x7f07d0073810 2024-03-30T00:56:22.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d8600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d01a0e50 con 0x7f07d0074df0 2024-03-30T00:56:22.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d6600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d0108c40 0x7f07d01a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:22.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d6600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d0108c40 0x7f07d01a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57600/0 (socket says 172.21.15.116:57600) 2024-03-30T00:56:22.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d6600700 1 -- 172.21.15.116:0/2056934801 learned_addr learned my addr 172.21.15.116:0/2056934801 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:22.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d5c00700 1 --2- 172.21.15.116:0/2056934801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07d0073810 0x7f07d0199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:22.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d5200700 1 --2- 172.21.15.116:0/2056934801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07d0074df0 0x7f07d0199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:22.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d6600700 1 -- 172.21.15.116:0/2056934801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07d0074df0 msgr2=0x7f07d0199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:22.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d6600700 1 --2- 172.21.15.116:0/2056934801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07d0074df0 0x7f07d0199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d6600700 1 -- 172.21.15.116:0/2056934801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07d0073810 msgr2=0x7f07d0199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:22.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d6600700 1 --2- 172.21.15.116:0/2056934801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07d0073810 0x7f07d0199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.368+0000 7f07d6600700 1 -- 172.21.15.116:0/2056934801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c8009d00 con 0x7f07d0108c40 2024-03-30T00:56:22.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.369+0000 7f07d6600700 1 --2- 172.21.15.116:0/2056934801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d0108c40 0x7f07d01a05b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f07c8007a60 tx=0x7f07c800bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:22.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.369+0000 7f07ce000700 1 -- 172.21.15.116:0/2056934801 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c800e040 con 0x7f07d0108c40 2024-03-30T00:56:22.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.369+0000 7f07ce000700 1 -- 172.21.15.116:0/2056934801 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f07c80094c0 con 0x7f07d0108c40 2024-03-30T00:56:22.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.369+0000 7f07d8600700 1 -- 172.21.15.116:0/2056934801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d01a10d0 con 0x7f07d0108c40 2024-03-30T00:56:22.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.369+0000 7f07ce000700 1 -- 172.21.15.116:0/2056934801 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c80052a0 con 0x7f07d0108c40 2024-03-30T00:56:22.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.369+0000 7f07d8600700 1 -- 172.21.15.116:0/2056934801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d01a15a0 con 0x7f07d0108c40 2024-03-30T00:56:22.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.370+0000 7f07d8600700 1 -- 172.21.15.116:0/2056934801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d0107a00 con 0x7f07d0108c40 2024-03-30T00:56:22.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.372+0000 7f07ce000700 1 -- 172.21.15.116:0/2056934801 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f07c801d470 con 0x7f07d0108c40 2024-03-30T00:56:22.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.373+0000 7f07ce000700 1 --2- 172.21.15.116:0/2056934801 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f07b80a11a0 0x7f07b80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:22.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.373+0000 7f07ce000700 1 -- 172.21.15.116:0/2056934801 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f07c80cdfe0 con 0x7f07d0108c40 2024-03-30T00:56:22.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.373+0000 7f07d5c00700 1 --2- 172.21.15.116:0/2056934801 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f07b80a11a0 0x7f07b80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:22.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.374+0000 7f07d5c00700 1 --2- 172.21.15.116:0/2056934801 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f07b80a11a0 0x7f07b80a3660 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f07c4007be0 tx=0x7f07c40073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:22.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.377+0000 7f07ce000700 1 -- 172.21.15.116:0/2056934801 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f07c80975f0 con 0x7f07d0108c40 2024-03-30T00:56:22.594 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 00:56:22 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-b[84904]: 2024-03-30T00:56:22.170+0000 7ffbd37aa700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-30T00:56:22.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:22 smithi130 ceph-mon[47626]: pgmap v244: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-30T00:56:22.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:22 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] up:active 2024-03-30T00:56:22.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:22 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1400382968' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-30T00:56:22.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:22 smithi130 ceph-mon[47626]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-30T00:56:22.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:22 smithi130 ceph-mon[47626]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-30T00:56:22.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:22 smithi130 ceph-mon[47626]: Cluster is now healthy 2024-03-30T00:56:22.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:22 smithi130 ceph-mon[47626]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-30T00:56:22.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:22 smithi130 ceph-mon[47626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-30T00:56:22.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:22 smithi130 ceph-mon[47626]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-30T00:56:22.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:22 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1345059362' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-30T00:56:22.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:22 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] up:active 2024-03-30T00:56:22.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:22 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1345059362' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-30T00:56:22.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:22 smithi130 ceph-mon[47626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T00:56:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.618+0000 7f07d8600700 1 -- 172.21.15.116:0/2056934801 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f07d00611f0 con 0x7f07b80a11a0 2024-03-30T00:56:22.632 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-mon[38086]: pgmap v244: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-30T00:56:22.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] up:active 2024-03-30T00:56:22.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1400382968' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-30T00:56:22.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-mon[38086]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-30T00:56:22.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-mon[38086]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-30T00:56:22.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-mon[38086]: Cluster is now healthy 2024-03-30T00:56:22.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-mon[38086]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-30T00:56:22.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-mon[38086]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-30T00:56:22.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-mon[38086]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-30T00:56:22.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1345059362' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-30T00:56:22.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] up:active 2024-03-30T00:56:22.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1345059362' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-30T00:56:22.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-mon[38086]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T00:56:22.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.647+0000 7f07ce000700 1 -- 172.21.15.116:0/2056934801 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f07d00611f0 con 0x7f07b80a11a0 2024-03-30T00:56:22.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.649+0000 7f07d8600700 1 -- 172.21.15.116:0/2056934801 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f07b80a11a0 msgr2=0x7f07b80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:22.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.649+0000 7f07d8600700 1 --2- 172.21.15.116:0/2056934801 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f07b80a11a0 0x7f07b80a3660 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f07c4007be0 tx=0x7f07c40073d0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.649+0000 7f07d8600700 1 -- 172.21.15.116:0/2056934801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d0108c40 msgr2=0x7f07d01a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:22.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.650+0000 7f07d8600700 1 --2- 172.21.15.116:0/2056934801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d0108c40 0x7f07d01a05b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f07c8007a60 tx=0x7f07c800bce0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.650+0000 7f07d8600700 1 -- 172.21.15.116:0/2056934801 shutdown_connections 2024-03-30T00:56:22.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.650+0000 7f07d8600700 1 --2- 172.21.15.116:0/2056934801 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f07b80a11a0 0x7f07b80a3660 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.650+0000 7f07d8600700 1 --2- 172.21.15.116:0/2056934801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d0108c40 0x7f07d01a05b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.650+0000 7f07d8600700 1 --2- 172.21.15.116:0/2056934801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07d0073810 0x7f07d0199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.650+0000 7f07d8600700 1 --2- 172.21.15.116:0/2056934801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07d0074df0 0x7f07d0199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.650+0000 7f07d8600700 1 -- 172.21.15.116:0/2056934801 >> 172.21.15.116:0/2056934801 conn(0x7f07d00fc2f0 msgr2=0x7f07d0104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:22.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.650+0000 7f07d8600700 1 -- 172.21.15.116:0/2056934801 shutdown_connections 2024-03-30T00:56:22.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.650+0000 7f07d8600700 1 -- 172.21.15.116:0/2056934801 wait complete. 2024-03-30T00:56:22.675 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-30T00:56:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:22 smithi181 ceph-mon[47622]: pgmap v244: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-30T00:56:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:22 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] up:active 2024-03-30T00:56:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:22 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1400382968' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-30T00:56:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:22 smithi181 ceph-mon[47622]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-30T00:56:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:22 smithi181 ceph-mon[47622]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-30T00:56:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:22 smithi181 ceph-mon[47622]: Cluster is now healthy 2024-03-30T00:56:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:22 smithi181 ceph-mon[47622]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-30T00:56:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:22 smithi181 ceph-mon[47622]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-30T00:56:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:22 smithi181 ceph-mon[47622]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-30T00:56:22.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:22 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1345059362' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-30T00:56:22.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:22 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] up:active 2024-03-30T00:56:22.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:22 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1345059362' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-30T00:56:22.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:22 smithi181 ceph-mon[47622]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T00:56:22.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.867+0000 7f528b000700 1 -- 172.21.15.116:0/3865058336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5284106b10 msgr2=0x7f5284108f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:22.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.867+0000 7f528b000700 1 --2- 172.21.15.116:0/3865058336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5284106b10 0x7f5284108f00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f527c00b0a0 tx=0x7f527c00b3b0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.867+0000 7f528b000700 1 -- 172.21.15.116:0/3865058336 shutdown_connections 2024-03-30T00:56:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.867+0000 7f528b000700 1 --2- 172.21.15.116:0/3865058336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5284106b10 0x7f5284108f00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.867+0000 7f528b000700 1 --2- 172.21.15.116:0/3865058336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f52841018b0 0x7f5284103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.867+0000 7f528b000700 1 --2- 172.21.15.116:0/3865058336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52841041e0 0x7f52841065d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.867+0000 7f528b000700 1 -- 172.21.15.116:0/3865058336 >> 172.21.15.116:0/3865058336 conn(0x7f52840fb200 msgr2=0x7f52840fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.867+0000 7f528b000700 1 -- 172.21.15.116:0/3865058336 shutdown_connections 2024-03-30T00:56:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.867+0000 7f528b000700 1 -- 172.21.15.116:0/3865058336 wait complete. 2024-03-30T00:56:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.868+0000 7f528b000700 1 Processor -- start 2024-03-30T00:56:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.868+0000 7f528b000700 1 -- start start 2024-03-30T00:56:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.868+0000 7f528b000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52841018b0 0x7f52841011d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.868+0000 7f528b000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52841041e0 0x7f52840ff860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.868+0000 7f528b000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5284106b10 0x7f52840ffda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.868+0000 7f528b000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5284100540 con 0x7f52841041e0 2024-03-30T00:56:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.868+0000 7f528b000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52841006b0 con 0x7f5284106b10 2024-03-30T00:56:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.868+0000 7f528b000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52841a0e30 con 0x7f52841018b0 2024-03-30T00:56:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.869+0000 7f5283400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52841041e0 0x7f52840ff860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.869+0000 7f5283e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52841018b0 0x7f52841011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.869+0000 7f5283400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52841041e0 0x7f52840ff860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57646/0 (socket says 172.21.15.116:57646) 2024-03-30T00:56:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.869+0000 7f5283400700 1 -- 172.21.15.116:0/4169981953 learned_addr learned my addr 172.21.15.116:0/4169981953 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.869+0000 7f5289000700 1 --2- 172.21.15.116:0/4169981953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5284106b10 0x7f52840ffda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.869+0000 7f5283400700 1 -- 172.21.15.116:0/4169981953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52841018b0 msgr2=0x7f52841011d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.869+0000 7f5283400700 1 --2- 172.21.15.116:0/4169981953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52841018b0 0x7f52841011d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.869+0000 7f5283400700 1 -- 172.21.15.116:0/4169981953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5284106b10 msgr2=0x7f52840ffda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.869+0000 7f5283400700 1 --2- 172.21.15.116:0/4169981953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5284106b10 0x7f52840ffda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.869+0000 7f5283400700 1 -- 172.21.15.116:0/4169981953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c009d00 con 0x7f52841041e0 2024-03-30T00:56:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.869+0000 7f5283400700 1 --2- 172.21.15.116:0/4169981953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52841041e0 0x7f52840ff860 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f527000e9a0 tx=0x7f527000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.870+0000 7f5280c00700 1 -- 172.21.15.116:0/4169981953 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270009790 con 0x7f52841041e0 2024-03-30T00:56:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.870+0000 7f5280c00700 1 -- 172.21.15.116:0/4169981953 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f527000ce70 con 0x7f52841041e0 2024-03-30T00:56:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.870+0000 7f528b000700 1 -- 172.21.15.116:0/4169981953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52841a10c0 con 0x7f52841041e0 2024-03-30T00:56:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.870+0000 7f5280c00700 1 -- 172.21.15.116:0/4169981953 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270018420 con 0x7f52841041e0 2024-03-30T00:56:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.870+0000 7f528b000700 1 -- 172.21.15.116:0/4169981953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52841a1620 con 0x7f52841041e0 2024-03-30T00:56:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.871+0000 7f528b000700 1 -- 172.21.15.116:0/4169981953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52840fcd80 con 0x7f52841041e0 2024-03-30T00:56:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.872+0000 7f5280c00700 1 -- 172.21.15.116:0/4169981953 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f52700106f0 con 0x7f52841041e0 2024-03-30T00:56:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.874+0000 7f5280c00700 1 --2- 172.21.15.116:0/4169981953 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f52680a11a0 0x7f52680a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.875+0000 7f5280c00700 1 -- 172.21.15.116:0/4169981953 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5270014070 con 0x7f52841041e0 2024-03-30T00:56:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.875+0000 7f5283e00700 1 --2- 172.21.15.116:0/4169981953 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f52680a11a0 0x7f52680a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.875+0000 7f5283e00700 1 --2- 172.21.15.116:0/4169981953 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f52680a11a0 0x7f52680a3660 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5274007c60 tx=0x7f52740073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:22.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:22.879+0000 7f5280c00700 1 -- 172.21.15.116:0/4169981953 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f52700529f0 con 0x7f52841041e0 2024-03-30T00:56:22.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:56:22.624+0000 7fc73e200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T00:56:22.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:56:22.624+0000 7fc73e200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T00:56:22.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:56:22.624+0000 7fc73e200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T00:56:22.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:56:22.624+0000 7fc73e200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T00:56:22.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 00:56:22 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T00:56:22.624+0000 7fc73e200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T00:56:23.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.123+0000 7f528b000700 1 -- 172.21.15.116:0/4169981953 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f52840611f0 con 0x7f52680a11a0 2024-03-30T00:56:23.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.146+0000 7f5280c00700 1 -- 172.21.15.116:0/4169981953 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f52840611f0 con 0x7f52680a11a0 2024-03-30T00:56:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.149+0000 7f528b000700 1 -- 172.21.15.116:0/4169981953 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f52680a11a0 msgr2=0x7f52680a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.149+0000 7f528b000700 1 --2- 172.21.15.116:0/4169981953 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f52680a11a0 0x7f52680a3660 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5274007c60 tx=0x7f52740073d0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.149+0000 7f528b000700 1 -- 172.21.15.116:0/4169981953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52841041e0 msgr2=0x7f52840ff860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.149+0000 7f528b000700 1 --2- 172.21.15.116:0/4169981953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52841041e0 0x7f52840ff860 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f527000e9a0 tx=0x7f527000ecb0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.149+0000 7f528b000700 1 -- 172.21.15.116:0/4169981953 shutdown_connections 2024-03-30T00:56:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.149+0000 7f528b000700 1 --2- 172.21.15.116:0/4169981953 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f52680a11a0 0x7f52680a3660 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.149+0000 7f528b000700 1 --2- 172.21.15.116:0/4169981953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52841018b0 0x7f52841011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.149+0000 7f528b000700 1 --2- 172.21.15.116:0/4169981953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5284106b10 0x7f52840ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.149+0000 7f528b000700 1 --2- 172.21.15.116:0/4169981953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52841041e0 0x7f52840ff860 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.149+0000 7f528b000700 1 -- 172.21.15.116:0/4169981953 >> 172.21.15.116:0/4169981953 conn(0x7f52840fb200 msgr2=0x7f52841022d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.149+0000 7f528b000700 1 -- 172.21.15.116:0/4169981953 shutdown_connections 2024-03-30T00:56:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.149+0000 7f528b000700 1 -- 172.21.15.116:0/4169981953 wait complete. 2024-03-30T00:56:23.174 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-30T00:56:23.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.367+0000 7fbc5b600700 1 -- 172.21.15.116:0/2392642995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc54101f70 msgr2=0x7fbc541023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:23.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.367+0000 7fbc5b600700 1 --2- 172.21.15.116:0/2392642995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc54101f70 0x7fbc541023d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbc400099c0 tx=0x7fbc40009cd0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.367+0000 7fbc5b600700 1 -- 172.21.15.116:0/2392642995 shutdown_connections 2024-03-30T00:56:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.367+0000 7fbc5b600700 1 --2- 172.21.15.116:0/2392642995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc54100d80 0x7fbc541011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.367+0000 7fbc5b600700 1 --2- 172.21.15.116:0/2392642995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc54102910 0x7fbc541091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.367+0000 7fbc5b600700 1 --2- 172.21.15.116:0/2392642995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc54101f70 0x7fbc541023d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.367+0000 7fbc5b600700 1 -- 172.21.15.116:0/2392642995 >> 172.21.15.116:0/2392642995 conn(0x7fbc540fc2f0 msgr2=0x7fbc540fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.367+0000 7fbc5b600700 1 -- 172.21.15.116:0/2392642995 shutdown_connections 2024-03-30T00:56:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.368+0000 7fbc5b600700 1 -- 172.21.15.116:0/2392642995 wait complete. 2024-03-30T00:56:23.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.368+0000 7fbc5b600700 1 Processor -- start 2024-03-30T00:56:23.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.369+0000 7fbc5b600700 1 -- start start 2024-03-30T00:56:23.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.369+0000 7fbc5b600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc54100d80 0x7fbc54074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:23.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.369+0000 7fbc5b600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc54101f70 0x7fbc540731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:23.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.369+0000 7fbc5b600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc54102910 0x7fbc540736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:23.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.369+0000 7fbc5b600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc54073e80 con 0x7fbc54101f70 2024-03-30T00:56:23.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.369+0000 7fbc5b600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc54073ff0 con 0x7fbc54102910 2024-03-30T00:56:23.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.369+0000 7fbc5b600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc541a0d70 con 0x7fbc54100d80 2024-03-30T00:56:23.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.369+0000 7fbc53e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc54101f70 0x7fbc540731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:23.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.369+0000 7fbc53e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc54101f70 0x7fbc540731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57650/0 (socket says 172.21.15.116:57650) 2024-03-30T00:56:23.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.369+0000 7fbc53e00700 1 -- 172.21.15.116:0/3866725647 learned_addr learned my addr 172.21.15.116:0/3866725647 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:23.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.369+0000 7fbc59600700 1 --2- 172.21.15.116:0/3866725647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc54102910 0x7fbc540736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:23.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.369+0000 7fbc58c00700 1 --2- 172.21.15.116:0/3866725647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc54100d80 0x7fbc54074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.370+0000 7fbc53e00700 1 -- 172.21.15.116:0/3866725647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc54100d80 msgr2=0x7fbc54074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.370+0000 7fbc53e00700 1 --2- 172.21.15.116:0/3866725647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc54100d80 0x7fbc54074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.370+0000 7fbc53e00700 1 -- 172.21.15.116:0/3866725647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc54102910 msgr2=0x7fbc540736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.370+0000 7fbc53e00700 1 --2- 172.21.15.116:0/3866725647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc54102910 0x7fbc540736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.370+0000 7fbc53e00700 1 -- 172.21.15.116:0/3866725647 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc400095f0 con 0x7fbc54101f70 2024-03-30T00:56:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.370+0000 7fbc53e00700 1 --2- 172.21.15.116:0/3866725647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc54101f70 0x7fbc540731a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbc40003730 tx=0x7fbc4000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.370+0000 7fbc51600700 1 -- 172.21.15.116:0/3866725647 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4001d070 con 0x7fbc54101f70 2024-03-30T00:56:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.370+0000 7fbc51600700 1 -- 172.21.15.116:0/3866725647 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc40005050 con 0x7fbc54101f70 2024-03-30T00:56:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.370+0000 7fbc5b600700 1 -- 172.21.15.116:0/3866725647 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc541a0fa0 con 0x7fbc54101f70 2024-03-30T00:56:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.370+0000 7fbc51600700 1 -- 172.21.15.116:0/3866725647 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc40004b10 con 0x7fbc54101f70 2024-03-30T00:56:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.370+0000 7fbc5b600700 1 -- 172.21.15.116:0/3866725647 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc541a1310 con 0x7fbc54101f70 2024-03-30T00:56:23.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.371+0000 7fbc5b600700 1 -- 172.21.15.116:0/3866725647 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc54108250 con 0x7fbc54101f70 2024-03-30T00:56:23.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.373+0000 7fbc51600700 1 -- 172.21.15.116:0/3866725647 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fbc400051c0 con 0x7fbc54101f70 2024-03-30T00:56:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.374+0000 7fbc51600700 1 --2- 172.21.15.116:0/3866725647 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbc3c0a1120 0x7fbc3c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.375+0000 7fbc51600700 1 -- 172.21.15.116:0/3866725647 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbc400cd890 con 0x7fbc54101f70 2024-03-30T00:56:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.375+0000 7fbc58c00700 1 --2- 172.21.15.116:0/3866725647 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbc3c0a1120 0x7fbc3c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:23.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.375+0000 7fbc58c00700 1 --2- 172.21.15.116:0/3866725647 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbc3c0a1120 0x7fbc3c0a35e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbc54101dd0 tx=0x7fbc48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:23.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.378+0000 7fbc51600700 1 -- 172.21.15.116:0/3866725647 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbc40096e60 con 0x7fbc54101f70 2024-03-30T00:56:23.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.411+0000 7fbc51600700 1 -- 172.21.15.116:0/3866725647 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fbc40097230 con 0x7fbc54101f70 2024-03-30T00:56:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.627+0000 7fbc5b600700 1 -- 172.21.15.116:0/3866725647 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc540611f0 con 0x7fbc3c0a1120 2024-03-30T00:56:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.631+0000 7fbc51600700 1 -- 172.21.15.116:0/3866725647 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fbc540611f0 con 0x7fbc3c0a1120 2024-03-30T00:56:23.638 INFO:teuthology.orchestra.run.smithi116.stdout:/volumes/qa/sv_0/960eea1e-6b47-46a8-908c-341b47bb61c0 2024-03-30T00:56:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.633+0000 7fbc5b600700 1 -- 172.21.15.116:0/3866725647 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbc3c0a1120 msgr2=0x7fbc3c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.633+0000 7fbc5b600700 1 --2- 172.21.15.116:0/3866725647 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbc3c0a1120 0x7fbc3c0a35e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbc54101dd0 tx=0x7fbc48009260 comp rx=0 tx=0).stop 2024-03-30T00:56:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.633+0000 7fbc5b600700 1 -- 172.21.15.116:0/3866725647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc54101f70 msgr2=0x7fbc540731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.633+0000 7fbc5b600700 1 --2- 172.21.15.116:0/3866725647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc54101f70 0x7fbc540731a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbc40003730 tx=0x7fbc4000f690 comp rx=0 tx=0).stop 2024-03-30T00:56:23.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.633+0000 7fbc5b600700 1 -- 172.21.15.116:0/3866725647 shutdown_connections 2024-03-30T00:56:23.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.633+0000 7fbc5b600700 1 --2- 172.21.15.116:0/3866725647 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbc3c0a1120 0x7fbc3c0a35e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.633+0000 7fbc5b600700 1 --2- 172.21.15.116:0/3866725647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc54100d80 0x7fbc54074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.633+0000 7fbc5b600700 1 --2- 172.21.15.116:0/3866725647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc54102910 0x7fbc540736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.633+0000 7fbc5b600700 1 --2- 172.21.15.116:0/3866725647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc54101f70 0x7fbc540731a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.633+0000 7fbc5b600700 1 -- 172.21.15.116:0/3866725647 >> 172.21.15.116:0/3866725647 conn(0x7fbc540fc2f0 msgr2=0x7fbc54105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:23.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.634+0000 7fbc5b600700 1 -- 172.21.15.116:0/3866725647 shutdown_connections 2024-03-30T00:56:23.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.634+0000 7fbc5b600700 1 -- 172.21.15.116:0/3866725647 wait complete. 2024-03-30T00:56:23.660 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-30T00:56:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.854+0000 7f53b8800700 1 -- 172.21.15.116:0/3885391944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b0074df0 msgr2=0x7f53b0073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.854+0000 7f53b8800700 1 --2- 172.21.15.116:0/3885391944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b0074df0 0x7f53b0073240 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009990 tx=0x7f53a4009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.854+0000 7f53b8800700 1 -- 172.21.15.116:0/3885391944 shutdown_connections 2024-03-30T00:56:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.854+0000 7f53b8800700 1 --2- 172.21.15.116:0/3885391944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53b0108c40 0x7f53b010b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.854+0000 7f53b8800700 1 --2- 172.21.15.116:0/3885391944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b0074df0 0x7f53b0073240 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.854+0000 7f53b8800700 1 --2- 172.21.15.116:0/3885391944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53b0073810 0x7f53b0073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.854+0000 7f53b8800700 1 -- 172.21.15.116:0/3885391944 >> 172.21.15.116:0/3885391944 conn(0x7f53b00fc2f0 msgr2=0x7f53b00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.855+0000 7f53b8800700 1 -- 172.21.15.116:0/3885391944 shutdown_connections 2024-03-30T00:56:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.855+0000 7f53b8800700 1 -- 172.21.15.116:0/3885391944 wait complete. 2024-03-30T00:56:23.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.856+0000 7f53b8800700 1 Processor -- start 2024-03-30T00:56:23.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.856+0000 7f53b8800700 1 -- start start 2024-03-30T00:56:23.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.856+0000 7f53b8800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53b0073810 0x7f53b0198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.856+0000 7f53b8800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53b0074df0 0x7f53b0199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.856+0000 7f53b8800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b0108c40 0x7f53b01a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.856+0000 7f53b8800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b01a0a20 con 0x7f53b0108c40 2024-03-30T00:56:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.856+0000 7f53b8800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b01a0b60 con 0x7f53b0074df0 2024-03-30T00:56:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.856+0000 7f53b8800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b01a0cf0 con 0x7f53b0073810 2024-03-30T00:56:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.856+0000 7f53b6800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b0108c40 0x7f53b01a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.856+0000 7f53b6800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b0108c40 0x7f53b01a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57656/0 (socket says 172.21.15.116:57656) 2024-03-30T00:56:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.856+0000 7f53b6800700 1 -- 172.21.15.116:0/2742055603 learned_addr learned my addr 172.21.15.116:0/2742055603 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.856+0000 7f53b5e00700 1 --2- 172.21.15.116:0/2742055603 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53b0073810 0x7f53b0198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.856+0000 7f53b5400700 1 --2- 172.21.15.116:0/2742055603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53b0074df0 0x7f53b0199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.857+0000 7f53b6800700 1 -- 172.21.15.116:0/2742055603 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53b0073810 msgr2=0x7f53b0198ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.857+0000 7f53b6800700 1 --2- 172.21.15.116:0/2742055603 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53b0073810 0x7f53b0198ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.857+0000 7f53b6800700 1 -- 172.21.15.116:0/2742055603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53b0074df0 msgr2=0x7f53b0199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.857+0000 7f53b6800700 1 --2- 172.21.15.116:0/2742055603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53b0074df0 0x7f53b0199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.857+0000 7f53b6800700 1 -- 172.21.15.116:0/2742055603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a40095f0 con 0x7f53b0108c40 2024-03-30T00:56:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.857+0000 7f53b6800700 1 --2- 172.21.15.116:0/2742055603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b0108c40 0x7f53b01a0450 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f53ac007ee0 tx=0x7f53ac00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.857+0000 7f53aaa00700 1 -- 172.21.15.116:0/2742055603 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac00dd20 con 0x7f53b0108c40 2024-03-30T00:56:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.857+0000 7f53aaa00700 1 -- 172.21.15.116:0/2742055603 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53ac00f040 con 0x7f53b0108c40 2024-03-30T00:56:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.857+0000 7f53aaa00700 1 -- 172.21.15.116:0/2742055603 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac0048a0 con 0x7f53b0108c40 2024-03-30T00:56:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.857+0000 7f53b8800700 1 -- 172.21.15.116:0/2742055603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b01a0fd0 con 0x7f53b0108c40 2024-03-30T00:56:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.857+0000 7f53b8800700 1 -- 172.21.15.116:0/2742055603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b01a1400 con 0x7f53b0108c40 2024-03-30T00:56:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.858+0000 7f53b8800700 1 -- 172.21.15.116:0/2742055603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b0107a00 con 0x7f53b0108c40 2024-03-30T00:56:23.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.860+0000 7f53aaa00700 1 -- 172.21.15.116:0/2742055603 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f53ac003a00 con 0x7f53b0108c40 2024-03-30T00:56:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.861+0000 7f53aaa00700 1 --2- 172.21.15.116:0/2742055603 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f53980a11b0 0x7f53980a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.862+0000 7f53aaa00700 1 -- 172.21.15.116:0/2742055603 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f53ac0cc440 con 0x7f53b0108c40 2024-03-30T00:56:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.862+0000 7f53b5e00700 1 --2- 172.21.15.116:0/2742055603 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f53980a11b0 0x7f53980a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.862+0000 7f53b5e00700 1 --2- 172.21.15.116:0/2742055603 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f53980a11b0 0x7f53980a3670 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f53a400b5c0 tx=0x7f53a4003790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:23.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:23.866+0000 7f53aaa00700 1 -- 172.21.15.116:0/2742055603 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f53ac095820 con 0x7f53b0108c40 2024-03-30T00:56:24.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.112+0000 7f53b8800700 1 -- 172.21.15.116:0/2742055603 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f53b002cd20 con 0x7f53980a11b0 2024-03-30T00:56:24.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.137+0000 7f53aaa00700 1 -- 172.21.15.116:0/2742055603 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f53b002cd20 con 0x7f53980a11b0 2024-03-30T00:56:24.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.139+0000 7f53b8800700 1 -- 172.21.15.116:0/2742055603 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f53980a11b0 msgr2=0x7f53980a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:24.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.139+0000 7f53b8800700 1 --2- 172.21.15.116:0/2742055603 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f53980a11b0 0x7f53980a3670 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f53a400b5c0 tx=0x7f53a4003790 comp rx=0 tx=0).stop 2024-03-30T00:56:24.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.139+0000 7f53b8800700 1 -- 172.21.15.116:0/2742055603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b0108c40 msgr2=0x7f53b01a0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:24.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.139+0000 7f53b8800700 1 --2- 172.21.15.116:0/2742055603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b0108c40 0x7f53b01a0450 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f53ac007ee0 tx=0x7f53ac00d3b0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.139+0000 7f53b8800700 1 -- 172.21.15.116:0/2742055603 shutdown_connections 2024-03-30T00:56:24.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.139+0000 7f53b8800700 1 --2- 172.21.15.116:0/2742055603 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f53980a11b0 0x7f53980a3670 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.139+0000 7f53b8800700 1 --2- 172.21.15.116:0/2742055603 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53b0073810 0x7f53b0198ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.139+0000 7f53b8800700 1 --2- 172.21.15.116:0/2742055603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b0108c40 0x7f53b01a0450 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.139+0000 7f53b8800700 1 --2- 172.21.15.116:0/2742055603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53b0074df0 0x7f53b0199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.139+0000 7f53b8800700 1 -- 172.21.15.116:0/2742055603 >> 172.21.15.116:0/2742055603 conn(0x7f53b00fc2f0 msgr2=0x7f53b0104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:24.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.139+0000 7f53b8800700 1 -- 172.21.15.116:0/2742055603 shutdown_connections 2024-03-30T00:56:24.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.140+0000 7f53b8800700 1 -- 172.21.15.116:0/2742055603 wait complete. 2024-03-30T00:56:24.165 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-30T00:56:24.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.358+0000 7f3eeb200700 1 -- 172.21.15.116:0/2242703676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ee4102910 msgr2=0x7f3ee41091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:24.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.358+0000 7f3eeb200700 1 --2- 172.21.15.116:0/2242703676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ee4102910 0x7f3ee41091c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00b0d0 tx=0x7f3edc00b3e0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.358+0000 7f3eeb200700 1 -- 172.21.15.116:0/2242703676 shutdown_connections 2024-03-30T00:56:24.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.358+0000 7f3eeb200700 1 --2- 172.21.15.116:0/2242703676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ee4102910 0x7f3ee41091c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.358+0000 7f3eeb200700 1 --2- 172.21.15.116:0/2242703676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ee4100d80 0x7f3ee41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.358+0000 7f3eeb200700 1 --2- 172.21.15.116:0/2242703676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee4101f70 0x7f3ee41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.358+0000 7f3eeb200700 1 -- 172.21.15.116:0/2242703676 >> 172.21.15.116:0/2242703676 conn(0x7f3ee40fc2f0 msgr2=0x7f3ee40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:24.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.358+0000 7f3eeb200700 1 -- 172.21.15.116:0/2242703676 shutdown_connections 2024-03-30T00:56:24.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.358+0000 7f3eeb200700 1 -- 172.21.15.116:0/2242703676 wait complete. 2024-03-30T00:56:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.359+0000 7f3eeb200700 1 Processor -- start 2024-03-30T00:56:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.359+0000 7f3eeb200700 1 -- start start 2024-03-30T00:56:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3eeb200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ee4100d80 0x7f3ee4074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3eeb200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee4101f70 0x7f3ee40731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3eeb200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ee4102910 0x7f3ee40736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3eeb200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee4073e80 con 0x7f3ee4102910 2024-03-30T00:56:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3eeb200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee4073ff0 con 0x7f3ee4100d80 2024-03-30T00:56:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3eeb200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee41a0e90 con 0x7f3ee4101f70 2024-03-30T00:56:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3ee9200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ee4102910 0x7f3ee40736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3ee9200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ee4102910 0x7f3ee40736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57672/0 (socket says 172.21.15.116:57672) 2024-03-30T00:56:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3ee9200700 1 -- 172.21.15.116:0/768553428 learned_addr learned my addr 172.21.15.116:0/768553428 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3ee3e00700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ee4100d80 0x7f3ee4074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3ee3400700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee4101f70 0x7f3ee40731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3ee9200700 1 -- 172.21.15.116:0/768553428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee4101f70 msgr2=0x7f3ee40731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3ee9200700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee4101f70 0x7f3ee40731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3ee9200700 1 -- 172.21.15.116:0/768553428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ee4100d80 msgr2=0x7f3ee4074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3ee9200700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ee4100d80 0x7f3ee4074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3ee9200700 1 -- 172.21.15.116:0/768553428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed0009520 con 0x7f3ee4102910 2024-03-30T00:56:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.360+0000 7f3ee3e00700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ee4100d80 0x7f3ee4074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.361+0000 7f3ee9200700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ee4102910 0x7f3ee40736e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3edc007b60 tx=0x7f3edc008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.361+0000 7f3ee0c00700 1 -- 172.21.15.116:0/768553428 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc00e040 con 0x7f3ee4102910 2024-03-30T00:56:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.361+0000 7f3eeb200700 1 -- 172.21.15.116:0/768553428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc009d00 con 0x7f3ee4102910 2024-03-30T00:56:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.361+0000 7f3ee0c00700 1 -- 172.21.15.116:0/768553428 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3edc01d930 con 0x7f3ee4102910 2024-03-30T00:56:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.361+0000 7f3eeb200700 1 -- 172.21.15.116:0/768553428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee41a1350 con 0x7f3ee4102910 2024-03-30T00:56:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.361+0000 7f3ee0c00700 1 -- 172.21.15.116:0/768553428 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc01b990 con 0x7f3ee4102910 2024-03-30T00:56:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.362+0000 7f3eeb200700 1 -- 172.21.15.116:0/768553428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee4108130 con 0x7f3ee4102910 2024-03-30T00:56:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.364+0000 7f3ee0c00700 1 -- 172.21.15.116:0/768553428 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f3edc01daa0 con 0x7f3ee4102910 2024-03-30T00:56:24.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.366+0000 7f3ee0c00700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3ec80a11b0 0x7f3ec80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.366+0000 7f3ee0c00700 1 -- 172.21.15.116:0/768553428 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3edc0cde00 con 0x7f3ee4102910 2024-03-30T00:56:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.366+0000 7f3ee3e00700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3ec80a11b0 0x7f3ec80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.367+0000 7f3ee3e00700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3ec80a11b0 0x7f3ec80a3670 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4101dd0 tx=0x7f3ed4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:24.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.370+0000 7f3ee0c00700 1 -- 172.21.15.116:0/768553428 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3edc097270 con 0x7f3ee4102910 2024-03-30T00:56:24.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:24 smithi116 ceph-mon[38086]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:24.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:24 smithi116 ceph-mon[38086]: pgmap v245: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-03-30T00:56:24.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:24 smithi116 ceph-mon[38086]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:24.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:24 smithi116 ceph-mon[38086]: mgrmap e17: x(active, since 5m), standbys: z, y 2024-03-30T00:56:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.623+0000 7f3eeb200700 1 -- 172.21.15.116:0/768553428 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ee40611f0 con 0x7f3ec80a11b0 2024-03-30T00:56:24.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.627+0000 7f3ee0c00700 1 -- 172.21.15.116:0/768553428 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f3ee40611f0 con 0x7f3ec80a11b0 2024-03-30T00:56:24.634 INFO:teuthology.orchestra.run.smithi116.stdout:/volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64 2024-03-30T00:56:24.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.629+0000 7f3eeb200700 1 -- 172.21.15.116:0/768553428 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3ec80a11b0 msgr2=0x7f3ec80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:24.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.629+0000 7f3eeb200700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3ec80a11b0 0x7f3ec80a3670 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4101dd0 tx=0x7f3ed4009260 comp rx=0 tx=0).stop 2024-03-30T00:56:24.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.629+0000 7f3eeb200700 1 -- 172.21.15.116:0/768553428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ee4102910 msgr2=0x7f3ee40736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.629+0000 7f3eeb200700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ee4102910 0x7f3ee40736e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3edc007b60 tx=0x7f3edc008e40 comp rx=0 tx=0).stop 2024-03-30T00:56:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.629+0000 7f3eeb200700 1 -- 172.21.15.116:0/768553428 shutdown_connections 2024-03-30T00:56:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.629+0000 7f3eeb200700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3ec80a11b0 0x7f3ec80a3670 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.629+0000 7f3eeb200700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ee4102910 0x7f3ee40736e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.629+0000 7f3eeb200700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ee4100d80 0x7f3ee4074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.629+0000 7f3eeb200700 1 --2- 172.21.15.116:0/768553428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee4101f70 0x7f3ee40731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.629+0000 7f3eeb200700 1 -- 172.21.15.116:0/768553428 >> 172.21.15.116:0/768553428 conn(0x7f3ee40fc2f0 msgr2=0x7f3ee4105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.629+0000 7f3eeb200700 1 -- 172.21.15.116:0/768553428 shutdown_connections 2024-03-30T00:56:24.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.629+0000 7f3eeb200700 1 -- 172.21.15.116:0/768553428 wait complete. 2024-03-30T00:56:24.655 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:56:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:24 smithi181 ceph-mon[47622]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:24 smithi181 ceph-mon[47622]: pgmap v245: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-03-30T00:56:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:24 smithi181 ceph-mon[47622]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:24 smithi181 ceph-mon[47622]: mgrmap e17: x(active, since 5m), standbys: z, y 2024-03-30T00:56:24.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:24 smithi130 ceph-mon[47626]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:24.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:24 smithi130 ceph-mon[47626]: pgmap v245: 129 pgs: 129 active+clean; 581 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-03-30T00:56:24.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:24 smithi130 ceph-mon[47626]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:24.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:24 smithi130 ceph-mon[47626]: mgrmap e17: x(active, since 5m), standbys: z, y 2024-03-30T00:56:24.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.844+0000 7f9393400700 1 -- 172.21.15.116:0/1278410126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f938c1040a0 msgr2=0x7f938c106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:24.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.844+0000 7f9393400700 1 --2- 172.21.15.116:0/1278410126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f938c1040a0 0x7f938c106490 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9378009990 tx=0x7f9378009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.844+0000 7f9393400700 1 -- 172.21.15.116:0/1278410126 shutdown_connections 2024-03-30T00:56:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.844+0000 7f9393400700 1 --2- 172.21.15.116:0/1278410126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f938c1069d0 0x7f938c108de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.844+0000 7f9393400700 1 --2- 172.21.15.116:0/1278410126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f938c101770 0x7f938c103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.844+0000 7f9393400700 1 --2- 172.21.15.116:0/1278410126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f938c1040a0 0x7f938c106490 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.844+0000 7f9393400700 1 -- 172.21.15.116:0/1278410126 >> 172.21.15.116:0/1278410126 conn(0x7f938c0fb100 msgr2=0x7f938c0fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.844+0000 7f9393400700 1 -- 172.21.15.116:0/1278410126 shutdown_connections 2024-03-30T00:56:24.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.844+0000 7f9393400700 1 -- 172.21.15.116:0/1278410126 wait complete. 2024-03-30T00:56:24.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.845+0000 7f9393400700 1 Processor -- start 2024-03-30T00:56:24.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.845+0000 7f9393400700 1 -- start start 2024-03-30T00:56:24.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.845+0000 7f9393400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f938c101770 0x7f938c198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:24.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.845+0000 7f9393400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f938c1040a0 0x7f938c199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.845+0000 7f9393400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f938c1069d0 0x7f938c1a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.845+0000 7f9393400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c1a08e0 con 0x7f938c1040a0 2024-03-30T00:56:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.845+0000 7f9393400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c1a0a50 con 0x7f938c101770 2024-03-30T00:56:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.845+0000 7f9393400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c1a0c10 con 0x7f938c1069d0 2024-03-30T00:56:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.846+0000 7f938be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f938c1040a0 0x7f938c199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.846+0000 7f938be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f938c1040a0 0x7f938c199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37884/0 (socket says 172.21.15.116:37884) 2024-03-30T00:56:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.846+0000 7f938be00700 1 -- 172.21.15.116:0/1235122043 learned_addr learned my addr 172.21.15.116:0/1235122043 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.846+0000 7f9390a00700 1 --2- 172.21.15.116:0/1235122043 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f938c101770 0x7f938c198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.846+0000 7f9391400700 1 --2- 172.21.15.116:0/1235122043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f938c1069d0 0x7f938c1a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.846+0000 7f938be00700 1 -- 172.21.15.116:0/1235122043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f938c1069d0 msgr2=0x7f938c1a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.846+0000 7f938be00700 1 --2- 172.21.15.116:0/1235122043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f938c1069d0 0x7f938c1a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.846+0000 7f938be00700 1 -- 172.21.15.116:0/1235122043 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f938c101770 msgr2=0x7f938c198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.846+0000 7f938be00700 1 --2- 172.21.15.116:0/1235122043 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f938c101770 0x7f938c198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.846+0000 7f938be00700 1 -- 172.21.15.116:0/1235122043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93780095f0 con 0x7f938c1040a0 2024-03-30T00:56:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.846+0000 7f938be00700 1 --2- 172.21.15.116:0/1235122043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f938c1040a0 0x7f938c199330 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9378005910 tx=0x7f9378009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.846+0000 7f9389600700 1 -- 172.21.15.116:0/1235122043 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937801d070 con 0x7f938c1040a0 2024-03-30T00:56:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.847+0000 7f9389600700 1 -- 172.21.15.116:0/1235122043 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9378021410 con 0x7f938c1040a0 2024-03-30T00:56:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.847+0000 7f9393400700 1 -- 172.21.15.116:0/1235122043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f938c1a0e90 con 0x7f938c1040a0 2024-03-30T00:56:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.847+0000 7f9389600700 1 -- 172.21.15.116:0/1235122043 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9378017650 con 0x7f938c1040a0 2024-03-30T00:56:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.847+0000 7f9393400700 1 -- 172.21.15.116:0/1235122043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f938c1a1200 con 0x7f938c1040a0 2024-03-30T00:56:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.848+0000 7f9393400700 1 -- 172.21.15.116:0/1235122043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f938c108570 con 0x7f938c1040a0 2024-03-30T00:56:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.849+0000 7f9389600700 1 -- 172.21.15.116:0/1235122043 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f9378021580 con 0x7f938c1040a0 2024-03-30T00:56:24.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.851+0000 7f9389600700 1 --2- 172.21.15.116:0/1235122043 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f93740a11b0 0x7f93740a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:24.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.851+0000 7f9389600700 1 -- 172.21.15.116:0/1235122043 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f93780cdef0 con 0x7f938c1040a0 2024-03-30T00:56:24.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.851+0000 7f9390a00700 1 --2- 172.21.15.116:0/1235122043 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f93740a11b0 0x7f93740a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:24.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.851+0000 7f9390a00700 1 --2- 172.21.15.116:0/1235122043 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f93740a11b0 0x7f93740a3670 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f938c19a270 tx=0x7f9380009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:24.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:24.854+0000 7f9389600700 1 -- 172.21.15.116:0/1235122043 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9378097360 con 0x7f938c1040a0 2024-03-30T00:56:25.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.107+0000 7f9393400700 1 -- 172.21.15.116:0/1235122043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f938c10c480 con 0x7f938c1040a0 2024-03-30T00:56:25.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.107+0000 7f9389600700 1 -- 172.21.15.116:0/1235122043 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+10886 (secure 0 0 0) 0x7f93780aee60 con 0x7f938c1040a0 2024-03-30T00:56:25.114 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:56:25.114 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:56:22.144079+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T00:56:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.110+0000 7f9393400700 1 -- 172.21.15.116:0/1235122043 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f93740a11b0 msgr2=0x7f93740a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.110+0000 7f9393400700 1 --2- 172.21.15.116:0/1235122043 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f93740a11b0 0x7f93740a3670 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f938c19a270 tx=0x7f9380009260 comp rx=0 tx=0).stop 2024-03-30T00:56:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.110+0000 7f9393400700 1 -- 172.21.15.116:0/1235122043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f938c1040a0 msgr2=0x7f938c199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.110+0000 7f9393400700 1 --2- 172.21.15.116:0/1235122043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f938c1040a0 0x7f938c199330 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9378005910 tx=0x7f9378009580 comp rx=0 tx=0).stop 2024-03-30T00:56:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.110+0000 7f9393400700 1 -- 172.21.15.116:0/1235122043 shutdown_connections 2024-03-30T00:56:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.110+0000 7f9393400700 1 --2- 172.21.15.116:0/1235122043 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f93740a11b0 0x7f93740a3670 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.110+0000 7f9393400700 1 --2- 172.21.15.116:0/1235122043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f938c1069d0 0x7f938c1a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.110+0000 7f9393400700 1 --2- 172.21.15.116:0/1235122043 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f938c101770 0x7f938c198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.110+0000 7f9393400700 1 --2- 172.21.15.116:0/1235122043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f938c1040a0 0x7f938c199330 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.110+0000 7f9393400700 1 -- 172.21.15.116:0/1235122043 >> 172.21.15.116:0/1235122043 conn(0x7f938c0fb100 msgr2=0x7f938c101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.110+0000 7f9393400700 1 -- 172.21.15.116:0/1235122043 shutdown_connections 2024-03-30T00:56:25.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.110+0000 7f9393400700 1 -- 172.21.15.116:0/1235122043 wait complete. 2024-03-30T00:56:25.118 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 22 2024-03-30T00:56:25.136 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:56:25.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.320+0000 7fe5b8800700 1 -- 172.21.15.116:0/346815523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5b0100c80 msgr2=0x7fe5b01010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:25.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.320+0000 7fe5b8800700 1 --2- 172.21.15.116:0/346815523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5b0100c80 0x7fe5b01010a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009920 tx=0x7fe5a4009c30 comp rx=0 tx=0).stop 2024-03-30T00:56:25.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.320+0000 7fe5b8800700 1 -- 172.21.15.116:0/346815523 shutdown_connections 2024-03-30T00:56:25.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.320+0000 7fe5b8800700 1 --2- 172.21.15.116:0/346815523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5b0102810 0x7fe5b01090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.320+0000 7fe5b8800700 1 --2- 172.21.15.116:0/346815523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5b0100c80 0x7fe5b01010a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.320+0000 7fe5b8800700 1 --2- 172.21.15.116:0/346815523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5b0101e70 0x7fe5b01022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.320+0000 7fe5b8800700 1 -- 172.21.15.116:0/346815523 >> 172.21.15.116:0/346815523 conn(0x7fe5b00fc1f0 msgr2=0x7fe5b00fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:25.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.320+0000 7fe5b8800700 1 -- 172.21.15.116:0/346815523 shutdown_connections 2024-03-30T00:56:25.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.320+0000 7fe5b8800700 1 -- 172.21.15.116:0/346815523 wait complete. 2024-03-30T00:56:25.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.321+0000 7fe5b8800700 1 Processor -- start 2024-03-30T00:56:25.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b8800700 1 -- start start 2024-03-30T00:56:25.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b8800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5b0100c80 0x7fe5b0196d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:25.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b8800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5b0101e70 0x7fe5b0197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:25.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b8800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5b0102810 0x7fe5b019e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:25.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b8800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b019e7b0 con 0x7fe5b0100c80 2024-03-30T00:56:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b8800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b019e920 con 0x7fe5b0102810 2024-03-30T00:56:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b8800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b019eae0 con 0x7fe5b0101e70 2024-03-30T00:56:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b5e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5b0100c80 0x7fe5b0196d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b5e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5b0100c80 0x7fe5b0196d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37906/0 (socket says 172.21.15.116:37906) 2024-03-30T00:56:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b5e00700 1 -- 172.21.15.116:0/331555480 learned_addr learned my addr 172.21.15.116:0/331555480 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b6800700 1 --2- 172.21.15.116:0/331555480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5b0102810 0x7fe5b019e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b5400700 1 --2- 172.21.15.116:0/331555480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5b0101e70 0x7fe5b0197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b5e00700 1 -- 172.21.15.116:0/331555480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5b0101e70 msgr2=0x7fe5b0197250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b5e00700 1 --2- 172.21.15.116:0/331555480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5b0101e70 0x7fe5b0197250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b5e00700 1 -- 172.21.15.116:0/331555480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5b0102810 msgr2=0x7fe5b019e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.322+0000 7fe5b5e00700 1 --2- 172.21.15.116:0/331555480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5b0102810 0x7fe5b019e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.323+0000 7fe5b5e00700 1 -- 172.21.15.116:0/331555480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a00095f0 con 0x7fe5b0100c80 2024-03-30T00:56:25.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.323+0000 7fe5b5e00700 1 --2- 172.21.15.116:0/331555480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5b0100c80 0x7fe5b0196d10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009230 tx=0x7fe5a400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:25.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.323+0000 7fe5aaa00700 1 -- 172.21.15.116:0/331555480 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a401c070 con 0x7fe5b0100c80 2024-03-30T00:56:25.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.323+0000 7fe5aaa00700 1 -- 172.21.15.116:0/331555480 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5a400f7d0 con 0x7fe5b0100c80 2024-03-30T00:56:25.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.323+0000 7fe5b8800700 1 -- 172.21.15.116:0/331555480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a4009520 con 0x7fe5b0100c80 2024-03-30T00:56:25.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.323+0000 7fe5aaa00700 1 -- 172.21.15.116:0/331555480 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a40049e0 con 0x7fe5b0100c80 2024-03-30T00:56:25.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.323+0000 7fe5b8800700 1 -- 172.21.15.116:0/331555480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b019efa0 con 0x7fe5b0100c80 2024-03-30T00:56:25.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.324+0000 7fe5b8800700 1 -- 172.21.15.116:0/331555480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b01081f0 con 0x7fe5b0100c80 2024-03-30T00:56:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.326+0000 7fe5aaa00700 1 -- 172.21.15.116:0/331555480 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fe5a4020070 con 0x7fe5b0100c80 2024-03-30T00:56:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.327+0000 7fe5aaa00700 1 --2- 172.21.15.116:0/331555480 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe5980a1230 0x7fe5980a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.327+0000 7fe5aaa00700 1 -- 172.21.15.116:0/331555480 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe5a40cda10 con 0x7fe5b0100c80 2024-03-30T00:56:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.327+0000 7fe5b5400700 1 --2- 172.21.15.116:0/331555480 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe5980a1230 0x7fe5980a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.328+0000 7fe5b5400700 1 --2- 172.21.15.116:0/331555480 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe5980a1230 0x7fe5980a36f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0005fd0 tx=0x7fe5a0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.331+0000 7fe5aaa00700 1 -- 172.21.15.116:0/331555480 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe5a4096e80 con 0x7fe5b0100c80 2024-03-30T00:56:25.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.569+0000 7fe5b8800700 1 -- 172.21.15.116:0/331555480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5b0067a10 con 0x7fe5b0100c80 2024-03-30T00:56:25.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.571+0000 7fe5aaa00700 1 -- 172.21.15.116:0/331555480 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+24028 (secure 0 0 0) 0x7fe5a409a4a0 con 0x7fe5b0100c80 2024-03-30T00:56:25.578 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:56:25.579 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:55:56.324776+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:56:25.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.573+0000 7fe5b8800700 1 -- 172.21.15.116:0/331555480 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe5980a1230 msgr2=0x7fe5980a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:25.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.573+0000 7fe5b8800700 1 --2- 172.21.15.116:0/331555480 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe5980a1230 0x7fe5980a36f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0005fd0 tx=0x7fe5a0009310 comp rx=0 tx=0).stop 2024-03-30T00:56:25.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.573+0000 7fe5b8800700 1 -- 172.21.15.116:0/331555480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5b0100c80 msgr2=0x7fe5b0196d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:25.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.573+0000 7fe5b8800700 1 --2- 172.21.15.116:0/331555480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5b0100c80 0x7fe5b0196d10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009230 tx=0x7fe5a400b790 comp rx=0 tx=0).stop 2024-03-30T00:56:25.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.574+0000 7fe5b8800700 1 -- 172.21.15.116:0/331555480 shutdown_connections 2024-03-30T00:56:25.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.574+0000 7fe5b8800700 1 --2- 172.21.15.116:0/331555480 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe5980a1230 0x7fe5980a36f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.574+0000 7fe5b8800700 1 --2- 172.21.15.116:0/331555480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5b0102810 0x7fe5b019e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.574+0000 7fe5b8800700 1 --2- 172.21.15.116:0/331555480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5b0100c80 0x7fe5b0196d10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.574+0000 7fe5b8800700 1 --2- 172.21.15.116:0/331555480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5b0101e70 0x7fe5b0197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.574+0000 7fe5b8800700 1 -- 172.21.15.116:0/331555480 >> 172.21.15.116:0/331555480 conn(0x7fe5b00fc1f0 msgr2=0x7fe5b0105330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:25.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.574+0000 7fe5b8800700 1 -- 172.21.15.116:0/331555480 shutdown_connections 2024-03-30T00:56:25.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.574+0000 7fe5b8800700 1 -- 172.21.15.116:0/331555480 wait complete. 2024-03-30T00:56:25.599 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-30T00:56:25.599 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T00:56:25.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:25 smithi116 ceph-mon[38086]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:25.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:25 smithi116 ceph-mon[38086]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:25.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:25 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1235122043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:56:25.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:25 smithi181 ceph-mon[47622]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:25.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:25 smithi181 ceph-mon[47622]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:25.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:25 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1235122043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:56:25.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.784+0000 7f7226a00700 1 -- 172.21.15.116:0/3572300621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220102810 msgr2=0x7f72201090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:25.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.784+0000 7f7226a00700 1 --2- 172.21.15.116:0/3572300621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220102810 0x7f72201090c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f721800b0d0 tx=0x7f721800b3e0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.784+0000 7f7226a00700 1 -- 172.21.15.116:0/3572300621 shutdown_connections 2024-03-30T00:56:25.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.784+0000 7f7226a00700 1 --2- 172.21.15.116:0/3572300621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220102810 0x7f72201090c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.784+0000 7f7226a00700 1 --2- 172.21.15.116:0/3572300621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7220100c80 0x7f72201010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.784+0000 7f7226a00700 1 --2- 172.21.15.116:0/3572300621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7220101e70 0x7f72201022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.784+0000 7f7226a00700 1 -- 172.21.15.116:0/3572300621 >> 172.21.15.116:0/3572300621 conn(0x7f72200fc1f0 msgr2=0x7f72200fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:25.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.785+0000 7f7226a00700 1 -- 172.21.15.116:0/3572300621 shutdown_connections 2024-03-30T00:56:25.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.785+0000 7f7226a00700 1 -- 172.21.15.116:0/3572300621 wait complete. 2024-03-30T00:56:25.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.785+0000 7f7226a00700 1 Processor -- start 2024-03-30T00:56:25.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.786+0000 7f7226a00700 1 -- start start 2024-03-30T00:56:25.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.786+0000 7f7226a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7220100c80 0x7f7220074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:25.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.786+0000 7f7226a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7220101e70 0x7f72200731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:25.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.786+0000 7f7226a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220102810 0x7f72200736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:25.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.786+0000 7f7226a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7220073e80 con 0x7f7220102810 2024-03-30T00:56:25.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.786+0000 7f7226a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7220073ff0 con 0x7f7220100c80 2024-03-30T00:56:25.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.786+0000 7f7226a00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72201a0d70 con 0x7f7220101e70 2024-03-30T00:56:25.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.786+0000 7f7224a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220102810 0x7f72200736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:25.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.786+0000 7f721f400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7220101e70 0x7f72200731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:25.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.786+0000 7f7224a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220102810 0x7f72200736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37918/0 (socket says 172.21.15.116:37918) 2024-03-30T00:56:25.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.786+0000 7f7224a00700 1 -- 172.21.15.116:0/2848800006 learned_addr learned my addr 172.21.15.116:0/2848800006 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:25.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.786+0000 7f721fe00700 1 --2- 172.21.15.116:0/2848800006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7220100c80 0x7f7220074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:25.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.787+0000 7f7224a00700 1 -- 172.21.15.116:0/2848800006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7220101e70 msgr2=0x7f72200731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:25.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.787+0000 7f7224a00700 1 --2- 172.21.15.116:0/2848800006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7220101e70 0x7f72200731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.787+0000 7f7224a00700 1 -- 172.21.15.116:0/2848800006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7220100c80 msgr2=0x7f7220074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.787+0000 7f7224a00700 1 --2- 172.21.15.116:0/2848800006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7220100c80 0x7f7220074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.787+0000 7f7224a00700 1 -- 172.21.15.116:0/2848800006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7218009d00 con 0x7f7220102810 2024-03-30T00:56:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.787+0000 7f721fe00700 1 --2- 172.21.15.116:0/2848800006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7220100c80 0x7f7220074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:56:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.787+0000 7f7224a00700 1 --2- 172.21.15.116:0/2848800006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220102810 0x7f72200736e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7218007b60 tx=0x7f7218008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.787+0000 7f721cc00700 1 -- 172.21.15.116:0/2848800006 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721800e040 con 0x7f7220102810 2024-03-30T00:56:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.787+0000 7f721cc00700 1 -- 172.21.15.116:0/2848800006 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f721801d930 con 0x7f7220102810 2024-03-30T00:56:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.787+0000 7f7226a00700 1 -- 172.21.15.116:0/2848800006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72201a0fa0 con 0x7f7220102810 2024-03-30T00:56:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.787+0000 7f721cc00700 1 -- 172.21.15.116:0/2848800006 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721801ba40 con 0x7f7220102810 2024-03-30T00:56:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.787+0000 7f7226a00700 1 -- 172.21.15.116:0/2848800006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72201a1360 con 0x7f7220102810 2024-03-30T00:56:25.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.788+0000 7f7226a00700 1 -- 172.21.15.116:0/2848800006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72201080d0 con 0x7f7220102810 2024-03-30T00:56:25.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.790+0000 7f721cc00700 1 -- 172.21.15.116:0/2848800006 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f721801daa0 con 0x7f7220102810 2024-03-30T00:56:25.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.792+0000 7f721cc00700 1 --2- 172.21.15.116:0/2848800006 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f72040a1230 0x7f72040a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:25.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.793+0000 7f721cc00700 1 -- 172.21.15.116:0/2848800006 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f72180ce1b0 con 0x7f7220102810 2024-03-30T00:56:25.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.793+0000 7f721fe00700 1 --2- 172.21.15.116:0/2848800006 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f72040a1230 0x7f72040a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:25.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.793+0000 7f721fe00700 1 --2- 172.21.15.116:0/2848800006 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f72040a1230 0x7f72040a36f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7220101cd0 tx=0x7f7210009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:25.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.796+0000 7f721cc00700 1 -- 172.21.15.116:0/2848800006 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7218097620 con 0x7f7220102810 2024-03-30T00:56:25.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:25 smithi130 ceph-mon[47626]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:25.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:25 smithi130 ceph-mon[47626]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:25.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:25 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1235122043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:56:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:25.998+0000 7f7226a00700 1 -- 172.21.15.116:0/2848800006 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f722002ce10 con 0x7f72040a1230 2024-03-30T00:56:26.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.013+0000 7f721cc00700 1 -- 172.21.15.116:0/2848800006 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422678 (secure 0 0 0) 0x7f722002ce10 con 0x7f72040a1230 2024-03-30T00:56:26.022 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:56:26.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.019+0000 7f7226a00700 1 -- 172.21.15.116:0/2848800006 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f72040a1230 msgr2=0x7f72040a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:26.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.019+0000 7f7226a00700 1 --2- 172.21.15.116:0/2848800006 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f72040a1230 0x7f72040a36f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7220101cd0 tx=0x7f7210009260 comp rx=0 tx=0).stop 2024-03-30T00:56:26.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.019+0000 7f7226a00700 1 -- 172.21.15.116:0/2848800006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220102810 msgr2=0x7f72200736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:26.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.019+0000 7f7226a00700 1 --2- 172.21.15.116:0/2848800006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220102810 0x7f72200736e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7218007b60 tx=0x7f7218008e40 comp rx=0 tx=0).stop 2024-03-30T00:56:26.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.019+0000 7f7226a00700 1 -- 172.21.15.116:0/2848800006 shutdown_connections 2024-03-30T00:56:26.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.019+0000 7f7226a00700 1 --2- 172.21.15.116:0/2848800006 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f72040a1230 0x7f72040a36f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:26.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.019+0000 7f7226a00700 1 --2- 172.21.15.116:0/2848800006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7220102810 0x7f72200736e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:26.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.019+0000 7f7226a00700 1 --2- 172.21.15.116:0/2848800006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7220100c80 0x7f7220074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:26.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.019+0000 7f7226a00700 1 --2- 172.21.15.116:0/2848800006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7220101e70 0x7f72200731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:26.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.019+0000 7f7226a00700 1 -- 172.21.15.116:0/2848800006 >> 172.21.15.116:0/2848800006 conn(0x7f72200fc1f0 msgr2=0x7f7220105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:26.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.019+0000 7f7226a00700 1 -- 172.21.15.116:0/2848800006 shutdown_connections 2024-03-30T00:56:26.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.019+0000 7f7226a00700 1 -- 172.21.15.116:0/2848800006 wait complete. 2024-03-30T00:56:26.086 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-30T00:56:26.087 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-03-30T00:56:25.107783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":633461,"num_objects":51,"num_object_clones":0,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":150,"num_read_kb":165,"num_write":302,"num_write_kb":3011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":351248,"kb_used_data":10636,"kb_used_omap":18,"kb_used_meta":340461,"kb_avail":1124590576,"statfs":{"total":1151940427776,"available":1151580749824,"internally_reserved":0,"allocated":10891264,"data_stored":7139697,"data_compressed":248961,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":19128,"internal_metadata":348632392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38640,"num_objects":27,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":24,"num_read_kb":56,"num_write":74,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004216"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815586+0000","last_change":"2024-03-30T00:55:51.809659+0000","last_active":"2024-03-30T00:55:55.815586+0000","last_peered":"2024-03-30T00:55:55.815586+0000","last_clean":"2024-03-30T00:55:55.815586+0000","last_became_active":"2024-03-30T00:55:51.809337+0000","last_became_peered":"2024-03-30T00:55:51.809337+0000","last_unstale":"2024-03-30T00:55:55.815586+0000","last_undegraded":"2024-03-30T00:55:55.815586+0000","last_fullsized":"2024-03-30T00:55:55.815586+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:20:50.640033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.818896+0000","last_change":"2024-03-30T00:55:51.813778+0000","last_active":"2024-03-30T00:55:55.818896+0000","last_peered":"2024-03-30T00:55:55.818896+0000","last_clean":"2024-03-30T00:55:55.818896+0000","last_became_active":"2024-03-30T00:55:51.810347+0000","last_became_peered":"2024-03-30T00:55:51.810347+0000","last_unstale":"2024-03-30T00:55:55.818896+0000","last_undegraded":"2024-03-30T00:55:55.818896+0000","last_fullsized":"2024-03-30T00:55:55.818896+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:49:07.031477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.818797+0000","last_change":"2024-03-30T00:55:51.817647+0000","last_active":"2024-03-30T00:55:55.818797+0000","last_peered":"2024-03-30T00:55:55.818797+0000","last_clean":"2024-03-30T00:55:55.818797+0000","last_became_active":"2024-03-30T00:55:51.816495+0000","last_became_peered":"2024-03-30T00:55:51.816495+0000","last_unstale":"2024-03-30T00:55:55.818797+0000","last_undegraded":"2024-03-30T00:55:55.818797+0000","last_fullsized":"2024-03-30T00:55:55.818797+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:59:42.965346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817411+0000","last_change":"2024-03-30T00:55:51.817581+0000","last_active":"2024-03-30T00:55:55.817411+0000","last_peered":"2024-03-30T00:55:55.817411+0000","last_clean":"2024-03-30T00:55:55.817411+0000","last_became_active":"2024-03-30T00:55:51.816223+0000","last_became_peered":"2024-03-30T00:55:51.816223+0000","last_unstale":"2024-03-30T00:55:55.817411+0000","last_undegraded":"2024-03-30T00:55:55.817411+0000","last_fullsized":"2024-03-30T00:55:55.817411+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:48:32.203975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815122+0000","last_change":"2024-03-30T00:55:51.811707+0000","last_active":"2024-03-30T00:55:55.815122+0000","last_peered":"2024-03-30T00:55:55.815122+0000","last_clean":"2024-03-30T00:55:55.815122+0000","last_became_active":"2024-03-30T00:55:51.811366+0000","last_became_peered":"2024-03-30T00:55:51.811366+0000","last_unstale":"2024-03-30T00:55:55.815122+0000","last_undegraded":"2024-03-30T00:55:55.815122+0000","last_fullsized":"2024-03-30T00:55:55.815122+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:20.438505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.855012+0000","last_change":"2024-03-30T00:55:51.826849+0000","last_active":"2024-03-30T00:55:55.855012+0000","last_peered":"2024-03-30T00:55:55.855012+0000","last_clean":"2024-03-30T00:55:55.855012+0000","last_became_active":"2024-03-30T00:55:51.826520+0000","last_became_peered":"2024-03-30T00:55:51.826520+0000","last_unstale":"2024-03-30T00:55:55.855012+0000","last_undegraded":"2024-03-30T00:55:55.855012+0000","last_fullsized":"2024-03-30T00:55:55.855012+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:54:03.423725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.823087+0000","last_change":"2024-03-30T00:55:51.812880+0000","last_active":"2024-03-30T00:55:55.823087+0000","last_peered":"2024-03-30T00:55:55.823087+0000","last_clean":"2024-03-30T00:55:55.823087+0000","last_became_active":"2024-03-30T00:55:51.812572+0000","last_became_peered":"2024-03-30T00:55:51.812572+0000","last_unstale":"2024-03-30T00:55:55.823087+0000","last_undegraded":"2024-03-30T00:55:55.823087+0000","last_fullsized":"2024-03-30T00:55:55.823087+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:44:49.670641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141676+0000","last_change":"2024-03-30T00:55:51.815480+0000","last_active":"2024-03-30T00:55:56.141676+0000","last_peered":"2024-03-30T00:55:56.141676+0000","last_clean":"2024-03-30T00:55:56.141676+0000","last_became_active":"2024-03-30T00:55:51.814052+0000","last_became_peered":"2024-03-30T00:55:51.814052+0000","last_unstale":"2024-03-30T00:55:56.141676+0000","last_undegraded":"2024-03-30T00:55:56.141676+0000","last_fullsized":"2024-03-30T00:55:56.141676+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:27:49.930928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816450+0000","last_change":"2024-03-30T00:55:51.814127+0000","last_active":"2024-03-30T00:55:55.816450+0000","last_peered":"2024-03-30T00:55:55.816450+0000","last_clean":"2024-03-30T00:55:55.816450+0000","last_became_active":"2024-03-30T00:55:51.813820+0000","last_became_peered":"2024-03-30T00:55:51.813820+0000","last_unstale":"2024-03-30T00:55:55.816450+0000","last_undegraded":"2024-03-30T00:55:55.816450+0000","last_fullsized":"2024-03-30T00:55:55.816450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:21:31.605935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.855477+0000","last_change":"2024-03-30T00:55:51.815198+0000","last_active":"2024-03-30T00:55:55.855477+0000","last_peered":"2024-03-30T00:55:55.855477+0000","last_clean":"2024-03-30T00:55:55.855477+0000","last_became_active":"2024-03-30T00:55:51.814901+0000","last_became_peered":"2024-03-30T00:55:51.814901+0000","last_unstale":"2024-03-30T00:55:55.855477+0000","last_undegraded":"2024-03-30T00:55:55.855477+0000","last_fullsized":"2024-03-30T00:55:55.855477+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:45:51.224346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.825549+0000","last_change":"2024-03-30T00:55:51.809884+0000","last_active":"2024-03-30T00:55:55.825549+0000","last_peered":"2024-03-30T00:55:55.825549+0000","last_clean":"2024-03-30T00:55:55.825549+0000","last_became_active":"2024-03-30T00:55:51.809576+0000","last_became_peered":"2024-03-30T00:55:51.809576+0000","last_unstale":"2024-03-30T00:55:55.825549+0000","last_undegraded":"2024-03-30T00:55:55.825549+0000","last_fullsized":"2024-03-30T00:55:55.825549+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:39:51.767744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817315+0000","last_change":"2024-03-30T00:55:51.817384+0000","last_active":"2024-03-30T00:55:55.817315+0000","last_peered":"2024-03-30T00:55:55.817315+0000","last_clean":"2024-03-30T00:55:55.817315+0000","last_became_active":"2024-03-30T00:55:51.816564+0000","last_became_peered":"2024-03-30T00:55:51.816564+0000","last_unstale":"2024-03-30T00:55:55.817315+0000","last_undegraded":"2024-03-30T00:55:55.817315+0000","last_fullsized":"2024-03-30T00:55:55.817315+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:47:07.195650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131413+0000","last_change":"2024-03-30T00:55:51.814310+0000","last_active":"2024-03-30T00:55:56.131413+0000","last_peered":"2024-03-30T00:55:56.131413+0000","last_clean":"2024-03-30T00:55:56.131413+0000","last_became_active":"2024-03-30T00:55:51.813968+0000","last_became_peered":"2024-03-30T00:55:51.813968+0000","last_unstale":"2024-03-30T00:55:56.131413+0000","last_undegraded":"2024-03-30T00:55:56.131413+0000","last_fullsized":"2024-03-30T00:55:56.131413+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:59:55.955502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817322+0000","last_change":"2024-03-30T00:55:51.815655+0000","last_active":"2024-03-30T00:55:55.817322+0000","last_peered":"2024-03-30T00:55:55.817322+0000","last_clean":"2024-03-30T00:55:55.817322+0000","last_became_active":"2024-03-30T00:55:51.814099+0000","last_became_peered":"2024-03-30T00:55:51.814099+0000","last_unstale":"2024-03-30T00:55:55.817322+0000","last_undegraded":"2024-03-30T00:55:55.817322+0000","last_fullsized":"2024-03-30T00:55:55.817322+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:42.958153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817198+0000","last_change":"2024-03-30T00:55:51.819772+0000","last_active":"2024-03-30T00:55:55.817198+0000","last_peered":"2024-03-30T00:55:55.817198+0000","last_clean":"2024-03-30T00:55:55.817198+0000","last_became_active":"2024-03-30T00:55:51.816764+0000","last_became_peered":"2024-03-30T00:55:51.816764+0000","last_unstale":"2024-03-30T00:55:55.817198+0000","last_undegraded":"2024-03-30T00:55:55.817198+0000","last_fullsized":"2024-03-30T00:55:55.817198+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:26:26.763396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816489+0000","last_change":"2024-03-30T00:55:51.822278+0000","last_active":"2024-03-30T00:55:55.816489+0000","last_peered":"2024-03-30T00:55:55.816489+0000","last_clean":"2024-03-30T00:55:55.816489+0000","last_became_active":"2024-03-30T00:55:51.821996+0000","last_became_peered":"2024-03-30T00:55:51.821996+0000","last_unstale":"2024-03-30T00:55:55.816489+0000","last_undegraded":"2024-03-30T00:55:55.816489+0000","last_fullsized":"2024-03-30T00:55:55.816489+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:37:45.841853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815484+0000","last_change":"2024-03-30T00:55:51.809547+0000","last_active":"2024-03-30T00:55:55.815484+0000","last_peered":"2024-03-30T00:55:55.815484+0000","last_clean":"2024-03-30T00:55:55.815484+0000","last_became_active":"2024-03-30T00:55:51.809314+0000","last_became_peered":"2024-03-30T00:55:51.809314+0000","last_unstale":"2024-03-30T00:55:55.815484+0000","last_undegraded":"2024-03-30T00:55:55.815484+0000","last_fullsized":"2024-03-30T00:55:55.815484+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:39:31.661362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.835459+0000","last_change":"2024-03-30T00:55:51.810857+0000","last_active":"2024-03-30T00:55:55.835459+0000","last_peered":"2024-03-30T00:55:55.835459+0000","last_clean":"2024-03-30T00:55:55.835459+0000","last_became_active":"2024-03-30T00:55:51.809420+0000","last_became_peered":"2024-03-30T00:55:51.809420+0000","last_unstale":"2024-03-30T00:55:55.835459+0000","last_undegraded":"2024-03-30T00:55:55.835459+0000","last_fullsized":"2024-03-30T00:55:55.835459+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:36:48.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.820658+0000","last_change":"2024-03-30T00:55:51.814693+0000","last_active":"2024-03-30T00:55:55.820658+0000","last_peered":"2024-03-30T00:55:55.820658+0000","last_clean":"2024-03-30T00:55:55.820658+0000","last_became_active":"2024-03-30T00:55:51.814359+0000","last_became_peered":"2024-03-30T00:55:51.814359+0000","last_unstale":"2024-03-30T00:55:55.820658+0000","last_undegraded":"2024-03-30T00:55:55.820658+0000","last_fullsized":"2024-03-30T00:55:55.820658+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:20:50.856313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.097403+0000","last_change":"2024-03-30T00:55:55.839089+0000","last_active":"2024-03-30T00:55:57.097403+0000","last_peered":"2024-03-30T00:55:57.097403+0000","last_clean":"2024-03-30T00:55:57.097403+0000","last_became_active":"2024-03-30T00:55:55.838705+0000","last_became_peered":"2024-03-30T00:55:55.838705+0000","last_unstale":"2024-03-30T00:55:57.097403+0000","last_undegraded":"2024-03-30T00:55:57.097403+0000","last_fullsized":"2024-03-30T00:55:57.097403+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:33:56.010272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'9","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:21.412206+0000","last_change":"2024-03-30T00:55:51.298577+0000","last_active":"2024-03-30T00:56:21.412206+0000","last_peered":"2024-03-30T00:56:21.412206+0000","last_clean":"2024-03-30T00:56:21.412206+0000","last_became_active":"2024-03-30T00:55:51.298269+0000","last_became_peered":"2024-03-30T00:55:51.298269+0000","last_unstale":"2024-03-30T00:56:21.412206+0000","last_undegraded":"2024-03-30T00:56:21.412206+0000","last_fullsized":"2024-03-30T00:56:21.412206+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:10:04.998621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141755+0000","last_change":"2024-03-30T00:55:51.818517+0000","last_active":"2024-03-30T00:55:56.141755+0000","last_peered":"2024-03-30T00:55:56.141755+0000","last_clean":"2024-03-30T00:55:56.141755+0000","last_became_active":"2024-03-30T00:55:51.818174+0000","last_became_peered":"2024-03-30T00:55:51.818174+0000","last_unstale":"2024-03-30T00:55:56.141755+0000","last_undegraded":"2024-03-30T00:55:56.141755+0000","last_fullsized":"2024-03-30T00:55:56.141755+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:03:01.604135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.329627+0000","last_change":"2024-03-30T00:55:55.836374+0000","last_active":"2024-03-30T00:55:56.329627+0000","last_peered":"2024-03-30T00:55:56.329627+0000","last_clean":"2024-03-30T00:55:56.329627+0000","last_became_active":"2024-03-30T00:55:55.835993+0000","last_became_peered":"2024-03-30T00:55:55.835993+0000","last_unstale":"2024-03-30T00:55:56.329627+0000","last_undegraded":"2024-03-30T00:55:56.329627+0000","last_fullsized":"2024-03-30T00:55:56.329627+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:52:12.165344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"76'1","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.833854+0000","last_change":"2024-03-30T00:55:51.312686+0000","last_active":"2024-03-30T00:55:55.833854+0000","last_peered":"2024-03-30T00:55:55.833854+0000","last_clean":"2024-03-30T00:55:55.833854+0000","last_became_active":"2024-03-30T00:55:51.312296+0000","last_became_peered":"2024-03-30T00:55:51.312296+0000","last_unstale":"2024-03-30T00:55:55.833854+0000","last_undegraded":"2024-03-30T00:55:55.833854+0000","last_fullsized":"2024-03-30T00:55:55.833854+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:28:58.398577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817450+0000","last_change":"2024-03-30T00:55:51.812197+0000","last_active":"2024-03-30T00:55:55.817450+0000","last_peered":"2024-03-30T00:55:55.817450+0000","last_clean":"2024-03-30T00:55:55.817450+0000","last_became_active":"2024-03-30T00:55:51.811892+0000","last_became_peered":"2024-03-30T00:55:51.811892+0000","last_unstale":"2024-03-30T00:55:55.817450+0000","last_undegraded":"2024-03-30T00:55:55.817450+0000","last_fullsized":"2024-03-30T00:55:55.817450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:06:53.459760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.329818+0000","last_change":"2024-03-30T00:55:55.839015+0000","last_active":"2024-03-30T00:55:56.329818+0000","last_peered":"2024-03-30T00:55:56.329818+0000","last_clean":"2024-03-30T00:55:56.329818+0000","last_became_active":"2024-03-30T00:55:55.838608+0000","last_became_peered":"2024-03-30T00:55:55.838608+0000","last_unstale":"2024-03-30T00:55:56.329818+0000","last_undegraded":"2024-03-30T00:55:56.329818+0000","last_fullsized":"2024-03-30T00:55:56.329818+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:24:23.804754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"76'2","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816967+0000","last_change":"2024-03-30T00:55:51.304199+0000","last_active":"2024-03-30T00:55:55.816967+0000","last_peered":"2024-03-30T00:55:55.816967+0000","last_clean":"2024-03-30T00:55:55.816967+0000","last_became_active":"2024-03-30T00:55:51.303837+0000","last_became_peered":"2024-03-30T00:55:51.303837+0000","last_unstale":"2024-03-30T00:55:55.816967+0000","last_undegraded":"2024-03-30T00:55:55.816967+0000","last_fullsized":"2024-03-30T00:55:55.816967+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:19.154957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816615+0000","last_change":"2024-03-30T00:55:51.817984+0000","last_active":"2024-03-30T00:55:55.816615+0000","last_peered":"2024-03-30T00:55:55.816615+0000","last_clean":"2024-03-30T00:55:55.816615+0000","last_became_active":"2024-03-30T00:55:51.817767+0000","last_became_peered":"2024-03-30T00:55:51.817767+0000","last_unstale":"2024-03-30T00:55:55.816615+0000","last_undegraded":"2024-03-30T00:55:55.816615+0000","last_fullsized":"2024-03-30T00:55:55.816615+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:12:48.558024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.329638+0000","last_change":"2024-03-30T00:55:55.836420+0000","last_active":"2024-03-30T00:55:56.329638+0000","last_peered":"2024-03-30T00:55:56.329638+0000","last_clean":"2024-03-30T00:55:56.329638+0000","last_became_active":"2024-03-30T00:55:55.836055+0000","last_became_peered":"2024-03-30T00:55:55.836055+0000","last_unstale":"2024-03-30T00:55:56.329638+0000","last_undegraded":"2024-03-30T00:55:56.329638+0000","last_fullsized":"2024-03-30T00:55:56.329638+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:40:53.920703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"76'2","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817465+0000","last_change":"2024-03-30T00:55:51.300535+0000","last_active":"2024-03-30T00:55:55.817465+0000","last_peered":"2024-03-30T00:55:55.817465+0000","last_clean":"2024-03-30T00:55:55.817465+0000","last_became_active":"2024-03-30T00:55:51.300007+0000","last_became_peered":"2024-03-30T00:55:51.300007+0000","last_unstale":"2024-03-30T00:55:55.817465+0000","last_undegraded":"2024-03-30T00:55:55.817465+0000","last_fullsized":"2024-03-30T00:55:55.817465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:44:01.862562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812416+0000","last_change":"2024-03-30T00:55:51.827985+0000","last_active":"2024-03-30T00:55:55.812416+0000","last_peered":"2024-03-30T00:55:55.812416+0000","last_clean":"2024-03-30T00:55:55.812416+0000","last_became_active":"2024-03-30T00:55:51.827678+0000","last_became_peered":"2024-03-30T00:55:51.827678+0000","last_unstale":"2024-03-30T00:55:55.812416+0000","last_undegraded":"2024-03-30T00:55:55.812416+0000","last_fullsized":"2024-03-30T00:55:55.812416+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:47.088009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.096240+0000","last_change":"2024-03-30T00:55:55.827267+0000","last_active":"2024-03-30T00:55:57.096240+0000","last_peered":"2024-03-30T00:55:57.096240+0000","last_clean":"2024-03-30T00:55:57.096240+0000","last_became_active":"2024-03-30T00:55:55.826821+0000","last_became_peered":"2024-03-30T00:55:55.826821+0000","last_unstale":"2024-03-30T00:55:57.096240+0000","last_undegraded":"2024-03-30T00:55:57.096240+0000","last_fullsized":"2024-03-30T00:55:57.096240+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:15:49.084359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"76'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812263+0000","last_change":"2024-03-30T00:55:51.305177+0000","last_active":"2024-03-30T00:55:55.812263+0000","last_peered":"2024-03-30T00:55:55.812263+0000","last_clean":"2024-03-30T00:55:55.812263+0000","last_became_active":"2024-03-30T00:55:51.304813+0000","last_became_peered":"2024-03-30T00:55:51.304813+0000","last_unstale":"2024-03-30T00:55:55.812263+0000","last_undegraded":"2024-03-30T00:55:55.812263+0000","last_fullsized":"2024-03-30T00:55:55.812263+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:05:20.578897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815997+0000","last_change":"2024-03-30T00:55:51.812714+0000","last_active":"2024-03-30T00:55:55.815997+0000","last_peered":"2024-03-30T00:55:55.815997+0000","last_clean":"2024-03-30T00:55:55.815997+0000","last_became_active":"2024-03-30T00:55:51.812406+0000","last_became_peered":"2024-03-30T00:55:51.812406+0000","last_unstale":"2024-03-30T00:55:55.815997+0000","last_undegraded":"2024-03-30T00:55:55.815997+0000","last_fullsized":"2024-03-30T00:55:55.815997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:47:01.691412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.096477+0000","last_change":"2024-03-30T00:55:55.831994+0000","last_active":"2024-03-30T00:55:57.096477+0000","last_peered":"2024-03-30T00:55:57.096477+0000","last_clean":"2024-03-30T00:55:57.096477+0000","last_became_active":"2024-03-30T00:55:55.831537+0000","last_became_peered":"2024-03-30T00:55:55.831537+0000","last_unstale":"2024-03-30T00:55:57.096477+0000","last_undegraded":"2024-03-30T00:55:57.096477+0000","last_fullsized":"2024-03-30T00:55:57.096477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:44:47.240981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:21.410078+0000","last_change":"2024-03-30T00:55:51.303468+0000","last_active":"2024-03-30T00:56:21.410078+0000","last_peered":"2024-03-30T00:56:21.410078+0000","last_clean":"2024-03-30T00:56:21.410078+0000","last_became_active":"2024-03-30T00:55:51.303068+0000","last_became_peered":"2024-03-30T00:55:51.303068+0000","last_unstale":"2024-03-30T00:56:21.410078+0000","last_undegraded":"2024-03-30T00:56:21.410078+0000","last_fullsized":"2024-03-30T00:56:21.410078+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:45:25.278196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812380+0000","last_change":"2024-03-30T00:55:51.825099+0000","last_active":"2024-03-30T00:55:55.812380+0000","last_peered":"2024-03-30T00:55:55.812380+0000","last_clean":"2024-03-30T00:55:55.812380+0000","last_became_active":"2024-03-30T00:55:51.824717+0000","last_became_peered":"2024-03-30T00:55:51.824717+0000","last_unstale":"2024-03-30T00:55:55.812380+0000","last_undegraded":"2024-03-30T00:55:55.812380+0000","last_fullsized":"2024-03-30T00:55:55.812380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:49:31.542430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.096641+0000","last_change":"2024-03-30T00:55:55.826106+0000","last_active":"2024-03-30T00:55:57.096641+0000","last_peered":"2024-03-30T00:55:57.096641+0000","last_clean":"2024-03-30T00:55:57.096641+0000","last_became_active":"2024-03-30T00:55:55.825597+0000","last_became_peered":"2024-03-30T00:55:55.825597+0000","last_unstale":"2024-03-30T00:55:57.096641+0000","last_undegraded":"2024-03-30T00:55:57.096641+0000","last_fullsized":"2024-03-30T00:55:57.096641+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:00:18.991732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.811402+0000","last_change":"2024-03-30T00:55:51.294090+0000","last_active":"2024-03-30T00:55:55.811402+0000","last_peered":"2024-03-30T00:55:55.811402+0000","last_clean":"2024-03-30T00:55:55.811402+0000","last_became_active":"2024-03-30T00:55:51.293668+0000","last_became_peered":"2024-03-30T00:55:51.293668+0000","last_unstale":"2024-03-30T00:55:55.811402+0000","last_undegraded":"2024-03-30T00:55:55.811402+0000","last_fullsized":"2024-03-30T00:55:55.811402+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:33:36.661760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141658+0000","last_change":"2024-03-30T00:55:51.823428+0000","last_active":"2024-03-30T00:55:56.141658+0000","last_peered":"2024-03-30T00:55:56.141658+0000","last_clean":"2024-03-30T00:55:56.141658+0000","last_became_active":"2024-03-30T00:55:51.822222+0000","last_became_peered":"2024-03-30T00:55:51.822222+0000","last_unstale":"2024-03-30T00:55:56.141658+0000","last_undegraded":"2024-03-30T00:55:56.141658+0000","last_fullsized":"2024-03-30T00:55:56.141658+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:05:34.798940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.831707+0000","last_change":"2024-03-30T00:55:55.828115+0000","last_active":"2024-03-30T00:55:55.831707+0000","last_peered":"2024-03-30T00:55:55.831707+0000","last_clean":"2024-03-30T00:55:55.831707+0000","last_became_active":"2024-03-30T00:55:55.827681+0000","last_became_peered":"2024-03-30T00:55:55.827681+0000","last_unstale":"2024-03-30T00:55:55.831707+0000","last_undegraded":"2024-03-30T00:55:55.831707+0000","last_fullsized":"2024-03-30T00:55:55.831707+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:18:25.751350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812943+0000","last_change":"2024-03-30T00:55:51.305163+0000","last_active":"2024-03-30T00:55:55.812943+0000","last_peered":"2024-03-30T00:55:55.812943+0000","last_clean":"2024-03-30T00:55:55.812943+0000","last_became_active":"2024-03-30T00:55:51.304860+0000","last_became_peered":"2024-03-30T00:55:51.304860+0000","last_unstale":"2024-03-30T00:55:55.812943+0000","last_undegraded":"2024-03-30T00:55:55.812943+0000","last_fullsized":"2024-03-30T00:55:55.812943+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:44:41.850998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.832243+0000","last_change":"2024-03-30T00:55:51.819401+0000","last_active":"2024-03-30T00:55:55.832243+0000","last_peered":"2024-03-30T00:55:55.832243+0000","last_clean":"2024-03-30T00:55:55.832243+0000","last_became_active":"2024-03-30T00:55:51.819021+0000","last_became_peered":"2024-03-30T00:55:51.819021+0000","last_unstale":"2024-03-30T00:55:55.832243+0000","last_undegraded":"2024-03-30T00:55:55.832243+0000","last_fullsized":"2024-03-30T00:55:55.832243+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:01:45.195719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141690+0000","last_change":"2024-03-30T00:55:55.836806+0000","last_active":"2024-03-30T00:55:56.141690+0000","last_peered":"2024-03-30T00:55:56.141690+0000","last_clean":"2024-03-30T00:55:56.141690+0000","last_became_active":"2024-03-30T00:55:55.836403+0000","last_became_peered":"2024-03-30T00:55:55.836403+0000","last_unstale":"2024-03-30T00:55:56.141690+0000","last_undegraded":"2024-03-30T00:55:56.141690+0000","last_fullsized":"2024-03-30T00:55:56.141690+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:17:13.913675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'3","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:23.157955+0000","last_change":"2024-03-30T00:55:51.300833+0000","last_active":"2024-03-30T00:56:23.157955+0000","last_peered":"2024-03-30T00:56:23.157955+0000","last_clean":"2024-03-30T00:56:23.157955+0000","last_became_active":"2024-03-30T00:55:51.300473+0000","last_became_peered":"2024-03-30T00:55:51.300473+0000","last_unstale":"2024-03-30T00:56:23.157955+0000","last_undegraded":"2024-03-30T00:56:23.157955+0000","last_fullsized":"2024-03-30T00:56:23.157955+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:00:39.211981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816447+0000","last_change":"2024-03-30T00:55:51.810781+0000","last_active":"2024-03-30T00:55:55.816447+0000","last_peered":"2024-03-30T00:55:55.816447+0000","last_clean":"2024-03-30T00:55:55.816447+0000","last_became_active":"2024-03-30T00:55:51.810215+0000","last_became_peered":"2024-03-30T00:55:51.810215+0000","last_unstale":"2024-03-30T00:55:55.816447+0000","last_undegraded":"2024-03-30T00:55:55.816447+0000","last_fullsized":"2024-03-30T00:55:55.816447+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:53:50.283209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.540701+0000","last_change":"2024-03-30T00:55:55.834505+0000","last_active":"2024-03-30T00:55:56.540701+0000","last_peered":"2024-03-30T00:55:56.540701+0000","last_clean":"2024-03-30T00:55:56.540701+0000","last_became_active":"2024-03-30T00:55:55.829319+0000","last_became_peered":"2024-03-30T00:55:55.829319+0000","last_unstale":"2024-03-30T00:55:56.540701+0000","last_undegraded":"2024-03-30T00:55:56.540701+0000","last_fullsized":"2024-03-30T00:55:56.540701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:51:37.595523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'2","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:20.396339+0000","last_change":"2024-03-30T00:55:51.302070+0000","last_active":"2024-03-30T00:56:20.396339+0000","last_peered":"2024-03-30T00:56:20.396339+0000","last_clean":"2024-03-30T00:56:20.396339+0000","last_became_active":"2024-03-30T00:55:51.298940+0000","last_became_peered":"2024-03-30T00:55:51.298940+0000","last_unstale":"2024-03-30T00:56:20.396339+0000","last_undegraded":"2024-03-30T00:56:20.396339+0000","last_fullsized":"2024-03-30T00:56:20.396339+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:07:47.915506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817361+0000","last_change":"2024-03-30T00:55:51.819918+0000","last_active":"2024-03-30T00:55:55.817361+0000","last_peered":"2024-03-30T00:55:55.817361+0000","last_clean":"2024-03-30T00:55:55.817361+0000","last_became_active":"2024-03-30T00:55:51.819493+0000","last_became_peered":"2024-03-30T00:55:51.819493+0000","last_unstale":"2024-03-30T00:55:55.817361+0000","last_undegraded":"2024-03-30T00:55:55.817361+0000","last_fullsized":"2024-03-30T00:55:55.817361+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:43:43.067031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.330119+0000","last_change":"2024-03-30T00:55:55.827176+0000","last_active":"2024-03-30T00:55:56.330119+0000","last_peered":"2024-03-30T00:55:56.330119+0000","last_clean":"2024-03-30T00:55:56.330119+0000","last_became_active":"2024-03-30T00:55:55.826772+0000","last_became_peered":"2024-03-30T00:55:55.826772+0000","last_unstale":"2024-03-30T00:55:56.330119+0000","last_undegraded":"2024-03-30T00:55:56.330119+0000","last_fullsized":"2024-03-30T00:55:56.330119+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:02:04.512050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"76'3","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812239+0000","last_change":"2024-03-30T00:55:51.306688+0000","last_active":"2024-03-30T00:55:55.812239+0000","last_peered":"2024-03-30T00:55:55.812239+0000","last_clean":"2024-03-30T00:55:55.812239+0000","last_became_active":"2024-03-30T00:55:51.306333+0000","last_became_peered":"2024-03-30T00:55:51.306333+0000","last_unstale":"2024-03-30T00:55:55.812239+0000","last_undegraded":"2024-03-30T00:55:55.812239+0000","last_fullsized":"2024-03-30T00:55:55.812239+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:21:13.719947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816780+0000","last_change":"2024-03-30T00:55:51.824879+0000","last_active":"2024-03-30T00:55:55.816780+0000","last_peered":"2024-03-30T00:55:55.816780+0000","last_clean":"2024-03-30T00:55:55.816780+0000","last_became_active":"2024-03-30T00:55:51.824566+0000","last_became_peered":"2024-03-30T00:55:51.824566+0000","last_unstale":"2024-03-30T00:55:55.816780+0000","last_undegraded":"2024-03-30T00:55:55.816780+0000","last_fullsized":"2024-03-30T00:55:55.816780+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:04:28.505693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.540728+0000","last_change":"2024-03-30T00:55:55.826387+0000","last_active":"2024-03-30T00:55:56.540728+0000","last_peered":"2024-03-30T00:55:56.540728+0000","last_clean":"2024-03-30T00:55:56.540728+0000","last_became_active":"2024-03-30T00:55:55.825941+0000","last_became_peered":"2024-03-30T00:55:55.825941+0000","last_unstale":"2024-03-30T00:55:56.540728+0000","last_undegraded":"2024-03-30T00:55:56.540728+0000","last_fullsized":"2024-03-30T00:55:56.540728+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:34:43.432712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"81'23","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:24.160174+0000","last_change":"2024-03-30T00:55:51.304592+0000","last_active":"2024-03-30T00:56:24.160174+0000","last_peered":"2024-03-30T00:56:24.160174+0000","last_clean":"2024-03-30T00:56:24.160174+0000","last_became_active":"2024-03-30T00:55:51.303931+0000","last_became_peered":"2024-03-30T00:55:51.303931+0000","last_unstale":"2024-03-30T00:56:24.160174+0000","last_undegraded":"2024-03-30T00:56:24.160174+0000","last_fullsized":"2024-03-30T00:56:24.160174+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:53:40.976289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40427,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":44,"num_write":24,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816688+0000","last_change":"2024-03-30T00:55:51.815196+0000","last_active":"2024-03-30T00:55:55.816688+0000","last_peered":"2024-03-30T00:55:55.816688+0000","last_clean":"2024-03-30T00:55:55.816688+0000","last_became_active":"2024-03-30T00:55:51.813376+0000","last_became_peered":"2024-03-30T00:55:51.813376+0000","last_unstale":"2024-03-30T00:55:55.816688+0000","last_undegraded":"2024-03-30T00:55:55.816688+0000","last_fullsized":"2024-03-30T00:55:55.816688+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:40:26.350248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.837509+0000","last_change":"2024-03-30T00:55:55.836906+0000","last_active":"2024-03-30T00:55:55.837509+0000","last_peered":"2024-03-30T00:55:55.837509+0000","last_clean":"2024-03-30T00:55:55.837509+0000","last_became_active":"2024-03-30T00:55:55.836539+0000","last_became_peered":"2024-03-30T00:55:55.836539+0000","last_unstale":"2024-03-30T00:55:55.837509+0000","last_undegraded":"2024-03-30T00:55:55.837509+0000","last_fullsized":"2024-03-30T00:55:55.837509+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:21:39.320192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'6","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:21.411514+0000","last_change":"2024-03-30T00:55:51.316839+0000","last_active":"2024-03-30T00:56:21.411514+0000","last_peered":"2024-03-30T00:56:21.411514+0000","last_clean":"2024-03-30T00:56:21.411514+0000","last_became_active":"2024-03-30T00:55:51.316519+0000","last_became_peered":"2024-03-30T00:55:51.316519+0000","last_unstale":"2024-03-30T00:56:21.411514+0000","last_undegraded":"2024-03-30T00:56:21.411514+0000","last_fullsized":"2024-03-30T00:56:21.411514+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:25:32.038384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131378+0000","last_change":"2024-03-30T00:55:51.819825+0000","last_active":"2024-03-30T00:55:56.131378+0000","last_peered":"2024-03-30T00:55:56.131378+0000","last_clean":"2024-03-30T00:55:56.131378+0000","last_became_active":"2024-03-30T00:55:51.819481+0000","last_became_peered":"2024-03-30T00:55:51.819481+0000","last_unstale":"2024-03-30T00:55:56.131378+0000","last_undegraded":"2024-03-30T00:55:56.131378+0000","last_fullsized":"2024-03-30T00:55:56.131378+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:10:17.138541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.097020+0000","last_change":"2024-03-30T00:55:55.832675+0000","last_active":"2024-03-30T00:55:57.097020+0000","last_peered":"2024-03-30T00:55:57.097020+0000","last_clean":"2024-03-30T00:55:57.097020+0000","last_became_active":"2024-03-30T00:55:55.832325+0000","last_became_peered":"2024-03-30T00:55:55.832325+0000","last_unstale":"2024-03-30T00:55:57.097020+0000","last_undegraded":"2024-03-30T00:55:57.097020+0000","last_fullsized":"2024-03-30T00:55:57.097020+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:38:46.765473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812030+0000","last_change":"2024-03-30T00:55:51.313700+0000","last_active":"2024-03-30T00:55:55.812030+0000","last_peered":"2024-03-30T00:55:55.812030+0000","last_clean":"2024-03-30T00:55:55.812030+0000","last_became_active":"2024-03-30T00:55:51.313355+0000","last_became_peered":"2024-03-30T00:55:51.313355+0000","last_unstale":"2024-03-30T00:55:55.812030+0000","last_undegraded":"2024-03-30T00:55:55.812030+0000","last_fullsized":"2024-03-30T00:55:55.812030+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:43:04.461398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.823400+0000","last_change":"2024-03-30T00:55:51.819939+0000","last_active":"2024-03-30T00:55:55.823400+0000","last_peered":"2024-03-30T00:55:55.823400+0000","last_clean":"2024-03-30T00:55:55.823400+0000","last_became_active":"2024-03-30T00:55:51.819614+0000","last_became_peered":"2024-03-30T00:55:51.819614+0000","last_unstale":"2024-03-30T00:55:55.823400+0000","last_undegraded":"2024-03-30T00:55:55.823400+0000","last_fullsized":"2024-03-30T00:55:55.823400+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:20:37.362779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.540778+0000","last_change":"2024-03-30T00:55:55.841536+0000","last_active":"2024-03-30T00:55:56.540778+0000","last_peered":"2024-03-30T00:55:56.540778+0000","last_clean":"2024-03-30T00:55:56.540778+0000","last_became_active":"2024-03-30T00:55:55.839691+0000","last_became_peered":"2024-03-30T00:55:55.839691+0000","last_unstale":"2024-03-30T00:55:56.540778+0000","last_undegraded":"2024-03-30T00:55:56.540778+0000","last_fullsized":"2024-03-30T00:55:56.540778+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:52:53.769565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"81'4","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:23.159572+0000","last_change":"2024-03-30T00:55:51.299246+0000","last_active":"2024-03-30T00:56:23.159572+0000","last_peered":"2024-03-30T00:56:23.159572+0000","last_clean":"2024-03-30T00:56:23.159572+0000","last_became_active":"2024-03-30T00:55:51.298924+0000","last_became_peered":"2024-03-30T00:55:51.298924+0000","last_unstale":"2024-03-30T00:56:23.159572+0000","last_undegraded":"2024-03-30T00:56:23.159572+0000","last_fullsized":"2024-03-30T00:56:23.159572+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:50:05.897612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.823167+0000","last_change":"2024-03-30T00:55:51.813804+0000","last_active":"2024-03-30T00:55:55.823167+0000","last_peered":"2024-03-30T00:55:55.823167+0000","last_clean":"2024-03-30T00:55:55.823167+0000","last_became_active":"2024-03-30T00:55:51.813500+0000","last_became_peered":"2024-03-30T00:55:51.813500+0000","last_unstale":"2024-03-30T00:55:55.823167+0000","last_undegraded":"2024-03-30T00:55:55.823167+0000","last_fullsized":"2024-03-30T00:55:55.823167+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:42:21.614900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131528+0000","last_change":"2024-03-30T00:55:55.837737+0000","last_active":"2024-03-30T00:55:56.131528+0000","last_peered":"2024-03-30T00:55:56.131528+0000","last_clean":"2024-03-30T00:55:56.131528+0000","last_became_active":"2024-03-30T00:55:55.837312+0000","last_became_peered":"2024-03-30T00:55:55.837312+0000","last_unstale":"2024-03-30T00:55:56.131528+0000","last_undegraded":"2024-03-30T00:55:56.131528+0000","last_fullsized":"2024-03-30T00:55:56.131528+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:11:46.139979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'10","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:20.393517+0000","last_change":"2024-03-30T00:55:51.306398+0000","last_active":"2024-03-30T00:56:20.393517+0000","last_peered":"2024-03-30T00:56:20.393517+0000","last_clean":"2024-03-30T00:56:20.393517+0000","last_became_active":"2024-03-30T00:55:51.305878+0000","last_became_peered":"2024-03-30T00:55:51.305878+0000","last_unstale":"2024-03-30T00:56:20.393517+0000","last_undegraded":"2024-03-30T00:56:20.393517+0000","last_fullsized":"2024-03-30T00:56:20.393517+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:20:32.742423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812237+0000","last_change":"2024-03-30T00:55:51.824294+0000","last_active":"2024-03-30T00:55:55.812237+0000","last_peered":"2024-03-30T00:55:55.812237+0000","last_clean":"2024-03-30T00:55:55.812237+0000","last_became_active":"2024-03-30T00:55:51.823957+0000","last_became_peered":"2024-03-30T00:55:51.823957+0000","last_unstale":"2024-03-30T00:55:55.812237+0000","last_undegraded":"2024-03-30T00:55:55.812237+0000","last_fullsized":"2024-03-30T00:55:55.812237+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:23:16.873140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.837548+0000","last_change":"2024-03-30T00:55:55.837023+0000","last_active":"2024-03-30T00:55:55.837548+0000","last_peered":"2024-03-30T00:55:55.837548+0000","last_clean":"2024-03-30T00:55:55.837548+0000","last_became_active":"2024-03-30T00:55:55.836541+0000","last_became_peered":"2024-03-30T00:55:55.836541+0000","last_unstale":"2024-03-30T00:55:55.837548+0000","last_undegraded":"2024-03-30T00:55:55.837548+0000","last_fullsized":"2024-03-30T00:55:55.837548+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:58:40.323953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"76'1","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815282+0000","last_change":"2024-03-30T00:55:51.300883+0000","last_active":"2024-03-30T00:55:55.815282+0000","last_peered":"2024-03-30T00:55:55.815282+0000","last_clean":"2024-03-30T00:55:55.815282+0000","last_became_active":"2024-03-30T00:55:51.300531+0000","last_became_peered":"2024-03-30T00:55:51.300531+0000","last_unstale":"2024-03-30T00:55:55.815282+0000","last_undegraded":"2024-03-30T00:55:55.815282+0000","last_fullsized":"2024-03-30T00:55:55.815282+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:38:38.874075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815671+0000","last_change":"2024-03-30T00:55:51.822970+0000","last_active":"2024-03-30T00:55:55.815671+0000","last_peered":"2024-03-30T00:55:55.815671+0000","last_clean":"2024-03-30T00:55:55.815671+0000","last_became_active":"2024-03-30T00:55:51.822653+0000","last_became_peered":"2024-03-30T00:55:51.822653+0000","last_unstale":"2024-03-30T00:55:55.815671+0000","last_undegraded":"2024-03-30T00:55:55.815671+0000","last_fullsized":"2024-03-30T00:55:55.815671+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:59:26.306672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.857191+0000","last_change":"2024-03-30T00:55:55.856605+0000","last_active":"2024-03-30T00:55:55.857191+0000","last_peered":"2024-03-30T00:55:55.857191+0000","last_clean":"2024-03-30T00:55:55.857191+0000","last_became_active":"2024-03-30T00:55:55.856061+0000","last_became_peered":"2024-03-30T00:55:55.856061+0000","last_unstale":"2024-03-30T00:55:55.857191+0000","last_undegraded":"2024-03-30T00:55:55.857191+0000","last_fullsized":"2024-03-30T00:55:55.857191+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:58:41.010206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.834443+0000","last_change":"2024-03-30T00:55:51.301961+0000","last_active":"2024-03-30T00:55:55.834443+0000","last_peered":"2024-03-30T00:55:55.834443+0000","last_clean":"2024-03-30T00:55:55.834443+0000","last_became_active":"2024-03-30T00:55:51.301618+0000","last_became_peered":"2024-03-30T00:55:51.301618+0000","last_unstale":"2024-03-30T00:55:55.834443+0000","last_undegraded":"2024-03-30T00:55:55.834443+0000","last_fullsized":"2024-03-30T00:55:55.834443+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:38:46.681490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.855479+0000","last_change":"2024-03-30T00:55:51.821331+0000","last_active":"2024-03-30T00:55:55.855479+0000","last_peered":"2024-03-30T00:55:55.855479+0000","last_clean":"2024-03-30T00:55:55.855479+0000","last_became_active":"2024-03-30T00:55:51.820988+0000","last_became_peered":"2024-03-30T00:55:51.820988+0000","last_unstale":"2024-03-30T00:55:55.855479+0000","last_undegraded":"2024-03-30T00:55:55.855479+0000","last_fullsized":"2024-03-30T00:55:55.855479+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:25:20.150789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131430+0000","last_change":"2024-03-30T00:55:55.826251+0000","last_active":"2024-03-30T00:55:56.131430+0000","last_peered":"2024-03-30T00:55:56.131430+0000","last_clean":"2024-03-30T00:55:56.131430+0000","last_became_active":"2024-03-30T00:55:55.822695+0000","last_became_peered":"2024-03-30T00:55:55.822695+0000","last_unstale":"2024-03-30T00:55:56.131430+0000","last_undegraded":"2024-03-30T00:55:56.131430+0000","last_fullsized":"2024-03-30T00:55:56.131430+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:32:19.011454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'1","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.017740+0000","last_change":"2024-03-30T00:55:51.309412+0000","last_active":"2024-03-30T00:55:57.017740+0000","last_peered":"2024-03-30T00:55:57.017740+0000","last_clean":"2024-03-30T00:55:57.017740+0000","last_became_active":"2024-03-30T00:55:51.309028+0000","last_became_peered":"2024-03-30T00:55:51.309028+0000","last_unstale":"2024-03-30T00:55:57.017740+0000","last_undegraded":"2024-03-30T00:55:57.017740+0000","last_fullsized":"2024-03-30T00:55:57.017740+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:00:49.022772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.855266+0000","last_change":"2024-03-30T00:55:51.825164+0000","last_active":"2024-03-30T00:55:55.855266+0000","last_peered":"2024-03-30T00:55:55.855266+0000","last_clean":"2024-03-30T00:55:55.855266+0000","last_became_active":"2024-03-30T00:55:51.824823+0000","last_became_peered":"2024-03-30T00:55:51.824823+0000","last_unstale":"2024-03-30T00:55:55.855266+0000","last_undegraded":"2024-03-30T00:55:55.855266+0000","last_fullsized":"2024-03-30T00:55:55.855266+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:29:11.471111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131539+0000","last_change":"2024-03-30T00:55:55.856377+0000","last_active":"2024-03-30T00:55:56.131539+0000","last_peered":"2024-03-30T00:55:56.131539+0000","last_clean":"2024-03-30T00:55:56.131539+0000","last_became_active":"2024-03-30T00:55:55.855986+0000","last_became_peered":"2024-03-30T00:55:55.855986+0000","last_unstale":"2024-03-30T00:55:56.131539+0000","last_undegraded":"2024-03-30T00:55:56.131539+0000","last_fullsized":"2024-03-30T00:55:56.131539+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:45:47.303446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.813157+0000","last_change":"2024-03-30T00:55:51.824176+0000","last_active":"2024-03-30T00:55:55.813157+0000","last_peered":"2024-03-30T00:55:55.813157+0000","last_clean":"2024-03-30T00:55:55.813157+0000","last_became_active":"2024-03-30T00:55:51.823674+0000","last_became_peered":"2024-03-30T00:55:51.823674+0000","last_unstale":"2024-03-30T00:55:55.813157+0000","last_undegraded":"2024-03-30T00:55:55.813157+0000","last_fullsized":"2024-03-30T00:55:55.813157+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:34:51.051973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'1","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.331029+0000","last_change":"2024-03-30T00:55:51.314834+0000","last_active":"2024-03-30T00:55:56.331029+0000","last_peered":"2024-03-30T00:55:56.331029+0000","last_clean":"2024-03-30T00:55:56.331029+0000","last_became_active":"2024-03-30T00:55:51.314429+0000","last_became_peered":"2024-03-30T00:55:51.314429+0000","last_unstale":"2024-03-30T00:55:56.331029+0000","last_undegraded":"2024-03-30T00:55:56.331029+0000","last_fullsized":"2024-03-30T00:55:56.331029+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:17:13.987180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:59.380272+0000","last_change":"2024-03-30T00:55:55.840721+0000","last_active":"2024-03-30T00:55:59.380272+0000","last_peered":"2024-03-30T00:55:59.380272+0000","last_clean":"2024-03-30T00:55:59.380272+0000","last_became_active":"2024-03-30T00:55:55.840269+0000","last_became_peered":"2024-03-30T00:55:55.840269+0000","last_unstale":"2024-03-30T00:55:59.380272+0000","last_undegraded":"2024-03-30T00:55:59.380272+0000","last_fullsized":"2024-03-30T00:55:59.380272+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:28:55.366411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141714+0000","last_change":"2024-03-30T00:55:51.812609+0000","last_active":"2024-03-30T00:55:56.141714+0000","last_peered":"2024-03-30T00:55:56.141714+0000","last_clean":"2024-03-30T00:55:56.141714+0000","last_became_active":"2024-03-30T00:55:51.812325+0000","last_became_peered":"2024-03-30T00:55:51.812325+0000","last_unstale":"2024-03-30T00:55:56.141714+0000","last_undegraded":"2024-03-30T00:55:56.141714+0000","last_fullsized":"2024-03-30T00:55:56.141714+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:22:22.074405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'3","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:22.154189+0000","last_change":"2024-03-30T00:55:51.296173+0000","last_active":"2024-03-30T00:56:22.154189+0000","last_peered":"2024-03-30T00:56:22.154189+0000","last_clean":"2024-03-30T00:56:22.154189+0000","last_became_active":"2024-03-30T00:55:51.295331+0000","last_became_peered":"2024-03-30T00:55:51.295331+0000","last_unstale":"2024-03-30T00:56:22.154189+0000","last_undegraded":"2024-03-30T00:56:22.154189+0000","last_fullsized":"2024-03-30T00:56:22.154189+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:51:58.343765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":324,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.017477+0000","last_change":"2024-03-30T00:55:02.487656+0000","last_active":"2024-03-30T00:55:57.017477+0000","last_peered":"2024-03-30T00:55:57.017477+0000","last_clean":"2024-03-30T00:55:57.017477+0000","last_became_active":"2024-03-30T00:55:02.487263+0000","last_became_peered":"2024-03-30T00:55:02.487263+0000","last_unstale":"2024-03-30T00:55:57.017477+0000","last_undegraded":"2024-03-30T00:55:57.017477+0000","last_fullsized":"2024-03-30T00:55:57.017477+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:52:29.691461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:52:29.691461+0000","last_clean_scrub_stamp":"2024-03-30T00:52:29.691461+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:19:20.825038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.857268+0000","last_change":"2024-03-30T00:55:55.856487+0000","last_active":"2024-03-30T00:55:55.857268+0000","last_peered":"2024-03-30T00:55:55.857268+0000","last_clean":"2024-03-30T00:55:55.857268+0000","last_became_active":"2024-03-30T00:55:55.856106+0000","last_became_peered":"2024-03-30T00:55:55.856106+0000","last_unstale":"2024-03-30T00:55:55.857268+0000","last_undegraded":"2024-03-30T00:55:55.857268+0000","last_fullsized":"2024-03-30T00:55:55.857268+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:14.504218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131308+0000","last_change":"2024-03-30T00:55:51.821519+0000","last_active":"2024-03-30T00:55:56.131308+0000","last_peered":"2024-03-30T00:55:56.131308+0000","last_clean":"2024-03-30T00:55:56.131308+0000","last_became_active":"2024-03-30T00:55:51.821095+0000","last_became_peered":"2024-03-30T00:55:51.821095+0000","last_unstale":"2024-03-30T00:55:56.131308+0000","last_undegraded":"2024-03-30T00:55:56.131308+0000","last_fullsized":"2024-03-30T00:55:56.131308+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:50:22.297450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'3","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:21.409836+0000","last_change":"2024-03-30T00:55:51.303047+0000","last_active":"2024-03-30T00:56:21.409836+0000","last_peered":"2024-03-30T00:56:21.409836+0000","last_clean":"2024-03-30T00:56:21.409836+0000","last_became_active":"2024-03-30T00:55:51.302712+0000","last_became_peered":"2024-03-30T00:55:51.302712+0000","last_unstale":"2024-03-30T00:56:21.409836+0000","last_undegraded":"2024-03-30T00:56:21.409836+0000","last_fullsized":"2024-03-30T00:56:21.409836+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:07:59.451674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.096266+0000","last_change":"2024-03-30T00:55:55.823692+0000","last_active":"2024-03-30T00:55:57.096266+0000","last_peered":"2024-03-30T00:55:57.096266+0000","last_clean":"2024-03-30T00:55:57.096266+0000","last_became_active":"2024-03-30T00:55:55.823188+0000","last_became_peered":"2024-03-30T00:55:55.823188+0000","last_unstale":"2024-03-30T00:55:57.096266+0000","last_undegraded":"2024-03-30T00:55:57.096266+0000","last_fullsized":"2024-03-30T00:55:57.096266+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:32:32.089166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'1","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.333575+0000","last_change":"2024-03-30T00:55:51.306935+0000","last_active":"2024-03-30T00:55:56.333575+0000","last_peered":"2024-03-30T00:55:56.333575+0000","last_clean":"2024-03-30T00:55:56.333575+0000","last_became_active":"2024-03-30T00:55:51.306595+0000","last_became_peered":"2024-03-30T00:55:51.306595+0000","last_unstale":"2024-03-30T00:55:56.333575+0000","last_undegraded":"2024-03-30T00:55:56.333575+0000","last_fullsized":"2024-03-30T00:55:56.333575+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:50:30.396762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131446+0000","last_change":"2024-03-30T00:55:51.807084+0000","last_active":"2024-03-30T00:55:56.131446+0000","last_peered":"2024-03-30T00:55:56.131446+0000","last_clean":"2024-03-30T00:55:56.131446+0000","last_became_active":"2024-03-30T00:55:51.806505+0000","last_became_peered":"2024-03-30T00:55:51.806505+0000","last_unstale":"2024-03-30T00:55:56.131446+0000","last_undegraded":"2024-03-30T00:55:56.131446+0000","last_fullsized":"2024-03-30T00:55:56.131446+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:38:44.759863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141634+0000","last_change":"2024-03-30T00:55:55.856457+0000","last_active":"2024-03-30T00:55:56.141634+0000","last_peered":"2024-03-30T00:55:56.141634+0000","last_clean":"2024-03-30T00:55:56.141634+0000","last_became_active":"2024-03-30T00:55:55.855982+0000","last_became_peered":"2024-03-30T00:55:55.855982+0000","last_unstale":"2024-03-30T00:55:56.141634+0000","last_undegraded":"2024-03-30T00:55:56.141634+0000","last_fullsized":"2024-03-30T00:55:56.141634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:18:25.154634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"81'3","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:21.409721+0000","last_change":"2024-03-30T00:55:51.312581+0000","last_active":"2024-03-30T00:56:21.409721+0000","last_peered":"2024-03-30T00:56:21.409721+0000","last_clean":"2024-03-30T00:56:21.409721+0000","last_became_active":"2024-03-30T00:55:51.312225+0000","last_became_peered":"2024-03-30T00:55:51.312225+0000","last_unstale":"2024-03-30T00:56:21.409721+0000","last_undegraded":"2024-03-30T00:56:21.409721+0000","last_fullsized":"2024-03-30T00:56:21.409721+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:27:12.615250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141773+0000","last_change":"2024-03-30T00:55:51.821527+0000","last_active":"2024-03-30T00:55:56.141773+0000","last_peered":"2024-03-30T00:55:56.141773+0000","last_clean":"2024-03-30T00:55:56.141773+0000","last_became_active":"2024-03-30T00:55:51.820689+0000","last_became_peered":"2024-03-30T00:55:51.820689+0000","last_unstale":"2024-03-30T00:55:56.141773+0000","last_undegraded":"2024-03-30T00:55:56.141773+0000","last_fullsized":"2024-03-30T00:55:56.141773+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:14:35.735985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.097219+0000","last_change":"2024-03-30T00:55:55.824307+0000","last_active":"2024-03-30T00:55:57.097219+0000","last_peered":"2024-03-30T00:55:57.097219+0000","last_clean":"2024-03-30T00:55:57.097219+0000","last_became_active":"2024-03-30T00:55:55.823900+0000","last_became_peered":"2024-03-30T00:55:55.823900+0000","last_unstale":"2024-03-30T00:55:57.097219+0000","last_undegraded":"2024-03-30T00:55:57.097219+0000","last_fullsized":"2024-03-30T00:55:57.097219+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:44:22.139120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:20.392961+0000","last_change":"2024-03-30T00:55:51.296965+0000","last_active":"2024-03-30T00:56:20.392961+0000","last_peered":"2024-03-30T00:56:20.392961+0000","last_clean":"2024-03-30T00:56:20.392961+0000","last_became_active":"2024-03-30T00:55:51.296645+0000","last_became_peered":"2024-03-30T00:55:51.296645+0000","last_unstale":"2024-03-30T00:56:20.392961+0000","last_undegraded":"2024-03-30T00:56:20.392961+0000","last_fullsized":"2024-03-30T00:56:20.392961+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:53:46.519972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.818968+0000","last_change":"2024-03-30T00:55:51.826282+0000","last_active":"2024-03-30T00:55:55.818968+0000","last_peered":"2024-03-30T00:55:55.818968+0000","last_clean":"2024-03-30T00:55:55.818968+0000","last_became_active":"2024-03-30T00:55:51.823910+0000","last_became_peered":"2024-03-30T00:55:51.823910+0000","last_unstale":"2024-03-30T00:55:55.818968+0000","last_undegraded":"2024-03-30T00:55:55.818968+0000","last_fullsized":"2024-03-30T00:55:55.818968+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:47.454280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.330002+0000","last_change":"2024-03-30T00:55:55.856750+0000","last_active":"2024-03-30T00:55:56.330002+0000","last_peered":"2024-03-30T00:55:56.330002+0000","last_clean":"2024-03-30T00:55:56.330002+0000","last_became_active":"2024-03-30T00:55:55.856288+0000","last_became_peered":"2024-03-30T00:55:55.856288+0000","last_unstale":"2024-03-30T00:55:56.330002+0000","last_undegraded":"2024-03-30T00:55:56.330002+0000","last_fullsized":"2024-03-30T00:55:56.330002+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:43:28.085910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:20.395225+0000","last_change":"2024-03-30T00:55:51.306357+0000","last_active":"2024-03-30T00:56:20.395225+0000","last_peered":"2024-03-30T00:56:20.395225+0000","last_clean":"2024-03-30T00:56:20.395225+0000","last_became_active":"2024-03-30T00:55:51.305815+0000","last_became_peered":"2024-03-30T00:55:51.305815+0000","last_unstale":"2024-03-30T00:56:20.395225+0000","last_undegraded":"2024-03-30T00:56:20.395225+0000","last_fullsized":"2024-03-30T00:56:20.395225+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:16:21.753446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817389+0000","last_change":"2024-03-30T00:55:51.810686+0000","last_active":"2024-03-30T00:55:55.817389+0000","last_peered":"2024-03-30T00:55:55.817389+0000","last_clean":"2024-03-30T00:55:55.817389+0000","last_became_active":"2024-03-30T00:55:51.810331+0000","last_became_peered":"2024-03-30T00:55:51.810331+0000","last_unstale":"2024-03-30T00:55:55.817389+0000","last_undegraded":"2024-03-30T00:55:55.817389+0000","last_fullsized":"2024-03-30T00:55:55.817389+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:24:13.750905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.540794+0000","last_change":"2024-03-30T00:55:55.842894+0000","last_active":"2024-03-30T00:55:56.540794+0000","last_peered":"2024-03-30T00:55:56.540794+0000","last_clean":"2024-03-30T00:55:56.540794+0000","last_became_active":"2024-03-30T00:55:55.842118+0000","last_became_peered":"2024-03-30T00:55:55.842118+0000","last_unstale":"2024-03-30T00:55:56.540794+0000","last_undegraded":"2024-03-30T00:55:56.540794+0000","last_fullsized":"2024-03-30T00:55:56.540794+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:39.246228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'3","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:20.395176+0000","last_change":"2024-03-30T00:55:51.313188+0000","last_active":"2024-03-30T00:56:20.395176+0000","last_peered":"2024-03-30T00:56:20.395176+0000","last_clean":"2024-03-30T00:56:20.395176+0000","last_became_active":"2024-03-30T00:55:51.312879+0000","last_became_peered":"2024-03-30T00:55:51.312879+0000","last_unstale":"2024-03-30T00:56:20.395176+0000","last_undegraded":"2024-03-30T00:56:20.395176+0000","last_fullsized":"2024-03-30T00:56:20.395176+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:41:23.420427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815027+0000","last_change":"2024-03-30T00:55:51.808998+0000","last_active":"2024-03-30T00:55:55.815027+0000","last_peered":"2024-03-30T00:55:55.815027+0000","last_clean":"2024-03-30T00:55:55.815027+0000","last_became_active":"2024-03-30T00:55:51.808636+0000","last_became_peered":"2024-03-30T00:55:51.808636+0000","last_unstale":"2024-03-30T00:55:55.815027+0000","last_undegraded":"2024-03-30T00:55:55.815027+0000","last_fullsized":"2024-03-30T00:55:55.815027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:02.379294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.329776+0000","last_change":"2024-03-30T00:55:55.833209+0000","last_active":"2024-03-30T00:55:56.329776+0000","last_peered":"2024-03-30T00:55:56.329776+0000","last_clean":"2024-03-30T00:55:56.329776+0000","last_became_active":"2024-03-30T00:55:55.832799+0000","last_became_peered":"2024-03-30T00:55:55.832799+0000","last_unstale":"2024-03-30T00:55:56.329776+0000","last_undegraded":"2024-03-30T00:55:56.329776+0000","last_fullsized":"2024-03-30T00:55:56.329776+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:38:45.496406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'1","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.095937+0000","last_change":"2024-03-30T00:55:51.313708+0000","last_active":"2024-03-30T00:55:57.095937+0000","last_peered":"2024-03-30T00:55:57.095937+0000","last_clean":"2024-03-30T00:55:57.095937+0000","last_became_active":"2024-03-30T00:55:51.313362+0000","last_became_peered":"2024-03-30T00:55:51.313362+0000","last_unstale":"2024-03-30T00:55:57.095937+0000","last_undegraded":"2024-03-30T00:55:57.095937+0000","last_fullsized":"2024-03-30T00:55:57.095937+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:03:17.859640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141746+0000","last_change":"2024-03-30T00:55:51.819638+0000","last_active":"2024-03-30T00:55:56.141746+0000","last_peered":"2024-03-30T00:55:56.141746+0000","last_clean":"2024-03-30T00:55:56.141746+0000","last_became_active":"2024-03-30T00:55:51.819282+0000","last_became_peered":"2024-03-30T00:55:51.819282+0000","last_unstale":"2024-03-30T00:55:56.141746+0000","last_undegraded":"2024-03-30T00:55:56.141746+0000","last_fullsized":"2024-03-30T00:55:56.141746+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:13:39.249550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.329910+0000","last_change":"2024-03-30T00:55:55.856498+0000","last_active":"2024-03-30T00:55:56.329910+0000","last_peered":"2024-03-30T00:55:56.329910+0000","last_clean":"2024-03-30T00:55:56.329910+0000","last_became_active":"2024-03-30T00:55:55.856015+0000","last_became_peered":"2024-03-30T00:55:55.856015+0000","last_unstale":"2024-03-30T00:55:56.329910+0000","last_undegraded":"2024-03-30T00:55:56.329910+0000","last_fullsized":"2024-03-30T00:55:56.329910+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:14:00.248812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:20.393163+0000","last_change":"2024-03-30T00:55:51.302995+0000","last_active":"2024-03-30T00:56:20.393163+0000","last_peered":"2024-03-30T00:56:20.393163+0000","last_clean":"2024-03-30T00:56:20.393163+0000","last_became_active":"2024-03-30T00:55:51.302650+0000","last_became_peered":"2024-03-30T00:55:51.302650+0000","last_unstale":"2024-03-30T00:56:20.393163+0000","last_undegraded":"2024-03-30T00:56:20.393163+0000","last_fullsized":"2024-03-30T00:56:20.393163+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:17:32.556550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.823393+0000","last_change":"2024-03-30T00:55:51.808969+0000","last_active":"2024-03-30T00:55:55.823393+0000","last_peered":"2024-03-30T00:55:55.823393+0000","last_clean":"2024-03-30T00:55:55.823393+0000","last_became_active":"2024-03-30T00:55:51.808633+0000","last_became_peered":"2024-03-30T00:55:51.808633+0000","last_unstale":"2024-03-30T00:55:55.823393+0000","last_undegraded":"2024-03-30T00:55:55.823393+0000","last_fullsized":"2024-03-30T00:55:55.823393+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:16:46.128066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.095798+0000","last_change":"2024-03-30T00:55:55.824964+0000","last_active":"2024-03-30T00:55:57.095798+0000","last_peered":"2024-03-30T00:55:57.095798+0000","last_clean":"2024-03-30T00:55:57.095798+0000","last_became_active":"2024-03-30T00:55:55.824559+0000","last_became_peered":"2024-03-30T00:55:55.824559+0000","last_unstale":"2024-03-30T00:55:57.095798+0000","last_undegraded":"2024-03-30T00:55:57.095798+0000","last_fullsized":"2024-03-30T00:55:57.095798+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:19:24.746963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'3","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:21.412462+0000","last_change":"2024-03-30T00:55:51.301602+0000","last_active":"2024-03-30T00:56:21.412462+0000","last_peered":"2024-03-30T00:56:21.412462+0000","last_clean":"2024-03-30T00:56:21.412462+0000","last_became_active":"2024-03-30T00:55:51.301221+0000","last_became_peered":"2024-03-30T00:55:51.301221+0000","last_unstale":"2024-03-30T00:56:21.412462+0000","last_undegraded":"2024-03-30T00:56:21.412462+0000","last_fullsized":"2024-03-30T00:56:21.412462+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:37:36.727603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.825582+0000","last_change":"2024-03-30T00:55:51.810197+0000","last_active":"2024-03-30T00:55:55.825582+0000","last_peered":"2024-03-30T00:55:55.825582+0000","last_clean":"2024-03-30T00:55:55.825582+0000","last_became_active":"2024-03-30T00:55:51.809193+0000","last_became_peered":"2024-03-30T00:55:51.809193+0000","last_unstale":"2024-03-30T00:55:55.825582+0000","last_undegraded":"2024-03-30T00:55:55.825582+0000","last_fullsized":"2024-03-30T00:55:55.825582+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:52:41.078425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.833443+0000","last_change":"2024-03-30T00:55:55.830549+0000","last_active":"2024-03-30T00:55:55.833443+0000","last_peered":"2024-03-30T00:55:55.833443+0000","last_clean":"2024-03-30T00:55:55.833443+0000","last_became_active":"2024-03-30T00:55:55.827966+0000","last_became_peered":"2024-03-30T00:55:55.827966+0000","last_unstale":"2024-03-30T00:55:55.833443+0000","last_undegraded":"2024-03-30T00:55:55.833443+0000","last_fullsized":"2024-03-30T00:55:55.833443+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:30:36.531960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"76'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812537+0000","last_change":"2024-03-30T00:55:51.311998+0000","last_active":"2024-03-30T00:55:55.812537+0000","last_peered":"2024-03-30T00:55:55.812537+0000","last_clean":"2024-03-30T00:55:55.812537+0000","last_became_active":"2024-03-30T00:55:51.311627+0000","last_became_peered":"2024-03-30T00:55:51.311627+0000","last_unstale":"2024-03-30T00:55:55.812537+0000","last_undegraded":"2024-03-30T00:55:55.812537+0000","last_fullsized":"2024-03-30T00:55:55.812537+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:23:06.939919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.823632+0000","last_change":"2024-03-30T00:55:51.819457+0000","last_active":"2024-03-30T00:55:55.823632+0000","last_peered":"2024-03-30T00:55:55.823632+0000","last_clean":"2024-03-30T00:55:55.823632+0000","last_became_active":"2024-03-30T00:55:51.819099+0000","last_became_peered":"2024-03-30T00:55:51.819099+0000","last_unstale":"2024-03-30T00:55:55.823632+0000","last_undegraded":"2024-03-30T00:55:55.823632+0000","last_fullsized":"2024-03-30T00:55:55.823632+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:31:57.363328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.826513+0000","last_change":"2024-03-30T00:55:55.824882+0000","last_active":"2024-03-30T00:55:55.826513+0000","last_peered":"2024-03-30T00:55:55.826513+0000","last_clean":"2024-03-30T00:55:55.826513+0000","last_became_active":"2024-03-30T00:55:55.824493+0000","last_became_peered":"2024-03-30T00:55:55.824493+0000","last_unstale":"2024-03-30T00:55:55.826513+0000","last_undegraded":"2024-03-30T00:55:55.826513+0000","last_fullsized":"2024-03-30T00:55:55.826513+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:24:48.825776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"76'2","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.811456+0000","last_change":"2024-03-30T00:55:51.311787+0000","last_active":"2024-03-30T00:55:55.811456+0000","last_peered":"2024-03-30T00:55:55.811456+0000","last_clean":"2024-03-30T00:55:55.811456+0000","last_became_active":"2024-03-30T00:55:51.311402+0000","last_became_peered":"2024-03-30T00:55:51.311402+0000","last_unstale":"2024-03-30T00:55:55.811456+0000","last_undegraded":"2024-03-30T00:55:55.811456+0000","last_fullsized":"2024-03-30T00:55:55.811456+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:37:13.960453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.824838+0000","last_change":"2024-03-30T00:55:51.809742+0000","last_active":"2024-03-30T00:55:55.824838+0000","last_peered":"2024-03-30T00:55:55.824838+0000","last_clean":"2024-03-30T00:55:55.824838+0000","last_became_active":"2024-03-30T00:55:51.809396+0000","last_became_peered":"2024-03-30T00:55:51.809396+0000","last_unstale":"2024-03-30T00:55:55.824838+0000","last_undegraded":"2024-03-30T00:55:55.824838+0000","last_fullsized":"2024-03-30T00:55:55.824838+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:22.535472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816499+0000","last_change":"2024-03-30T00:55:51.814429+0000","last_active":"2024-03-30T00:55:55.816499+0000","last_peered":"2024-03-30T00:55:55.816499+0000","last_clean":"2024-03-30T00:55:55.816499+0000","last_became_active":"2024-03-30T00:55:51.814044+0000","last_became_peered":"2024-03-30T00:55:51.814044+0000","last_unstale":"2024-03-30T00:55:55.816499+0000","last_undegraded":"2024-03-30T00:55:55.816499+0000","last_fullsized":"2024-03-30T00:55:55.816499+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:24:58.830738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817590+0000","last_change":"2024-03-30T00:55:51.828977+0000","last_active":"2024-03-30T00:55:55.817590+0000","last_peered":"2024-03-30T00:55:55.817590+0000","last_clean":"2024-03-30T00:55:55.817590+0000","last_became_active":"2024-03-30T00:55:51.828656+0000","last_became_peered":"2024-03-30T00:55:51.828656+0000","last_unstale":"2024-03-30T00:55:55.817590+0000","last_undegraded":"2024-03-30T00:55:55.817590+0000","last_fullsized":"2024-03-30T00:55:55.817590+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:10:23.463770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816065+0000","last_change":"2024-03-30T00:55:51.821511+0000","last_active":"2024-03-30T00:55:55.816065+0000","last_peered":"2024-03-30T00:55:55.816065+0000","last_clean":"2024-03-30T00:55:55.816065+0000","last_became_active":"2024-03-30T00:55:51.820876+0000","last_became_peered":"2024-03-30T00:55:51.820876+0000","last_unstale":"2024-03-30T00:55:55.816065+0000","last_undegraded":"2024-03-30T00:55:55.816065+0000","last_fullsized":"2024-03-30T00:55:55.816065+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:14:48.299376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817918+0000","last_change":"2024-03-30T00:55:51.823867+0000","last_active":"2024-03-30T00:55:55.817918+0000","last_peered":"2024-03-30T00:55:55.817918+0000","last_clean":"2024-03-30T00:55:55.817918+0000","last_became_active":"2024-03-30T00:55:51.820230+0000","last_became_peered":"2024-03-30T00:55:51.820230+0000","last_unstale":"2024-03-30T00:55:55.817918+0000","last_undegraded":"2024-03-30T00:55:55.817918+0000","last_fullsized":"2024-03-30T00:55:55.817918+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:38:51.257855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131462+0000","last_change":"2024-03-30T00:55:51.821413+0000","last_active":"2024-03-30T00:55:56.131462+0000","last_peered":"2024-03-30T00:55:56.131462+0000","last_clean":"2024-03-30T00:55:56.131462+0000","last_became_active":"2024-03-30T00:55:51.820833+0000","last_became_peered":"2024-03-30T00:55:51.820833+0000","last_unstale":"2024-03-30T00:55:56.131462+0000","last_undegraded":"2024-03-30T00:55:56.131462+0000","last_fullsized":"2024-03-30T00:55:56.131462+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:34:11.969849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815114+0000","last_change":"2024-03-30T00:55:51.813674+0000","last_active":"2024-03-30T00:55:55.815114+0000","last_peered":"2024-03-30T00:55:55.815114+0000","last_clean":"2024-03-30T00:55:55.815114+0000","last_became_active":"2024-03-30T00:55:51.813341+0000","last_became_peered":"2024-03-30T00:55:51.813341+0000","last_unstale":"2024-03-30T00:55:55.815114+0000","last_undegraded":"2024-03-30T00:55:55.815114+0000","last_fullsized":"2024-03-30T00:55:55.815114+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:54:17.285997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816461+0000","last_change":"2024-03-30T00:55:51.812472+0000","last_active":"2024-03-30T00:55:55.816461+0000","last_peered":"2024-03-30T00:55:55.816461+0000","last_clean":"2024-03-30T00:55:55.816461+0000","last_became_active":"2024-03-30T00:55:51.812061+0000","last_became_peered":"2024-03-30T00:55:51.812061+0000","last_unstale":"2024-03-30T00:55:55.816461+0000","last_undegraded":"2024-03-30T00:55:55.816461+0000","last_fullsized":"2024-03-30T00:55:55.816461+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:46.550201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141783+0000","last_change":"2024-03-30T00:55:51.829179+0000","last_active":"2024-03-30T00:55:56.141783+0000","last_peered":"2024-03-30T00:55:56.141783+0000","last_clean":"2024-03-30T00:55:56.141783+0000","last_became_active":"2024-03-30T00:55:51.828805+0000","last_became_peered":"2024-03-30T00:55:51.828805+0000","last_unstale":"2024-03-30T00:55:56.141783+0000","last_undegraded":"2024-03-30T00:55:56.141783+0000","last_fullsized":"2024-03-30T00:55:56.141783+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:23.576125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.813043+0000","last_change":"2024-03-30T00:55:51.823170+0000","last_active":"2024-03-30T00:55:55.813043+0000","last_peered":"2024-03-30T00:55:55.813043+0000","last_clean":"2024-03-30T00:55:55.813043+0000","last_became_active":"2024-03-30T00:55:51.822785+0000","last_became_peered":"2024-03-30T00:55:51.822785+0000","last_unstale":"2024-03-30T00:55:55.813043+0000","last_undegraded":"2024-03-30T00:55:55.813043+0000","last_fullsized":"2024-03-30T00:55:55.813043+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:34:43.393540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141705+0000","last_change":"2024-03-30T00:55:51.817160+0000","last_active":"2024-03-30T00:55:56.141705+0000","last_peered":"2024-03-30T00:55:56.141705+0000","last_clean":"2024-03-30T00:55:56.141705+0000","last_became_active":"2024-03-30T00:55:51.816881+0000","last_became_peered":"2024-03-30T00:55:51.816881+0000","last_unstale":"2024-03-30T00:55:56.141705+0000","last_undegraded":"2024-03-30T00:55:56.141705+0000","last_fullsized":"2024-03-30T00:55:56.141705+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:09:05.211242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816796+0000","last_change":"2024-03-30T00:55:51.817670+0000","last_active":"2024-03-30T00:55:55.816796+0000","last_peered":"2024-03-30T00:55:55.816796+0000","last_clean":"2024-03-30T00:55:55.816796+0000","last_became_active":"2024-03-30T00:55:51.817380+0000","last_became_peered":"2024-03-30T00:55:51.817380+0000","last_unstale":"2024-03-30T00:55:55.816796+0000","last_undegraded":"2024-03-30T00:55:55.816796+0000","last_fullsized":"2024-03-30T00:55:55.816796+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:24.582557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131396+0000","last_change":"2024-03-30T00:55:51.813071+0000","last_active":"2024-03-30T00:55:56.131396+0000","last_peered":"2024-03-30T00:55:56.131396+0000","last_clean":"2024-03-30T00:55:56.131396+0000","last_became_active":"2024-03-30T00:55:51.811681+0000","last_became_peered":"2024-03-30T00:55:51.811681+0000","last_unstale":"2024-03-30T00:55:56.131396+0000","last_undegraded":"2024-03-30T00:55:56.131396+0000","last_fullsized":"2024-03-30T00:55:56.131396+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:10:53.840055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816334+0000","last_change":"2024-03-30T00:55:51.816716+0000","last_active":"2024-03-30T00:55:55.816334+0000","last_peered":"2024-03-30T00:55:55.816334+0000","last_clean":"2024-03-30T00:55:55.816334+0000","last_became_active":"2024-03-30T00:55:51.816383+0000","last_became_peered":"2024-03-30T00:55:51.816383+0000","last_unstale":"2024-03-30T00:55:55.816334+0000","last_undegraded":"2024-03-30T00:55:55.816334+0000","last_fullsized":"2024-03-30T00:55:55.816334+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:12:20.044088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":212960,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":42976,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":24,"num_read_kb":56,"num_write":120,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":113532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645328,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":439510,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.40300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.624},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.49399999999999999}]},{"osd":5,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.50600000000000001}]},{"osd":6,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.66300000000000003}]},{"osd":7,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.68000000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":9,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.76700000000000002}]},{"osd":10,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.80800000000000005}]}]},{"osd":10,"up_from":65,"seq":279172874259,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":1071315,"data_compressed":47572,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.60499999999999998}]},{"osd":4,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.57399999999999995}]},{"osd":5,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.62}]},{"osd":6,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":8,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.72699999999999998}]},{"osd":9,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.77400000000000002}]},{"osd":11,"last update":"Sat Mar 30 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.76200000000000001}]}]},{"osd":9,"up_from":60,"seq":257698037781,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":439712,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.51900000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.63400000000000001}]},{"osd":2,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.74399999999999999}]},{"osd":3,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.72099999999999997}]},{"osd":4,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.79500000000000004}]},{"osd":5,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":0.93500000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.623}]},{"osd":7,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.94599999999999995}]},{"osd":8,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.0109999999999999}]},{"osd":10,"last update":"Sat Mar 30 00:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.98899999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]}]},{"osd":8,"up_from":53,"seq":227633266714,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":880640,"data_stored":439201,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.46500000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.746},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.68300000000000005}]},{"osd":2,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.496},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.52700000000000002}]},{"osd":3,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.883,"5min":2.883,"15min":2.883},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.65300000000000002}]},{"osd":4,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.77700000000000002}]},{"osd":5,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.623}]},{"osd":6,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.8940000000000001,"5min":2.8940000000000001,"15min":2.8940000000000001},"last":0.64200000000000002}]},{"osd":7,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.70799999999999996}]},{"osd":9,"last update":"Sat Mar 30 00:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.79700000000000004}]},{"osd":10,"last update":"Sat Mar 30 00:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.85999999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]}]},{"osd":7,"up_from":46,"seq":197568495646,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":1035543,"data_compressed":47572,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60199999999999998}]},{"osd":1,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.45200000000000001,"15min":0.44800000000000001},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.60599999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.44700000000000001,"15min":0.441},"min":{"1min":0.32500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68799999999999994,"5min":0.747,"15min":0.747},"last":0.68700000000000006}]},{"osd":2,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.46700000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.57399999999999995,"15min":0.56100000000000005},"min":{"1min":0.51800000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.58399999999999996}]},{"osd":3,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.40400000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.89600000000000002,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61599999999999999,"15min":0.60999999999999999},"min":{"1min":0.53200000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89400000000000002,"5min":0.997,"15min":0.997},"last":0.65200000000000002}]},{"osd":4,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.36599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.39200000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.629,"15min":0.61899999999999999},"min":{"1min":0.505,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.63200000000000001,"15min":0.62},"min":{"1min":0.58099999999999996,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86199999999999999,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.80600000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.58799999999999997,"5min":0.372,"15min":0.372},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.745,"15min":0.72099999999999997},"min":{"1min":0.55300000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":4.117,"5min":4.117,"15min":4.117},"last":0.66900000000000004}]},{"osd":8,"last update":"Sat Mar 30 00:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.56299999999999994,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95199999999999996,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.755,"15min":0.752},"min":{"1min":0.53400000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.84199999999999997}]},{"osd":9,"last update":"Sat Mar 30 00:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":4.0519999999999996,"5min":4.0519999999999996,"15min":4.0519999999999996},"last":0.78500000000000003}]},{"osd":10,"last update":"Sat Mar 30 00:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.755},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.85599999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]}]},{"osd":6,"up_from":42,"seq":180388626466,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31968,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713184,"statfs":{"total":95995035648,"available":95962300416,"internally_reserved":0,"allocated":876544,"data_stored":438928,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.38900000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.505},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.49199999999999999,"15min":0.48399999999999999},"min":{"1min":0.42299999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69999999999999996,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.64500000000000002,"15min":0.629},"min":{"1min":0.58599999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.59599999999999997,"15min":0.59299999999999997},"min":{"1min":0.47599999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83999999999999997,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.61799999999999999}]},{"osd":2,"last update":"Sat Mar 30 00:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.69699999999999995,"15min":0.67900000000000005},"min":{"1min":0.56599999999999995,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.0229999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.65000000000000002,"15min":0.63800000000000001},"min":{"1min":0.57499999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85499999999999998,"5min":1.006,"15min":1.006},"last":0.69199999999999995}]},{"osd":3,"last update":"Sat Mar 30 00:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.49199999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.996,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.40799999999999997,"5min":0.375,"15min":0.375},"max":{"1min":0.84399999999999997,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.54900000000000004}]},{"osd":4,"last update":"Sat Mar 30 00:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.54900000000000004,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.82099999999999995,"5min":1.006,"15min":1.006},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.75700000000000001,"15min":0.749},"min":{"1min":0.60299999999999998,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.006,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.77800000000000002}]},{"osd":5,"last update":"Sat Mar 30 00:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.69299999999999995,"15min":0.68200000000000005},"min":{"1min":0.54900000000000004,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.93400000000000005,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.73999999999999999,"15min":0.72999999999999998},"min":{"1min":0.53400000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.93799999999999994,"5min":1.091,"15min":1.091},"last":0.79400000000000004}]},{"osd":7,"last update":"Sat Mar 30 00:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81799999999999995,"15min":0.81399999999999995},"min":{"1min":0.64300000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.1200000000000001,"5min":1.127,"15min":1.127},"last":0.877},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.77600000000000002,"15min":0.77200000000000002},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.66600000000000004}]},{"osd":8,"last update":"Sat Mar 30 00:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.96199999999999997}]},{"osd":9,"last update":"Sat Mar 30 00:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.72999999999999998}]},{"osd":10,"last update":"Sat Mar 30 00:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.94299999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":1,"up_from":12,"seq":51539607604,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":449053,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55500000000000005,"15min":0.53500000000000003},"min":{"1min":0.42399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72099999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.499},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.58099999999999996,"15min":0.55300000000000005},"min":{"1min":0.47199999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.79100000000000004,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.59899999999999998}]},{"osd":2,"last update":"Sat Mar 30 00:55:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.505,"15min":0.48499999999999999},"min":{"1min":0.39400000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.50700000000000001,"15min":0.49099999999999999},"min":{"1min":0.39300000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.56200000000000006}]},{"osd":3,"last update":"Sat Mar 30 00:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.55000000000000004,"15min":0.52400000000000002},"min":{"1min":0.40799999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82299999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.61899999999999999}]},{"osd":4,"last update":"Sat Mar 30 00:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.41799999999999998,"15min":0.40799999999999997},"min":{"1min":0.33500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.61899999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.48199999999999998,"15min":0.46600000000000003},"min":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.57599999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:55:30 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.60899999999999999,"15min":0.61299999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63500000000000001,"15min":0.63400000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.63300000000000001}]},{"osd":6,"last update":"Sat Mar 30 00:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.877,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67900000000000005,"15min":0.68200000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.72799999999999998}]},{"osd":7,"last update":"Sat Mar 30 00:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55400000000000005,"15min":0.55100000000000005},"min":{"1min":0.35799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79400000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.84899999999999998,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.68700000000000006}]},{"osd":8,"last update":"Sat Mar 30 00:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.81599999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.89000000000000001}]},{"osd":10,"last update":"Sat Mar 30 00:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.80400000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31980,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713172,"statfs":{"total":95995035648,"available":95962288128,"internally_reserved":0,"allocated":876544,"data_stored":438745,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 00:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.504,"5min":0.372,"15min":0.372},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.42099999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.752}]},{"osd":2,"last update":"Sat Mar 30 00:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.53600000000000003,"15min":0.52300000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.627},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.57399999999999995,"15min":0.55600000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.72499999999999998}]},{"osd":3,"last update":"Sat Mar 30 00:54:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45200000000000001,"15min":0.436},"min":{"1min":0.35899999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.379,"5min":0.317,"15min":0.317},"max":{"1min":0.69499999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.56699999999999995}]},{"osd":4,"last update":"Sat Mar 30 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65100000000000002,"15min":0.623},"min":{"1min":0.503,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90000000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.64100000000000001,"15min":0.60999999999999999},"min":{"1min":0.51400000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.93300000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.93100000000000005}]},{"osd":5,"last update":"Sat Mar 30 00:55:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.65000000000000002,"15min":0.63500000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.63200000000000001,"15min":0.61799999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.84499999999999997}]},{"osd":6,"last update":"Sat Mar 30 00:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.41999999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64200000000000002,"15min":0.65000000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.998}]},{"osd":7,"last update":"Sat Mar 30 00:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.96799999999999997}]},{"osd":8,"last update":"Sat Mar 30 00:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":1.026}]},{"osd":9,"last update":"Sat Mar 30 00:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":1.0760000000000001}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":1.016},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":1.0369999999999999}]},{"osd":11,"last update":"Sat Mar 30 00:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":1.107},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":1.0620000000000001}]}]},{"osd":2,"up_from":17,"seq":73014444082,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":480381,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.39000000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78100000000000003,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54000000000000004,"15min":0.52500000000000002},"min":{"1min":0.38,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.82199999999999995,"5min":1.097,"15min":1.097},"last":0.38300000000000001}]},{"osd":1,"last update":"Sat Mar 30 00:55:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53500000000000003,"15min":0.52600000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.50600000000000001}]},{"osd":3,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57899999999999996,"15min":0.55900000000000005},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.58899999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.42199999999999999}]},{"osd":4,"last update":"Sat Mar 30 00:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.39700000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.52000000000000002,"15min":0.51500000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Mar 30 00:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.439,"15min":0.42999999999999999},"min":{"1min":0.317,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.46100000000000002,"15min":0.45400000000000001},"min":{"1min":0.28599999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.32500000000000001}]},{"osd":6,"last update":"Sat Mar 30 00:55:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64700000000000002,"15min":0.64900000000000002},"min":{"1min":0.44500000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.40899999999999997}]},{"osd":7,"last update":"Sat Mar 30 00:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.66500000000000004}]},{"osd":8,"last update":"Sat Mar 30 00:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.52200000000000002}]},{"osd":9,"last update":"Sat Mar 30 00:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.59599999999999997}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.67400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]}]},{"osd":3,"up_from":24,"seq":103079215149,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":438849,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.437,"15min":0.40600000000000003},"min":{"1min":0.35399999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.45000000000000001,"15min":0.42799999999999999},"min":{"1min":0.35199999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.76800000000000002,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.497}]},{"osd":1,"last update":"Sat Mar 30 00:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.63,"15min":0.59599999999999997},"min":{"1min":0.51400000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Mar 30 00:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.64900000000000002,"15min":0.621},"min":{"1min":0.45800000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.752},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63400000000000001,"15min":0.61699999999999999},"min":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.64600000000000002}]},{"osd":4,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.69499999999999995,"15min":0.66300000000000003},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0600000000000001,"5min":1.113,"15min":1.113},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67500000000000004,"15min":0.66400000000000003},"min":{"1min":0.501,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.83399999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.46999999999999997}]},{"osd":5,"last update":"Sat Mar 30 00:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.65600000000000003,"15min":0.64600000000000002},"min":{"1min":0.48699999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.68500000000000005,"15min":0.65900000000000003},"min":{"1min":0.56499999999999995,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64000000000000001,"15min":0.63500000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64800000000000002,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.57099999999999995}]},{"osd":7,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.97899999999999998,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.621},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83999999999999997,"15min":0.84099999999999997},"min":{"1min":0.66000000000000003,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.1890000000000001,"5min":1.292,"15min":1.292},"last":0.72399999999999998}]},{"osd":8,"last update":"Sat Mar 30 00:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.57399999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.74399999999999999}]},{"osd":10,"last update":"Sat Mar 30 00:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.81399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986218,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":880640,"data_stored":439313,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.66700000000000004,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.79300000000000004,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73099999999999998,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39700000000000002,"15min":0.39100000000000001},"min":{"1min":0.29999999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.61199999999999999,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.32000000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.46899999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.439}]},{"osd":2,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.55400000000000005,"15min":0.54500000000000004},"min":{"1min":0.36399999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81799999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.55400000000000005,"15min":0.53900000000000003},"min":{"1min":0.441,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76000000000000001,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.59799999999999998}]},{"osd":3,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.66000000000000003,"15min":0.69799999999999995},"min":{"1min":0.42399999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.72999999999999998,"5min":3.887,"15min":3.887},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.39600000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78400000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.80300000000000005}]},{"osd":5,"last update":"Sat Mar 30 00:55:27 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.57099999999999995,"15min":0.56299999999999994},"min":{"1min":0.38200000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.65200000000000002}]},{"osd":6,"last update":"Sat Mar 30 00:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.627,"15min":0.626},"min":{"1min":0.47399999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65400000000000003,"15min":0.65300000000000002},"min":{"1min":0.51100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.82599999999999996}]},{"osd":7,"last update":"Sat Mar 30 00:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54400000000000004,"15min":0.54000000000000004},"min":{"1min":0.35199999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.79500000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52700000000000002,"15min":0.52100000000000002},"min":{"1min":0.34599999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57799999999999996}]},{"osd":8,"last update":"Sat Mar 30 00:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.624},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.73999999999999999}]},{"osd":9,"last update":"Sat Mar 30 00:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.77100000000000002}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.79200000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]}]},{"osd":5,"up_from":35,"seq":150323855397,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":933888,"data_stored":1029147,"data_compressed":47572,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.50600000000000001,"15min":0.499},"min":{"1min":0.40799999999999997,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.751,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.504},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.5,"15min":0.49399999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:55:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.747},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.40200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.72899999999999998}]},{"osd":2,"last update":"Sat Mar 30 00:55:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.28299999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38900000000000001,"15min":0.39300000000000002},"min":{"1min":0.30099999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.60999999999999999,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.59499999999999997}]},{"osd":3,"last update":"Sat Mar 30 00:55:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.40799999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53100000000000003,"15min":0.52700000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.65100000000000002}]},{"osd":4,"last update":"Sat Mar 30 00:55:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.68300000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.40200000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.85399999999999998,"5min":1.038,"15min":1.038},"last":0.67600000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.46500000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89700000000000002,"5min":0.997,"15min":0.997},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.49199999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.80500000000000005,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.70799999999999996}]},{"osd":7,"last update":"Sat Mar 30 00:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.42699999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82199999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.63600000000000001,"15min":0.63700000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89500000000000002,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.82299999999999995}]},{"osd":8,"last update":"Sat Mar 30 00:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.81200000000000006}]},{"osd":9,"last update":"Sat Mar 30 00:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.86099999999999999}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.874},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.89800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":42122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":42350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T00:56:26.097 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T00:56:26.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.292+0000 7f205ec00700 1 -- 172.21.15.116:0/3060276131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2058074df0 msgr2=0x7f2058073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:26.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.292+0000 7f205ec00700 1 --2- 172.21.15.116:0/3060276131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2058074df0 0x7f2058073240 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f20480098f0 tx=0x7f2048009c00 comp rx=0 tx=0).stop 2024-03-30T00:56:26.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.292+0000 7f205ec00700 1 -- 172.21.15.116:0/3060276131 shutdown_connections 2024-03-30T00:56:26.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.292+0000 7f205ec00700 1 --2- 172.21.15.116:0/3060276131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2058074df0 0x7f2058073240 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:26.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.292+0000 7f205ec00700 1 --2- 172.21.15.116:0/3060276131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2058108b40 0x7f205810afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:26.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.292+0000 7f205ec00700 1 --2- 172.21.15.116:0/3060276131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2058073810 0x7f2058073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:26.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.292+0000 7f205ec00700 1 -- 172.21.15.116:0/3060276131 >> 172.21.15.116:0/3060276131 conn(0x7f20580fc1f0 msgr2=0x7f20580fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:26.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.292+0000 7f205ec00700 1 -- 172.21.15.116:0/3060276131 shutdown_connections 2024-03-30T00:56:26.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.292+0000 7f205ec00700 1 -- 172.21.15.116:0/3060276131 wait complete. 2024-03-30T00:56:26.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.293+0000 7f205ec00700 1 Processor -- start 2024-03-30T00:56:26.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.293+0000 7f205ec00700 1 -- start start 2024-03-30T00:56:26.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.293+0000 7f205ec00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2058073810 0x7f2058198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:26.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.293+0000 7f205ec00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2058074df0 0x7f2058199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:26.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.293+0000 7f205ec00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2058108b40 0x7f20581a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:26.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.293+0000 7f205ec00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20581a09f0 con 0x7f2058108b40 2024-03-30T00:56:26.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.293+0000 7f205ec00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20581a0b30 con 0x7f2058073810 2024-03-30T00:56:26.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.293+0000 7f205ec00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20581a0cc0 con 0x7f2058074df0 2024-03-30T00:56:26.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.294+0000 7f205cc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2058108b40 0x7f20581a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:26.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.294+0000 7f205cc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2058108b40 0x7f20581a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37936/0 (socket says 172.21.15.116:37936) 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.294+0000 7f205cc00700 1 -- 172.21.15.116:0/4037998460 learned_addr learned my addr 172.21.15.116:0/4037998460 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.294+0000 7f2057400700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2058074df0 0x7f2058199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.294+0000 7f2057e00700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2058073810 0x7f2058198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.294+0000 7f2057400700 1 -- 172.21.15.116:0/4037998460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2058073810 msgr2=0x7f2058198ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.294+0000 7f2057400700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2058073810 0x7f2058198ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.294+0000 7f2057400700 1 -- 172.21.15.116:0/4037998460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2058108b40 msgr2=0x7f20581a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.294+0000 7f2057400700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2058108b40 0x7f20581a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.294+0000 7f2057400700 1 -- 172.21.15.116:0/4037998460 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2050009d00 con 0x7f2058074df0 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.294+0000 7f205cc00700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2058108b40 0x7f20581a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.294+0000 7f2057400700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2058074df0 0x7f2058199400 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f204400d720 tx=0x7f204400da30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.295+0000 7f2054c00700 1 -- 172.21.15.116:0/4037998460 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204400f690 con 0x7f2058074df0 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.295+0000 7f2054c00700 1 -- 172.21.15.116:0/4037998460 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f204400fcb0 con 0x7f2058074df0 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.295+0000 7f2054c00700 1 -- 172.21.15.116:0/4037998460 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2044010e60 con 0x7f2058074df0 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.295+0000 7f205ec00700 1 -- 172.21.15.116:0/4037998460 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2048009520 con 0x7f2058074df0 2024-03-30T00:56:26.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.295+0000 7f205ec00700 1 -- 172.21.15.116:0/4037998460 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20581a1240 con 0x7f2058074df0 2024-03-30T00:56:26.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.296+0000 7f205ec00700 1 -- 172.21.15.116:0/4037998460 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2058107900 con 0x7f2058074df0 2024-03-30T00:56:26.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.298+0000 7f2054c00700 1 -- 172.21.15.116:0/4037998460 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f204400f7f0 con 0x7f2058074df0 2024-03-30T00:56:26.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.299+0000 7f2054c00700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f203c0a11b0 0x7f203c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:26.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.299+0000 7f2054c00700 1 -- 172.21.15.116:0/4037998460 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f20440ccca0 con 0x7f2058074df0 2024-03-30T00:56:26.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.299+0000 7f2057e00700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f203c0a11b0 0x7f203c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:26.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.300+0000 7f2057e00700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f203c0a11b0 0x7f203c0a3670 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f204800fa40 tx=0x7f2048019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:26.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.301+0000 7f2054c00700 1 -- 172.21.15.116:0/4037998460 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2044096080 con 0x7f2058074df0 2024-03-30T00:56:26.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.505+0000 7f205ec00700 1 -- 172.21.15.116:0/4037998460 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f205802cdc0 con 0x7f203c0a11b0 2024-03-30T00:56:26.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.518+0000 7f2054c00700 1 -- 172.21.15.116:0/4037998460 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422678 (secure 0 0 0) 0x7f205802cdc0 con 0x7f203c0a11b0 2024-03-30T00:56:26.530 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:56:26.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.523+0000 7f205ec00700 1 -- 172.21.15.116:0/4037998460 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f203c0a11b0 msgr2=0x7f203c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:26.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.523+0000 7f205ec00700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f203c0a11b0 0x7f203c0a3670 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f204800fa40 tx=0x7f2048019040 comp rx=0 tx=0).stop 2024-03-30T00:56:26.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.523+0000 7f205ec00700 1 -- 172.21.15.116:0/4037998460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2058074df0 msgr2=0x7f2058199400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:26.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.523+0000 7f205ec00700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2058074df0 0x7f2058199400 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f204400d720 tx=0x7f204400da30 comp rx=0 tx=0).stop 2024-03-30T00:56:26.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.523+0000 7f205ec00700 1 -- 172.21.15.116:0/4037998460 shutdown_connections 2024-03-30T00:56:26.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.523+0000 7f205ec00700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f203c0a11b0 0x7f203c0a3670 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:26.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.523+0000 7f205ec00700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2058108b40 0x7f20581a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:26.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.523+0000 7f205ec00700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2058073810 0x7f2058198ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:26.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.523+0000 7f205ec00700 1 --2- 172.21.15.116:0/4037998460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2058074df0 0x7f2058199400 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:26.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.523+0000 7f205ec00700 1 -- 172.21.15.116:0/4037998460 >> 172.21.15.116:0/4037998460 conn(0x7f20580fc1f0 msgr2=0x7f2058104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:26.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.523+0000 7f205ec00700 1 -- 172.21.15.116:0/4037998460 shutdown_connections 2024-03-30T00:56:26.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:26.523+0000 7f205ec00700 1 -- 172.21.15.116:0/4037998460 wait complete. 2024-03-30T00:56:26.536 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-30T00:56:26.550 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-03-30T00:56:25.107783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":633461,"num_objects":51,"num_object_clones":0,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":150,"num_read_kb":165,"num_write":302,"num_write_kb":3011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":351248,"kb_used_data":10636,"kb_used_omap":18,"kb_used_meta":340461,"kb_avail":1124590576,"statfs":{"total":1151940427776,"available":1151580749824,"internally_reserved":0,"allocated":10891264,"data_stored":7139697,"data_compressed":248961,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":19128,"internal_metadata":348632392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38640,"num_objects":27,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":24,"num_read_kb":56,"num_write":74,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004216"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815586+0000","last_change":"2024-03-30T00:55:51.809659+0000","last_active":"2024-03-30T00:55:55.815586+0000","last_peered":"2024-03-30T00:55:55.815586+0000","last_clean":"2024-03-30T00:55:55.815586+0000","last_became_active":"2024-03-30T00:55:51.809337+0000","last_became_peered":"2024-03-30T00:55:51.809337+0000","last_unstale":"2024-03-30T00:55:55.815586+0000","last_undegraded":"2024-03-30T00:55:55.815586+0000","last_fullsized":"2024-03-30T00:55:55.815586+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:20:50.640033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.818896+0000","last_change":"2024-03-30T00:55:51.813778+0000","last_active":"2024-03-30T00:55:55.818896+0000","last_peered":"2024-03-30T00:55:55.818896+0000","last_clean":"2024-03-30T00:55:55.818896+0000","last_became_active":"2024-03-30T00:55:51.810347+0000","last_became_peered":"2024-03-30T00:55:51.810347+0000","last_unstale":"2024-03-30T00:55:55.818896+0000","last_undegraded":"2024-03-30T00:55:55.818896+0000","last_fullsized":"2024-03-30T00:55:55.818896+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:49:07.031477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.818797+0000","last_change":"2024-03-30T00:55:51.817647+0000","last_active":"2024-03-30T00:55:55.818797+0000","last_peered":"2024-03-30T00:55:55.818797+0000","last_clean":"2024-03-30T00:55:55.818797+0000","last_became_active":"2024-03-30T00:55:51.816495+0000","last_became_peered":"2024-03-30T00:55:51.816495+0000","last_unstale":"2024-03-30T00:55:55.818797+0000","last_undegraded":"2024-03-30T00:55:55.818797+0000","last_fullsized":"2024-03-30T00:55:55.818797+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:59:42.965346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817411+0000","last_change":"2024-03-30T00:55:51.817581+0000","last_active":"2024-03-30T00:55:55.817411+0000","last_peered":"2024-03-30T00:55:55.817411+0000","last_clean":"2024-03-30T00:55:55.817411+0000","last_became_active":"2024-03-30T00:55:51.816223+0000","last_became_peered":"2024-03-30T00:55:51.816223+0000","last_unstale":"2024-03-30T00:55:55.817411+0000","last_undegraded":"2024-03-30T00:55:55.817411+0000","last_fullsized":"2024-03-30T00:55:55.817411+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:48:32.203975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815122+0000","last_change":"2024-03-30T00:55:51.811707+0000","last_active":"2024-03-30T00:55:55.815122+0000","last_peered":"2024-03-30T00:55:55.815122+0000","last_clean":"2024-03-30T00:55:55.815122+0000","last_became_active":"2024-03-30T00:55:51.811366+0000","last_became_peered":"2024-03-30T00:55:51.811366+0000","last_unstale":"2024-03-30T00:55:55.815122+0000","last_undegraded":"2024-03-30T00:55:55.815122+0000","last_fullsized":"2024-03-30T00:55:55.815122+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:20.438505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.855012+0000","last_change":"2024-03-30T00:55:51.826849+0000","last_active":"2024-03-30T00:55:55.855012+0000","last_peered":"2024-03-30T00:55:55.855012+0000","last_clean":"2024-03-30T00:55:55.855012+0000","last_became_active":"2024-03-30T00:55:51.826520+0000","last_became_peered":"2024-03-30T00:55:51.826520+0000","last_unstale":"2024-03-30T00:55:55.855012+0000","last_undegraded":"2024-03-30T00:55:55.855012+0000","last_fullsized":"2024-03-30T00:55:55.855012+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:54:03.423725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.823087+0000","last_change":"2024-03-30T00:55:51.812880+0000","last_active":"2024-03-30T00:55:55.823087+0000","last_peered":"2024-03-30T00:55:55.823087+0000","last_clean":"2024-03-30T00:55:55.823087+0000","last_became_active":"2024-03-30T00:55:51.812572+0000","last_became_peered":"2024-03-30T00:55:51.812572+0000","last_unstale":"2024-03-30T00:55:55.823087+0000","last_undegraded":"2024-03-30T00:55:55.823087+0000","last_fullsized":"2024-03-30T00:55:55.823087+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:44:49.670641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141676+0000","last_change":"2024-03-30T00:55:51.815480+0000","last_active":"2024-03-30T00:55:56.141676+0000","last_peered":"2024-03-30T00:55:56.141676+0000","last_clean":"2024-03-30T00:55:56.141676+0000","last_became_active":"2024-03-30T00:55:51.814052+0000","last_became_peered":"2024-03-30T00:55:51.814052+0000","last_unstale":"2024-03-30T00:55:56.141676+0000","last_undegraded":"2024-03-30T00:55:56.141676+0000","last_fullsized":"2024-03-30T00:55:56.141676+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:27:49.930928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816450+0000","last_change":"2024-03-30T00:55:51.814127+0000","last_active":"2024-03-30T00:55:55.816450+0000","last_peered":"2024-03-30T00:55:55.816450+0000","last_clean":"2024-03-30T00:55:55.816450+0000","last_became_active":"2024-03-30T00:55:51.813820+0000","last_became_peered":"2024-03-30T00:55:51.813820+0000","last_unstale":"2024-03-30T00:55:55.816450+0000","last_undegraded":"2024-03-30T00:55:55.816450+0000","last_fullsized":"2024-03-30T00:55:55.816450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:21:31.605935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.855477+0000","last_change":"2024-03-30T00:55:51.815198+0000","last_active":"2024-03-30T00:55:55.855477+0000","last_peered":"2024-03-30T00:55:55.855477+0000","last_clean":"2024-03-30T00:55:55.855477+0000","last_became_active":"2024-03-30T00:55:51.814901+0000","last_became_peered":"2024-03-30T00:55:51.814901+0000","last_unstale":"2024-03-30T00:55:55.855477+0000","last_undegraded":"2024-03-30T00:55:55.855477+0000","last_fullsized":"2024-03-30T00:55:55.855477+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:45:51.224346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.825549+0000","last_change":"2024-03-30T00:55:51.809884+0000","last_active":"2024-03-30T00:55:55.825549+0000","last_peered":"2024-03-30T00:55:55.825549+0000","last_clean":"2024-03-30T00:55:55.825549+0000","last_became_active":"2024-03-30T00:55:51.809576+0000","last_became_peered":"2024-03-30T00:55:51.809576+0000","last_unstale":"2024-03-30T00:55:55.825549+0000","last_undegraded":"2024-03-30T00:55:55.825549+0000","last_fullsized":"2024-03-30T00:55:55.825549+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:39:51.767744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817315+0000","last_change":"2024-03-30T00:55:51.817384+0000","last_active":"2024-03-30T00:55:55.817315+0000","last_peered":"2024-03-30T00:55:55.817315+0000","last_clean":"2024-03-30T00:55:55.817315+0000","last_became_active":"2024-03-30T00:55:51.816564+0000","last_became_peered":"2024-03-30T00:55:51.816564+0000","last_unstale":"2024-03-30T00:55:55.817315+0000","last_undegraded":"2024-03-30T00:55:55.817315+0000","last_fullsized":"2024-03-30T00:55:55.817315+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:47:07.195650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131413+0000","last_change":"2024-03-30T00:55:51.814310+0000","last_active":"2024-03-30T00:55:56.131413+0000","last_peered":"2024-03-30T00:55:56.131413+0000","last_clean":"2024-03-30T00:55:56.131413+0000","last_became_active":"2024-03-30T00:55:51.813968+0000","last_became_peered":"2024-03-30T00:55:51.813968+0000","last_unstale":"2024-03-30T00:55:56.131413+0000","last_undegraded":"2024-03-30T00:55:56.131413+0000","last_fullsized":"2024-03-30T00:55:56.131413+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:59:55.955502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817322+0000","last_change":"2024-03-30T00:55:51.815655+0000","last_active":"2024-03-30T00:55:55.817322+0000","last_peered":"2024-03-30T00:55:55.817322+0000","last_clean":"2024-03-30T00:55:55.817322+0000","last_became_active":"2024-03-30T00:55:51.814099+0000","last_became_peered":"2024-03-30T00:55:51.814099+0000","last_unstale":"2024-03-30T00:55:55.817322+0000","last_undegraded":"2024-03-30T00:55:55.817322+0000","last_fullsized":"2024-03-30T00:55:55.817322+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:42.958153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817198+0000","last_change":"2024-03-30T00:55:51.819772+0000","last_active":"2024-03-30T00:55:55.817198+0000","last_peered":"2024-03-30T00:55:55.817198+0000","last_clean":"2024-03-30T00:55:55.817198+0000","last_became_active":"2024-03-30T00:55:51.816764+0000","last_became_peered":"2024-03-30T00:55:51.816764+0000","last_unstale":"2024-03-30T00:55:55.817198+0000","last_undegraded":"2024-03-30T00:55:55.817198+0000","last_fullsized":"2024-03-30T00:55:55.817198+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:26:26.763396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816489+0000","last_change":"2024-03-30T00:55:51.822278+0000","last_active":"2024-03-30T00:55:55.816489+0000","last_peered":"2024-03-30T00:55:55.816489+0000","last_clean":"2024-03-30T00:55:55.816489+0000","last_became_active":"2024-03-30T00:55:51.821996+0000","last_became_peered":"2024-03-30T00:55:51.821996+0000","last_unstale":"2024-03-30T00:55:55.816489+0000","last_undegraded":"2024-03-30T00:55:55.816489+0000","last_fullsized":"2024-03-30T00:55:55.816489+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:37:45.841853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815484+0000","last_change":"2024-03-30T00:55:51.809547+0000","last_active":"2024-03-30T00:55:55.815484+0000","last_peered":"2024-03-30T00:55:55.815484+0000","last_clean":"2024-03-30T00:55:55.815484+0000","last_became_active":"2024-03-30T00:55:51.809314+0000","last_became_peered":"2024-03-30T00:55:51.809314+0000","last_unstale":"2024-03-30T00:55:55.815484+0000","last_undegraded":"2024-03-30T00:55:55.815484+0000","last_fullsized":"2024-03-30T00:55:55.815484+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:39:31.661362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.835459+0000","last_change":"2024-03-30T00:55:51.810857+0000","last_active":"2024-03-30T00:55:55.835459+0000","last_peered":"2024-03-30T00:55:55.835459+0000","last_clean":"2024-03-30T00:55:55.835459+0000","last_became_active":"2024-03-30T00:55:51.809420+0000","last_became_peered":"2024-03-30T00:55:51.809420+0000","last_unstale":"2024-03-30T00:55:55.835459+0000","last_undegraded":"2024-03-30T00:55:55.835459+0000","last_fullsized":"2024-03-30T00:55:55.835459+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:36:48.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.820658+0000","last_change":"2024-03-30T00:55:51.814693+0000","last_active":"2024-03-30T00:55:55.820658+0000","last_peered":"2024-03-30T00:55:55.820658+0000","last_clean":"2024-03-30T00:55:55.820658+0000","last_became_active":"2024-03-30T00:55:51.814359+0000","last_became_peered":"2024-03-30T00:55:51.814359+0000","last_unstale":"2024-03-30T00:55:55.820658+0000","last_undegraded":"2024-03-30T00:55:55.820658+0000","last_fullsized":"2024-03-30T00:55:55.820658+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:20:50.856313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.097403+0000","last_change":"2024-03-30T00:55:55.839089+0000","last_active":"2024-03-30T00:55:57.097403+0000","last_peered":"2024-03-30T00:55:57.097403+0000","last_clean":"2024-03-30T00:55:57.097403+0000","last_became_active":"2024-03-30T00:55:55.838705+0000","last_became_peered":"2024-03-30T00:55:55.838705+0000","last_unstale":"2024-03-30T00:55:57.097403+0000","last_undegraded":"2024-03-30T00:55:57.097403+0000","last_fullsized":"2024-03-30T00:55:57.097403+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:33:56.010272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'9","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:21.412206+0000","last_change":"2024-03-30T00:55:51.298577+0000","last_active":"2024-03-30T00:56:21.412206+0000","last_peered":"2024-03-30T00:56:21.412206+0000","last_clean":"2024-03-30T00:56:21.412206+0000","last_became_active":"2024-03-30T00:55:51.298269+0000","last_became_peered":"2024-03-30T00:55:51.298269+0000","last_unstale":"2024-03-30T00:56:21.412206+0000","last_undegraded":"2024-03-30T00:56:21.412206+0000","last_fullsized":"2024-03-30T00:56:21.412206+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:10:04.998621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141755+0000","last_change":"2024-03-30T00:55:51.818517+0000","last_active":"2024-03-30T00:55:56.141755+0000","last_peered":"2024-03-30T00:55:56.141755+0000","last_clean":"2024-03-30T00:55:56.141755+0000","last_became_active":"2024-03-30T00:55:51.818174+0000","last_became_peered":"2024-03-30T00:55:51.818174+0000","last_unstale":"2024-03-30T00:55:56.141755+0000","last_undegraded":"2024-03-30T00:55:56.141755+0000","last_fullsized":"2024-03-30T00:55:56.141755+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:03:01.604135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.329627+0000","last_change":"2024-03-30T00:55:55.836374+0000","last_active":"2024-03-30T00:55:56.329627+0000","last_peered":"2024-03-30T00:55:56.329627+0000","last_clean":"2024-03-30T00:55:56.329627+0000","last_became_active":"2024-03-30T00:55:55.835993+0000","last_became_peered":"2024-03-30T00:55:55.835993+0000","last_unstale":"2024-03-30T00:55:56.329627+0000","last_undegraded":"2024-03-30T00:55:56.329627+0000","last_fullsized":"2024-03-30T00:55:56.329627+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:52:12.165344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"76'1","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.833854+0000","last_change":"2024-03-30T00:55:51.312686+0000","last_active":"2024-03-30T00:55:55.833854+0000","last_peered":"2024-03-30T00:55:55.833854+0000","last_clean":"2024-03-30T00:55:55.833854+0000","last_became_active":"2024-03-30T00:55:51.312296+0000","last_became_peered":"2024-03-30T00:55:51.312296+0000","last_unstale":"2024-03-30T00:55:55.833854+0000","last_undegraded":"2024-03-30T00:55:55.833854+0000","last_fullsized":"2024-03-30T00:55:55.833854+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:28:58.398577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817450+0000","last_change":"2024-03-30T00:55:51.812197+0000","last_active":"2024-03-30T00:55:55.817450+0000","last_peered":"2024-03-30T00:55:55.817450+0000","last_clean":"2024-03-30T00:55:55.817450+0000","last_became_active":"2024-03-30T00:55:51.811892+0000","last_became_peered":"2024-03-30T00:55:51.811892+0000","last_unstale":"2024-03-30T00:55:55.817450+0000","last_undegraded":"2024-03-30T00:55:55.817450+0000","last_fullsized":"2024-03-30T00:55:55.817450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:06:53.459760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.329818+0000","last_change":"2024-03-30T00:55:55.839015+0000","last_active":"2024-03-30T00:55:56.329818+0000","last_peered":"2024-03-30T00:55:56.329818+0000","last_clean":"2024-03-30T00:55:56.329818+0000","last_became_active":"2024-03-30T00:55:55.838608+0000","last_became_peered":"2024-03-30T00:55:55.838608+0000","last_unstale":"2024-03-30T00:55:56.329818+0000","last_undegraded":"2024-03-30T00:55:56.329818+0000","last_fullsized":"2024-03-30T00:55:56.329818+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:24:23.804754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"76'2","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816967+0000","last_change":"2024-03-30T00:55:51.304199+0000","last_active":"2024-03-30T00:55:55.816967+0000","last_peered":"2024-03-30T00:55:55.816967+0000","last_clean":"2024-03-30T00:55:55.816967+0000","last_became_active":"2024-03-30T00:55:51.303837+0000","last_became_peered":"2024-03-30T00:55:51.303837+0000","last_unstale":"2024-03-30T00:55:55.816967+0000","last_undegraded":"2024-03-30T00:55:55.816967+0000","last_fullsized":"2024-03-30T00:55:55.816967+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:19.154957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816615+0000","last_change":"2024-03-30T00:55:51.817984+0000","last_active":"2024-03-30T00:55:55.816615+0000","last_peered":"2024-03-30T00:55:55.816615+0000","last_clean":"2024-03-30T00:55:55.816615+0000","last_became_active":"2024-03-30T00:55:51.817767+0000","last_became_peered":"2024-03-30T00:55:51.817767+0000","last_unstale":"2024-03-30T00:55:55.816615+0000","last_undegraded":"2024-03-30T00:55:55.816615+0000","last_fullsized":"2024-03-30T00:55:55.816615+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:12:48.558024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.329638+0000","last_change":"2024-03-30T00:55:55.836420+0000","last_active":"2024-03-30T00:55:56.329638+0000","last_peered":"2024-03-30T00:55:56.329638+0000","last_clean":"2024-03-30T00:55:56.329638+0000","last_became_active":"2024-03-30T00:55:55.836055+0000","last_became_peered":"2024-03-30T00:55:55.836055+0000","last_unstale":"2024-03-30T00:55:56.329638+0000","last_undegraded":"2024-03-30T00:55:56.329638+0000","last_fullsized":"2024-03-30T00:55:56.329638+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:40:53.920703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"76'2","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817465+0000","last_change":"2024-03-30T00:55:51.300535+0000","last_active":"2024-03-30T00:55:55.817465+0000","last_peered":"2024-03-30T00:55:55.817465+0000","last_clean":"2024-03-30T00:55:55.817465+0000","last_became_active":"2024-03-30T00:55:51.300007+0000","last_became_peered":"2024-03-30T00:55:51.300007+0000","last_unstale":"2024-03-30T00:55:55.817465+0000","last_undegraded":"2024-03-30T00:55:55.817465+0000","last_fullsized":"2024-03-30T00:55:55.817465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:44:01.862562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812416+0000","last_change":"2024-03-30T00:55:51.827985+0000","last_active":"2024-03-30T00:55:55.812416+0000","last_peered":"2024-03-30T00:55:55.812416+0000","last_clean":"2024-03-30T00:55:55.812416+0000","last_became_active":"2024-03-30T00:55:51.827678+0000","last_became_peered":"2024-03-30T00:55:51.827678+0000","last_unstale":"2024-03-30T00:55:55.812416+0000","last_undegraded":"2024-03-30T00:55:55.812416+0000","last_fullsized":"2024-03-30T00:55:55.812416+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:47.088009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.096240+0000","last_change":"2024-03-30T00:55:55.827267+0000","last_active":"2024-03-30T00:55:57.096240+0000","last_peered":"2024-03-30T00:55:57.096240+0000","last_clean":"2024-03-30T00:55:57.096240+0000","last_became_active":"2024-03-30T00:55:55.826821+0000","last_became_peered":"2024-03-30T00:55:55.826821+0000","last_unstale":"2024-03-30T00:55:57.096240+0000","last_undegraded":"2024-03-30T00:55:57.096240+0000","last_fullsized":"2024-03-30T00:55:57.096240+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:15:49.084359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"76'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812263+0000","last_change":"2024-03-30T00:55:51.305177+0000","last_active":"2024-03-30T00:55:55.812263+0000","last_peered":"2024-03-30T00:55:55.812263+0000","last_clean":"2024-03-30T00:55:55.812263+0000","last_became_active":"2024-03-30T00:55:51.304813+0000","last_became_peered":"2024-03-30T00:55:51.304813+0000","last_unstale":"2024-03-30T00:55:55.812263+0000","last_undegraded":"2024-03-30T00:55:55.812263+0000","last_fullsized":"2024-03-30T00:55:55.812263+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:05:20.578897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815997+0000","last_change":"2024-03-30T00:55:51.812714+0000","last_active":"2024-03-30T00:55:55.815997+0000","last_peered":"2024-03-30T00:55:55.815997+0000","last_clean":"2024-03-30T00:55:55.815997+0000","last_became_active":"2024-03-30T00:55:51.812406+0000","last_became_peered":"2024-03-30T00:55:51.812406+0000","last_unstale":"2024-03-30T00:55:55.815997+0000","last_undegraded":"2024-03-30T00:55:55.815997+0000","last_fullsized":"2024-03-30T00:55:55.815997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:47:01.691412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.096477+0000","last_change":"2024-03-30T00:55:55.831994+0000","last_active":"2024-03-30T00:55:57.096477+0000","last_peered":"2024-03-30T00:55:57.096477+0000","last_clean":"2024-03-30T00:55:57.096477+0000","last_became_active":"2024-03-30T00:55:55.831537+0000","last_became_peered":"2024-03-30T00:55:55.831537+0000","last_unstale":"2024-03-30T00:55:57.096477+0000","last_undegraded":"2024-03-30T00:55:57.096477+0000","last_fullsized":"2024-03-30T00:55:57.096477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:44:47.240981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:21.410078+0000","last_change":"2024-03-30T00:55:51.303468+0000","last_active":"2024-03-30T00:56:21.410078+0000","last_peered":"2024-03-30T00:56:21.410078+0000","last_clean":"2024-03-30T00:56:21.410078+0000","last_became_active":"2024-03-30T00:55:51.303068+0000","last_became_peered":"2024-03-30T00:55:51.303068+0000","last_unstale":"2024-03-30T00:56:21.410078+0000","last_undegraded":"2024-03-30T00:56:21.410078+0000","last_fullsized":"2024-03-30T00:56:21.410078+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:45:25.278196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812380+0000","last_change":"2024-03-30T00:55:51.825099+0000","last_active":"2024-03-30T00:55:55.812380+0000","last_peered":"2024-03-30T00:55:55.812380+0000","last_clean":"2024-03-30T00:55:55.812380+0000","last_became_active":"2024-03-30T00:55:51.824717+0000","last_became_peered":"2024-03-30T00:55:51.824717+0000","last_unstale":"2024-03-30T00:55:55.812380+0000","last_undegraded":"2024-03-30T00:55:55.812380+0000","last_fullsized":"2024-03-30T00:55:55.812380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:49:31.542430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.096641+0000","last_change":"2024-03-30T00:55:55.826106+0000","last_active":"2024-03-30T00:55:57.096641+0000","last_peered":"2024-03-30T00:55:57.096641+0000","last_clean":"2024-03-30T00:55:57.096641+0000","last_became_active":"2024-03-30T00:55:55.825597+0000","last_became_peered":"2024-03-30T00:55:55.825597+0000","last_unstale":"2024-03-30T00:55:57.096641+0000","last_undegraded":"2024-03-30T00:55:57.096641+0000","last_fullsized":"2024-03-30T00:55:57.096641+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:00:18.991732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.811402+0000","last_change":"2024-03-30T00:55:51.294090+0000","last_active":"2024-03-30T00:55:55.811402+0000","last_peered":"2024-03-30T00:55:55.811402+0000","last_clean":"2024-03-30T00:55:55.811402+0000","last_became_active":"2024-03-30T00:55:51.293668+0000","last_became_peered":"2024-03-30T00:55:51.293668+0000","last_unstale":"2024-03-30T00:55:55.811402+0000","last_undegraded":"2024-03-30T00:55:55.811402+0000","last_fullsized":"2024-03-30T00:55:55.811402+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:33:36.661760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141658+0000","last_change":"2024-03-30T00:55:51.823428+0000","last_active":"2024-03-30T00:55:56.141658+0000","last_peered":"2024-03-30T00:55:56.141658+0000","last_clean":"2024-03-30T00:55:56.141658+0000","last_became_active":"2024-03-30T00:55:51.822222+0000","last_became_peered":"2024-03-30T00:55:51.822222+0000","last_unstale":"2024-03-30T00:55:56.141658+0000","last_undegraded":"2024-03-30T00:55:56.141658+0000","last_fullsized":"2024-03-30T00:55:56.141658+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:05:34.798940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.831707+0000","last_change":"2024-03-30T00:55:55.828115+0000","last_active":"2024-03-30T00:55:55.831707+0000","last_peered":"2024-03-30T00:55:55.831707+0000","last_clean":"2024-03-30T00:55:55.831707+0000","last_became_active":"2024-03-30T00:55:55.827681+0000","last_became_peered":"2024-03-30T00:55:55.827681+0000","last_unstale":"2024-03-30T00:55:55.831707+0000","last_undegraded":"2024-03-30T00:55:55.831707+0000","last_fullsized":"2024-03-30T00:55:55.831707+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:18:25.751350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812943+0000","last_change":"2024-03-30T00:55:51.305163+0000","last_active":"2024-03-30T00:55:55.812943+0000","last_peered":"2024-03-30T00:55:55.812943+0000","last_clean":"2024-03-30T00:55:55.812943+0000","last_became_active":"2024-03-30T00:55:51.304860+0000","last_became_peered":"2024-03-30T00:55:51.304860+0000","last_unstale":"2024-03-30T00:55:55.812943+0000","last_undegraded":"2024-03-30T00:55:55.812943+0000","last_fullsized":"2024-03-30T00:55:55.812943+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:44:41.850998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.832243+0000","last_change":"2024-03-30T00:55:51.819401+0000","last_active":"2024-03-30T00:55:55.832243+0000","last_peered":"2024-03-30T00:55:55.832243+0000","last_clean":"2024-03-30T00:55:55.832243+0000","last_became_active":"2024-03-30T00:55:51.819021+0000","last_became_peered":"2024-03-30T00:55:51.819021+0000","last_unstale":"2024-03-30T00:55:55.832243+0000","last_undegraded":"2024-03-30T00:55:55.832243+0000","last_fullsized":"2024-03-30T00:55:55.832243+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:01:45.195719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141690+0000","last_change":"2024-03-30T00:55:55.836806+0000","last_active":"2024-03-30T00:55:56.141690+0000","last_peered":"2024-03-30T00:55:56.141690+0000","last_clean":"2024-03-30T00:55:56.141690+0000","last_became_active":"2024-03-30T00:55:55.836403+0000","last_became_peered":"2024-03-30T00:55:55.836403+0000","last_unstale":"2024-03-30T00:55:56.141690+0000","last_undegraded":"2024-03-30T00:55:56.141690+0000","last_fullsized":"2024-03-30T00:55:56.141690+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:17:13.913675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'3","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:23.157955+0000","last_change":"2024-03-30T00:55:51.300833+0000","last_active":"2024-03-30T00:56:23.157955+0000","last_peered":"2024-03-30T00:56:23.157955+0000","last_clean":"2024-03-30T00:56:23.157955+0000","last_became_active":"2024-03-30T00:55:51.300473+0000","last_became_peered":"2024-03-30T00:55:51.300473+0000","last_unstale":"2024-03-30T00:56:23.157955+0000","last_undegraded":"2024-03-30T00:56:23.157955+0000","last_fullsized":"2024-03-30T00:56:23.157955+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:00:39.211981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816447+0000","last_change":"2024-03-30T00:55:51.810781+0000","last_active":"2024-03-30T00:55:55.816447+0000","last_peered":"2024-03-30T00:55:55.816447+0000","last_clean":"2024-03-30T00:55:55.816447+0000","last_became_active":"2024-03-30T00:55:51.810215+0000","last_became_peered":"2024-03-30T00:55:51.810215+0000","last_unstale":"2024-03-30T00:55:55.816447+0000","last_undegraded":"2024-03-30T00:55:55.816447+0000","last_fullsized":"2024-03-30T00:55:55.816447+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:53:50.283209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.540701+0000","last_change":"2024-03-30T00:55:55.834505+0000","last_active":"2024-03-30T00:55:56.540701+0000","last_peered":"2024-03-30T00:55:56.540701+0000","last_clean":"2024-03-30T00:55:56.540701+0000","last_became_active":"2024-03-30T00:55:55.829319+0000","last_became_peered":"2024-03-30T00:55:55.829319+0000","last_unstale":"2024-03-30T00:55:56.540701+0000","last_undegraded":"2024-03-30T00:55:56.540701+0000","last_fullsized":"2024-03-30T00:55:56.540701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:51:37.595523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'2","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:20.396339+0000","last_change":"2024-03-30T00:55:51.302070+0000","last_active":"2024-03-30T00:56:20.396339+0000","last_peered":"2024-03-30T00:56:20.396339+0000","last_clean":"2024-03-30T00:56:20.396339+0000","last_became_active":"2024-03-30T00:55:51.298940+0000","last_became_peered":"2024-03-30T00:55:51.298940+0000","last_unstale":"2024-03-30T00:56:20.396339+0000","last_undegraded":"2024-03-30T00:56:20.396339+0000","last_fullsized":"2024-03-30T00:56:20.396339+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:07:47.915506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817361+0000","last_change":"2024-03-30T00:55:51.819918+0000","last_active":"2024-03-30T00:55:55.817361+0000","last_peered":"2024-03-30T00:55:55.817361+0000","last_clean":"2024-03-30T00:55:55.817361+0000","last_became_active":"2024-03-30T00:55:51.819493+0000","last_became_peered":"2024-03-30T00:55:51.819493+0000","last_unstale":"2024-03-30T00:55:55.817361+0000","last_undegraded":"2024-03-30T00:55:55.817361+0000","last_fullsized":"2024-03-30T00:55:55.817361+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:43:43.067031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.330119+0000","last_change":"2024-03-30T00:55:55.827176+0000","last_active":"2024-03-30T00:55:56.330119+0000","last_peered":"2024-03-30T00:55:56.330119+0000","last_clean":"2024-03-30T00:55:56.330119+0000","last_became_active":"2024-03-30T00:55:55.826772+0000","last_became_peered":"2024-03-30T00:55:55.826772+0000","last_unstale":"2024-03-30T00:55:56.330119+0000","last_undegraded":"2024-03-30T00:55:56.330119+0000","last_fullsized":"2024-03-30T00:55:56.330119+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:02:04.512050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"76'3","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812239+0000","last_change":"2024-03-30T00:55:51.306688+0000","last_active":"2024-03-30T00:55:55.812239+0000","last_peered":"2024-03-30T00:55:55.812239+0000","last_clean":"2024-03-30T00:55:55.812239+0000","last_became_active":"2024-03-30T00:55:51.306333+0000","last_became_peered":"2024-03-30T00:55:51.306333+0000","last_unstale":"2024-03-30T00:55:55.812239+0000","last_undegraded":"2024-03-30T00:55:55.812239+0000","last_fullsized":"2024-03-30T00:55:55.812239+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:21:13.719947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816780+0000","last_change":"2024-03-30T00:55:51.824879+0000","last_active":"2024-03-30T00:55:55.816780+0000","last_peered":"2024-03-30T00:55:55.816780+0000","last_clean":"2024-03-30T00:55:55.816780+0000","last_became_active":"2024-03-30T00:55:51.824566+0000","last_became_peered":"2024-03-30T00:55:51.824566+0000","last_unstale":"2024-03-30T00:55:55.816780+0000","last_undegraded":"2024-03-30T00:55:55.816780+0000","last_fullsized":"2024-03-30T00:55:55.816780+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:04:28.505693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.540728+0000","last_change":"2024-03-30T00:55:55.826387+0000","last_active":"2024-03-30T00:55:56.540728+0000","last_peered":"2024-03-30T00:55:56.540728+0000","last_clean":"2024-03-30T00:55:56.540728+0000","last_became_active":"2024-03-30T00:55:55.825941+0000","last_became_peered":"2024-03-30T00:55:55.825941+0000","last_unstale":"2024-03-30T00:55:56.540728+0000","last_undegraded":"2024-03-30T00:55:56.540728+0000","last_fullsized":"2024-03-30T00:55:56.540728+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:34:43.432712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"81'23","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:24.160174+0000","last_change":"2024-03-30T00:55:51.304592+0000","last_active":"2024-03-30T00:56:24.160174+0000","last_peered":"2024-03-30T00:56:24.160174+0000","last_clean":"2024-03-30T00:56:24.160174+0000","last_became_active":"2024-03-30T00:55:51.303931+0000","last_became_peered":"2024-03-30T00:55:51.303931+0000","last_unstale":"2024-03-30T00:56:24.160174+0000","last_undegraded":"2024-03-30T00:56:24.160174+0000","last_fullsized":"2024-03-30T00:56:24.160174+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:53:40.976289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40427,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":44,"num_write":24,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816688+0000","last_change":"2024-03-30T00:55:51.815196+0000","last_active":"2024-03-30T00:55:55.816688+0000","last_peered":"2024-03-30T00:55:55.816688+0000","last_clean":"2024-03-30T00:55:55.816688+0000","last_became_active":"2024-03-30T00:55:51.813376+0000","last_became_peered":"2024-03-30T00:55:51.813376+0000","last_unstale":"2024-03-30T00:55:55.816688+0000","last_undegraded":"2024-03-30T00:55:55.816688+0000","last_fullsized":"2024-03-30T00:55:55.816688+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:40:26.350248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.837509+0000","last_change":"2024-03-30T00:55:55.836906+0000","last_active":"2024-03-30T00:55:55.837509+0000","last_peered":"2024-03-30T00:55:55.837509+0000","last_clean":"2024-03-30T00:55:55.837509+0000","last_became_active":"2024-03-30T00:55:55.836539+0000","last_became_peered":"2024-03-30T00:55:55.836539+0000","last_unstale":"2024-03-30T00:55:55.837509+0000","last_undegraded":"2024-03-30T00:55:55.837509+0000","last_fullsized":"2024-03-30T00:55:55.837509+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:21:39.320192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'6","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:21.411514+0000","last_change":"2024-03-30T00:55:51.316839+0000","last_active":"2024-03-30T00:56:21.411514+0000","last_peered":"2024-03-30T00:56:21.411514+0000","last_clean":"2024-03-30T00:56:21.411514+0000","last_became_active":"2024-03-30T00:55:51.316519+0000","last_became_peered":"2024-03-30T00:55:51.316519+0000","last_unstale":"2024-03-30T00:56:21.411514+0000","last_undegraded":"2024-03-30T00:56:21.411514+0000","last_fullsized":"2024-03-30T00:56:21.411514+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:25:32.038384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131378+0000","last_change":"2024-03-30T00:55:51.819825+0000","last_active":"2024-03-30T00:55:56.131378+0000","last_peered":"2024-03-30T00:55:56.131378+0000","last_clean":"2024-03-30T00:55:56.131378+0000","last_became_active":"2024-03-30T00:55:51.819481+0000","last_became_peered":"2024-03-30T00:55:51.819481+0000","last_unstale":"2024-03-30T00:55:56.131378+0000","last_undegraded":"2024-03-30T00:55:56.131378+0000","last_fullsized":"2024-03-30T00:55:56.131378+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:10:17.138541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.097020+0000","last_change":"2024-03-30T00:55:55.832675+0000","last_active":"2024-03-30T00:55:57.097020+0000","last_peered":"2024-03-30T00:55:57.097020+0000","last_clean":"2024-03-30T00:55:57.097020+0000","last_became_active":"2024-03-30T00:55:55.832325+0000","last_became_peered":"2024-03-30T00:55:55.832325+0000","last_unstale":"2024-03-30T00:55:57.097020+0000","last_undegraded":"2024-03-30T00:55:57.097020+0000","last_fullsized":"2024-03-30T00:55:57.097020+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:38:46.765473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812030+0000","last_change":"2024-03-30T00:55:51.313700+0000","last_active":"2024-03-30T00:55:55.812030+0000","last_peered":"2024-03-30T00:55:55.812030+0000","last_clean":"2024-03-30T00:55:55.812030+0000","last_became_active":"2024-03-30T00:55:51.313355+0000","last_became_peered":"2024-03-30T00:55:51.313355+0000","last_unstale":"2024-03-30T00:55:55.812030+0000","last_undegraded":"2024-03-30T00:55:55.812030+0000","last_fullsized":"2024-03-30T00:55:55.812030+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:43:04.461398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.823400+0000","last_change":"2024-03-30T00:55:51.819939+0000","last_active":"2024-03-30T00:55:55.823400+0000","last_peered":"2024-03-30T00:55:55.823400+0000","last_clean":"2024-03-30T00:55:55.823400+0000","last_became_active":"2024-03-30T00:55:51.819614+0000","last_became_peered":"2024-03-30T00:55:51.819614+0000","last_unstale":"2024-03-30T00:55:55.823400+0000","last_undegraded":"2024-03-30T00:55:55.823400+0000","last_fullsized":"2024-03-30T00:55:55.823400+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:20:37.362779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.540778+0000","last_change":"2024-03-30T00:55:55.841536+0000","last_active":"2024-03-30T00:55:56.540778+0000","last_peered":"2024-03-30T00:55:56.540778+0000","last_clean":"2024-03-30T00:55:56.540778+0000","last_became_active":"2024-03-30T00:55:55.839691+0000","last_became_peered":"2024-03-30T00:55:55.839691+0000","last_unstale":"2024-03-30T00:55:56.540778+0000","last_undegraded":"2024-03-30T00:55:56.540778+0000","last_fullsized":"2024-03-30T00:55:56.540778+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:52:53.769565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"81'4","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:23.159572+0000","last_change":"2024-03-30T00:55:51.299246+0000","last_active":"2024-03-30T00:56:23.159572+0000","last_peered":"2024-03-30T00:56:23.159572+0000","last_clean":"2024-03-30T00:56:23.159572+0000","last_became_active":"2024-03-30T00:55:51.298924+0000","last_became_peered":"2024-03-30T00:55:51.298924+0000","last_unstale":"2024-03-30T00:56:23.159572+0000","last_undegraded":"2024-03-30T00:56:23.159572+0000","last_fullsized":"2024-03-30T00:56:23.159572+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:50:05.897612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.823167+0000","last_change":"2024-03-30T00:55:51.813804+0000","last_active":"2024-03-30T00:55:55.823167+0000","last_peered":"2024-03-30T00:55:55.823167+0000","last_clean":"2024-03-30T00:55:55.823167+0000","last_became_active":"2024-03-30T00:55:51.813500+0000","last_became_peered":"2024-03-30T00:55:51.813500+0000","last_unstale":"2024-03-30T00:55:55.823167+0000","last_undegraded":"2024-03-30T00:55:55.823167+0000","last_fullsized":"2024-03-30T00:55:55.823167+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:42:21.614900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131528+0000","last_change":"2024-03-30T00:55:55.837737+0000","last_active":"2024-03-30T00:55:56.131528+0000","last_peered":"2024-03-30T00:55:56.131528+0000","last_clean":"2024-03-30T00:55:56.131528+0000","last_became_active":"2024-03-30T00:55:55.837312+0000","last_became_peered":"2024-03-30T00:55:55.837312+0000","last_unstale":"2024-03-30T00:55:56.131528+0000","last_undegraded":"2024-03-30T00:55:56.131528+0000","last_fullsized":"2024-03-30T00:55:56.131528+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:11:46.139979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'10","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:20.393517+0000","last_change":"2024-03-30T00:55:51.306398+0000","last_active":"2024-03-30T00:56:20.393517+0000","last_peered":"2024-03-30T00:56:20.393517+0000","last_clean":"2024-03-30T00:56:20.393517+0000","last_became_active":"2024-03-30T00:55:51.305878+0000","last_became_peered":"2024-03-30T00:55:51.305878+0000","last_unstale":"2024-03-30T00:56:20.393517+0000","last_undegraded":"2024-03-30T00:56:20.393517+0000","last_fullsized":"2024-03-30T00:56:20.393517+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:20:32.742423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812237+0000","last_change":"2024-03-30T00:55:51.824294+0000","last_active":"2024-03-30T00:55:55.812237+0000","last_peered":"2024-03-30T00:55:55.812237+0000","last_clean":"2024-03-30T00:55:55.812237+0000","last_became_active":"2024-03-30T00:55:51.823957+0000","last_became_peered":"2024-03-30T00:55:51.823957+0000","last_unstale":"2024-03-30T00:55:55.812237+0000","last_undegraded":"2024-03-30T00:55:55.812237+0000","last_fullsized":"2024-03-30T00:55:55.812237+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:23:16.873140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.837548+0000","last_change":"2024-03-30T00:55:55.837023+0000","last_active":"2024-03-30T00:55:55.837548+0000","last_peered":"2024-03-30T00:55:55.837548+0000","last_clean":"2024-03-30T00:55:55.837548+0000","last_became_active":"2024-03-30T00:55:55.836541+0000","last_became_peered":"2024-03-30T00:55:55.836541+0000","last_unstale":"2024-03-30T00:55:55.837548+0000","last_undegraded":"2024-03-30T00:55:55.837548+0000","last_fullsized":"2024-03-30T00:55:55.837548+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:58:40.323953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"76'1","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815282+0000","last_change":"2024-03-30T00:55:51.300883+0000","last_active":"2024-03-30T00:55:55.815282+0000","last_peered":"2024-03-30T00:55:55.815282+0000","last_clean":"2024-03-30T00:55:55.815282+0000","last_became_active":"2024-03-30T00:55:51.300531+0000","last_became_peered":"2024-03-30T00:55:51.300531+0000","last_unstale":"2024-03-30T00:55:55.815282+0000","last_undegraded":"2024-03-30T00:55:55.815282+0000","last_fullsized":"2024-03-30T00:55:55.815282+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:38:38.874075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815671+0000","last_change":"2024-03-30T00:55:51.822970+0000","last_active":"2024-03-30T00:55:55.815671+0000","last_peered":"2024-03-30T00:55:55.815671+0000","last_clean":"2024-03-30T00:55:55.815671+0000","last_became_active":"2024-03-30T00:55:51.822653+0000","last_became_peered":"2024-03-30T00:55:51.822653+0000","last_unstale":"2024-03-30T00:55:55.815671+0000","last_undegraded":"2024-03-30T00:55:55.815671+0000","last_fullsized":"2024-03-30T00:55:55.815671+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:59:26.306672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.857191+0000","last_change":"2024-03-30T00:55:55.856605+0000","last_active":"2024-03-30T00:55:55.857191+0000","last_peered":"2024-03-30T00:55:55.857191+0000","last_clean":"2024-03-30T00:55:55.857191+0000","last_became_active":"2024-03-30T00:55:55.856061+0000","last_became_peered":"2024-03-30T00:55:55.856061+0000","last_unstale":"2024-03-30T00:55:55.857191+0000","last_undegraded":"2024-03-30T00:55:55.857191+0000","last_fullsized":"2024-03-30T00:55:55.857191+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:58:41.010206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.834443+0000","last_change":"2024-03-30T00:55:51.301961+0000","last_active":"2024-03-30T00:55:55.834443+0000","last_peered":"2024-03-30T00:55:55.834443+0000","last_clean":"2024-03-30T00:55:55.834443+0000","last_became_active":"2024-03-30T00:55:51.301618+0000","last_became_peered":"2024-03-30T00:55:51.301618+0000","last_unstale":"2024-03-30T00:55:55.834443+0000","last_undegraded":"2024-03-30T00:55:55.834443+0000","last_fullsized":"2024-03-30T00:55:55.834443+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:38:46.681490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.855479+0000","last_change":"2024-03-30T00:55:51.821331+0000","last_active":"2024-03-30T00:55:55.855479+0000","last_peered":"2024-03-30T00:55:55.855479+0000","last_clean":"2024-03-30T00:55:55.855479+0000","last_became_active":"2024-03-30T00:55:51.820988+0000","last_became_peered":"2024-03-30T00:55:51.820988+0000","last_unstale":"2024-03-30T00:55:55.855479+0000","last_undegraded":"2024-03-30T00:55:55.855479+0000","last_fullsized":"2024-03-30T00:55:55.855479+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:25:20.150789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131430+0000","last_change":"2024-03-30T00:55:55.826251+0000","last_active":"2024-03-30T00:55:56.131430+0000","last_peered":"2024-03-30T00:55:56.131430+0000","last_clean":"2024-03-30T00:55:56.131430+0000","last_became_active":"2024-03-30T00:55:55.822695+0000","last_became_peered":"2024-03-30T00:55:55.822695+0000","last_unstale":"2024-03-30T00:55:56.131430+0000","last_undegraded":"2024-03-30T00:55:56.131430+0000","last_fullsized":"2024-03-30T00:55:56.131430+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:32:19.011454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'1","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.017740+0000","last_change":"2024-03-30T00:55:51.309412+0000","last_active":"2024-03-30T00:55:57.017740+0000","last_peered":"2024-03-30T00:55:57.017740+0000","last_clean":"2024-03-30T00:55:57.017740+0000","last_became_active":"2024-03-30T00:55:51.309028+0000","last_became_peered":"2024-03-30T00:55:51.309028+0000","last_unstale":"2024-03-30T00:55:57.017740+0000","last_undegraded":"2024-03-30T00:55:57.017740+0000","last_fullsized":"2024-03-30T00:55:57.017740+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:00:49.022772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.855266+0000","last_change":"2024-03-30T00:55:51.825164+0000","last_active":"2024-03-30T00:55:55.855266+0000","last_peered":"2024-03-30T00:55:55.855266+0000","last_clean":"2024-03-30T00:55:55.855266+0000","last_became_active":"2024-03-30T00:55:51.824823+0000","last_became_peered":"2024-03-30T00:55:51.824823+0000","last_unstale":"2024-03-30T00:55:55.855266+0000","last_undegraded":"2024-03-30T00:55:55.855266+0000","last_fullsized":"2024-03-30T00:55:55.855266+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:29:11.471111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131539+0000","last_change":"2024-03-30T00:55:55.856377+0000","last_active":"2024-03-30T00:55:56.131539+0000","last_peered":"2024-03-30T00:55:56.131539+0000","last_clean":"2024-03-30T00:55:56.131539+0000","last_became_active":"2024-03-30T00:55:55.855986+0000","last_became_peered":"2024-03-30T00:55:55.855986+0000","last_unstale":"2024-03-30T00:55:56.131539+0000","last_undegraded":"2024-03-30T00:55:56.131539+0000","last_fullsized":"2024-03-30T00:55:56.131539+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:45:47.303446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.813157+0000","last_change":"2024-03-30T00:55:51.824176+0000","last_active":"2024-03-30T00:55:55.813157+0000","last_peered":"2024-03-30T00:55:55.813157+0000","last_clean":"2024-03-30T00:55:55.813157+0000","last_became_active":"2024-03-30T00:55:51.823674+0000","last_became_peered":"2024-03-30T00:55:51.823674+0000","last_unstale":"2024-03-30T00:55:55.813157+0000","last_undegraded":"2024-03-30T00:55:55.813157+0000","last_fullsized":"2024-03-30T00:55:55.813157+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:34:51.051973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'1","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.331029+0000","last_change":"2024-03-30T00:55:51.314834+0000","last_active":"2024-03-30T00:55:56.331029+0000","last_peered":"2024-03-30T00:55:56.331029+0000","last_clean":"2024-03-30T00:55:56.331029+0000","last_became_active":"2024-03-30T00:55:51.314429+0000","last_became_peered":"2024-03-30T00:55:51.314429+0000","last_unstale":"2024-03-30T00:55:56.331029+0000","last_undegraded":"2024-03-30T00:55:56.331029+0000","last_fullsized":"2024-03-30T00:55:56.331029+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:17:13.987180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:59.380272+0000","last_change":"2024-03-30T00:55:55.840721+0000","last_active":"2024-03-30T00:55:59.380272+0000","last_peered":"2024-03-30T00:55:59.380272+0000","last_clean":"2024-03-30T00:55:59.380272+0000","last_became_active":"2024-03-30T00:55:55.840269+0000","last_became_peered":"2024-03-30T00:55:55.840269+0000","last_unstale":"2024-03-30T00:55:59.380272+0000","last_undegraded":"2024-03-30T00:55:59.380272+0000","last_fullsized":"2024-03-30T00:55:59.380272+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:28:55.366411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141714+0000","last_change":"2024-03-30T00:55:51.812609+0000","last_active":"2024-03-30T00:55:56.141714+0000","last_peered":"2024-03-30T00:55:56.141714+0000","last_clean":"2024-03-30T00:55:56.141714+0000","last_became_active":"2024-03-30T00:55:51.812325+0000","last_became_peered":"2024-03-30T00:55:51.812325+0000","last_unstale":"2024-03-30T00:55:56.141714+0000","last_undegraded":"2024-03-30T00:55:56.141714+0000","last_fullsized":"2024-03-30T00:55:56.141714+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:22:22.074405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'3","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:22.154189+0000","last_change":"2024-03-30T00:55:51.296173+0000","last_active":"2024-03-30T00:56:22.154189+0000","last_peered":"2024-03-30T00:56:22.154189+0000","last_clean":"2024-03-30T00:56:22.154189+0000","last_became_active":"2024-03-30T00:55:51.295331+0000","last_became_peered":"2024-03-30T00:55:51.295331+0000","last_unstale":"2024-03-30T00:56:22.154189+0000","last_undegraded":"2024-03-30T00:56:22.154189+0000","last_fullsized":"2024-03-30T00:56:22.154189+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:51:58.343765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":324,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.017477+0000","last_change":"2024-03-30T00:55:02.487656+0000","last_active":"2024-03-30T00:55:57.017477+0000","last_peered":"2024-03-30T00:55:57.017477+0000","last_clean":"2024-03-30T00:55:57.017477+0000","last_became_active":"2024-03-30T00:55:02.487263+0000","last_became_peered":"2024-03-30T00:55:02.487263+0000","last_unstale":"2024-03-30T00:55:57.017477+0000","last_undegraded":"2024-03-30T00:55:57.017477+0000","last_fullsized":"2024-03-30T00:55:57.017477+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:52:29.691461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:52:29.691461+0000","last_clean_scrub_stamp":"2024-03-30T00:52:29.691461+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:19:20.825038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.857268+0000","last_change":"2024-03-30T00:55:55.856487+0000","last_active":"2024-03-30T00:55:55.857268+0000","last_peered":"2024-03-30T00:55:55.857268+0000","last_clean":"2024-03-30T00:55:55.857268+0000","last_became_active":"2024-03-30T00:55:55.856106+0000","last_became_peered":"2024-03-30T00:55:55.856106+0000","last_unstale":"2024-03-30T00:55:55.857268+0000","last_undegraded":"2024-03-30T00:55:55.857268+0000","last_fullsized":"2024-03-30T00:55:55.857268+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:14.504218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131308+0000","last_change":"2024-03-30T00:55:51.821519+0000","last_active":"2024-03-30T00:55:56.131308+0000","last_peered":"2024-03-30T00:55:56.131308+0000","last_clean":"2024-03-30T00:55:56.131308+0000","last_became_active":"2024-03-30T00:55:51.821095+0000","last_became_peered":"2024-03-30T00:55:51.821095+0000","last_unstale":"2024-03-30T00:55:56.131308+0000","last_undegraded":"2024-03-30T00:55:56.131308+0000","last_fullsized":"2024-03-30T00:55:56.131308+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:50:22.297450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'3","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:21.409836+0000","last_change":"2024-03-30T00:55:51.303047+0000","last_active":"2024-03-30T00:56:21.409836+0000","last_peered":"2024-03-30T00:56:21.409836+0000","last_clean":"2024-03-30T00:56:21.409836+0000","last_became_active":"2024-03-30T00:55:51.302712+0000","last_became_peered":"2024-03-30T00:55:51.302712+0000","last_unstale":"2024-03-30T00:56:21.409836+0000","last_undegraded":"2024-03-30T00:56:21.409836+0000","last_fullsized":"2024-03-30T00:56:21.409836+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:07:59.451674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.096266+0000","last_change":"2024-03-30T00:55:55.823692+0000","last_active":"2024-03-30T00:55:57.096266+0000","last_peered":"2024-03-30T00:55:57.096266+0000","last_clean":"2024-03-30T00:55:57.096266+0000","last_became_active":"2024-03-30T00:55:55.823188+0000","last_became_peered":"2024-03-30T00:55:55.823188+0000","last_unstale":"2024-03-30T00:55:57.096266+0000","last_undegraded":"2024-03-30T00:55:57.096266+0000","last_fullsized":"2024-03-30T00:55:57.096266+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:32:32.089166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'1","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.333575+0000","last_change":"2024-03-30T00:55:51.306935+0000","last_active":"2024-03-30T00:55:56.333575+0000","last_peered":"2024-03-30T00:55:56.333575+0000","last_clean":"2024-03-30T00:55:56.333575+0000","last_became_active":"2024-03-30T00:55:51.306595+0000","last_became_peered":"2024-03-30T00:55:51.306595+0000","last_unstale":"2024-03-30T00:55:56.333575+0000","last_undegraded":"2024-03-30T00:55:56.333575+0000","last_fullsized":"2024-03-30T00:55:56.333575+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:50:30.396762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131446+0000","last_change":"2024-03-30T00:55:51.807084+0000","last_active":"2024-03-30T00:55:56.131446+0000","last_peered":"2024-03-30T00:55:56.131446+0000","last_clean":"2024-03-30T00:55:56.131446+0000","last_became_active":"2024-03-30T00:55:51.806505+0000","last_became_peered":"2024-03-30T00:55:51.806505+0000","last_unstale":"2024-03-30T00:55:56.131446+0000","last_undegraded":"2024-03-30T00:55:56.131446+0000","last_fullsized":"2024-03-30T00:55:56.131446+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:38:44.759863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141634+0000","last_change":"2024-03-30T00:55:55.856457+0000","last_active":"2024-03-30T00:55:56.141634+0000","last_peered":"2024-03-30T00:55:56.141634+0000","last_clean":"2024-03-30T00:55:56.141634+0000","last_became_active":"2024-03-30T00:55:55.855982+0000","last_became_peered":"2024-03-30T00:55:55.855982+0000","last_unstale":"2024-03-30T00:55:56.141634+0000","last_undegraded":"2024-03-30T00:55:56.141634+0000","last_fullsized":"2024-03-30T00:55:56.141634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:18:25.154634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"81'3","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:21.409721+0000","last_change":"2024-03-30T00:55:51.312581+0000","last_active":"2024-03-30T00:56:21.409721+0000","last_peered":"2024-03-30T00:56:21.409721+0000","last_clean":"2024-03-30T00:56:21.409721+0000","last_became_active":"2024-03-30T00:55:51.312225+0000","last_became_peered":"2024-03-30T00:55:51.312225+0000","last_unstale":"2024-03-30T00:56:21.409721+0000","last_undegraded":"2024-03-30T00:56:21.409721+0000","last_fullsized":"2024-03-30T00:56:21.409721+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:27:12.615250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141773+0000","last_change":"2024-03-30T00:55:51.821527+0000","last_active":"2024-03-30T00:55:56.141773+0000","last_peered":"2024-03-30T00:55:56.141773+0000","last_clean":"2024-03-30T00:55:56.141773+0000","last_became_active":"2024-03-30T00:55:51.820689+0000","last_became_peered":"2024-03-30T00:55:51.820689+0000","last_unstale":"2024-03-30T00:55:56.141773+0000","last_undegraded":"2024-03-30T00:55:56.141773+0000","last_fullsized":"2024-03-30T00:55:56.141773+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:14:35.735985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.097219+0000","last_change":"2024-03-30T00:55:55.824307+0000","last_active":"2024-03-30T00:55:57.097219+0000","last_peered":"2024-03-30T00:55:57.097219+0000","last_clean":"2024-03-30T00:55:57.097219+0000","last_became_active":"2024-03-30T00:55:55.823900+0000","last_became_peered":"2024-03-30T00:55:55.823900+0000","last_unstale":"2024-03-30T00:55:57.097219+0000","last_undegraded":"2024-03-30T00:55:57.097219+0000","last_fullsized":"2024-03-30T00:55:57.097219+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:44:22.139120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:20.392961+0000","last_change":"2024-03-30T00:55:51.296965+0000","last_active":"2024-03-30T00:56:20.392961+0000","last_peered":"2024-03-30T00:56:20.392961+0000","last_clean":"2024-03-30T00:56:20.392961+0000","last_became_active":"2024-03-30T00:55:51.296645+0000","last_became_peered":"2024-03-30T00:55:51.296645+0000","last_unstale":"2024-03-30T00:56:20.392961+0000","last_undegraded":"2024-03-30T00:56:20.392961+0000","last_fullsized":"2024-03-30T00:56:20.392961+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:53:46.519972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.818968+0000","last_change":"2024-03-30T00:55:51.826282+0000","last_active":"2024-03-30T00:55:55.818968+0000","last_peered":"2024-03-30T00:55:55.818968+0000","last_clean":"2024-03-30T00:55:55.818968+0000","last_became_active":"2024-03-30T00:55:51.823910+0000","last_became_peered":"2024-03-30T00:55:51.823910+0000","last_unstale":"2024-03-30T00:55:55.818968+0000","last_undegraded":"2024-03-30T00:55:55.818968+0000","last_fullsized":"2024-03-30T00:55:55.818968+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:47.454280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.330002+0000","last_change":"2024-03-30T00:55:55.856750+0000","last_active":"2024-03-30T00:55:56.330002+0000","last_peered":"2024-03-30T00:55:56.330002+0000","last_clean":"2024-03-30T00:55:56.330002+0000","last_became_active":"2024-03-30T00:55:55.856288+0000","last_became_peered":"2024-03-30T00:55:55.856288+0000","last_unstale":"2024-03-30T00:55:56.330002+0000","last_undegraded":"2024-03-30T00:55:56.330002+0000","last_fullsized":"2024-03-30T00:55:56.330002+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:43:28.085910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:20.395225+0000","last_change":"2024-03-30T00:55:51.306357+0000","last_active":"2024-03-30T00:56:20.395225+0000","last_peered":"2024-03-30T00:56:20.395225+0000","last_clean":"2024-03-30T00:56:20.395225+0000","last_became_active":"2024-03-30T00:55:51.305815+0000","last_became_peered":"2024-03-30T00:55:51.305815+0000","last_unstale":"2024-03-30T00:56:20.395225+0000","last_undegraded":"2024-03-30T00:56:20.395225+0000","last_fullsized":"2024-03-30T00:56:20.395225+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:16:21.753446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817389+0000","last_change":"2024-03-30T00:55:51.810686+0000","last_active":"2024-03-30T00:55:55.817389+0000","last_peered":"2024-03-30T00:55:55.817389+0000","last_clean":"2024-03-30T00:55:55.817389+0000","last_became_active":"2024-03-30T00:55:51.810331+0000","last_became_peered":"2024-03-30T00:55:51.810331+0000","last_unstale":"2024-03-30T00:55:55.817389+0000","last_undegraded":"2024-03-30T00:55:55.817389+0000","last_fullsized":"2024-03-30T00:55:55.817389+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:24:13.750905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.540794+0000","last_change":"2024-03-30T00:55:55.842894+0000","last_active":"2024-03-30T00:55:56.540794+0000","last_peered":"2024-03-30T00:55:56.540794+0000","last_clean":"2024-03-30T00:55:56.540794+0000","last_became_active":"2024-03-30T00:55:55.842118+0000","last_became_peered":"2024-03-30T00:55:55.842118+0000","last_unstale":"2024-03-30T00:55:56.540794+0000","last_undegraded":"2024-03-30T00:55:56.540794+0000","last_fullsized":"2024-03-30T00:55:56.540794+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:39.246228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'3","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:20.395176+0000","last_change":"2024-03-30T00:55:51.313188+0000","last_active":"2024-03-30T00:56:20.395176+0000","last_peered":"2024-03-30T00:56:20.395176+0000","last_clean":"2024-03-30T00:56:20.395176+0000","last_became_active":"2024-03-30T00:55:51.312879+0000","last_became_peered":"2024-03-30T00:55:51.312879+0000","last_unstale":"2024-03-30T00:56:20.395176+0000","last_undegraded":"2024-03-30T00:56:20.395176+0000","last_fullsized":"2024-03-30T00:56:20.395176+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:41:23.420427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815027+0000","last_change":"2024-03-30T00:55:51.808998+0000","last_active":"2024-03-30T00:55:55.815027+0000","last_peered":"2024-03-30T00:55:55.815027+0000","last_clean":"2024-03-30T00:55:55.815027+0000","last_became_active":"2024-03-30T00:55:51.808636+0000","last_became_peered":"2024-03-30T00:55:51.808636+0000","last_unstale":"2024-03-30T00:55:55.815027+0000","last_undegraded":"2024-03-30T00:55:55.815027+0000","last_fullsized":"2024-03-30T00:55:55.815027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:02.379294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.329776+0000","last_change":"2024-03-30T00:55:55.833209+0000","last_active":"2024-03-30T00:55:56.329776+0000","last_peered":"2024-03-30T00:55:56.329776+0000","last_clean":"2024-03-30T00:55:56.329776+0000","last_became_active":"2024-03-30T00:55:55.832799+0000","last_became_peered":"2024-03-30T00:55:55.832799+0000","last_unstale":"2024-03-30T00:55:56.329776+0000","last_undegraded":"2024-03-30T00:55:56.329776+0000","last_fullsized":"2024-03-30T00:55:56.329776+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:38:45.496406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'1","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.095937+0000","last_change":"2024-03-30T00:55:51.313708+0000","last_active":"2024-03-30T00:55:57.095937+0000","last_peered":"2024-03-30T00:55:57.095937+0000","last_clean":"2024-03-30T00:55:57.095937+0000","last_became_active":"2024-03-30T00:55:51.313362+0000","last_became_peered":"2024-03-30T00:55:51.313362+0000","last_unstale":"2024-03-30T00:55:57.095937+0000","last_undegraded":"2024-03-30T00:55:57.095937+0000","last_fullsized":"2024-03-30T00:55:57.095937+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:03:17.859640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141746+0000","last_change":"2024-03-30T00:55:51.819638+0000","last_active":"2024-03-30T00:55:56.141746+0000","last_peered":"2024-03-30T00:55:56.141746+0000","last_clean":"2024-03-30T00:55:56.141746+0000","last_became_active":"2024-03-30T00:55:51.819282+0000","last_became_peered":"2024-03-30T00:55:51.819282+0000","last_unstale":"2024-03-30T00:55:56.141746+0000","last_undegraded":"2024-03-30T00:55:56.141746+0000","last_fullsized":"2024-03-30T00:55:56.141746+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:13:39.249550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.329910+0000","last_change":"2024-03-30T00:55:55.856498+0000","last_active":"2024-03-30T00:55:56.329910+0000","last_peered":"2024-03-30T00:55:56.329910+0000","last_clean":"2024-03-30T00:55:56.329910+0000","last_became_active":"2024-03-30T00:55:55.856015+0000","last_became_peered":"2024-03-30T00:55:55.856015+0000","last_unstale":"2024-03-30T00:55:56.329910+0000","last_undegraded":"2024-03-30T00:55:56.329910+0000","last_fullsized":"2024-03-30T00:55:56.329910+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:14:00.248812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:20.393163+0000","last_change":"2024-03-30T00:55:51.302995+0000","last_active":"2024-03-30T00:56:20.393163+0000","last_peered":"2024-03-30T00:56:20.393163+0000","last_clean":"2024-03-30T00:56:20.393163+0000","last_became_active":"2024-03-30T00:55:51.302650+0000","last_became_peered":"2024-03-30T00:55:51.302650+0000","last_unstale":"2024-03-30T00:56:20.393163+0000","last_undegraded":"2024-03-30T00:56:20.393163+0000","last_fullsized":"2024-03-30T00:56:20.393163+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:17:32.556550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.823393+0000","last_change":"2024-03-30T00:55:51.808969+0000","last_active":"2024-03-30T00:55:55.823393+0000","last_peered":"2024-03-30T00:55:55.823393+0000","last_clean":"2024-03-30T00:55:55.823393+0000","last_became_active":"2024-03-30T00:55:51.808633+0000","last_became_peered":"2024-03-30T00:55:51.808633+0000","last_unstale":"2024-03-30T00:55:55.823393+0000","last_undegraded":"2024-03-30T00:55:55.823393+0000","last_fullsized":"2024-03-30T00:55:55.823393+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:16:46.128066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:55:57.095798+0000","last_change":"2024-03-30T00:55:55.824964+0000","last_active":"2024-03-30T00:55:57.095798+0000","last_peered":"2024-03-30T00:55:57.095798+0000","last_clean":"2024-03-30T00:55:57.095798+0000","last_became_active":"2024-03-30T00:55:55.824559+0000","last_became_peered":"2024-03-30T00:55:55.824559+0000","last_unstale":"2024-03-30T00:55:57.095798+0000","last_undegraded":"2024-03-30T00:55:57.095798+0000","last_fullsized":"2024-03-30T00:55:57.095798+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:19:24.746963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'3","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T00:56:21.412462+0000","last_change":"2024-03-30T00:55:51.301602+0000","last_active":"2024-03-30T00:56:21.412462+0000","last_peered":"2024-03-30T00:56:21.412462+0000","last_clean":"2024-03-30T00:56:21.412462+0000","last_became_active":"2024-03-30T00:55:51.301221+0000","last_became_peered":"2024-03-30T00:55:51.301221+0000","last_unstale":"2024-03-30T00:56:21.412462+0000","last_undegraded":"2024-03-30T00:56:21.412462+0000","last_fullsized":"2024-03-30T00:56:21.412462+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:37:36.727603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.825582+0000","last_change":"2024-03-30T00:55:51.810197+0000","last_active":"2024-03-30T00:55:55.825582+0000","last_peered":"2024-03-30T00:55:55.825582+0000","last_clean":"2024-03-30T00:55:55.825582+0000","last_became_active":"2024-03-30T00:55:51.809193+0000","last_became_peered":"2024-03-30T00:55:51.809193+0000","last_unstale":"2024-03-30T00:55:55.825582+0000","last_undegraded":"2024-03-30T00:55:55.825582+0000","last_fullsized":"2024-03-30T00:55:55.825582+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:52:41.078425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.833443+0000","last_change":"2024-03-30T00:55:55.830549+0000","last_active":"2024-03-30T00:55:55.833443+0000","last_peered":"2024-03-30T00:55:55.833443+0000","last_clean":"2024-03-30T00:55:55.833443+0000","last_became_active":"2024-03-30T00:55:55.827966+0000","last_became_peered":"2024-03-30T00:55:55.827966+0000","last_unstale":"2024-03-30T00:55:55.833443+0000","last_undegraded":"2024-03-30T00:55:55.833443+0000","last_fullsized":"2024-03-30T00:55:55.833443+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:30:36.531960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"76'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.812537+0000","last_change":"2024-03-30T00:55:51.311998+0000","last_active":"2024-03-30T00:55:55.812537+0000","last_peered":"2024-03-30T00:55:55.812537+0000","last_clean":"2024-03-30T00:55:55.812537+0000","last_became_active":"2024-03-30T00:55:51.311627+0000","last_became_peered":"2024-03-30T00:55:51.311627+0000","last_unstale":"2024-03-30T00:55:55.812537+0000","last_undegraded":"2024-03-30T00:55:55.812537+0000","last_fullsized":"2024-03-30T00:55:55.812537+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:23:06.939919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.823632+0000","last_change":"2024-03-30T00:55:51.819457+0000","last_active":"2024-03-30T00:55:55.823632+0000","last_peered":"2024-03-30T00:55:55.823632+0000","last_clean":"2024-03-30T00:55:55.823632+0000","last_became_active":"2024-03-30T00:55:51.819099+0000","last_became_peered":"2024-03-30T00:55:51.819099+0000","last_unstale":"2024-03-30T00:55:55.823632+0000","last_undegraded":"2024-03-30T00:55:55.823632+0000","last_fullsized":"2024-03-30T00:55:55.823632+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:31:57.363328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.826513+0000","last_change":"2024-03-30T00:55:55.824882+0000","last_active":"2024-03-30T00:55:55.826513+0000","last_peered":"2024-03-30T00:55:55.826513+0000","last_clean":"2024-03-30T00:55:55.826513+0000","last_became_active":"2024-03-30T00:55:55.824493+0000","last_became_peered":"2024-03-30T00:55:55.824493+0000","last_unstale":"2024-03-30T00:55:55.826513+0000","last_undegraded":"2024-03-30T00:55:55.826513+0000","last_fullsized":"2024-03-30T00:55:55.826513+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:24:48.825776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"76'2","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.811456+0000","last_change":"2024-03-30T00:55:51.311787+0000","last_active":"2024-03-30T00:55:55.811456+0000","last_peered":"2024-03-30T00:55:55.811456+0000","last_clean":"2024-03-30T00:55:55.811456+0000","last_became_active":"2024-03-30T00:55:51.311402+0000","last_became_peered":"2024-03-30T00:55:51.311402+0000","last_unstale":"2024-03-30T00:55:55.811456+0000","last_undegraded":"2024-03-30T00:55:55.811456+0000","last_fullsized":"2024-03-30T00:55:55.811456+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:37:13.960453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.824838+0000","last_change":"2024-03-30T00:55:51.809742+0000","last_active":"2024-03-30T00:55:55.824838+0000","last_peered":"2024-03-30T00:55:55.824838+0000","last_clean":"2024-03-30T00:55:55.824838+0000","last_became_active":"2024-03-30T00:55:51.809396+0000","last_became_peered":"2024-03-30T00:55:51.809396+0000","last_unstale":"2024-03-30T00:55:55.824838+0000","last_undegraded":"2024-03-30T00:55:55.824838+0000","last_fullsized":"2024-03-30T00:55:55.824838+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:22.535472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816499+0000","last_change":"2024-03-30T00:55:51.814429+0000","last_active":"2024-03-30T00:55:55.816499+0000","last_peered":"2024-03-30T00:55:55.816499+0000","last_clean":"2024-03-30T00:55:55.816499+0000","last_became_active":"2024-03-30T00:55:51.814044+0000","last_became_peered":"2024-03-30T00:55:51.814044+0000","last_unstale":"2024-03-30T00:55:55.816499+0000","last_undegraded":"2024-03-30T00:55:55.816499+0000","last_fullsized":"2024-03-30T00:55:55.816499+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:24:58.830738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817590+0000","last_change":"2024-03-30T00:55:51.828977+0000","last_active":"2024-03-30T00:55:55.817590+0000","last_peered":"2024-03-30T00:55:55.817590+0000","last_clean":"2024-03-30T00:55:55.817590+0000","last_became_active":"2024-03-30T00:55:51.828656+0000","last_became_peered":"2024-03-30T00:55:51.828656+0000","last_unstale":"2024-03-30T00:55:55.817590+0000","last_undegraded":"2024-03-30T00:55:55.817590+0000","last_fullsized":"2024-03-30T00:55:55.817590+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:10:23.463770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816065+0000","last_change":"2024-03-30T00:55:51.821511+0000","last_active":"2024-03-30T00:55:55.816065+0000","last_peered":"2024-03-30T00:55:55.816065+0000","last_clean":"2024-03-30T00:55:55.816065+0000","last_became_active":"2024-03-30T00:55:51.820876+0000","last_became_peered":"2024-03-30T00:55:51.820876+0000","last_unstale":"2024-03-30T00:55:55.816065+0000","last_undegraded":"2024-03-30T00:55:55.816065+0000","last_fullsized":"2024-03-30T00:55:55.816065+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:14:48.299376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.817918+0000","last_change":"2024-03-30T00:55:51.823867+0000","last_active":"2024-03-30T00:55:55.817918+0000","last_peered":"2024-03-30T00:55:55.817918+0000","last_clean":"2024-03-30T00:55:55.817918+0000","last_became_active":"2024-03-30T00:55:51.820230+0000","last_became_peered":"2024-03-30T00:55:51.820230+0000","last_unstale":"2024-03-30T00:55:55.817918+0000","last_undegraded":"2024-03-30T00:55:55.817918+0000","last_fullsized":"2024-03-30T00:55:55.817918+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:38:51.257855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131462+0000","last_change":"2024-03-30T00:55:51.821413+0000","last_active":"2024-03-30T00:55:56.131462+0000","last_peered":"2024-03-30T00:55:56.131462+0000","last_clean":"2024-03-30T00:55:56.131462+0000","last_became_active":"2024-03-30T00:55:51.820833+0000","last_became_peered":"2024-03-30T00:55:51.820833+0000","last_unstale":"2024-03-30T00:55:56.131462+0000","last_undegraded":"2024-03-30T00:55:56.131462+0000","last_fullsized":"2024-03-30T00:55:56.131462+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:34:11.969849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.815114+0000","last_change":"2024-03-30T00:55:51.813674+0000","last_active":"2024-03-30T00:55:55.815114+0000","last_peered":"2024-03-30T00:55:55.815114+0000","last_clean":"2024-03-30T00:55:55.815114+0000","last_became_active":"2024-03-30T00:55:51.813341+0000","last_became_peered":"2024-03-30T00:55:51.813341+0000","last_unstale":"2024-03-30T00:55:55.815114+0000","last_undegraded":"2024-03-30T00:55:55.815114+0000","last_fullsized":"2024-03-30T00:55:55.815114+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:54:17.285997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816461+0000","last_change":"2024-03-30T00:55:51.812472+0000","last_active":"2024-03-30T00:55:55.816461+0000","last_peered":"2024-03-30T00:55:55.816461+0000","last_clean":"2024-03-30T00:55:55.816461+0000","last_became_active":"2024-03-30T00:55:51.812061+0000","last_became_peered":"2024-03-30T00:55:51.812061+0000","last_unstale":"2024-03-30T00:55:55.816461+0000","last_undegraded":"2024-03-30T00:55:55.816461+0000","last_fullsized":"2024-03-30T00:55:55.816461+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:46.550201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141783+0000","last_change":"2024-03-30T00:55:51.829179+0000","last_active":"2024-03-30T00:55:56.141783+0000","last_peered":"2024-03-30T00:55:56.141783+0000","last_clean":"2024-03-30T00:55:56.141783+0000","last_became_active":"2024-03-30T00:55:51.828805+0000","last_became_peered":"2024-03-30T00:55:51.828805+0000","last_unstale":"2024-03-30T00:55:56.141783+0000","last_undegraded":"2024-03-30T00:55:56.141783+0000","last_fullsized":"2024-03-30T00:55:56.141783+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:23.576125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.813043+0000","last_change":"2024-03-30T00:55:51.823170+0000","last_active":"2024-03-30T00:55:55.813043+0000","last_peered":"2024-03-30T00:55:55.813043+0000","last_clean":"2024-03-30T00:55:55.813043+0000","last_became_active":"2024-03-30T00:55:51.822785+0000","last_became_peered":"2024-03-30T00:55:51.822785+0000","last_unstale":"2024-03-30T00:55:55.813043+0000","last_undegraded":"2024-03-30T00:55:55.813043+0000","last_fullsized":"2024-03-30T00:55:55.813043+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:34:43.393540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.141705+0000","last_change":"2024-03-30T00:55:51.817160+0000","last_active":"2024-03-30T00:55:56.141705+0000","last_peered":"2024-03-30T00:55:56.141705+0000","last_clean":"2024-03-30T00:55:56.141705+0000","last_became_active":"2024-03-30T00:55:51.816881+0000","last_became_peered":"2024-03-30T00:55:51.816881+0000","last_unstale":"2024-03-30T00:55:56.141705+0000","last_undegraded":"2024-03-30T00:55:56.141705+0000","last_fullsized":"2024-03-30T00:55:56.141705+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:09:05.211242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816796+0000","last_change":"2024-03-30T00:55:51.817670+0000","last_active":"2024-03-30T00:55:55.816796+0000","last_peered":"2024-03-30T00:55:55.816796+0000","last_clean":"2024-03-30T00:55:55.816796+0000","last_became_active":"2024-03-30T00:55:51.817380+0000","last_became_peered":"2024-03-30T00:55:51.817380+0000","last_unstale":"2024-03-30T00:55:55.816796+0000","last_undegraded":"2024-03-30T00:55:55.816796+0000","last_fullsized":"2024-03-30T00:55:55.816796+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:24.582557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:56.131396+0000","last_change":"2024-03-30T00:55:51.813071+0000","last_active":"2024-03-30T00:55:56.131396+0000","last_peered":"2024-03-30T00:55:56.131396+0000","last_clean":"2024-03-30T00:55:56.131396+0000","last_became_active":"2024-03-30T00:55:51.811681+0000","last_became_peered":"2024-03-30T00:55:51.811681+0000","last_unstale":"2024-03-30T00:55:56.131396+0000","last_undegraded":"2024-03-30T00:55:56.131396+0000","last_fullsized":"2024-03-30T00:55:56.131396+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:10:53.840055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-30T00:55:55.816334+0000","last_change":"2024-03-30T00:55:51.816716+0000","last_active":"2024-03-30T00:55:55.816334+0000","last_peered":"2024-03-30T00:55:55.816334+0000","last_clean":"2024-03-30T00:55:55.816334+0000","last_became_active":"2024-03-30T00:55:51.816383+0000","last_became_peered":"2024-03-30T00:55:51.816383+0000","last_unstale":"2024-03-30T00:55:55.816334+0000","last_undegraded":"2024-03-30T00:55:55.816334+0000","last_fullsized":"2024-03-30T00:55:55.816334+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:12:20.044088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":212960,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":42976,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":24,"num_read_kb":56,"num_write":120,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":113532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645328,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":439510,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.40300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.624},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.49399999999999999}]},{"osd":5,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.50600000000000001}]},{"osd":6,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.66300000000000003}]},{"osd":7,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.68000000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":9,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.76700000000000002}]},{"osd":10,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.80800000000000005}]}]},{"osd":10,"up_from":65,"seq":279172874259,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":1071315,"data_compressed":47572,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.60499999999999998}]},{"osd":4,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.57399999999999995}]},{"osd":5,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.62}]},{"osd":6,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.71299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":8,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.72699999999999998}]},{"osd":9,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.77400000000000002}]},{"osd":11,"last update":"Sat Mar 30 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.76200000000000001}]}]},{"osd":9,"up_from":60,"seq":257698037781,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":439712,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.51900000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.63400000000000001}]},{"osd":2,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.74399999999999999}]},{"osd":3,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.72099999999999997}]},{"osd":4,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.79500000000000004}]},{"osd":5,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":0.93500000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.623}]},{"osd":7,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.94599999999999995}]},{"osd":8,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.0109999999999999}]},{"osd":10,"last update":"Sat Mar 30 00:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.98899999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]}]},{"osd":8,"up_from":53,"seq":227633266714,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":880640,"data_stored":439201,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.46500000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.746},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.68300000000000005}]},{"osd":2,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.496},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.52700000000000002}]},{"osd":3,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.883,"5min":2.883,"15min":2.883},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.65300000000000002}]},{"osd":4,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.77700000000000002}]},{"osd":5,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.623}]},{"osd":6,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.8940000000000001,"5min":2.8940000000000001,"15min":2.8940000000000001},"last":0.64200000000000002}]},{"osd":7,"last update":"Sat Mar 30 00:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.70799999999999996}]},{"osd":9,"last update":"Sat Mar 30 00:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.79700000000000004}]},{"osd":10,"last update":"Sat Mar 30 00:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.85999999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]}]},{"osd":7,"up_from":46,"seq":197568495646,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":1035543,"data_compressed":47572,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60199999999999998}]},{"osd":1,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.45200000000000001,"15min":0.44800000000000001},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.60599999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.44700000000000001,"15min":0.441},"min":{"1min":0.32500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68799999999999994,"5min":0.747,"15min":0.747},"last":0.68700000000000006}]},{"osd":2,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.46700000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.57399999999999995,"15min":0.56100000000000005},"min":{"1min":0.51800000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.58399999999999996}]},{"osd":3,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.40400000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.89600000000000002,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61599999999999999,"15min":0.60999999999999999},"min":{"1min":0.53200000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89400000000000002,"5min":0.997,"15min":0.997},"last":0.65200000000000002}]},{"osd":4,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.36599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.39200000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.629,"15min":0.61899999999999999},"min":{"1min":0.505,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.63200000000000001,"15min":0.62},"min":{"1min":0.58099999999999996,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86199999999999999,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.80600000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.58799999999999997,"5min":0.372,"15min":0.372},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.745,"15min":0.72099999999999997},"min":{"1min":0.55300000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":4.117,"5min":4.117,"15min":4.117},"last":0.66900000000000004}]},{"osd":8,"last update":"Sat Mar 30 00:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.56299999999999994,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95199999999999996,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.755,"15min":0.752},"min":{"1min":0.53400000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.84199999999999997}]},{"osd":9,"last update":"Sat Mar 30 00:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":4.0519999999999996,"5min":4.0519999999999996,"15min":4.0519999999999996},"last":0.78500000000000003}]},{"osd":10,"last update":"Sat Mar 30 00:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.755},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.85599999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]}]},{"osd":6,"up_from":42,"seq":180388626466,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31968,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713184,"statfs":{"total":95995035648,"available":95962300416,"internally_reserved":0,"allocated":876544,"data_stored":438928,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.38900000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.505},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.49199999999999999,"15min":0.48399999999999999},"min":{"1min":0.42299999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69999999999999996,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.64500000000000002,"15min":0.629},"min":{"1min":0.58599999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.59599999999999997,"15min":0.59299999999999997},"min":{"1min":0.47599999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83999999999999997,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.61799999999999999}]},{"osd":2,"last update":"Sat Mar 30 00:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.69699999999999995,"15min":0.67900000000000005},"min":{"1min":0.56599999999999995,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.0229999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.65000000000000002,"15min":0.63800000000000001},"min":{"1min":0.57499999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85499999999999998,"5min":1.006,"15min":1.006},"last":0.69199999999999995}]},{"osd":3,"last update":"Sat Mar 30 00:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.49199999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.996,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.40799999999999997,"5min":0.375,"15min":0.375},"max":{"1min":0.84399999999999997,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.54900000000000004}]},{"osd":4,"last update":"Sat Mar 30 00:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.54900000000000004,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.82099999999999995,"5min":1.006,"15min":1.006},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.75700000000000001,"15min":0.749},"min":{"1min":0.60299999999999998,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.006,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.77800000000000002}]},{"osd":5,"last update":"Sat Mar 30 00:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.69299999999999995,"15min":0.68200000000000005},"min":{"1min":0.54900000000000004,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.93400000000000005,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.73999999999999999,"15min":0.72999999999999998},"min":{"1min":0.53400000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.93799999999999994,"5min":1.091,"15min":1.091},"last":0.79400000000000004}]},{"osd":7,"last update":"Sat Mar 30 00:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81799999999999995,"15min":0.81399999999999995},"min":{"1min":0.64300000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.1200000000000001,"5min":1.127,"15min":1.127},"last":0.877},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.77600000000000002,"15min":0.77200000000000002},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.66600000000000004}]},{"osd":8,"last update":"Sat Mar 30 00:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.96199999999999997}]},{"osd":9,"last update":"Sat Mar 30 00:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.72999999999999998}]},{"osd":10,"last update":"Sat Mar 30 00:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.94299999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":1,"up_from":12,"seq":51539607604,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":449053,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55500000000000005,"15min":0.53500000000000003},"min":{"1min":0.42399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72099999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.499},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.58099999999999996,"15min":0.55300000000000005},"min":{"1min":0.47199999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.79100000000000004,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.59899999999999998}]},{"osd":2,"last update":"Sat Mar 30 00:55:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.505,"15min":0.48499999999999999},"min":{"1min":0.39400000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.50700000000000001,"15min":0.49099999999999999},"min":{"1min":0.39300000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.56200000000000006}]},{"osd":3,"last update":"Sat Mar 30 00:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.55000000000000004,"15min":0.52400000000000002},"min":{"1min":0.40799999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82299999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.61899999999999999}]},{"osd":4,"last update":"Sat Mar 30 00:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.41799999999999998,"15min":0.40799999999999997},"min":{"1min":0.33500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.61899999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.48199999999999998,"15min":0.46600000000000003},"min":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.57599999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:55:30 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.60899999999999999,"15min":0.61299999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63500000000000001,"15min":0.63400000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.63300000000000001}]},{"osd":6,"last update":"Sat Mar 30 00:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.877,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67900000000000005,"15min":0.68200000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.72799999999999998}]},{"osd":7,"last update":"Sat Mar 30 00:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55400000000000005,"15min":0.55100000000000005},"min":{"1min":0.35799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79400000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.84899999999999998,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.68700000000000006}]},{"osd":8,"last update":"Sat Mar 30 00:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.81599999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.89000000000000001}]},{"osd":10,"last update":"Sat Mar 30 00:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.80400000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31980,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713172,"statfs":{"total":95995035648,"available":95962288128,"internally_reserved":0,"allocated":876544,"data_stored":438745,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 00:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.504,"5min":0.372,"15min":0.372},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.42099999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.752}]},{"osd":2,"last update":"Sat Mar 30 00:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.53600000000000003,"15min":0.52300000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.627},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.57399999999999995,"15min":0.55600000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.72499999999999998}]},{"osd":3,"last update":"Sat Mar 30 00:54:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45200000000000001,"15min":0.436},"min":{"1min":0.35899999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.379,"5min":0.317,"15min":0.317},"max":{"1min":0.69499999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.56699999999999995}]},{"osd":4,"last update":"Sat Mar 30 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65100000000000002,"15min":0.623},"min":{"1min":0.503,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90000000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.64100000000000001,"15min":0.60999999999999999},"min":{"1min":0.51400000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.93300000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.93100000000000005}]},{"osd":5,"last update":"Sat Mar 30 00:55:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.65000000000000002,"15min":0.63500000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.63200000000000001,"15min":0.61799999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.84499999999999997}]},{"osd":6,"last update":"Sat Mar 30 00:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.41999999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64200000000000002,"15min":0.65000000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.998}]},{"osd":7,"last update":"Sat Mar 30 00:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.96799999999999997}]},{"osd":8,"last update":"Sat Mar 30 00:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":1.026}]},{"osd":9,"last update":"Sat Mar 30 00:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":1.0760000000000001}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":1.016},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":1.0369999999999999}]},{"osd":11,"last update":"Sat Mar 30 00:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":1.107},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":1.0620000000000001}]}]},{"osd":2,"up_from":17,"seq":73014444082,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":480381,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.39000000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78100000000000003,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54000000000000004,"15min":0.52500000000000002},"min":{"1min":0.38,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.82199999999999995,"5min":1.097,"15min":1.097},"last":0.38300000000000001}]},{"osd":1,"last update":"Sat Mar 30 00:55:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53500000000000003,"15min":0.52600000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.50600000000000001}]},{"osd":3,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57899999999999996,"15min":0.55900000000000005},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.58899999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.42199999999999999}]},{"osd":4,"last update":"Sat Mar 30 00:55:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.39700000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.52000000000000002,"15min":0.51500000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Mar 30 00:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.439,"15min":0.42999999999999999},"min":{"1min":0.317,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.46100000000000002,"15min":0.45400000000000001},"min":{"1min":0.28599999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.32500000000000001}]},{"osd":6,"last update":"Sat Mar 30 00:55:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64700000000000002,"15min":0.64900000000000002},"min":{"1min":0.44500000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.40899999999999997}]},{"osd":7,"last update":"Sat Mar 30 00:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.66500000000000004}]},{"osd":8,"last update":"Sat Mar 30 00:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.52200000000000002}]},{"osd":9,"last update":"Sat Mar 30 00:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.59599999999999997}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.67400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]}]},{"osd":3,"up_from":24,"seq":103079215149,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":438849,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.437,"15min":0.40600000000000003},"min":{"1min":0.35399999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.45000000000000001,"15min":0.42799999999999999},"min":{"1min":0.35199999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.76800000000000002,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.497}]},{"osd":1,"last update":"Sat Mar 30 00:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.63,"15min":0.59599999999999997},"min":{"1min":0.51400000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Mar 30 00:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.64900000000000002,"15min":0.621},"min":{"1min":0.45800000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.752},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63400000000000001,"15min":0.61699999999999999},"min":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.64600000000000002}]},{"osd":4,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.69499999999999995,"15min":0.66300000000000003},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0600000000000001,"5min":1.113,"15min":1.113},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67500000000000004,"15min":0.66400000000000003},"min":{"1min":0.501,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.83399999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.46999999999999997}]},{"osd":5,"last update":"Sat Mar 30 00:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.65600000000000003,"15min":0.64600000000000002},"min":{"1min":0.48699999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.68500000000000005,"15min":0.65900000000000003},"min":{"1min":0.56499999999999995,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64000000000000001,"15min":0.63500000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64800000000000002,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.57099999999999995}]},{"osd":7,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.97899999999999998,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.621},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83999999999999997,"15min":0.84099999999999997},"min":{"1min":0.66000000000000003,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.1890000000000001,"5min":1.292,"15min":1.292},"last":0.72399999999999998}]},{"osd":8,"last update":"Sat Mar 30 00:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.57399999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.74399999999999999}]},{"osd":10,"last update":"Sat Mar 30 00:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.81399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986218,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":880640,"data_stored":439313,"data_compressed":11805,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.66700000000000004,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.79300000000000004,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73099999999999998,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39700000000000002,"15min":0.39100000000000001},"min":{"1min":0.29999999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.61199999999999999,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.32000000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.46899999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.439}]},{"osd":2,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.55400000000000005,"15min":0.54500000000000004},"min":{"1min":0.36399999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81799999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.55400000000000005,"15min":0.53900000000000003},"min":{"1min":0.441,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76000000000000001,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.59799999999999998}]},{"osd":3,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.66000000000000003,"15min":0.69799999999999995},"min":{"1min":0.42399999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.72999999999999998,"5min":3.887,"15min":3.887},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.39600000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78400000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.80300000000000005}]},{"osd":5,"last update":"Sat Mar 30 00:55:27 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.57099999999999995,"15min":0.56299999999999994},"min":{"1min":0.38200000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.65200000000000002}]},{"osd":6,"last update":"Sat Mar 30 00:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.627,"15min":0.626},"min":{"1min":0.47399999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65400000000000003,"15min":0.65300000000000002},"min":{"1min":0.51100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.82599999999999996}]},{"osd":7,"last update":"Sat Mar 30 00:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54400000000000004,"15min":0.54000000000000004},"min":{"1min":0.35199999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.79500000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52700000000000002,"15min":0.52100000000000002},"min":{"1min":0.34599999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57799999999999996}]},{"osd":8,"last update":"Sat Mar 30 00:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.624},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.73999999999999999}]},{"osd":9,"last update":"Sat Mar 30 00:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.77100000000000002}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.79200000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]}]},{"osd":5,"up_from":35,"seq":150323855397,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":933888,"data_stored":1029147,"data_compressed":47572,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.50600000000000001,"15min":0.499},"min":{"1min":0.40799999999999997,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.751,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.504},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.5,"15min":0.49399999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:55:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.747},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.40200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.72899999999999998}]},{"osd":2,"last update":"Sat Mar 30 00:55:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.28299999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38900000000000001,"15min":0.39300000000000002},"min":{"1min":0.30099999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.60999999999999999,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.59499999999999997}]},{"osd":3,"last update":"Sat Mar 30 00:55:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.40799999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53100000000000003,"15min":0.52700000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.65100000000000002}]},{"osd":4,"last update":"Sat Mar 30 00:55:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.68300000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.40200000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.85399999999999998,"5min":1.038,"15min":1.038},"last":0.67600000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.46500000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89700000000000002,"5min":0.997,"15min":0.997},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.49199999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.80500000000000005,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.70799999999999996}]},{"osd":7,"last update":"Sat Mar 30 00:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.42699999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82199999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.63600000000000001,"15min":0.63700000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89500000000000002,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.82299999999999995}]},{"osd":8,"last update":"Sat Mar 30 00:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.81200000000000006}]},{"osd":9,"last update":"Sat Mar 30 00:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.86099999999999999}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.874},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.89800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":42122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":42350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T00:56:26.558 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-30T00:56:26.562 INFO:tasks.cephadm:Setting up client nodes... 2024-03-30T00:56:26.563 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-30T00:56:26.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:26 smithi116 ceph-mon[38086]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:26.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:26 smithi116 ceph-mon[38086]: pgmap v246: 129 pgs: 129 active+clean; 619 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2024-03-30T00:56:26.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:26 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/331555480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:26.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:26 smithi181 ceph-mon[47622]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:26.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:26 smithi181 ceph-mon[47622]: pgmap v246: 129 pgs: 129 active+clean; 619 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2024-03-30T00:56:26.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:26 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/331555480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:26.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:26 smithi130 ceph-mon[47626]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:56:26.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:26 smithi130 ceph-mon[47626]: pgmap v246: 129 pgs: 129 active+clean; 619 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.7 KiB/s wr, 8 op/s 2024-03-30T00:56:26.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:26 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/331555480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:26.952 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:27.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:27 smithi181 ceph-mon[47622]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:56:27.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:27 smithi181 ceph-mon[47622]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:56:27.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:27 smithi130 ceph-mon[47626]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:56:27.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:27 smithi130 ceph-mon[47626]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:56:27.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:27 smithi116 ceph-mon[38086]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:56:27.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:27 smithi116 ceph-mon[38086]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:56:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.110+0000 7f454a600700 1 -- 172.21.15.116:0/2243798192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4544100dc0 msgr2=0x7f45441011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.110+0000 7f454a600700 1 --2- 172.21.15.116:0/2243798192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4544100dc0 0x7f45441011e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f45340099c0 tx=0x7f4534009cd0 comp rx=0 tx=0).stop 2024-03-30T00:56:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.110+0000 7f454a600700 1 -- 172.21.15.116:0/2243798192 shutdown_connections 2024-03-30T00:56:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.110+0000 7f454a600700 1 --2- 172.21.15.116:0/2243798192 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4544102960 0x7f4544109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.110+0000 7f454a600700 1 --2- 172.21.15.116:0/2243798192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4544100dc0 0x7f45441011e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.110+0000 7f454a600700 1 --2- 172.21.15.116:0/2243798192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4544101fc0 0x7f4544102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.110+0000 7f454a600700 1 -- 172.21.15.116:0/2243798192 >> 172.21.15.116:0/2243798192 conn(0x7f45440fc360 msgr2=0x7f45440fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.110+0000 7f454a600700 1 -- 172.21.15.116:0/2243798192 shutdown_connections 2024-03-30T00:56:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.110+0000 7f454a600700 1 -- 172.21.15.116:0/2243798192 wait complete. 2024-03-30T00:56:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.111+0000 7f454a600700 1 Processor -- start 2024-03-30T00:56:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.111+0000 7f454a600700 1 -- start start 2024-03-30T00:56:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.111+0000 7f454a600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4544100dc0 0x7f4544074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.111+0000 7f454a600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4544101fc0 0x7f45440731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.111+0000 7f454a600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4544102960 0x7f45440736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.111+0000 7f454a600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4544073e80 con 0x7f4544102960 2024-03-30T00:56:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.111+0000 7f454a600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4544073ff0 con 0x7f4544100dc0 2024-03-30T00:56:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.111+0000 7f454a600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441a0dd0 con 0x7f4544101fc0 2024-03-30T00:56:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.112+0000 7f4542a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4544101fc0 0x7f45440731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.112+0000 7f4543e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4544102960 0x7f45440736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.112+0000 7f4542a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4544101fc0 0x7f45440731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59002/0 (socket says 172.21.15.116:59002) 2024-03-30T00:56:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.112+0000 7f4543400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4544100dc0 0x7f4544074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.112+0000 7f4542a00700 1 -- 172.21.15.116:0/490310293 learned_addr learned my addr 172.21.15.116:0/490310293 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.112+0000 7f4542a00700 1 -- 172.21.15.116:0/490310293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4544100dc0 msgr2=0x7f4544074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:28.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.112+0000 7f4542a00700 1 --2- 172.21.15.116:0/490310293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4544100dc0 0x7f4544074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:28.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.112+0000 7f4542a00700 1 -- 172.21.15.116:0/490310293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4544102960 msgr2=0x7f45440736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:28.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.112+0000 7f4542a00700 1 --2- 172.21.15.116:0/490310293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4544102960 0x7f45440736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:28.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.112+0000 7f4542a00700 1 -- 172.21.15.116:0/490310293 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45340095f0 con 0x7f4544101fc0 2024-03-30T00:56:28.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.112+0000 7f4542a00700 1 --2- 172.21.15.116:0/490310293 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4544101fc0 0x7f45440731a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f453000e9a0 tx=0x7f453000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:28.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.113+0000 7f453be00700 1 -- 172.21.15.116:0/490310293 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530009790 con 0x7f4544101fc0 2024-03-30T00:56:28.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.113+0000 7f453be00700 1 -- 172.21.15.116:0/490310293 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4530010430 con 0x7f4544101fc0 2024-03-30T00:56:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.113+0000 7f454a600700 1 -- 172.21.15.116:0/490310293 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45441a1060 con 0x7f4544101fc0 2024-03-30T00:56:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.113+0000 7f453be00700 1 -- 172.21.15.116:0/490310293 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530018680 con 0x7f4544101fc0 2024-03-30T00:56:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.113+0000 7f454a600700 1 -- 172.21.15.116:0/490310293 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45441a1560 con 0x7f4544101fc0 2024-03-30T00:56:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.114+0000 7f454a600700 1 -- 172.21.15.116:0/490310293 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45441082c0 con 0x7f4544101fc0 2024-03-30T00:56:28.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.116+0000 7f453be00700 1 -- 172.21.15.116:0/490310293 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f45300105a0 con 0x7f4544101fc0 2024-03-30T00:56:28.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.117+0000 7f453be00700 1 --2- 172.21.15.116:0/490310293 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f45280a11b0 0x7f45280a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:28.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.117+0000 7f453be00700 1 -- 172.21.15.116:0/490310293 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4530014070 con 0x7f4544101fc0 2024-03-30T00:56:28.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.117+0000 7f4543400700 1 --2- 172.21.15.116:0/490310293 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f45280a11b0 0x7f45280a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:28.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.118+0000 7f4543400700 1 --2- 172.21.15.116:0/490310293 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f45280a11b0 0x7f45280a3670 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4534009990 tx=0x7f4534000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:28.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.119+0000 7f453be00700 1 -- 172.21.15.116:0/490310293 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4530096420 con 0x7f4544101fc0 2024-03-30T00:56:28.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.403+0000 7f454a600700 1 -- 172.21.15.116:0/490310293 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f454404ea80 con 0x7f4544101fc0 2024-03-30T00:56:28.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.409+0000 7f453be00700 1 -- 172.21.15.116:0/490310293 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7f45300260f0 con 0x7f4544101fc0 2024-03-30T00:56:28.416 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-03-30T00:56:28.416 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA8YwdmTiYxGBAAbMEkLjoyPJuSR2o8Y4HOYA== 2024-03-30T00:56:28.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.411+0000 7f454a600700 1 -- 172.21.15.116:0/490310293 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f45280a11b0 msgr2=0x7f45280a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.411+0000 7f454a600700 1 --2- 172.21.15.116:0/490310293 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f45280a11b0 0x7f45280a3670 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4534009990 tx=0x7f4534000bc0 comp rx=0 tx=0).stop 2024-03-30T00:56:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.411+0000 7f454a600700 1 -- 172.21.15.116:0/490310293 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4544101fc0 msgr2=0x7f45440731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.411+0000 7f454a600700 1 --2- 172.21.15.116:0/490310293 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4544101fc0 0x7f45440731a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f453000e9a0 tx=0x7f453000ecb0 comp rx=0 tx=0).stop 2024-03-30T00:56:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.411+0000 7f454a600700 1 -- 172.21.15.116:0/490310293 shutdown_connections 2024-03-30T00:56:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.411+0000 7f454a600700 1 --2- 172.21.15.116:0/490310293 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f45280a11b0 0x7f45280a3670 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.411+0000 7f454a600700 1 --2- 172.21.15.116:0/490310293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4544102960 0x7f45440736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.411+0000 7f454a600700 1 --2- 172.21.15.116:0/490310293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4544100dc0 0x7f4544074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.411+0000 7f454a600700 1 --2- 172.21.15.116:0/490310293 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4544101fc0 0x7f45440731a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.411+0000 7f454a600700 1 -- 172.21.15.116:0/490310293 >> 172.21.15.116:0/490310293 conn(0x7f45440fc360 msgr2=0x7f4544105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.411+0000 7f454a600700 1 -- 172.21.15.116:0/490310293 shutdown_connections 2024-03-30T00:56:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:28.412+0000 7f454a600700 1 -- 172.21.15.116:0/490310293 wait complete. 2024-03-30T00:56:28.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:28 smithi181 ceph-mon[47622]: pgmap v247: 129 pgs: 129 active+clean; 620 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2024-03-30T00:56:28.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:28 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/490310293' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T00:56:28.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:28 smithi181 ceph-mon[47622]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T00:56:28.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:28 smithi181 ceph-mon[47622]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T00:56:28.794 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:28 smithi116 ceph-mon[38086]: pgmap v247: 129 pgs: 129 active+clean; 620 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2024-03-30T00:56:28.794 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:28 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/490310293' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T00:56:28.794 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:28 smithi116 ceph-mon[38086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T00:56:28.794 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:28 smithi116 ceph-mon[38086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T00:56:28.821 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:56:28.821 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-30T00:56:28.821 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-30T00:56:28.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:28 smithi130 ceph-mon[47626]: pgmap v247: 129 pgs: 129 active+clean; 620 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2024-03-30T00:56:28.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:28 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/490310293' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T00:56:28.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:28 smithi130 ceph-mon[47626]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T00:56:28.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:28 smithi130 ceph-mon[47626]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T00:56:28.916 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-30T00:56:28.916 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-30T00:56:28.916 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph mgr dump --format=json 2024-03-30T00:56:29.320 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:30.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.481+0000 7f6093800700 1 -- 172.21.15.116:0/1163411042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f608c106ef0 msgr2=0x7f608c1072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:30.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.481+0000 7f6093800700 1 --2- 172.21.15.116:0/1163411042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f608c106ef0 0x7f608c1072d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f60800099c0 tx=0x7f6080009cd0 comp rx=0 tx=0).stop 2024-03-30T00:56:30.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.481+0000 7f6093800700 1 -- 172.21.15.116:0/1163411042 shutdown_connections 2024-03-30T00:56:30.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.481+0000 7f6093800700 1 --2- 172.21.15.116:0/1163411042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f608c101820 0x7f608c10dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:30.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.481+0000 7f6093800700 1 --2- 172.21.15.116:0/1163411042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f608c106ef0 0x7f608c1072d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:30.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.481+0000 7f6093800700 1 --2- 172.21.15.116:0/1163411042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f608c100e60 0x7f608c1012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:30.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.481+0000 7f6093800700 1 -- 172.21.15.116:0/1163411042 >> 172.21.15.116:0/1163411042 conn(0x7f608c077fa0 msgr2=0x7f608c0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:30.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.481+0000 7f6093800700 1 -- 172.21.15.116:0/1163411042 shutdown_connections 2024-03-30T00:56:30.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.481+0000 7f6093800700 1 -- 172.21.15.116:0/1163411042 wait complete. 2024-03-30T00:56:30.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.482+0000 7f6093800700 1 Processor -- start 2024-03-30T00:56:30.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.482+0000 7f6093800700 1 -- start start 2024-03-30T00:56:30.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.483+0000 7f6093800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f608c100e60 0x7f608c1994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:30.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.483+0000 7f6093800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f608c101820 0x7f608c1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:30.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.483+0000 7f6093800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f608c106ef0 0x7f608c19dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:30.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.483+0000 7f6093800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c19e3a0 con 0x7f608c106ef0 2024-03-30T00:56:30.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.483+0000 7f6093800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c19e510 con 0x7f608c101820 2024-03-30T00:56:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.483+0000 7f6093800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c19e6d0 con 0x7f608c100e60 2024-03-30T00:56:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.483+0000 7f6090e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f608c100e60 0x7f608c1994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.483+0000 7f6090e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f608c100e60 0x7f608c1994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59014/0 (socket says 172.21.15.116:59014) 2024-03-30T00:56:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.483+0000 7f6090e00700 1 -- 172.21.15.116:0/886481577 learned_addr learned my addr 172.21.15.116:0/886481577 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.483+0000 7f608be00700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f608c101820 0x7f608c1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.483+0000 7f6091800700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f608c106ef0 0x7f608c19dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.483+0000 7f6090e00700 1 -- 172.21.15.116:0/886481577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f608c101820 msgr2=0x7f608c1999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.483+0000 7f6090e00700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f608c101820 0x7f608c1999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.484+0000 7f6090e00700 1 -- 172.21.15.116:0/886481577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f608c106ef0 msgr2=0x7f608c19dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.484+0000 7f6090e00700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f608c106ef0 0x7f608c19dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.484+0000 7f6090e00700 1 -- 172.21.15.116:0/886481577 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60800095f0 con 0x7f608c100e60 2024-03-30T00:56:30.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.484+0000 7f608be00700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f608c101820 0x7f608c1999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.484+0000 7f6090e00700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f608c100e60 0x7f608c1994a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f608000b580 tx=0x7f6080005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.484+0000 7f6089600700 1 -- 172.21.15.116:0/886481577 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608001d070 con 0x7f608c100e60 2024-03-30T00:56:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.484+0000 7f6089600700 1 -- 172.21.15.116:0/886481577 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f608000fb30 con 0x7f608c100e60 2024-03-30T00:56:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.484+0000 7f6093800700 1 -- 172.21.15.116:0/886481577 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f608c19e950 con 0x7f608c100e60 2024-03-30T00:56:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.484+0000 7f6089600700 1 -- 172.21.15.116:0/886481577 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6080022ab0 con 0x7f608c100e60 2024-03-30T00:56:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.484+0000 7f6093800700 1 -- 172.21.15.116:0/886481577 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f608c1a6210 con 0x7f608c100e60 2024-03-30T00:56:30.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.485+0000 7f6093800700 1 -- 172.21.15.116:0/886481577 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608c104180 con 0x7f608c100e60 2024-03-30T00:56:30.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.487+0000 7f6089600700 1 -- 172.21.15.116:0/886481577 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f608000fca0 con 0x7f608c100e60 2024-03-30T00:56:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.488+0000 7f6089600700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f60740a11b0 0x7f60740a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.488+0000 7f6089600700 1 -- 172.21.15.116:0/886481577 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f60800ce3a0 con 0x7f608c100e60 2024-03-30T00:56:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.488+0000 7f608be00700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f60740a11b0 0x7f60740a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.489+0000 7f608be00700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f60740a11b0 0x7f60740a3670 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f608c19aa00 tx=0x7f607800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:30.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.491+0000 7f6089600700 1 -- 172.21.15.116:0/886481577 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6080097930 con 0x7f608c100e60 2024-03-30T00:56:30.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:30 smithi116 ceph-mon[38086]: pgmap v248: 129 pgs: 129 active+clean; 620 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-30T00:56:30.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:30 smithi181 ceph-mon[47622]: pgmap v248: 129 pgs: 129 active+clean; 620 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-30T00:56:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.749+0000 7f6093800700 1 -- 172.21.15.116:0/886481577 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f608c04ea80 con 0x7f608c100e60 2024-03-30T00:56:30.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.754+0000 7f6089600700 1 -- 172.21.15.116:0/886481577 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260283 (secure 0 0 0) 0x7f6080027070 con 0x7f608c100e60 2024-03-30T00:56:30.765 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:56:30.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.758+0000 7f6093800700 1 -- 172.21.15.116:0/886481577 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f60740a11b0 msgr2=0x7f60740a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:30.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.758+0000 7f6093800700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f60740a11b0 0x7f60740a3670 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f608c19aa00 tx=0x7f607800b410 comp rx=0 tx=0).stop 2024-03-30T00:56:30.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.758+0000 7f6093800700 1 -- 172.21.15.116:0/886481577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f608c100e60 msgr2=0x7f608c1994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:30.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.758+0000 7f6093800700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f608c100e60 0x7f608c1994a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f608000b580 tx=0x7f6080005740 comp rx=0 tx=0).stop 2024-03-30T00:56:30.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.758+0000 7f6093800700 1 -- 172.21.15.116:0/886481577 shutdown_connections 2024-03-30T00:56:30.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.758+0000 7f6093800700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f60740a11b0 0x7f60740a3670 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:30.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.758+0000 7f6093800700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f608c100e60 0x7f608c1994a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:30.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.758+0000 7f6093800700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f608c106ef0 0x7f608c19dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:30.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.758+0000 7f6093800700 1 --2- 172.21.15.116:0/886481577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f608c101820 0x7f608c1999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:30.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.758+0000 7f6093800700 1 -- 172.21.15.116:0/886481577 >> 172.21.15.116:0/886481577 conn(0x7f608c077fa0 msgr2=0x7f608c10c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:30.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.758+0000 7f6093800700 1 -- 172.21.15.116:0/886481577 shutdown_connections 2024-03-30T00:56:30.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:30.758+0000 7f6093800700 1 -- 172.21.15.116:0/886481577 wait complete. 2024-03-30T00:56:30.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:30 smithi130 ceph-mon[47626]: pgmap v248: 129 pgs: 129 active+clean; 620 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-30T00:56:31.166 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3072243062},{"type":"v1","addr":"172.21.15.116:6801","nonce":3072243062}]},"active_addr":"172.21.15.116:6801/3072243062","active_change":"2024-03-30T00:50:35.029597+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.116:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":1638454568}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":1330843846}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":721879338}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":2494915144}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":838025850}]}]} 2024-03-30T00:56:31.172 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-30T00:56:31.172 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-30T00:56:31.173 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T00:56:31.536 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:31.694 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:31 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/886481577' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T00:56:31.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:31 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/886481577' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T00:56:31.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:31 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/886481577' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T00:56:32.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.649+0000 7f9a22600700 1 -- 172.21.15.116:0/2175315508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a1c073900 msgr2=0x7f9a1c10d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:32.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.649+0000 7f9a22600700 1 --2- 172.21.15.116:0/2175315508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a1c073900 0x7f9a1c10d300 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009990 tx=0x7f9a04009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:32.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.649+0000 7f9a22600700 1 -- 172.21.15.116:0/2175315508 shutdown_connections 2024-03-30T00:56:32.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.649+0000 7f9a22600700 1 --2- 172.21.15.116:0/2175315508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a1c10d840 0x7f9a1c10fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:32.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.649+0000 7f9a22600700 1 --2- 172.21.15.116:0/2175315508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a1c072fe0 0x7f9a1c0733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:32.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.649+0000 7f9a22600700 1 --2- 172.21.15.116:0/2175315508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a1c073900 0x7f9a1c10d300 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:32.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.649+0000 7f9a22600700 1 -- 172.21.15.116:0/2175315508 >> 172.21.15.116:0/2175315508 conn(0x7f9a1c077e90 msgr2=0x7f9a1c0782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:32.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.649+0000 7f9a22600700 1 -- 172.21.15.116:0/2175315508 shutdown_connections 2024-03-30T00:56:32.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.649+0000 7f9a22600700 1 -- 172.21.15.116:0/2175315508 wait complete. 2024-03-30T00:56:32.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.650+0000 7f9a22600700 1 Processor -- start 2024-03-30T00:56:32.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.650+0000 7f9a22600700 1 -- start start 2024-03-30T00:56:32.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.650+0000 7f9a22600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a1c072fe0 0x7f9a1c1993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:32.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a22600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a1c073900 0x7f9a1c199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:32.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a22600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a1c10d840 0x7f9a1c19dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:32.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a22600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c19e2d0 con 0x7f9a1c072fe0 2024-03-30T00:56:32.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a22600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c19e440 con 0x7f9a1c073900 2024-03-30T00:56:32.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a22600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c19e600 con 0x7f9a1c10d840 2024-03-30T00:56:32.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a1b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a1c072fe0 0x7f9a1c1993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:32.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a1b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a1c072fe0 0x7f9a1c1993d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37974/0 (socket says 172.21.15.116:37974) 2024-03-30T00:56:32.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a1b400700 1 -- 172.21.15.116:0/1545582365 learned_addr learned my addr 172.21.15.116:0/1545582365 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:32.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a1be00700 1 --2- 172.21.15.116:0/1545582365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a1c10d840 0x7f9a1c19dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:32.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a13e00700 1 --2- 172.21.15.116:0/1545582365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a1c073900 0x7f9a1c199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:32.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a1b400700 1 -- 172.21.15.116:0/1545582365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a1c10d840 msgr2=0x7f9a1c19dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:32.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a1b400700 1 --2- 172.21.15.116:0/1545582365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a1c10d840 0x7f9a1c19dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:32.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a1b400700 1 -- 172.21.15.116:0/1545582365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a1c073900 msgr2=0x7f9a1c199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:32.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a1b400700 1 --2- 172.21.15.116:0/1545582365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a1c073900 0x7f9a1c199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:32.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a1b400700 1 -- 172.21.15.116:0/1545582365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a040095f0 con 0x7f9a1c072fe0 2024-03-30T00:56:32.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.651+0000 7f9a1b400700 1 --2- 172.21.15.116:0/1545582365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a1c072fe0 0x7f9a1c1993d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c00d980 tx=0x7f9a0c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:32.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.652+0000 7f9a18c00700 1 -- 172.21.15.116:0/1545582365 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c004220 con 0x7f9a1c072fe0 2024-03-30T00:56:32.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.652+0000 7f9a18c00700 1 -- 172.21.15.116:0/1545582365 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a0c007640 con 0x7f9a1c072fe0 2024-03-30T00:56:32.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.652+0000 7f9a22600700 1 -- 172.21.15.116:0/1545582365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1c19e8e0 con 0x7f9a1c072fe0 2024-03-30T00:56:32.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.652+0000 7f9a18c00700 1 -- 172.21.15.116:0/1545582365 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c005690 con 0x7f9a1c072fe0 2024-03-30T00:56:32.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.652+0000 7f9a22600700 1 -- 172.21.15.116:0/1545582365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1c1a6140 con 0x7f9a1c072fe0 2024-03-30T00:56:32.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.653+0000 7f9a22600700 1 -- 172.21.15.116:0/1545582365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a1c108a50 con 0x7f9a1c072fe0 2024-03-30T00:56:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.655+0000 7f9a18c00700 1 -- 172.21.15.116:0/1545582365 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f9a0c004750 con 0x7f9a1c072fe0 2024-03-30T00:56:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.656+0000 7f9a18c00700 1 --2- 172.21.15.116:0/1545582365 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9a000a11b0 0x7f9a000a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.656+0000 7f9a18c00700 1 -- 172.21.15.116:0/1545582365 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9a0c0cca40 con 0x7f9a1c072fe0 2024-03-30T00:56:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.656+0000 7f9a13e00700 1 --2- 172.21.15.116:0/1545582365 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9a000a11b0 0x7f9a000a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.657+0000 7f9a13e00700 1 --2- 172.21.15.116:0/1545582365 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9a000a11b0 0x7f9a000a3670 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c19a930 tx=0x7f9a0400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.660+0000 7f9a18c00700 1 -- 172.21.15.116:0/1545582365 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9a0c095e20 con 0x7f9a1c072fe0 2024-03-30T00:56:32.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:32 smithi116 ceph-mon[38086]: pgmap v249: 129 pgs: 129 active+clean; 626 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-30T00:56:32.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:32 smithi181 ceph-mon[47622]: pgmap v249: 129 pgs: 129 active+clean; 626 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-30T00:56:32.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:32 smithi130 ceph-mon[47626]: pgmap v249: 129 pgs: 129 active+clean; 626 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-30T00:56:32.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.863+0000 7f9a22600700 1 -- 172.21.15.116:0/1545582365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a1c19a6e0 con 0x7f9a1c072fe0 2024-03-30T00:56:32.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.865+0000 7f9a18c00700 1 -- 172.21.15.116:0/1545582365 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+24028 (secure 0 0 0) 0x7f9a0c016020 con 0x7f9a1c072fe0 2024-03-30T00:56:32.872 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:56:32.872 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:55:56.324776+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:56:32.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.867+0000 7f9a22600700 1 -- 172.21.15.116:0/1545582365 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9a000a11b0 msgr2=0x7f9a000a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:32.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.867+0000 7f9a22600700 1 --2- 172.21.15.116:0/1545582365 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9a000a11b0 0x7f9a000a3670 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c19a930 tx=0x7f9a0400b260 comp rx=0 tx=0).stop 2024-03-30T00:56:32.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.867+0000 7f9a22600700 1 -- 172.21.15.116:0/1545582365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a1c072fe0 msgr2=0x7f9a1c1993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:32.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.867+0000 7f9a22600700 1 --2- 172.21.15.116:0/1545582365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a1c072fe0 0x7f9a1c1993d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c00d980 tx=0x7f9a0c00dc90 comp rx=0 tx=0).stop 2024-03-30T00:56:32.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.867+0000 7f9a22600700 1 -- 172.21.15.116:0/1545582365 shutdown_connections 2024-03-30T00:56:32.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.867+0000 7f9a22600700 1 --2- 172.21.15.116:0/1545582365 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9a000a11b0 0x7f9a000a3670 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:32.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.867+0000 7f9a22600700 1 --2- 172.21.15.116:0/1545582365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a1c10d840 0x7f9a1c19dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:32.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.867+0000 7f9a22600700 1 --2- 172.21.15.116:0/1545582365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a1c072fe0 0x7f9a1c1993d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:32.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.867+0000 7f9a22600700 1 --2- 172.21.15.116:0/1545582365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a1c073900 0x7f9a1c199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:32.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.867+0000 7f9a22600700 1 -- 172.21.15.116:0/1545582365 >> 172.21.15.116:0/1545582365 conn(0x7f9a1c077e90 msgr2=0x7f9a1c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:32.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.867+0000 7f9a22600700 1 -- 172.21.15.116:0/1545582365 shutdown_connections 2024-03-30T00:56:32.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:32.868+0000 7f9a22600700 1 -- 172.21.15.116:0/1545582365 wait complete. 2024-03-30T00:56:33.253 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-30T00:56:33.254 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T00:56:33.618 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:33.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:33 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1545582365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:33.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:33 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1545582365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:33.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:33 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1545582365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:34.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.691+0000 7f5fb6c00700 1 -- 172.21.15.116:0/526322050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb010d810 msgr2=0x7f5fb010fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:34.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.691+0000 7f5fb6c00700 1 --2- 172.21.15.116:0/526322050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb010d810 0x7f5fb010fc00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009f90 tx=0x7f5fa800b330 comp rx=0 tx=0).stop 2024-03-30T00:56:34.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.691+0000 7f5fb6c00700 1 -- 172.21.15.116:0/526322050 shutdown_connections 2024-03-30T00:56:34.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.691+0000 7f5fb6c00700 1 --2- 172.21.15.116:0/526322050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb010d810 0x7f5fb010fc00 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:34.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.691+0000 7f5fb6c00700 1 --2- 172.21.15.116:0/526322050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5fb0072fe0 0x7f5fb00733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:34.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.691+0000 7f5fb6c00700 1 --2- 172.21.15.116:0/526322050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fb0073900 0x7f5fb010d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:34.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.691+0000 7f5fb6c00700 1 -- 172.21.15.116:0/526322050 >> 172.21.15.116:0/526322050 conn(0x7f5fb0077fa0 msgr2=0x7f5fb00783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:34.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.691+0000 7f5fb6c00700 1 -- 172.21.15.116:0/526322050 shutdown_connections 2024-03-30T00:56:34.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.691+0000 7f5fb6c00700 1 -- 172.21.15.116:0/526322050 wait complete. 2024-03-30T00:56:34.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.692+0000 7f5fb6c00700 1 Processor -- start 2024-03-30T00:56:34.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.692+0000 7f5fb6c00700 1 -- start start 2024-03-30T00:56:34.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.692+0000 7f5fb6c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5fb0072fe0 0x7f5fb01995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:34.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5fb6c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fb0073900 0x7f5fb0199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:34.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5fb6c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb010d810 0x7f5fb019de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:34.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5fb6c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb019e460 con 0x7f5fb010d810 2024-03-30T00:56:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5fb6c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb019e5d0 con 0x7f5fb0072fe0 2024-03-30T00:56:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5fb6c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb019e790 con 0x7f5fb0073900 2024-03-30T00:56:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5fafe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5fb0072fe0 0x7f5fb01995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5fb4c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb010d810 0x7f5fb019de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5fafe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5fb0072fe0 0x7f5fb01995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:44572/0 (socket says 172.21.15.116:44572) 2024-03-30T00:56:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5fafe00700 1 -- 172.21.15.116:0/1411733733 learned_addr learned my addr 172.21.15.116:0/1411733733 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5faf400700 1 --2- 172.21.15.116:0/1411733733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fb0073900 0x7f5fb0199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5fb4c00700 1 -- 172.21.15.116:0/1411733733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fb0073900 msgr2=0x7f5fb0199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5fb4c00700 1 --2- 172.21.15.116:0/1411733733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fb0073900 0x7f5fb0199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5fb4c00700 1 -- 172.21.15.116:0/1411733733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5fb0072fe0 msgr2=0x7f5fb01995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5fb4c00700 1 --2- 172.21.15.116:0/1411733733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5fb0072fe0 0x7f5fb01995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.693+0000 7f5fb4c00700 1 -- 172.21.15.116:0/1411733733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa00095f0 con 0x7f5fb010d810 2024-03-30T00:56:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.694+0000 7f5fb4c00700 1 --2- 172.21.15.116:0/1411733733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb010d810 0x7f5fb019de50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5fb006a4b0 tx=0x7f5fa80093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.694+0000 7f5fafe00700 1 --2- 172.21.15.116:0/1411733733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5fb0072fe0 0x7f5fb01995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:34.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.694+0000 7f5facc00700 1 -- 172.21.15.116:0/1411733733 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa800e040 con 0x7f5fb010d810 2024-03-30T00:56:34.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.694+0000 7f5facc00700 1 -- 172.21.15.116:0/1411733733 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5fa801b450 con 0x7f5fb010d810 2024-03-30T00:56:34.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.694+0000 7f5fb6c00700 1 -- 172.21.15.116:0/1411733733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa8009c40 con 0x7f5fb010d810 2024-03-30T00:56:34.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.694+0000 7f5facc00700 1 -- 172.21.15.116:0/1411733733 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa80052a0 con 0x7f5fb010d810 2024-03-30T00:56:34.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.694+0000 7f5fb6c00700 1 -- 172.21.15.116:0/1411733733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb01a6100 con 0x7f5fb010d810 2024-03-30T00:56:34.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.695+0000 7f5fb6c00700 1 -- 172.21.15.116:0/1411733733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb0108a20 con 0x7f5fb010d810 2024-03-30T00:56:34.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.697+0000 7f5facc00700 1 -- 172.21.15.116:0/1411733733 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f5fa801b880 con 0x7f5fb010d810 2024-03-30T00:56:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.698+0000 7f5facc00700 1 --2- 172.21.15.116:0/1411733733 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5f940a11b0 0x7f5f940a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.699+0000 7f5facc00700 1 -- 172.21.15.116:0/1411733733 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5fa80cdd10 con 0x7f5fb010d810 2024-03-30T00:56:34.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.699+0000 7f5fafe00700 1 --2- 172.21.15.116:0/1411733733 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5f940a11b0 0x7f5f940a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:34.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.699+0000 7f5fafe00700 1 --2- 172.21.15.116:0/1411733733 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5f940a11b0 0x7f5f940a3670 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0005d10 tx=0x7f5fa0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.703+0000 7f5facc00700 1 -- 172.21.15.116:0/1411733733 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5fa80d3020 con 0x7f5fb010d810 2024-03-30T00:56:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:34 smithi181 ceph-mon[47622]: pgmap v250: 129 pgs: 129 active+clean; 626 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-30T00:56:34.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:34 smithi130 ceph-mon[47626]: pgmap v250: 129 pgs: 129 active+clean; 626 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-30T00:56:34.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.905+0000 7f5fb6c00700 1 -- 172.21.15.116:0/1411733733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fb004ea80 con 0x7f5fb010d810 2024-03-30T00:56:34.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.907+0000 7f5facc00700 1 -- 172.21.15.116:0/1411733733 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+24028 (secure 0 0 0) 0x7f5fa8097180 con 0x7f5fb010d810 2024-03-30T00:56:34.914 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:56:34.914 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:55:56.324776+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:56:34.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.909+0000 7f5fb6c00700 1 -- 172.21.15.116:0/1411733733 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5f940a11b0 msgr2=0x7f5f940a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:34.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.909+0000 7f5fb6c00700 1 --2- 172.21.15.116:0/1411733733 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5f940a11b0 0x7f5f940a3670 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0005d10 tx=0x7f5fa0009310 comp rx=0 tx=0).stop 2024-03-30T00:56:34.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.909+0000 7f5fb6c00700 1 -- 172.21.15.116:0/1411733733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb010d810 msgr2=0x7f5fb019de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:34.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.909+0000 7f5fb6c00700 1 --2- 172.21.15.116:0/1411733733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb010d810 0x7f5fb019de50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5fb006a4b0 tx=0x7f5fa80093f0 comp rx=0 tx=0).stop 2024-03-30T00:56:34.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.909+0000 7f5fb6c00700 1 -- 172.21.15.116:0/1411733733 shutdown_connections 2024-03-30T00:56:34.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.909+0000 7f5fb6c00700 1 --2- 172.21.15.116:0/1411733733 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5f940a11b0 0x7f5f940a3670 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:34.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.909+0000 7f5fb6c00700 1 --2- 172.21.15.116:0/1411733733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb010d810 0x7f5fb019de50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:34.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.909+0000 7f5fb6c00700 1 --2- 172.21.15.116:0/1411733733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5fb0072fe0 0x7f5fb01995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:34.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.909+0000 7f5fb6c00700 1 --2- 172.21.15.116:0/1411733733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fb0073900 0x7f5fb0199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:34.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.909+0000 7f5fb6c00700 1 -- 172.21.15.116:0/1411733733 >> 172.21.15.116:0/1411733733 conn(0x7f5fb0077fa0 msgr2=0x7f5fb00fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:34.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.909+0000 7f5fb6c00700 1 -- 172.21.15.116:0/1411733733 shutdown_connections 2024-03-30T00:56:34.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:34.910+0000 7f5fb6c00700 1 -- 172.21.15.116:0/1411733733 wait complete. 2024-03-30T00:56:34.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:34 smithi116 ceph-mon[38086]: pgmap v250: 129 pgs: 129 active+clean; 626 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-30T00:56:35.331 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-03-30T00:56:35.332 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-03-30T00:56:35.332 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-03-30T00:56:35.333 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-03-30T00:56:35.333 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-03-30T00:56:35.333 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-03-30T00:56:35.334 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-03-30T00:56:35.334 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-03-30T00:56:35.335 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph tell osd.8 flush_pg_stats 2024-03-30T00:56:35.335 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph tell osd.9 flush_pg_stats 2024-03-30T00:56:35.335 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph tell osd.10 flush_pg_stats 2024-03-30T00:56:35.336 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph tell osd.11 flush_pg_stats 2024-03-30T00:56:35.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:35 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1411733733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:35.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-30T00:56:35.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-30T00:56:35.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-03-30T00:56:35.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-30T00:56:35.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-30T00:56:35.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-03-30T00:56:35.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-03-30T00:56:35.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-30T00:56:35.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-30T00:56:35.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:35 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [5, 8]}]: dispatch 2024-03-30T00:56:35.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:35 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1411733733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:35.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-30T00:56:35.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-30T00:56:35.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-03-30T00:56:35.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-30T00:56:35.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-30T00:56:35.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-03-30T00:56:35.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-03-30T00:56:35.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-30T00:56:35.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-30T00:56:35.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:35 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [5, 8]}]: dispatch 2024-03-30T00:56:35.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:35 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1411733733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:56:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-30T00:56:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-30T00:56:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-03-30T00:56:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-30T00:56:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-30T00:56:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-03-30T00:56:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-03-30T00:56:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-30T00:56:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-30T00:56:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:35 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [5, 8]}]: dispatch 2024-03-30T00:56:36.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:36 smithi181 ceph-mon[47622]: pgmap v251: 129 pgs: 129 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.2 KiB/s wr, 14 op/s 2024-03-30T00:56:36.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:36 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-30T00:56:36.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:36 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-30T00:56:36.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:36 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-03-30T00:56:36.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:36 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-30T00:56:36.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:36 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-30T00:56:36.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:36 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-03-30T00:56:36.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:36 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-03-30T00:56:36.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:36 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-30T00:56:36.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:36 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-30T00:56:36.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:36 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [5, 8]}]': finished 2024-03-30T00:56:36.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:36 smithi181 ceph-mon[47622]: osdmap e82: 12 total, 12 up, 12 in 2024-03-30T00:56:36.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:36 smithi130 ceph-mon[47626]: pgmap v251: 129 pgs: 129 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.2 KiB/s wr, 14 op/s 2024-03-30T00:56:36.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:36 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-30T00:56:36.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:36 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-30T00:56:36.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:36 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-03-30T00:56:36.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:36 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-30T00:56:36.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:36 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-30T00:56:36.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:36 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-03-30T00:56:36.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:36 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-03-30T00:56:36.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:36 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-30T00:56:36.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:36 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-30T00:56:36.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:36 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [5, 8]}]': finished 2024-03-30T00:56:36.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:36 smithi130 ceph-mon[47626]: osdmap e82: 12 total, 12 up, 12 in 2024-03-30T00:56:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:36 smithi116 ceph-mon[38086]: pgmap v251: 129 pgs: 129 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.2 KiB/s wr, 14 op/s 2024-03-30T00:56:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-30T00:56:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-30T00:56:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-03-30T00:56:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-30T00:56:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-30T00:56:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-03-30T00:56:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-03-30T00:56:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-30T00:56:36.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-30T00:56:36.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:36 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [5, 8]}]': finished 2024-03-30T00:56:36.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:36 smithi116 ceph-mon[38086]: osdmap e82: 12 total, 12 up, 12 in 2024-03-30T00:56:37.259 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:37.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:37 smithi116 ceph-mon[38086]: osdmap e83: 12 total, 12 up, 12 in 2024-03-30T00:56:37.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:37 smithi181 ceph-mon[47622]: osdmap e83: 12 total, 12 up, 12 in 2024-03-30T00:56:37.770 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:37.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:37 smithi130 ceph-mon[47626]: osdmap e83: 12 total, 12 up, 12 in 2024-03-30T00:56:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:38.297 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:38.298 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:38.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:38 smithi116 ceph-mon[38086]: pgmap v254: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2024-03-30T00:56:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:38.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:38 smithi181 ceph-mon[47622]: pgmap v254: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2024-03-30T00:56:38.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:38 smithi130 ceph-mon[47626]: pgmap v254: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2024-03-30T00:56:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.996+0000 7f8692000700 1 -- 172.21.15.116:0/719157431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f868c106e10 msgr2=0x7f868c1071f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.996+0000 7f8692000700 1 --2- 172.21.15.116:0/719157431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f868c106e10 0x7f868c1071f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f867c009990 tx=0x7f867c009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.996+0000 7f8692000700 1 -- 172.21.15.116:0/719157431 shutdown_connections 2024-03-30T00:56:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.996+0000 7f8692000700 1 --2- 172.21.15.116:0/719157431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f868c0690f0 0x7f868c10dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.996+0000 7f8692000700 1 --2- 172.21.15.116:0/719157431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f868c106e10 0x7f868c1071f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.996+0000 7f8692000700 1 --2- 172.21.15.116:0/719157431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f868c068750 0x7f868c068bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.996+0000 7f8692000700 1 -- 172.21.15.116:0/719157431 >> 172.21.15.116:0/719157431 conn(0x7f868c075d50 msgr2=0x7f868c076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.996+0000 7f8692000700 1 -- 172.21.15.116:0/719157431 shutdown_connections 2024-03-30T00:56:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.996+0000 7f8692000700 1 -- 172.21.15.116:0/719157431 wait complete. 2024-03-30T00:56:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.997+0000 7f8692000700 1 Processor -- start 2024-03-30T00:56:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.998+0000 7f8692000700 1 -- start start 2024-03-30T00:56:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.998+0000 7f8692000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f868c068750 0x7f868c1993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.998+0000 7f8692000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f868c0690f0 0x7f868c1998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.998+0000 7f8692000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f868c106e10 0x7f868c19dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.998+0000 7f8692000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c19e2a0 con 0x7f868c068750 2024-03-30T00:56:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.998+0000 7f8692000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c19e410 con 0x7f868c0690f0 2024-03-30T00:56:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.998+0000 7f8692000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c19e5d0 con 0x7f868c106e10 2024-03-30T00:56:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.999+0000 7f868b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f868c068750 0x7f868c1993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.999+0000 7f868aa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f868c0690f0 0x7f868c1998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.999+0000 7f868aa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f868c0690f0 0x7f868c1998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:46334/0 (socket says 172.21.15.116:46334) 2024-03-30T00:56:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.999+0000 7f868aa00700 1 -- 172.21.15.116:0/55034741 learned_addr learned my addr 172.21.15.116:0/55034741 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.999+0000 7f868be00700 1 --2- 172.21.15.116:0/55034741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f868c106e10 0x7f868c19dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.999+0000 7f868aa00700 1 -- 172.21.15.116:0/55034741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f868c106e10 msgr2=0x7f868c19dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.999+0000 7f868aa00700 1 --2- 172.21.15.116:0/55034741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f868c106e10 0x7f868c19dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.999+0000 7f868aa00700 1 -- 172.21.15.116:0/55034741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f868c068750 msgr2=0x7f868c1993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.999+0000 7f868aa00700 1 --2- 172.21.15.116:0/55034741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f868c068750 0x7f868c1993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:38.999+0000 7f868aa00700 1 -- 172.21.15.116:0/55034741 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c0095f0 con 0x7f868c0690f0 2024-03-30T00:56:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.000+0000 7f868aa00700 1 --2- 172.21.15.116:0/55034741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f868c0690f0 0x7f868c1998e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f867800c9b0 tx=0x7f867800ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.000+0000 7f8683e00700 1 -- 172.21.15.116:0/55034741 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8678007b00 con 0x7f868c0690f0 2024-03-30T00:56:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.000+0000 7f8683e00700 1 -- 172.21.15.116:0/55034741 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8678007c60 con 0x7f868c0690f0 2024-03-30T00:56:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.000+0000 7f8692000700 1 -- 172.21.15.116:0/55034741 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868c19e8b0 con 0x7f868c0690f0 2024-03-30T00:56:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.000+0000 7f8683e00700 1 -- 172.21.15.116:0/55034741 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8678018660 con 0x7f868c0690f0 2024-03-30T00:56:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.000+0000 7f8692000700 1 -- 172.21.15.116:0/55034741 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868c1a6110 con 0x7f868c0690f0 2024-03-30T00:56:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.001+0000 7f8692000700 1 -- 172.21.15.116:0/55034741 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f868c110f60 con 0x7f868c0690f0 2024-03-30T00:56:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.005+0000 7f8683e00700 1 -- 172.21.15.116:0/55034741 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f867800f450 con 0x7f868c0690f0 2024-03-30T00:56:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.006+0000 7f8683e00700 1 --2- 172.21.15.116:0/55034741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f86700a11b0 0x7f86700a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.006+0000 7f8683e00700 1 -- 172.21.15.116:0/55034741 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f86780ccd20 con 0x7f868c0690f0 2024-03-30T00:56:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.007+0000 7f8683e00700 1 --2- 172.21.15.116:0/55034741 >> [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060] conn(0x7f86700ac1d0 0x7f86700ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.007+0000 7f8683e00700 1 -- 172.21.15.116:0/55034741 --> [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86700aed40 con 0x7f86700ac1d0 2024-03-30T00:56:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.007+0000 7f8683e00700 1 -- 172.21.15.116:0/55034741 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86780cf3a0 con 0x7f868c0690f0 2024-03-30T00:56:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.007+0000 7f868b400700 1 --2- 172.21.15.116:0/55034741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f86700a11b0 0x7f86700a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.007+0000 7f868be00700 1 --2- 172.21.15.116:0/55034741 >> [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060] conn(0x7f86700ac1d0 0x7f86700ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.007+0000 7f868b400700 1 --2- 172.21.15.116:0/55034741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f86700a11b0 0x7f86700a3670 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f867c00b2e0 tx=0x7f867c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.008+0000 7f868be00700 1 --2- 172.21.15.116:0/55034741 >> [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060] conn(0x7f86700ac1d0 0x7f86700ae670 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.009+0000 7f8683e00700 1 -- 172.21.15.116:0/55034741 <== osd.2 v2:172.21.15.181:6800/3331770060 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f86700aed40 con 0x7f86700ac1d0 2024-03-30T00:56:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.033+0000 7f8692000700 1 -- 172.21.15.116:0/55034741 --> [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f868c10c970 con 0x7f86700ac1d0 2024-03-30T00:56:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.034+0000 7f8683e00700 1 -- 172.21.15.116:0/55034741 <== osd.2 v2:172.21.15.181:6800/3331770060 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f868c10c970 con 0x7f86700ac1d0 2024-03-30T00:56:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.034+0000 7f8692000700 1 -- 172.21.15.116:0/55034741 >> [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060] conn(0x7f86700ac1d0 msgr2=0x7f86700ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.034+0000 7f8692000700 1 --2- 172.21.15.116:0/55034741 >> [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060] conn(0x7f86700ac1d0 0x7f86700ae670 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.034+0000 7f8692000700 1 -- 172.21.15.116:0/55034741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f86700a11b0 msgr2=0x7f86700a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.034+0000 7f8692000700 1 --2- 172.21.15.116:0/55034741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f86700a11b0 0x7f86700a3670 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f867c00b2e0 tx=0x7f867c005fd0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.034+0000 7f8692000700 1 -- 172.21.15.116:0/55034741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f868c0690f0 msgr2=0x7f868c1998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.034+0000 7f8692000700 1 --2- 172.21.15.116:0/55034741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f868c0690f0 0x7f868c1998e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f867800c9b0 tx=0x7f867800ccc0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.034+0000 7f868be00700 1 -- 172.21.15.116:0/55034741 reap_dead start 2024-03-30T00:56:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.034+0000 7f8692000700 1 -- 172.21.15.116:0/55034741 shutdown_connections 2024-03-30T00:56:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.034+0000 7f8692000700 1 -- 172.21.15.116:0/55034741 >> 172.21.15.116:0/55034741 conn(0x7f868c075d50 msgr2=0x7f868c10bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.036+0000 7f8692000700 1 -- 172.21.15.116:0/55034741 shutdown_connections 2024-03-30T00:56:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.036+0000 7f8692000700 1 -- 172.21.15.116:0/55034741 wait complete. 2024-03-30T00:56:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.484+0000 7f95a7600700 1 -- 172.21.15.116:0/3532704543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a0100e60 msgr2=0x7f95a01012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.484+0000 7f95a7600700 1 --2- 172.21.15.116:0/3532704543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a0100e60 0x7f95a01012e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f958c0099c0 tx=0x7f958c009cd0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.484+0000 7f95a7600700 1 -- 172.21.15.116:0/3532704543 shutdown_connections 2024-03-30T00:56:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.484+0000 7f95a7600700 1 --2- 172.21.15.116:0/3532704543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95a0101820 0x7f95a010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.484+0000 7f95a7600700 1 --2- 172.21.15.116:0/3532704543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a0106ef0 0x7f95a01072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.484+0000 7f95a7600700 1 --2- 172.21.15.116:0/3532704543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a0100e60 0x7f95a01012e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.484+0000 7f95a7600700 1 -- 172.21.15.116:0/3532704543 >> 172.21.15.116:0/3532704543 conn(0x7f95a0077fa0 msgr2=0x7f95a00783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.484+0000 7f95a7600700 1 -- 172.21.15.116:0/3532704543 shutdown_connections 2024-03-30T00:56:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.484+0000 7f95a7600700 1 -- 172.21.15.116:0/3532704543 wait complete. 2024-03-30T00:56:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.485+0000 7f95a7600700 1 Processor -- start 2024-03-30T00:56:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.486+0000 7f95a7600700 1 -- start start 2024-03-30T00:56:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.486+0000 7f95a7600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a0100e60 0x7f95a0199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.486+0000 7f95a7600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95a0101820 0x7f95a01999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.486+0000 7f95a7600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a0106ef0 0x7f95a019dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.486+0000 7f95a7600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a019e360 con 0x7f95a0100e60 2024-03-30T00:56:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.486+0000 7f95a7600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a019e4d0 con 0x7f95a0106ef0 2024-03-30T00:56:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.486+0000 7f95a7600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a019e690 con 0x7f95a0101820 2024-03-30T00:56:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.486+0000 7f95a4c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a0100e60 0x7f95a0199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.486+0000 7f95a4c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a0100e60 0x7f95a0199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60206/0 (socket says 172.21.15.116:60206) 2024-03-30T00:56:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.486+0000 7f959fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95a0101820 0x7f95a01999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.486+0000 7f95a4c00700 1 -- 172.21.15.116:0/882511265 learned_addr learned my addr 172.21.15.116:0/882511265 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.486+0000 7f95a5600700 1 --2- 172.21.15.116:0/882511265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a0106ef0 0x7f95a019dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.487+0000 7f959fe00700 1 -- 172.21.15.116:0/882511265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a0106ef0 msgr2=0x7f95a019dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.487+0000 7f959fe00700 1 --2- 172.21.15.116:0/882511265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a0106ef0 0x7f95a019dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.487+0000 7f959fe00700 1 -- 172.21.15.116:0/882511265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a0100e60 msgr2=0x7f95a0199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.487+0000 7f959fe00700 1 --2- 172.21.15.116:0/882511265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a0100e60 0x7f95a0199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.487+0000 7f959fe00700 1 -- 172.21.15.116:0/882511265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958c0095f0 con 0x7f95a0101820 2024-03-30T00:56:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.487+0000 7f959fe00700 1 --2- 172.21.15.116:0/882511265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95a0101820 0x7f95a01999a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f958c009f50 tx=0x7f958c005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.487+0000 7f959d600700 1 -- 172.21.15.116:0/882511265 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c01d070 con 0x7f95a0101820 2024-03-30T00:56:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.487+0000 7f95a7600700 1 -- 172.21.15.116:0/882511265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a019e910 con 0x7f95a0101820 2024-03-30T00:56:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.487+0000 7f959d600700 1 -- 172.21.15.116:0/882511265 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f958c00fb30 con 0x7f95a0101820 2024-03-30T00:56:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.488+0000 7f959d600700 1 -- 172.21.15.116:0/882511265 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c022ac0 con 0x7f95a0101820 2024-03-30T00:56:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.488+0000 7f95a7600700 1 -- 172.21.15.116:0/882511265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a01a61d0 con 0x7f95a0101820 2024-03-30T00:56:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.488+0000 7f95a7600700 1 -- 172.21.15.116:0/882511265 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f95a0111000 con 0x7f95a0101820 2024-03-30T00:56:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.491+0000 7f959d600700 1 -- 172.21.15.116:0/882511265 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f958c00fca0 con 0x7f95a0101820 2024-03-30T00:56:39.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.492+0000 7f959d600700 1 --2- 172.21.15.116:0/882511265 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f95880a1230 0x7f95880a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:39.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.492+0000 7f959d600700 1 -- 172.21.15.116:0/882511265 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f958c0ce280 con 0x7f95a0101820 2024-03-30T00:56:39.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.493+0000 7f959d600700 1 --2- 172.21.15.116:0/882511265 >> [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272] conn(0x7f95880ac250 0x7f95880ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:39.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.493+0000 7f959d600700 1 -- 172.21.15.116:0/882511265 --> [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95880aedc0 con 0x7f95880ac250 2024-03-30T00:56:39.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.493+0000 7f959d600700 1 -- 172.21.15.116:0/882511265 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f958c0ce690 con 0x7f95a0101820 2024-03-30T00:56:39.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.493+0000 7f95a4c00700 1 --2- 172.21.15.116:0/882511265 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f95880a1230 0x7f95880a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:39.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.493+0000 7f95a5600700 1 --2- 172.21.15.116:0/882511265 >> [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272] conn(0x7f95880ac250 0x7f95880ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:39.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.494+0000 7f95a5600700 1 --2- 172.21.15.116:0/882511265 >> [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272] conn(0x7f95880ac250 0x7f95880ae6f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:39.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.494+0000 7f95a4c00700 1 --2- 172.21.15.116:0/882511265 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f95880a1230 0x7f95880a36f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9594007c10 tx=0x7f9594007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.495+0000 7f959d600700 1 -- 172.21.15.116:0/882511265 <== osd.11 v2:172.21.15.181:6824/2545451272 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f95880aedc0 con 0x7f95880ac250 2024-03-30T00:56:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.513+0000 7f95a7600700 1 -- 172.21.15.116:0/882511265 --> [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f95a0061430 con 0x7f95880ac250 2024-03-30T00:56:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.514+0000 7f959d600700 1 -- 172.21.15.116:0/882511265 <== osd.11 v2:172.21.15.181:6824/2545451272 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f95a0061430 con 0x7f95880ac250 2024-03-30T00:56:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.514+0000 7f95a7600700 1 -- 172.21.15.116:0/882511265 >> [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272] conn(0x7f95880ac250 msgr2=0x7f95880ae6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.514+0000 7f95a7600700 1 --2- 172.21.15.116:0/882511265 >> [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272] conn(0x7f95880ac250 0x7f95880ae6f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.514+0000 7f95a7600700 1 -- 172.21.15.116:0/882511265 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f95880a1230 msgr2=0x7f95880a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.514+0000 7f95a7600700 1 --2- 172.21.15.116:0/882511265 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f95880a1230 0x7f95880a36f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9594007c10 tx=0x7f9594007480 comp rx=0 tx=0).stop 2024-03-30T00:56:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.514+0000 7f95a7600700 1 -- 172.21.15.116:0/882511265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95a0101820 msgr2=0x7f95a01999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.514+0000 7f95a7600700 1 --2- 172.21.15.116:0/882511265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95a0101820 0x7f95a01999a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f958c009f50 tx=0x7f958c005740 comp rx=0 tx=0).stop 2024-03-30T00:56:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.514+0000 7f95a5600700 1 -- 172.21.15.116:0/882511265 reap_dead start 2024-03-30T00:56:39.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.515+0000 7f95a7600700 1 -- 172.21.15.116:0/882511265 shutdown_connections 2024-03-30T00:56:39.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.515+0000 7f95a7600700 1 -- 172.21.15.116:0/882511265 >> 172.21.15.116:0/882511265 conn(0x7f95a0077fa0 msgr2=0x7f95a010c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:39.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.515+0000 7f95a7600700 1 -- 172.21.15.116:0/882511265 shutdown_connections 2024-03-30T00:56:39.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:39.515+0000 7f95a7600700 1 -- 172.21.15.116:0/882511265 wait complete. 2024-03-30T00:56:40.532 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:40 smithi116 ceph-mon[38086]: pgmap v255: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-03-30T00:56:40.693 INFO:teuthology.orchestra.run.smithi116.stdout:73014444085 2024-03-30T00:56:40.693 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-03-30T00:56:40.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:40 smithi181 ceph-mon[47622]: pgmap v255: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-03-30T00:56:40.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:40 smithi130 ceph-mon[47626]: pgmap v255: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-03-30T00:56:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.073+0000 7fad5ac00700 1 -- 172.21.15.116:0/2535393628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad54105ef0 msgr2=0x7fad5410dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.073+0000 7fad5ac00700 1 --2- 172.21.15.116:0/2535393628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad54105ef0 0x7fad5410dac0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00b0a0 tx=0x7fad4c00b3b0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.073+0000 7fad5ac00700 1 -- 172.21.15.116:0/2535393628 shutdown_connections 2024-03-30T00:56:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.073+0000 7fad5ac00700 1 --2- 172.21.15.116:0/2535393628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad54100e10 0x7fad541011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.073+0000 7fad5ac00700 1 --2- 172.21.15.116:0/2535393628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad54105ef0 0x7fad5410dac0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.073+0000 7fad5ac00700 1 --2- 172.21.15.116:0/2535393628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad541017c0 0x7fad54105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.073+0000 7fad5ac00700 1 -- 172.21.15.116:0/2535393628 >> 172.21.15.116:0/2535393628 conn(0x7fad540fc6e0 msgr2=0x7fad540feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.073+0000 7fad5ac00700 1 -- 172.21.15.116:0/2535393628 shutdown_connections 2024-03-30T00:56:41.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.073+0000 7fad5ac00700 1 -- 172.21.15.116:0/2535393628 wait complete. 2024-03-30T00:56:41.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.075+0000 7fad5ac00700 1 Processor -- start 2024-03-30T00:56:41.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.075+0000 7fad5ac00700 1 -- start start 2024-03-30T00:56:41.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.075+0000 7fad5ac00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad54100e10 0x7fad54197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:41.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.075+0000 7fad5ac00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad541017c0 0x7fad541976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:41.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.075+0000 7fad5ac00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad54105ef0 0x7fad5419ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:41.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.075+0000 7fad5ac00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5419c080 con 0x7fad54105ef0 2024-03-30T00:56:41.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.075+0000 7fad5ac00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5419c1f0 con 0x7fad541017c0 2024-03-30T00:56:41.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.075+0000 7fad5ac00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5419c3b0 con 0x7fad54100e10 2024-03-30T00:56:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.076+0000 7fad59800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad54100e10 0x7fad54197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.076+0000 7fad5a200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad54105ef0 0x7fad5419ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.076+0000 7fad59800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad54100e10 0x7fad54197180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:36260/0 (socket says 172.21.15.116:36260) 2024-03-30T00:56:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.076+0000 7fad59800700 1 -- 172.21.15.116:0/3431034001 learned_addr learned my addr 172.21.15.116:0/3431034001 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.076+0000 7fad58e00700 1 --2- 172.21.15.116:0/3431034001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad541017c0 0x7fad541976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.076+0000 7fad59800700 1 -- 172.21.15.116:0/3431034001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad541017c0 msgr2=0x7fad541976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.076+0000 7fad59800700 1 --2- 172.21.15.116:0/3431034001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad541017c0 0x7fad541976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.076+0000 7fad59800700 1 -- 172.21.15.116:0/3431034001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad54105ef0 msgr2=0x7fad5419ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.076+0000 7fad59800700 1 --2- 172.21.15.116:0/3431034001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad54105ef0 0x7fad5419ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.076+0000 7fad59800700 1 -- 172.21.15.116:0/3431034001 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c009d00 con 0x7fad54100e10 2024-03-30T00:56:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.076+0000 7fad5a200700 1 --2- 172.21.15.116:0/3431034001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad54105ef0 0x7fad5419ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.076+0000 7fad59800700 1 --2- 172.21.15.116:0/3431034001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad54100e10 0x7fad54197180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fad4800e9a0 tx=0x7fad4800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:41.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.077+0000 7fad52000700 1 -- 172.21.15.116:0/3431034001 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48009790 con 0x7fad54100e10 2024-03-30T00:56:41.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.077+0000 7fad5ac00700 1 -- 172.21.15.116:0/3431034001 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5419c690 con 0x7fad54100e10 2024-03-30T00:56:41.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.077+0000 7fad52000700 1 -- 172.21.15.116:0/3431034001 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad48010430 con 0x7fad54100e10 2024-03-30T00:56:41.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.077+0000 7fad52000700 1 -- 172.21.15.116:0/3431034001 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48018680 con 0x7fad54100e10 2024-03-30T00:56:41.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.077+0000 7fad5ac00700 1 -- 172.21.15.116:0/3431034001 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad541a3ef0 con 0x7fad54100e10 2024-03-30T00:56:41.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.078+0000 7fad5ac00700 1 -- 172.21.15.116:0/3431034001 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fad541022f0 con 0x7fad54100e10 2024-03-30T00:56:41.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.080+0000 7fad52000700 1 -- 172.21.15.116:0/3431034001 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fad48010810 con 0x7fad54100e10 2024-03-30T00:56:41.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.081+0000 7fad52000700 1 --2- 172.21.15.116:0/3431034001 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fad3c0a1230 0x7fad3c0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:41.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.081+0000 7fad52000700 1 -- 172.21.15.116:0/3431034001 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fad48014070 con 0x7fad54100e10 2024-03-30T00:56:41.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.081+0000 7fad58e00700 1 --2- 172.21.15.116:0/3431034001 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fad3c0a1230 0x7fad3c0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:41.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.082+0000 7fad52000700 1 --2- 172.21.15.116:0/3431034001 >> [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027] conn(0x7fad3c0ac250 0x7fad3c0ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:41.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.082+0000 7fad52000700 1 -- 172.21.15.116:0/3431034001 --> [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fad3c0aedc0 con 0x7fad3c0ac250 2024-03-30T00:56:41.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.082+0000 7fad52000700 1 -- 172.21.15.116:0/3431034001 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad480cd3e0 con 0x7fad54100e10 2024-03-30T00:56:41.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.082+0000 7fad58e00700 1 --2- 172.21.15.116:0/3431034001 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fad3c0a1230 0x7fad3c0a36f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fad541986e0 tx=0x7fad40009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:41.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.082+0000 7fad5a200700 1 --2- 172.21.15.116:0/3431034001 >> [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027] conn(0x7fad3c0ac250 0x7fad3c0ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:41.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.083+0000 7fad5a200700 1 --2- 172.21.15.116:0/3431034001 >> [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027] conn(0x7fad3c0ac250 0x7fad3c0ae6f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:41.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.084+0000 7fad52000700 1 -- 172.21.15.116:0/3431034001 <== osd.5 v2:172.21.15.181:6808/1074863027 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fad3c0aedc0 con 0x7fad3c0ac250 2024-03-30T00:56:41.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.100+0000 7fad5ac00700 1 -- 172.21.15.116:0/3431034001 --> [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fad54110db0 con 0x7fad3c0ac250 2024-03-30T00:56:41.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.101+0000 7fad52000700 1 -- 172.21.15.116:0/3431034001 <== osd.5 v2:172.21.15.181:6808/1074863027 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fad54110db0 con 0x7fad3c0ac250 2024-03-30T00:56:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.101+0000 7fad5ac00700 1 -- 172.21.15.116:0/3431034001 >> [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027] conn(0x7fad3c0ac250 msgr2=0x7fad3c0ae6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.101+0000 7fad5ac00700 1 --2- 172.21.15.116:0/3431034001 >> [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027] conn(0x7fad3c0ac250 0x7fad3c0ae6f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.102+0000 7fad5ac00700 1 -- 172.21.15.116:0/3431034001 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fad3c0a1230 msgr2=0x7fad3c0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.102+0000 7fad5ac00700 1 --2- 172.21.15.116:0/3431034001 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fad3c0a1230 0x7fad3c0a36f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fad541986e0 tx=0x7fad40009260 comp rx=0 tx=0).stop 2024-03-30T00:56:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.102+0000 7fad5ac00700 1 -- 172.21.15.116:0/3431034001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad54100e10 msgr2=0x7fad54197180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.102+0000 7fad5ac00700 1 --2- 172.21.15.116:0/3431034001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad54100e10 0x7fad54197180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fad4800e9a0 tx=0x7fad4800ecb0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.102+0000 7fad5a200700 1 -- 172.21.15.116:0/3431034001 reap_dead start 2024-03-30T00:56:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.102+0000 7fad5ac00700 1 -- 172.21.15.116:0/3431034001 shutdown_connections 2024-03-30T00:56:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.102+0000 7fad5ac00700 1 -- 172.21.15.116:0/3431034001 >> 172.21.15.116:0/3431034001 conn(0x7fad540fc6e0 msgr2=0x7fad540fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.102+0000 7fad5ac00700 1 -- 172.21.15.116:0/3431034001 shutdown_connections 2024-03-30T00:56:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.102+0000 7fad5ac00700 1 -- 172.21.15.116:0/3431034001 wait complete. 2024-03-30T00:56:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:41.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.750+0000 7f919ac00700 1 -- 172.21.15.116:0/3013008598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9194101820 msgr2=0x7f919410dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:41.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.750+0000 7f919ac00700 1 --2- 172.21.15.116:0/3013008598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9194101820 0x7f919410dc10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f918c00b0d0 tx=0x7f918c00b3e0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.750+0000 7f919ac00700 1 -- 172.21.15.116:0/3013008598 shutdown_connections 2024-03-30T00:56:41.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.750+0000 7f919ac00700 1 --2- 172.21.15.116:0/3013008598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9194101820 0x7f919410dc10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.750+0000 7f919ac00700 1 --2- 172.21.15.116:0/3013008598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9194106ef0 0x7f91941072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.750+0000 7f919ac00700 1 --2- 172.21.15.116:0/3013008598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9194100e60 0x7f91941012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.750+0000 7f919ac00700 1 -- 172.21.15.116:0/3013008598 >> 172.21.15.116:0/3013008598 conn(0x7f9194077fa0 msgr2=0x7f91940783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.750+0000 7f919ac00700 1 -- 172.21.15.116:0/3013008598 shutdown_connections 2024-03-30T00:56:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.750+0000 7f919ac00700 1 -- 172.21.15.116:0/3013008598 wait complete. 2024-03-30T00:56:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.751+0000 7f919ac00700 1 Processor -- start 2024-03-30T00:56:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.752+0000 7f919ac00700 1 -- start start 2024-03-30T00:56:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.752+0000 7f919ac00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9194100e60 0x7f91940730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.752+0000 7f919ac00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9194101820 0x7f91940735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.752+0000 7f919ac00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9194106ef0 0x7f9194073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.752+0000 7f919ac00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9194075070 con 0x7f9194100e60 2024-03-30T00:56:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.752+0000 7f919ac00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91941a9d30 con 0x7f9194101820 2024-03-30T00:56:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.752+0000 7f919ac00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91941a9ef0 con 0x7f9194106ef0 2024-03-30T00:56:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.752+0000 7f9193e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9194100e60 0x7f91940730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.752+0000 7f9193e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9194100e60 0x7f91940730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60240/0 (socket says 172.21.15.116:60240) 2024-03-30T00:56:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.752+0000 7f9193e00700 1 -- 172.21.15.116:0/3008477447 learned_addr learned my addr 172.21.15.116:0/3008477447 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.752+0000 7f9193400700 1 --2- 172.21.15.116:0/3008477447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9194101820 0x7f91940735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.752+0000 7f9198c00700 1 --2- 172.21.15.116:0/3008477447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9194106ef0 0x7f9194073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.753+0000 7f9193e00700 1 -- 172.21.15.116:0/3008477447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9194106ef0 msgr2=0x7f9194073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.753+0000 7f9193e00700 1 --2- 172.21.15.116:0/3008477447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9194106ef0 0x7f9194073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.753+0000 7f9193e00700 1 -- 172.21.15.116:0/3008477447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9194101820 msgr2=0x7f91940735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:41.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.753+0000 7f9193e00700 1 --2- 172.21.15.116:0/3008477447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9194101820 0x7f91940735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.753+0000 7f9193e00700 1 -- 172.21.15.116:0/3008477447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c009d00 con 0x7f9194100e60 2024-03-30T00:56:41.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.753+0000 7f9193e00700 1 --2- 172.21.15.116:0/3008477447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9194100e60 0x7f91940730b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f918400e9a0 tx=0x7f918400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:41.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.753+0000 7f9190c00700 1 -- 172.21.15.116:0/3008477447 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184009790 con 0x7f9194100e60 2024-03-30T00:56:41.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.753+0000 7f919ac00700 1 -- 172.21.15.116:0/3008477447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91941aa1d0 con 0x7f9194100e60 2024-03-30T00:56:41.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.753+0000 7f9190c00700 1 -- 172.21.15.116:0/3008477447 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9184010430 con 0x7f9194100e60 2024-03-30T00:56:41.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.753+0000 7f9190c00700 1 -- 172.21.15.116:0/3008477447 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184018680 con 0x7f9194100e60 2024-03-30T00:56:41.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.753+0000 7f919ac00700 1 -- 172.21.15.116:0/3008477447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91941aa720 con 0x7f9194100e60 2024-03-30T00:56:41.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.754+0000 7f919ac00700 1 -- 172.21.15.116:0/3008477447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9194111050 con 0x7f9194100e60 2024-03-30T00:56:41.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.756+0000 7f9190c00700 1 -- 172.21.15.116:0/3008477447 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f91840105a0 con 0x7f9194100e60 2024-03-30T00:56:41.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.757+0000 7f9190c00700 1 --2- 172.21.15.116:0/3008477447 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91780a1230 0x7f91780a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:41.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.758+0000 7f9190c00700 1 -- 172.21.15.116:0/3008477447 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9184014070 con 0x7f9194100e60 2024-03-30T00:56:41.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.758+0000 7f9193400700 1 --2- 172.21.15.116:0/3008477447 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91780a1230 0x7f91780a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:41.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.758+0000 7f9190c00700 1 --2- 172.21.15.116:0/3008477447 >> [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262] conn(0x7f91780ac250 0x7f91780ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:41.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.758+0000 7f9190c00700 1 -- 172.21.15.116:0/3008477447 --> [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91780aedc0 con 0x7f91780ac250 2024-03-30T00:56:41.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.758+0000 7f9190c00700 1 -- 172.21.15.116:0/3008477447 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91840cd490 con 0x7f9194100e60 2024-03-30T00:56:41.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.758+0000 7f9198c00700 1 --2- 172.21.15.116:0/3008477447 >> [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262] conn(0x7f91780ac250 0x7f91780ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:41.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.759+0000 7f9193400700 1 --2- 172.21.15.116:0/3008477447 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91780a1230 0x7f91780a36f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f91940747e0 tx=0x7f9180009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:41.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.759+0000 7f9198c00700 1 --2- 172.21.15.116:0/3008477447 >> [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262] conn(0x7f91780ac250 0x7f91780ae6f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:41.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.760+0000 7f9190c00700 1 -- 172.21.15.116:0/3008477447 <== osd.10 v2:172.21.15.130:6824/3569352262 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f91780aedc0 con 0x7f91780ac250 2024-03-30T00:56:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.785+0000 7f919ac00700 1 -- 172.21.15.116:0/3008477447 --> [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f91941024f0 con 0x7f91780ac250 2024-03-30T00:56:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.785+0000 7f9190c00700 1 -- 172.21.15.116:0/3008477447 <== osd.10 v2:172.21.15.130:6824/3569352262 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f91941024f0 con 0x7f91780ac250 2024-03-30T00:56:41.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.786+0000 7f919ac00700 1 -- 172.21.15.116:0/3008477447 >> [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262] conn(0x7f91780ac250 msgr2=0x7f91780ae6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:41.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.786+0000 7f919ac00700 1 --2- 172.21.15.116:0/3008477447 >> [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262] conn(0x7f91780ac250 0x7f91780ae6f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.786+0000 7f919ac00700 1 -- 172.21.15.116:0/3008477447 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91780a1230 msgr2=0x7f91780a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:41.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.786+0000 7f919ac00700 1 --2- 172.21.15.116:0/3008477447 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91780a1230 0x7f91780a36f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f91940747e0 tx=0x7f9180009260 comp rx=0 tx=0).stop 2024-03-30T00:56:41.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.786+0000 7f919ac00700 1 -- 172.21.15.116:0/3008477447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9194100e60 msgr2=0x7f91940730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:41.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.786+0000 7f919ac00700 1 --2- 172.21.15.116:0/3008477447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9194100e60 0x7f91940730b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f918400e9a0 tx=0x7f918400ecb0 comp rx=0 tx=0).stop 2024-03-30T00:56:41.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.786+0000 7f9198c00700 1 -- 172.21.15.116:0/3008477447 reap_dead start 2024-03-30T00:56:41.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.786+0000 7f919ac00700 1 -- 172.21.15.116:0/3008477447 shutdown_connections 2024-03-30T00:56:41.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.786+0000 7f919ac00700 1 -- 172.21.15.116:0/3008477447 >> 172.21.15.116:0/3008477447 conn(0x7f9194077fa0 msgr2=0x7f91940ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:41.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.786+0000 7f919ac00700 1 -- 172.21.15.116:0/3008477447 shutdown_connections 2024-03-30T00:56:41.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:41.786+0000 7f919ac00700 1 -- 172.21.15.116:0/3008477447 wait complete. 2024-03-30T00:56:42.615 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:42 smithi116 ceph-mon[38086]: pgmap v256: 129 pgs: 129 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-03-30T00:56:42.640 INFO:teuthology.orchestra.run.smithi116.stdout:309237645332 2024-03-30T00:56:42.641 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.11 2024-03-30T00:56:42.723 INFO:teuthology.orchestra.run.smithi116.stdout:150323855402 2024-03-30T00:56:42.724 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-03-30T00:56:42.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:42 smithi181 ceph-mon[47622]: pgmap v256: 129 pgs: 129 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-03-30T00:56:42.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:42 smithi130 ceph-mon[47626]: pgmap v256: 129 pgs: 129 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-03-30T00:56:43.155 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:43.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.332+0000 7f4e88c00700 1 -- 172.21.15.116:0/2493201819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e84106df0 msgr2=0x7f4e841071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:43.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.332+0000 7f4e88c00700 1 --2- 172.21.15.116:0/2493201819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e84106df0 0x7f4e841071d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4e740098f0 tx=0x7f4e74009c00 comp rx=0 tx=0).stop 2024-03-30T00:56:43.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.332+0000 7f4e88c00700 1 -- 172.21.15.116:0/2493201819 shutdown_connections 2024-03-30T00:56:43.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.332+0000 7f4e88c00700 1 --2- 172.21.15.116:0/2493201819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e840690f0 0x7f4e8410dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:43.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.332+0000 7f4e88c00700 1 --2- 172.21.15.116:0/2493201819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e84106df0 0x7f4e841071d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:43.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.332+0000 7f4e88c00700 1 --2- 172.21.15.116:0/2493201819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e84068750 0x7f4e84068bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:43.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.332+0000 7f4e88c00700 1 -- 172.21.15.116:0/2493201819 >> 172.21.15.116:0/2493201819 conn(0x7f4e84075d50 msgr2=0x7f4e84076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:43.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.332+0000 7f4e88c00700 1 -- 172.21.15.116:0/2493201819 shutdown_connections 2024-03-30T00:56:43.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.332+0000 7f4e88c00700 1 -- 172.21.15.116:0/2493201819 wait complete. 2024-03-30T00:56:43.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.333+0000 7f4e88c00700 1 Processor -- start 2024-03-30T00:56:43.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.333+0000 7f4e88c00700 1 -- start start 2024-03-30T00:56:43.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.333+0000 7f4e88c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e84068750 0x7f4e84199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:43.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.333+0000 7f4e88c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e840690f0 0x7f4e84199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:43.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.333+0000 7f4e88c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e84106df0 0x7f4e8419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:43.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.333+0000 7f4e88c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8419e3d0 con 0x7f4e840690f0 2024-03-30T00:56:43.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.333+0000 7f4e88c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8419e540 con 0x7f4e84068750 2024-03-30T00:56:43.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.333+0000 7f4e88c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8419e700 con 0x7f4e84106df0 2024-03-30T00:56:43.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.334+0000 7f4e81400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e840690f0 0x7f4e84199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:43.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.334+0000 7f4e81400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e840690f0 0x7f4e84199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60264/0 (socket says 172.21.15.116:60264) 2024-03-30T00:56:43.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.334+0000 7f4e81e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e84068750 0x7f4e84199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:43.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.334+0000 7f4e81400700 1 -- 172.21.15.116:0/877586553 learned_addr learned my addr 172.21.15.116:0/877586553 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:43.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.334+0000 7f4e82800700 1 --2- 172.21.15.116:0/877586553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e84106df0 0x7f4e8419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.334+0000 7f4e81400700 1 -- 172.21.15.116:0/877586553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e84106df0 msgr2=0x7f4e8419ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.334+0000 7f4e81400700 1 --2- 172.21.15.116:0/877586553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e84106df0 0x7f4e8419ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.334+0000 7f4e81400700 1 -- 172.21.15.116:0/877586553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e84068750 msgr2=0x7f4e84199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.334+0000 7f4e81400700 1 --2- 172.21.15.116:0/877586553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e84068750 0x7f4e84199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.334+0000 7f4e81400700 1 -- 172.21.15.116:0/877586553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7c009c40 con 0x7f4e840690f0 2024-03-30T00:56:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.334+0000 7f4e81400700 1 --2- 172.21.15.116:0/877586553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e840690f0 0x7f4e84199a60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000d980 tx=0x7f4e7000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.335+0000 7f4e7aa00700 1 -- 172.21.15.116:0/877586553 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70009b40 con 0x7f4e840690f0 2024-03-30T00:56:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.335+0000 7f4e7aa00700 1 -- 172.21.15.116:0/877586553 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e7000b6a0 con 0x7f4e840690f0 2024-03-30T00:56:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.335+0000 7f4e88c00700 1 -- 172.21.15.116:0/877586553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e74009520 con 0x7f4e840690f0 2024-03-30T00:56:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.335+0000 7f4e7aa00700 1 -- 172.21.15.116:0/877586553 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000f3c0 con 0x7f4e840690f0 2024-03-30T00:56:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.335+0000 7f4e88c00700 1 -- 172.21.15.116:0/877586553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e84107b70 con 0x7f4e840690f0 2024-03-30T00:56:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.336+0000 7f4e88c00700 1 -- 172.21.15.116:0/877586553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4e84110f80 con 0x7f4e840690f0 2024-03-30T00:56:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.338+0000 7f4e7aa00700 1 -- 172.21.15.116:0/877586553 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f4e70010460 con 0x7f4e840690f0 2024-03-30T00:56:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.340+0000 7f4e7aa00700 1 --2- 172.21.15.116:0/877586553 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4e680a11b0 0x7f4e680a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.340+0000 7f4e7aa00700 1 -- 172.21.15.116:0/877586553 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4e700cd0b0 con 0x7f4e840690f0 2024-03-30T00:56:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.340+0000 7f4e81e00700 1 --2- 172.21.15.116:0/877586553 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4e680a11b0 0x7f4e680a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.341+0000 7f4e7aa00700 1 --2- 172.21.15.116:0/877586553 >> [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130] conn(0x7f4e680ac1d0 0x7f4e680ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.341+0000 7f4e7aa00700 1 -- 172.21.15.116:0/877586553 --> [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e680aed40 con 0x7f4e680ac1d0 2024-03-30T00:56:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.341+0000 7f4e7aa00700 1 -- 172.21.15.116:0/877586553 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e700cd420 con 0x7f4e840690f0 2024-03-30T00:56:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.341+0000 7f4e81e00700 1 --2- 172.21.15.116:0/877586553 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4e680a11b0 0x7f4e680a3670 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4e7400c010 tx=0x7f4e74019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.341+0000 7f4e82800700 1 --2- 172.21.15.116:0/877586553 >> [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130] conn(0x7f4e680ac1d0 0x7f4e680ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.341+0000 7f4e82800700 1 --2- 172.21.15.116:0/877586553 >> [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130] conn(0x7f4e680ac1d0 0x7f4e680ae670 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.343+0000 7f4e7aa00700 1 -- 172.21.15.116:0/877586553 <== osd.6 v2:172.21.15.116:6818/1426239130 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4e680aed40 con 0x7f4e680ac1d0 2024-03-30T00:56:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.374+0000 7f4e88c00700 1 -- 172.21.15.116:0/877586553 --> [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4e8410c9f0 con 0x7f4e680ac1d0 2024-03-30T00:56:43.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.374+0000 7f4e7aa00700 1 -- 172.21.15.116:0/877586553 <== osd.6 v2:172.21.15.116:6818/1426239130 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4e8410c9f0 con 0x7f4e680ac1d0 2024-03-30T00:56:43.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.375+0000 7f4e88c00700 1 -- 172.21.15.116:0/877586553 >> [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130] conn(0x7f4e680ac1d0 msgr2=0x7f4e680ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:43.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.375+0000 7f4e88c00700 1 --2- 172.21.15.116:0/877586553 >> [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130] conn(0x7f4e680ac1d0 0x7f4e680ae670 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:43.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.375+0000 7f4e88c00700 1 -- 172.21.15.116:0/877586553 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4e680a11b0 msgr2=0x7f4e680a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.375+0000 7f4e88c00700 1 --2- 172.21.15.116:0/877586553 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4e680a11b0 0x7f4e680a3670 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4e7400c010 tx=0x7f4e74019040 comp rx=0 tx=0).stop 2024-03-30T00:56:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.375+0000 7f4e88c00700 1 -- 172.21.15.116:0/877586553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e840690f0 msgr2=0x7f4e84199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.375+0000 7f4e88c00700 1 --2- 172.21.15.116:0/877586553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e840690f0 0x7f4e84199a60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000d980 tx=0x7f4e7000dd40 comp rx=0 tx=0).stop 2024-03-30T00:56:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.375+0000 7f4e82800700 1 -- 172.21.15.116:0/877586553 reap_dead start 2024-03-30T00:56:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.375+0000 7f4e88c00700 1 -- 172.21.15.116:0/877586553 shutdown_connections 2024-03-30T00:56:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.375+0000 7f4e88c00700 1 -- 172.21.15.116:0/877586553 >> 172.21.15.116:0/877586553 conn(0x7f4e84075d50 msgr2=0x7f4e8410c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.375+0000 7f4e88c00700 1 -- 172.21.15.116:0/877586553 shutdown_connections 2024-03-30T00:56:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.375+0000 7f4e88c00700 1 -- 172.21.15.116:0/877586553 wait complete. 2024-03-30T00:56:43.774 INFO:teuthology.orchestra.run.smithi116.stdout:279172874263 2024-03-30T00:56:43.775 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.10 2024-03-30T00:56:44.109 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.112+0000 7f0d4f400700 1 -- 172.21.15.116:0/357486574 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d48072fe0 msgr2=0x7f0d480733c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.112+0000 7f0d4f400700 1 --2- 172.21.15.116:0/357486574 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d48072fe0 0x7f0d480733c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c009990 tx=0x7f0d3c009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.113+0000 7f0d4f400700 1 -- 172.21.15.116:0/357486574 shutdown_connections 2024-03-30T00:56:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.113+0000 7f0d4f400700 1 --2- 172.21.15.116:0/357486574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d4810d810 0x7f0d4810fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.113+0000 7f0d4f400700 1 --2- 172.21.15.116:0/357486574 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d48072fe0 0x7f0d480733c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.113+0000 7f0d4f400700 1 --2- 172.21.15.116:0/357486574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d48073900 0x7f0d4810d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.113+0000 7f0d4f400700 1 -- 172.21.15.116:0/357486574 >> 172.21.15.116:0/357486574 conn(0x7f0d48077fa0 msgr2=0x7f0d480783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.113+0000 7f0d4f400700 1 -- 172.21.15.116:0/357486574 shutdown_connections 2024-03-30T00:56:44.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.114+0000 7f0d4f400700 1 -- 172.21.15.116:0/357486574 wait complete. 2024-03-30T00:56:44.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.116+0000 7f0d4f400700 1 Processor -- start 2024-03-30T00:56:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.116+0000 7f0d4f400700 1 -- start start 2024-03-30T00:56:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.116+0000 7f0d4f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d48072fe0 0x7f0d48199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.116+0000 7f0d4f400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d48073900 0x7f0d481999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.116+0000 7f0d4f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d4810d810 0x7f0d4819dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.116+0000 7f0d4f400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4819e370 con 0x7f0d48072fe0 2024-03-30T00:56:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.116+0000 7f0d4f400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4819e4e0 con 0x7f0d4810d810 2024-03-30T00:56:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.116+0000 7f0d4f400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4819e6a0 con 0x7f0d48073900 2024-03-30T00:56:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.117+0000 7f0d4ca00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d48072fe0 0x7f0d48199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.117+0000 7f0d4d400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d4810d810 0x7f0d4819dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.117+0000 7f0d47e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d48073900 0x7f0d481999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.117+0000 7f0d47e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d48073900 0x7f0d481999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:36348/0 (socket says 172.21.15.116:36348) 2024-03-30T00:56:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.117+0000 7f0d47e00700 1 -- 172.21.15.116:0/798037925 learned_addr learned my addr 172.21.15.116:0/798037925 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.117+0000 7f0d4d400700 1 -- 172.21.15.116:0/798037925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d48073900 msgr2=0x7f0d481999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.117+0000 7f0d4d400700 1 --2- 172.21.15.116:0/798037925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d48073900 0x7f0d481999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.117+0000 7f0d4d400700 1 -- 172.21.15.116:0/798037925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d48072fe0 msgr2=0x7f0d48199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.117+0000 7f0d4d400700 1 --2- 172.21.15.116:0/798037925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d48072fe0 0x7f0d48199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.117+0000 7f0d4d400700 1 -- 172.21.15.116:0/798037925 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c0095f0 con 0x7f0d4810d810 2024-03-30T00:56:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.118+0000 7f0d4d400700 1 --2- 172.21.15.116:0/798037925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d4810d810 0x7f0d4819dd60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0d4000ef90 tx=0x7f0d4000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.118+0000 7f0d45600700 1 -- 172.21.15.116:0/798037925 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d40018070 con 0x7f0d4810d810 2024-03-30T00:56:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.118+0000 7f0d45600700 1 -- 172.21.15.116:0/798037925 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d4000f040 con 0x7f0d4810d810 2024-03-30T00:56:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.118+0000 7f0d4f400700 1 -- 172.21.15.116:0/798037925 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4819e920 con 0x7f0d4810d810 2024-03-30T00:56:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.118+0000 7f0d4f400700 1 -- 172.21.15.116:0/798037925 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d481a61e0 con 0x7f0d4810d810 2024-03-30T00:56:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.118+0000 7f0d45600700 1 -- 172.21.15.116:0/798037925 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4000cb80 con 0x7f0d4810d810 2024-03-30T00:56:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.119+0000 7f0d4f400700 1 -- 172.21.15.116:0/798037925 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0d481132e0 con 0x7f0d4810d810 2024-03-30T00:56:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.122+0000 7f0d45600700 1 -- 172.21.15.116:0/798037925 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f0d4000cce0 con 0x7f0d4810d810 2024-03-30T00:56:44.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.123+0000 7f0d45600700 1 --2- 172.21.15.116:0/798037925 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0d300a11b0 0x7f0d300a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.123+0000 7f0d45600700 1 -- 172.21.15.116:0/798037925 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0d400cd220 con 0x7f0d4810d810 2024-03-30T00:56:44.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.123+0000 7f0d4ca00700 1 --2- 172.21.15.116:0/798037925 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0d300a11b0 0x7f0d300a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.124+0000 7f0d45600700 1 --2- 172.21.15.116:0/798037925 >> [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389] conn(0x7f0d300ac1d0 0x7f0d300ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.124+0000 7f0d45600700 1 -- 172.21.15.116:0/798037925 --> [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d300aed40 con 0x7f0d300ac1d0 2024-03-30T00:56:44.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.124+0000 7f0d45600700 1 -- 172.21.15.116:0/798037925 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d400cd590 con 0x7f0d4810d810 2024-03-30T00:56:44.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.124+0000 7f0d47e00700 1 --2- 172.21.15.116:0/798037925 >> [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389] conn(0x7f0d300ac1d0 0x7f0d300ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.124+0000 7f0d4ca00700 1 --2- 172.21.15.116:0/798037925 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0d300a11b0 0x7f0d300a3670 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00b2e0 tx=0x7f0d3c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:44.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.124+0000 7f0d47e00700 1 --2- 172.21.15.116:0/798037925 >> [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389] conn(0x7f0d300ac1d0 0x7f0d300ae670 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:44.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.125+0000 7f0d45600700 1 -- 172.21.15.116:0/798037925 <== osd.8 v2:172.21.15.181:6816/3650979389 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f0d300aed40 con 0x7f0d300ac1d0 2024-03-30T00:56:44.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.142+0000 7f0d4f400700 1 -- 172.21.15.116:0/798037925 --> [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0d48074670 con 0x7f0d300ac1d0 2024-03-30T00:56:44.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.143+0000 7f0d45600700 1 -- 172.21.15.116:0/798037925 <== osd.8 v2:172.21.15.181:6816/3650979389 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0d48074670 con 0x7f0d300ac1d0 2024-03-30T00:56:44.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.143+0000 7f0d4f400700 1 -- 172.21.15.116:0/798037925 >> [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389] conn(0x7f0d300ac1d0 msgr2=0x7f0d300ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.143+0000 7f0d4f400700 1 --2- 172.21.15.116:0/798037925 >> [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389] conn(0x7f0d300ac1d0 0x7f0d300ae670 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.143+0000 7f0d4f400700 1 -- 172.21.15.116:0/798037925 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0d300a11b0 msgr2=0x7f0d300a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.143+0000 7f0d4f400700 1 --2- 172.21.15.116:0/798037925 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0d300a11b0 0x7f0d300a3670 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00b2e0 tx=0x7f0d3c005fd0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.143+0000 7f0d4f400700 1 -- 172.21.15.116:0/798037925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d4810d810 msgr2=0x7f0d4819dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.143+0000 7f0d4f400700 1 --2- 172.21.15.116:0/798037925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d4810d810 0x7f0d4819dd60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0d4000ef90 tx=0x7f0d4000c3a0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.143+0000 7f0d4d400700 1 -- 172.21.15.116:0/798037925 reap_dead start 2024-03-30T00:56:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.143+0000 7f0d4f400700 1 -- 172.21.15.116:0/798037925 shutdown_connections 2024-03-30T00:56:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.143+0000 7f0d4f400700 1 -- 172.21.15.116:0/798037925 >> 172.21.15.116:0/798037925 conn(0x7f0d48077fa0 msgr2=0x7f0d480fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:44.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.144+0000 7f0d4f400700 1 -- 172.21.15.116:0/798037925 shutdown_connections 2024-03-30T00:56:44.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.144+0000 7f0d4f400700 1 -- 172.21.15.116:0/798037925 wait complete. 2024-03-30T00:56:44.395 INFO:teuthology.orchestra.run.smithi116.stdout:180388626471 2024-03-30T00:56:44.395 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-03-30T00:56:44.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.575+0000 7f7db0400700 1 -- 172.21.15.116:0/4101114761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da8101820 msgr2=0x7f7da810dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.575+0000 7f7db0400700 1 --2- 172.21.15.116:0/4101114761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da8101820 0x7f7da810dc10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7da000b0d0 tx=0x7f7da000b3e0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.575+0000 7f7db0400700 1 -- 172.21.15.116:0/4101114761 shutdown_connections 2024-03-30T00:56:44.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.575+0000 7f7db0400700 1 --2- 172.21.15.116:0/4101114761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7da8106ef0 0x7f7da81072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.575+0000 7f7db0400700 1 --2- 172.21.15.116:0/4101114761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da8101820 0x7f7da810dc10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.575+0000 7f7db0400700 1 --2- 172.21.15.116:0/4101114761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7da8100e60 0x7f7da81012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.575+0000 7f7db0400700 1 -- 172.21.15.116:0/4101114761 >> 172.21.15.116:0/4101114761 conn(0x7f7da8077fa0 msgr2=0x7f7da80783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:44.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.575+0000 7f7db0400700 1 -- 172.21.15.116:0/4101114761 shutdown_connections 2024-03-30T00:56:44.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.575+0000 7f7db0400700 1 -- 172.21.15.116:0/4101114761 wait complete. 2024-03-30T00:56:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.576+0000 7f7db0400700 1 Processor -- start 2024-03-30T00:56:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.576+0000 7f7db0400700 1 -- start start 2024-03-30T00:56:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.576+0000 7f7db0400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da8100e60 0x7f7da81972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.576+0000 7f7db0400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7da8101820 0x7f7da8197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.576+0000 7f7db0400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7da8106ef0 0x7f7da819bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.576+0000 7f7db0400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da819c1c0 con 0x7f7da8100e60 2024-03-30T00:56:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.576+0000 7f7db0400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da819c330 con 0x7f7da8101820 2024-03-30T00:56:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.576+0000 7f7db0400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da819c4f0 con 0x7f7da8106ef0 2024-03-30T00:56:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.577+0000 7f7dada00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da8100e60 0x7f7da81972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.577+0000 7f7dada00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da8100e60 0x7f7da81972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60282/0 (socket says 172.21.15.116:60282) 2024-03-30T00:56:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.577+0000 7f7dada00700 1 -- 172.21.15.116:0/2169241657 learned_addr learned my addr 172.21.15.116:0/2169241657 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.577+0000 7f7dad000700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7da8101820 0x7f7da8197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.577+0000 7f7dae400700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7da8106ef0 0x7f7da819bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.577+0000 7f7dada00700 1 -- 172.21.15.116:0/2169241657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7da8106ef0 msgr2=0x7f7da819bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.577+0000 7f7dada00700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7da8106ef0 0x7f7da819bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.577+0000 7f7dada00700 1 -- 172.21.15.116:0/2169241657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7da8101820 msgr2=0x7f7da8197800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.577+0000 7f7dada00700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7da8101820 0x7f7da8197800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.577+0000 7f7dada00700 1 -- 172.21.15.116:0/2169241657 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da0009d00 con 0x7f7da8100e60 2024-03-30T00:56:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.577+0000 7f7dae400700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7da8106ef0 0x7f7da819bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:56:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.577+0000 7f7dada00700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da8100e60 0x7f7da81972c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c00e9a0 tx=0x7f7d9c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.578+0000 7f7da6000700 1 -- 172.21.15.116:0/2169241657 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c009790 con 0x7f7da8100e60 2024-03-30T00:56:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.578+0000 7f7db0400700 1 -- 172.21.15.116:0/2169241657 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da819c7d0 con 0x7f7da8100e60 2024-03-30T00:56:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.578+0000 7f7da6000700 1 -- 172.21.15.116:0/2169241657 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d9c00cbd0 con 0x7f7da8100e60 2024-03-30T00:56:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.578+0000 7f7da6000700 1 -- 172.21.15.116:0/2169241657 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c018680 con 0x7f7da8100e60 2024-03-30T00:56:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.578+0000 7f7db0400700 1 -- 172.21.15.116:0/2169241657 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da81a4030 con 0x7f7da8100e60 2024-03-30T00:56:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.579+0000 7f7db0400700 1 -- 172.21.15.116:0/2169241657 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7da81024f0 con 0x7f7da8100e60 2024-03-30T00:56:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.581+0000 7f7da6000700 1 -- 172.21.15.116:0/2169241657 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f7d9c00cd40 con 0x7f7da8100e60 2024-03-30T00:56:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.582+0000 7f7da6000700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7d900a11b0 0x7f7d900a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.582+0000 7f7da6000700 1 -- 172.21.15.116:0/2169241657 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7d9c014070 con 0x7f7da8100e60 2024-03-30T00:56:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.582+0000 7f7dad000700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7d900a11b0 0x7f7d900a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.582+0000 7f7da6000700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957] conn(0x7f7d900ac1d0 0x7f7d900ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.582+0000 7f7da6000700 1 -- 172.21.15.116:0/2169241657 --> [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d900aed40 con 0x7f7d900ac1d0 2024-03-30T00:56:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.582+0000 7f7da6000700 1 -- 172.21.15.116:0/2169241657 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d9c0cd7c0 con 0x7f7da8100e60 2024-03-30T00:56:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.583+0000 7f7dae400700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957] conn(0x7f7d900ac1d0 0x7f7d900ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.583+0000 7f7dad000700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7d900a11b0 0x7f7d900a3670 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7da8198820 tx=0x7f7d94009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.583+0000 7f7dae400700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957] conn(0x7f7d900ac1d0 0x7f7d900ae670 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.584+0000 7f7da6000700 1 -- 172.21.15.116:0/2169241657 <== osd.4 v2:172.21.15.130:6808/81879957 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7d900aed40 con 0x7f7d900ac1d0 2024-03-30T00:56:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.601+0000 7f7db0400700 1 -- 172.21.15.116:0/2169241657 --> [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7da8110fd0 con 0x7f7d900ac1d0 2024-03-30T00:56:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.602+0000 7f7da6000700 1 -- 172.21.15.116:0/2169241657 <== osd.4 v2:172.21.15.130:6808/81879957 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7da8110fd0 con 0x7f7d900ac1d0 2024-03-30T00:56:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.602+0000 7f7db0400700 1 -- 172.21.15.116:0/2169241657 >> [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957] conn(0x7f7d900ac1d0 msgr2=0x7f7d900ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.602+0000 7f7db0400700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957] conn(0x7f7d900ac1d0 0x7f7d900ae670 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.602+0000 7f7db0400700 1 -- 172.21.15.116:0/2169241657 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7d900a11b0 msgr2=0x7f7d900a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.602+0000 7f7db0400700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7d900a11b0 0x7f7d900a3670 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7da8198820 tx=0x7f7d94009260 comp rx=0 tx=0).stop 2024-03-30T00:56:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.602+0000 7f7db0400700 1 -- 172.21.15.116:0/2169241657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da8100e60 msgr2=0x7f7da81972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.602+0000 7f7db0400700 1 --2- 172.21.15.116:0/2169241657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da8100e60 0x7f7da81972c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c00e9a0 tx=0x7f7d9c00ecb0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.602+0000 7f7dae400700 1 -- 172.21.15.116:0/2169241657 reap_dead start 2024-03-30T00:56:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.602+0000 7f7db0400700 1 -- 172.21.15.116:0/2169241657 shutdown_connections 2024-03-30T00:56:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.602+0000 7f7db0400700 1 -- 172.21.15.116:0/2169241657 >> 172.21.15.116:0/2169241657 conn(0x7f7da8077fa0 msgr2=0x7f7da80ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.603+0000 7f7db0400700 1 -- 172.21.15.116:0/2169241657 shutdown_connections 2024-03-30T00:56:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:43.603+0000 7f7db0400700 1 -- 172.21.15.116:0/2169241657 wait complete. 2024-03-30T00:56:44.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:44 smithi116 ceph-mon[38086]: pgmap v257: 129 pgs: 129 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-03-30T00:56:44.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:44 smithi181 ceph-mon[47622]: pgmap v257: 129 pgs: 129 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-03-30T00:56:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.833+0000 7f73ede00700 1 -- 172.21.15.116:0/1924331651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e810d840 msgr2=0x7f73e810fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.833+0000 7f73ede00700 1 --2- 172.21.15.116:0/1924331651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e810d840 0x7f73e810fc30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f73e000b0a0 tx=0x7f73e000b3b0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.833+0000 7f73ede00700 1 -- 172.21.15.116:0/1924331651 shutdown_connections 2024-03-30T00:56:44.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.833+0000 7f73ede00700 1 --2- 172.21.15.116:0/1924331651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73e8072fe0 0x7f73e80733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.833+0000 7f73ede00700 1 --2- 172.21.15.116:0/1924331651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e810d840 0x7f73e810fc30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.833+0000 7f73ede00700 1 --2- 172.21.15.116:0/1924331651 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73e8073900 0x7f73e810d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.833+0000 7f73ede00700 1 -- 172.21.15.116:0/1924331651 >> 172.21.15.116:0/1924331651 conn(0x7f73e8077e90 msgr2=0x7f73e80782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:44.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.833+0000 7f73ede00700 1 -- 172.21.15.116:0/1924331651 shutdown_connections 2024-03-30T00:56:44.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.833+0000 7f73ede00700 1 -- 172.21.15.116:0/1924331651 wait complete. 2024-03-30T00:56:44.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.834+0000 7f73ede00700 1 Processor -- start 2024-03-30T00:56:44.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.834+0000 7f73ede00700 1 -- start start 2024-03-30T00:56:44.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.835+0000 7f73ede00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e8072fe0 0x7f73e8199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.835+0000 7f73ede00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73e8073900 0x7f73e8199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.835+0000 7f73ede00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73e810d840 0x7f73e819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.835+0000 7f73ede00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e819e3b0 con 0x7f73e8072fe0 2024-03-30T00:56:44.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.835+0000 7f73ede00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e819e520 con 0x7f73e8073900 2024-03-30T00:56:44.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.835+0000 7f73ede00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e819e6e0 con 0x7f73e810d840 2024-03-30T00:56:44.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.835+0000 7f73e6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e8072fe0 0x7f73e8199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.835+0000 7f73e6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e8072fe0 0x7f73e8199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38008/0 (socket says 172.21.15.116:38008) 2024-03-30T00:56:44.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.835+0000 7f73e6a00700 1 -- 172.21.15.116:0/2382262114 learned_addr learned my addr 172.21.15.116:0/2382262114 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:44.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.835+0000 7f73e6000700 1 --2- 172.21.15.116:0/2382262114 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73e8073900 0x7f73e8199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.835+0000 7f73e7400700 1 --2- 172.21.15.116:0/2382262114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73e810d840 0x7f73e819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.835+0000 7f73e6a00700 1 -- 172.21.15.116:0/2382262114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73e810d840 msgr2=0x7f73e819dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.835+0000 7f73e6a00700 1 --2- 172.21.15.116:0/2382262114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73e810d840 0x7f73e819dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.836+0000 7f73e6a00700 1 -- 172.21.15.116:0/2382262114 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73e8073900 msgr2=0x7f73e8199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.836+0000 7f73e6a00700 1 --2- 172.21.15.116:0/2382262114 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73e8073900 0x7f73e8199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.836+0000 7f73e6a00700 1 -- 172.21.15.116:0/2382262114 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e0009d00 con 0x7f73e8072fe0 2024-03-30T00:56:44.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.836+0000 7f73e6a00700 1 --2- 172.21.15.116:0/2382262114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e8072fe0 0x7f73e8199520 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f73d800e9a0 tx=0x7f73d800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:44.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.836+0000 7f73eca00700 1 -- 172.21.15.116:0/2382262114 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d8009790 con 0x7f73e8072fe0 2024-03-30T00:56:44.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.836+0000 7f73eca00700 1 -- 172.21.15.116:0/2382262114 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73d8010430 con 0x7f73e8072fe0 2024-03-30T00:56:44.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.836+0000 7f73ede00700 1 -- 172.21.15.116:0/2382262114 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e819e9c0 con 0x7f73e8072fe0 2024-03-30T00:56:44.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.836+0000 7f73eca00700 1 -- 172.21.15.116:0/2382262114 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d8018680 con 0x7f73e8072fe0 2024-03-30T00:56:44.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:44 smithi130 ceph-mon[47626]: pgmap v257: 129 pgs: 129 active+clean; 635 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-03-30T00:56:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.836+0000 7f73ede00700 1 -- 172.21.15.116:0/2382262114 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e81a6220 con 0x7f73e8072fe0 2024-03-30T00:56:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.837+0000 7f73ede00700 1 -- 172.21.15.116:0/2382262114 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f73e81131b0 con 0x7f73e8072fe0 2024-03-30T00:56:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.839+0000 7f73eca00700 1 -- 172.21.15.116:0/2382262114 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f73d80105a0 con 0x7f73e8072fe0 2024-03-30T00:56:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.840+0000 7f73eca00700 1 --2- 172.21.15.116:0/2382262114 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f73cc0a11b0 0x7f73cc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.841+0000 7f73eca00700 1 -- 172.21.15.116:0/2382262114 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f73d8014070 con 0x7f73e8072fe0 2024-03-30T00:56:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.841+0000 7f73e6000700 1 --2- 172.21.15.116:0/2382262114 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f73cc0a11b0 0x7f73cc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.841+0000 7f73eca00700 1 --2- 172.21.15.116:0/2382262114 >> [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018] conn(0x7f73cc0ac1d0 0x7f73cc0ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.841+0000 7f73eca00700 1 -- 172.21.15.116:0/2382262114 --> [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73cc0aed40 con 0x7f73cc0ac1d0 2024-03-30T00:56:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.841+0000 7f73eca00700 1 -- 172.21.15.116:0/2382262114 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73d80cd5f0 con 0x7f73e8072fe0 2024-03-30T00:56:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.841+0000 7f73e7400700 1 --2- 172.21.15.116:0/2382262114 >> [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018] conn(0x7f73cc0ac1d0 0x7f73cc0ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.841+0000 7f73e6000700 1 --2- 172.21.15.116:0/2382262114 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f73cc0a11b0 0x7f73cc0a3670 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f73e810f490 tx=0x7f73d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.842+0000 7f73e7400700 1 --2- 172.21.15.116:0/2382262114 >> [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018] conn(0x7f73cc0ac1d0 0x7f73cc0ae670 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.842+0000 7f73eca00700 1 -- 172.21.15.116:0/2382262114 <== osd.0 v2:172.21.15.116:6802/3901884018 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f73cc0aed40 con 0x7f73cc0ac1d0 2024-03-30T00:56:44.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.859+0000 7f73ede00700 1 -- 172.21.15.116:0/2382262114 --> [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f73e80614b0 con 0x7f73cc0ac1d0 2024-03-30T00:56:44.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.859+0000 7f73eca00700 1 -- 172.21.15.116:0/2382262114 <== osd.0 v2:172.21.15.116:6802/3901884018 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f73e80614b0 con 0x7f73cc0ac1d0 2024-03-30T00:56:44.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.860+0000 7f73ede00700 1 -- 172.21.15.116:0/2382262114 >> [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018] conn(0x7f73cc0ac1d0 msgr2=0x7f73cc0ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.860+0000 7f73ede00700 1 --2- 172.21.15.116:0/2382262114 >> [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018] conn(0x7f73cc0ac1d0 0x7f73cc0ae670 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.860+0000 7f73ede00700 1 -- 172.21.15.116:0/2382262114 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f73cc0a11b0 msgr2=0x7f73cc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.860+0000 7f73ede00700 1 --2- 172.21.15.116:0/2382262114 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f73cc0a11b0 0x7f73cc0a3670 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f73e810f490 tx=0x7f73d4009260 comp rx=0 tx=0).stop 2024-03-30T00:56:44.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.860+0000 7f73ede00700 1 -- 172.21.15.116:0/2382262114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e8072fe0 msgr2=0x7f73e8199520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:44.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.860+0000 7f73ede00700 1 --2- 172.21.15.116:0/2382262114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e8072fe0 0x7f73e8199520 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f73d800e9a0 tx=0x7f73d800ecb0 comp rx=0 tx=0).stop 2024-03-30T00:56:44.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.860+0000 7f73e7400700 1 -- 172.21.15.116:0/2382262114 reap_dead start 2024-03-30T00:56:44.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.860+0000 7f73ede00700 1 -- 172.21.15.116:0/2382262114 shutdown_connections 2024-03-30T00:56:44.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.860+0000 7f73ede00700 1 -- 172.21.15.116:0/2382262114 >> 172.21.15.116:0/2382262114 conn(0x7f73e8077e90 msgr2=0x7f73e80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:44.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.860+0000 7f73ede00700 1 -- 172.21.15.116:0/2382262114 shutdown_connections 2024-03-30T00:56:44.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:44.860+0000 7f73ede00700 1 -- 172.21.15.116:0/2382262114 wait complete. 2024-03-30T00:56:45.192 INFO:teuthology.orchestra.run.smithi116.stdout:227633266719 2024-03-30T00:56:45.193 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.8 2024-03-30T00:56:45.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.284+0000 7fcbd7400700 1 -- 172.21.15.116:0/1913740503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcbd0068750 msgr2=0x7fcbd0068bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:45.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.284+0000 7fcbd7400700 1 --2- 172.21.15.116:0/1913740503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcbd0068750 0x7fcbd0068bb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009990 tx=0x7fcbbc009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:45.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.284+0000 7fcbd7400700 1 -- 172.21.15.116:0/1913740503 shutdown_connections 2024-03-30T00:56:45.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.284+0000 7fcbd7400700 1 --2- 172.21.15.116:0/1913740503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbd00690f0 0x7fcbd010dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:45.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.284+0000 7fcbd7400700 1 --2- 172.21.15.116:0/1913740503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbd0106df0 0x7fcbd01071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:45.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.284+0000 7fcbd7400700 1 --2- 172.21.15.116:0/1913740503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcbd0068750 0x7fcbd0068bb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:45.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.284+0000 7fcbd7400700 1 -- 172.21.15.116:0/1913740503 >> 172.21.15.116:0/1913740503 conn(0x7fcbd0075d50 msgr2=0x7fcbd0076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:45.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.285+0000 7fcbd7400700 1 -- 172.21.15.116:0/1913740503 shutdown_connections 2024-03-30T00:56:45.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.285+0000 7fcbd7400700 1 -- 172.21.15.116:0/1913740503 wait complete. 2024-03-30T00:56:45.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.286+0000 7fcbd7400700 1 Processor -- start 2024-03-30T00:56:45.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.286+0000 7fcbd7400700 1 -- start start 2024-03-30T00:56:45.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.286+0000 7fcbd7400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbd0068750 0x7fcbd01993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:45.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.286+0000 7fcbd7400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcbd00690f0 0x7fcbd0199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:45.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.286+0000 7fcbd7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbd0106df0 0x7fcbd019dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:45.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.286+0000 7fcbd7400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd019e2f0 con 0x7fcbd0106df0 2024-03-30T00:56:45.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.286+0000 7fcbd7400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd019e460 con 0x7fcbd00690f0 2024-03-30T00:56:45.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.286+0000 7fcbd7400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd019e620 con 0x7fcbd0068750 2024-03-30T00:56:45.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.287+0000 7fcbd4a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbd0068750 0x7fcbd01993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:45.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.287+0000 7fcbd4a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbd0068750 0x7fcbd01993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:48638/0 (socket says 172.21.15.116:48638) 2024-03-30T00:56:45.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.287+0000 7fcbcfe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcbd00690f0 0x7fcbd0199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:45.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.287+0000 7fcbd4a00700 1 -- 172.21.15.116:0/2362627133 learned_addr learned my addr 172.21.15.116:0/2362627133 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:45.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.287+0000 7fcbd5400700 1 --2- 172.21.15.116:0/2362627133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbd0106df0 0x7fcbd019dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:45.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.287+0000 7fcbd4a00700 1 -- 172.21.15.116:0/2362627133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcbd00690f0 msgr2=0x7fcbd0199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.287+0000 7fcbd4a00700 1 --2- 172.21.15.116:0/2362627133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcbd00690f0 0x7fcbd0199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.287+0000 7fcbd4a00700 1 -- 172.21.15.116:0/2362627133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbd0106df0 msgr2=0x7fcbd019dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.287+0000 7fcbd4a00700 1 --2- 172.21.15.116:0/2362627133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbd0106df0 0x7fcbd019dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.287+0000 7fcbd4a00700 1 -- 172.21.15.116:0/2362627133 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc0095f0 con 0x7fcbd0068750 2024-03-30T00:56:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.287+0000 7fcbcfe00700 1 --2- 172.21.15.116:0/2362627133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcbd00690f0 0x7fcbd0199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.287+0000 7fcbd4a00700 1 --2- 172.21.15.116:0/2362627133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbd0068750 0x7fcbd01993f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcbc400d950 tx=0x7fcbc400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.288+0000 7fcbcd600700 1 -- 172.21.15.116:0/2362627133 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc400f890 con 0x7fcbd0068750 2024-03-30T00:56:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.288+0000 7fcbcd600700 1 -- 172.21.15.116:0/2362627133 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcbc4010460 con 0x7fcbd0068750 2024-03-30T00:56:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.288+0000 7fcbd7400700 1 -- 172.21.15.116:0/2362627133 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd019e900 con 0x7fcbd0068750 2024-03-30T00:56:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.288+0000 7fcbcd600700 1 -- 172.21.15.116:0/2362627133 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc400e610 con 0x7fcbd0068750 2024-03-30T00:56:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.288+0000 7fcbd7400700 1 -- 172.21.15.116:0/2362627133 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd01a6160 con 0x7fcbd0068750 2024-03-30T00:56:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.289+0000 7fcbd7400700 1 -- 172.21.15.116:0/2362627133 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcbd01024a0 con 0x7fcbd0068750 2024-03-30T00:56:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.291+0000 7fcbcd600700 1 -- 172.21.15.116:0/2362627133 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fcbc40105d0 con 0x7fcbd0068750 2024-03-30T00:56:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.292+0000 7fcbcd600700 1 --2- 172.21.15.116:0/2362627133 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcbb80a11b0 0x7fcbb80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.292+0000 7fcbcd600700 1 -- 172.21.15.116:0/2362627133 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcbc40ccfe0 con 0x7fcbd0068750 2024-03-30T00:56:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.293+0000 7fcbcfe00700 1 --2- 172.21.15.116:0/2362627133 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcbb80a11b0 0x7fcbb80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.293+0000 7fcbcd600700 1 --2- 172.21.15.116:0/2362627133 >> [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072] conn(0x7fcbb80ac1d0 0x7fcbb80ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.293+0000 7fcbcd600700 1 -- 172.21.15.116:0/2362627133 --> [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbb80aed40 con 0x7fcbb80ac1d0 2024-03-30T00:56:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.293+0000 7fcbcd600700 1 -- 172.21.15.116:0/2362627133 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbc40cd3f0 con 0x7fcbd0068750 2024-03-30T00:56:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.293+0000 7fcbd5400700 1 --2- 172.21.15.116:0/2362627133 >> [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072] conn(0x7fcbb80ac1d0 0x7fcbb80ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.294+0000 7fcbd5400700 1 --2- 172.21.15.116:0/2362627133 >> [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072] conn(0x7fcbb80ac1d0 0x7fcbb80ae670 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.294+0000 7fcbcfe00700 1 --2- 172.21.15.116:0/2362627133 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcbb80a11b0 0x7fcbb80a3670 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcbd019a950 tx=0x7fcbbc00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.295+0000 7fcbcd600700 1 -- 172.21.15.116:0/2362627133 <== osd.3 v2:172.21.15.116:6810/2931358072 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fcbb80aed40 con 0x7fcbb80ac1d0 2024-03-30T00:56:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.332+0000 7fcbd7400700 1 -- 172.21.15.116:0/2362627133 --> [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcbd0110f40 con 0x7fcbb80ac1d0 2024-03-30T00:56:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.333+0000 7fcbcd600700 1 -- 172.21.15.116:0/2362627133 <== osd.3 v2:172.21.15.116:6810/2931358072 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcbd0110f40 con 0x7fcbb80ac1d0 2024-03-30T00:56:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.334+0000 7fcbd7400700 1 -- 172.21.15.116:0/2362627133 >> [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072] conn(0x7fcbb80ac1d0 msgr2=0x7fcbb80ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.334+0000 7fcbd7400700 1 --2- 172.21.15.116:0/2362627133 >> [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072] conn(0x7fcbb80ac1d0 0x7fcbb80ae670 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.334+0000 7fcbd7400700 1 -- 172.21.15.116:0/2362627133 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcbb80a11b0 msgr2=0x7fcbb80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.334+0000 7fcbd7400700 1 --2- 172.21.15.116:0/2362627133 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcbb80a11b0 0x7fcbb80a3670 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcbd019a950 tx=0x7fcbbc00b260 comp rx=0 tx=0).stop 2024-03-30T00:56:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.334+0000 7fcbd7400700 1 -- 172.21.15.116:0/2362627133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbd0068750 msgr2=0x7fcbd01993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.334+0000 7fcbd7400700 1 --2- 172.21.15.116:0/2362627133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbd0068750 0x7fcbd01993f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcbc400d950 tx=0x7fcbc400dc60 comp rx=0 tx=0).stop 2024-03-30T00:56:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.335+0000 7fcbd5400700 1 -- 172.21.15.116:0/2362627133 reap_dead start 2024-03-30T00:56:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.335+0000 7fcbd7400700 1 -- 172.21.15.116:0/2362627133 shutdown_connections 2024-03-30T00:56:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.335+0000 7fcbd7400700 1 -- 172.21.15.116:0/2362627133 >> 172.21.15.116:0/2362627133 conn(0x7fcbd0075d50 msgr2=0x7fcbd010bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.335+0000 7fcbd7400700 1 -- 172.21.15.116:0/2362627133 shutdown_connections 2024-03-30T00:56:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:45.335+0000 7fcbd7400700 1 -- 172.21.15.116:0/2362627133 wait complete. 2024-03-30T00:56:45.881 INFO:teuthology.orchestra.run.smithi116.stdout:34359738429 2024-03-30T00:56:45.881 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-03-30T00:56:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.052+0000 7f98f9800700 1 -- 172.21.15.116:0/2105813158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f4100e60 msgr2=0x7f98f41012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.052+0000 7f98f9800700 1 --2- 172.21.15.116:0/2105813158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f4100e60 0x7f98f41012e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f98e00099c0 tx=0x7f98e0009cd0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.052+0000 7f98f9800700 1 -- 172.21.15.116:0/2105813158 shutdown_connections 2024-03-30T00:56:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.052+0000 7f98f9800700 1 --2- 172.21.15.116:0/2105813158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98f4106ef0 0x7f98f41072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.052+0000 7f98f9800700 1 --2- 172.21.15.116:0/2105813158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f4101820 0x7f98f410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.052+0000 7f98f9800700 1 --2- 172.21.15.116:0/2105813158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f4100e60 0x7f98f41012e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.052+0000 7f98f9800700 1 -- 172.21.15.116:0/2105813158 >> 172.21.15.116:0/2105813158 conn(0x7f98f4077fa0 msgr2=0x7f98f40783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.052+0000 7f98f9800700 1 -- 172.21.15.116:0/2105813158 shutdown_connections 2024-03-30T00:56:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.052+0000 7f98f9800700 1 -- 172.21.15.116:0/2105813158 wait complete. 2024-03-30T00:56:46.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.053+0000 7f98f9800700 1 Processor -- start 2024-03-30T00:56:46.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.053+0000 7f98f9800700 1 -- start start 2024-03-30T00:56:46.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.053+0000 7f98f9800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98f4100e60 0x7f98f4199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:46.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.053+0000 7f98f9800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f4101820 0x7f98f4199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.053+0000 7f98f9800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f4106ef0 0x7f98f419dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.053+0000 7f98f9800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f419e300 con 0x7f98f4106ef0 2024-03-30T00:56:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.053+0000 7f98f9800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f419e470 con 0x7f98f4101820 2024-03-30T00:56:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.053+0000 7f98f9800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f419e630 con 0x7f98f4100e60 2024-03-30T00:56:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.054+0000 7f98f3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f4106ef0 0x7f98f419dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.054+0000 7f98f2000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f4101820 0x7f98f4199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.054+0000 7f98f2000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f4101820 0x7f98f4199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:42172/0 (socket says 172.21.15.116:42172) 2024-03-30T00:56:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.054+0000 7f98f2a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98f4100e60 0x7f98f4199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.054+0000 7f98f2000700 1 -- 172.21.15.116:0/3100736952 learned_addr learned my addr 172.21.15.116:0/3100736952 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:46.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.054+0000 7f98f2000700 1 -- 172.21.15.116:0/3100736952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98f4100e60 msgr2=0x7f98f4199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:46.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.054+0000 7f98f2000700 1 --2- 172.21.15.116:0/3100736952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98f4100e60 0x7f98f4199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.054+0000 7f98f2000700 1 -- 172.21.15.116:0/3100736952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f4106ef0 msgr2=0x7f98f419dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:46.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.054+0000 7f98f2000700 1 --2- 172.21.15.116:0/3100736952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f4106ef0 0x7f98f419dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.054+0000 7f98f2000700 1 -- 172.21.15.116:0/3100736952 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e00095f0 con 0x7f98f4101820 2024-03-30T00:56:46.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.054+0000 7f98f2a00700 1 --2- 172.21.15.116:0/3100736952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98f4100e60 0x7f98f4199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:46.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.055+0000 7f98f2000700 1 --2- 172.21.15.116:0/3100736952 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f4101820 0x7f98f4199940 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f98e0005b40 tx=0x7f98e000b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:46.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.055+0000 7f98eb400700 1 -- 172.21.15.116:0/3100736952 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e001d070 con 0x7f98f4101820 2024-03-30T00:56:46.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.055+0000 7f98eb400700 1 -- 172.21.15.116:0/3100736952 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98e000be00 con 0x7f98f4101820 2024-03-30T00:56:46.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.055+0000 7f98eb400700 1 -- 172.21.15.116:0/3100736952 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e000fb00 con 0x7f98f4101820 2024-03-30T00:56:46.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.055+0000 7f98f9800700 1 -- 172.21.15.116:0/3100736952 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f419e8b0 con 0x7f98f4101820 2024-03-30T00:56:46.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.055+0000 7f98f9800700 1 -- 172.21.15.116:0/3100736952 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f41a6170 con 0x7f98f4101820 2024-03-30T00:56:46.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.056+0000 7f98f9800700 1 -- 172.21.15.116:0/3100736952 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f98f4111010 con 0x7f98f4101820 2024-03-30T00:56:46.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.058+0000 7f98eb400700 1 -- 172.21.15.116:0/3100736952 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f98e0003680 con 0x7f98f4101820 2024-03-30T00:56:46.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.058+0000 7f98eb400700 1 --2- 172.21.15.116:0/3100736952 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f98d80a1230 0x7f98d80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:46.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.058+0000 7f98eb400700 1 -- 172.21.15.116:0/3100736952 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f98e00cdc60 con 0x7f98f4101820 2024-03-30T00:56:46.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.058+0000 7f98f2a00700 1 --2- 172.21.15.116:0/3100736952 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f98d80a1230 0x7f98d80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:46.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.058+0000 7f98eb400700 1 --2- 172.21.15.116:0/3100736952 >> [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112] conn(0x7f98d80ac120 0x7f98d80ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:46.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.058+0000 7f98eb400700 1 -- 172.21.15.116:0/3100736952 --> [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f98d80aec90 con 0x7f98d80ac120 2024-03-30T00:56:46.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.058+0000 7f98eb400700 1 -- 172.21.15.116:0/3100736952 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98e00cdfd0 con 0x7f98f4101820 2024-03-30T00:56:46.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.058+0000 7f98f2a00700 1 --2- 172.21.15.116:0/3100736952 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f98d80a1230 0x7f98d80a36f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f98e4011490 tx=0x7f98e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:46.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.072+0000 7f98f3400700 1 --2- 172.21.15.116:0/3100736952 >> [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112] conn(0x7f98d80ac120 0x7f98d80ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:46.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.093+0000 7f98f3400700 1 --2- 172.21.15.116:0/3100736952 >> [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112] conn(0x7f98d80ac120 0x7f98d80ae5c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:46.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.093+0000 7f98eb400700 1 -- 172.21.15.116:0/3100736952 <== osd.1 v2:172.21.15.130:6800/3819551112 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f98d80aec90 con 0x7f98d80ac120 2024-03-30T00:56:46.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.110+0000 7f98f9800700 1 -- 172.21.15.116:0/3100736952 --> [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f98f4061430 con 0x7f98d80ac120 2024-03-30T00:56:46.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.111+0000 7f98eb400700 1 -- 172.21.15.116:0/3100736952 <== osd.1 v2:172.21.15.130:6800/3819551112 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f98f4061430 con 0x7f98d80ac120 2024-03-30T00:56:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.112+0000 7f98f9800700 1 -- 172.21.15.116:0/3100736952 >> [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112] conn(0x7f98d80ac120 msgr2=0x7f98d80ae5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.112+0000 7f98f9800700 1 --2- 172.21.15.116:0/3100736952 >> [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112] conn(0x7f98d80ac120 0x7f98d80ae5c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.112+0000 7f98f9800700 1 -- 172.21.15.116:0/3100736952 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f98d80a1230 msgr2=0x7f98d80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.112+0000 7f98f9800700 1 --2- 172.21.15.116:0/3100736952 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f98d80a1230 0x7f98d80a36f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f98e4011490 tx=0x7f98e4009260 comp rx=0 tx=0).stop 2024-03-30T00:56:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.112+0000 7f98f9800700 1 -- 172.21.15.116:0/3100736952 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f4101820 msgr2=0x7f98f4199940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.112+0000 7f98f9800700 1 --2- 172.21.15.116:0/3100736952 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f4101820 0x7f98f4199940 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f98e0005b40 tx=0x7f98e000b9c0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.112+0000 7f98f3400700 1 -- 172.21.15.116:0/3100736952 reap_dead start 2024-03-30T00:56:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.112+0000 7f98f9800700 1 -- 172.21.15.116:0/3100736952 shutdown_connections 2024-03-30T00:56:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.112+0000 7f98f9800700 1 -- 172.21.15.116:0/3100736952 >> 172.21.15.116:0/3100736952 conn(0x7f98f4077fa0 msgr2=0x7f98f410c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.112+0000 7f98f9800700 1 -- 172.21.15.116:0/3100736952 shutdown_connections 2024-03-30T00:56:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.112+0000 7f98f9800700 1 -- 172.21.15.116:0/3100736952 wait complete. 2024-03-30T00:56:46.563 INFO:teuthology.orchestra.run.smithi116.stdout:103079215154 2024-03-30T00:56:46.564 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-03-30T00:56:46.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.645+0000 7f1e3da00700 1 -- 172.21.15.116:0/2101275362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e38107270 msgr2=0x7f1e3810dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:46.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.645+0000 7f1e3da00700 1 --2- 172.21.15.116:0/2101275362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e38107270 0x7f1e3810dc40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1e3000b0a0 tx=0x7f1e3000b3b0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.645+0000 7f1e3da00700 1 -- 172.21.15.116:0/2101275362 shutdown_connections 2024-03-30T00:56:46.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.645+0000 7f1e3da00700 1 --2- 172.21.15.116:0/2101275362 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e38101da0 0x7f1e38102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.645+0000 7f1e3da00700 1 --2- 172.21.15.116:0/2101275362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e38107270 0x7f1e3810dc40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.645+0000 7f1e3da00700 1 --2- 172.21.15.116:0/2101275362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e38102750 0x7f1e38106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.645+0000 7f1e3da00700 1 -- 172.21.15.116:0/2101275362 >> 172.21.15.116:0/2101275362 conn(0x7f1e380fb5f0 msgr2=0x7f1e380fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:46.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.645+0000 7f1e3da00700 1 -- 172.21.15.116:0/2101275362 shutdown_connections 2024-03-30T00:56:46.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.645+0000 7f1e3da00700 1 -- 172.21.15.116:0/2101275362 wait complete. 2024-03-30T00:56:46.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.646+0000 7f1e3da00700 1 Processor -- start 2024-03-30T00:56:46.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.646+0000 7f1e3da00700 1 -- start start 2024-03-30T00:56:46.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e3da00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e38101da0 0x7f1e38199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:46.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e3da00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e38102750 0x7f1e38199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:46.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e3da00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e38107270 0x7f1e3819de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:46.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e3da00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3819e490 con 0x7f1e38102750 2024-03-30T00:56:46.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e3da00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3819e600 con 0x7f1e38107270 2024-03-30T00:56:46.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e3da00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3819e7c0 con 0x7f1e38101da0 2024-03-30T00:56:46.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e37400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e38102750 0x7f1e38199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:46.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e37400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e38102750 0x7f1e38199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38072/0 (socket says 172.21.15.116:38072) 2024-03-30T00:56:46.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e37400700 1 -- 172.21.15.116:0/2986855872 learned_addr learned my addr 172.21.15.116:0/2986855872 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:46.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e37e00700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e38101da0 0x7f1e38199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:46.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e3d000700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e38107270 0x7f1e3819de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:46.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e37e00700 1 -- 172.21.15.116:0/2986855872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e38107270 msgr2=0x7f1e3819de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:46.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e37e00700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e38107270 0x7f1e3819de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e37e00700 1 -- 172.21.15.116:0/2986855872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e38102750 msgr2=0x7f1e38199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:46.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e37e00700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e38102750 0x7f1e38199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e37e00700 1 -- 172.21.15.116:0/2986855872 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e30009d00 con 0x7f1e38101da0 2024-03-30T00:56:46.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.647+0000 7f1e37400700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e38102750 0x7f1e38199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:46.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.648+0000 7f1e37e00700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e38101da0 0x7f1e38199590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800e9f0 tx=0x7f1e2800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:46.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.648+0000 7f1e34c00700 1 -- 172.21.15.116:0/2986855872 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2800c9c0 con 0x7f1e38101da0 2024-03-30T00:56:46.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.648+0000 7f1e34c00700 1 -- 172.21.15.116:0/2986855872 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1e28010430 con 0x7f1e38101da0 2024-03-30T00:56:46.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.648+0000 7f1e3da00700 1 -- 172.21.15.116:0/2986855872 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e3819eaa0 con 0x7f1e38101da0 2024-03-30T00:56:46.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.648+0000 7f1e34c00700 1 -- 172.21.15.116:0/2986855872 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e28018680 con 0x7f1e38101da0 2024-03-30T00:56:46.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.648+0000 7f1e3da00700 1 -- 172.21.15.116:0/2986855872 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e381a6300 con 0x7f1e38101da0 2024-03-30T00:56:46.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.649+0000 7f1e3da00700 1 -- 172.21.15.116:0/2986855872 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1e38110f40 con 0x7f1e38101da0 2024-03-30T00:56:46.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.651+0000 7f1e34c00700 1 -- 172.21.15.116:0/2986855872 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f1e28010860 con 0x7f1e38101da0 2024-03-30T00:56:46.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.652+0000 7f1e34c00700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1e1c0a11b0 0x7f1e1c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:46.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.653+0000 7f1e34c00700 1 -- 172.21.15.116:0/2986855872 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1e28014070 con 0x7f1e38101da0 2024-03-30T00:56:46.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.653+0000 7f1e37400700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1e1c0a11b0 0x7f1e1c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:46.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.653+0000 7f1e34c00700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178] conn(0x7f1e1c0ac1d0 0x7f1e1c0ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:46.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.653+0000 7f1e34c00700 1 -- 172.21.15.116:0/2986855872 --> [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e1c0aed40 con 0x7f1e1c0ac1d0 2024-03-30T00:56:46.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.653+0000 7f1e34c00700 1 -- 172.21.15.116:0/2986855872 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e280cd090 con 0x7f1e38101da0 2024-03-30T00:56:46.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.653+0000 7f1e3d000700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178] conn(0x7f1e1c0ac1d0 0x7f1e1c0ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:46.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.653+0000 7f1e37400700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1e1c0a11b0 0x7f1e1c0a3670 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1e3819aaf0 tx=0x7f1e24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:46.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.654+0000 7f1e3d000700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178] conn(0x7f1e1c0ac1d0 0x7f1e1c0ae670 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:46.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.655+0000 7f1e34c00700 1 -- 172.21.15.116:0/2986855872 <== osd.9 v2:172.21.15.116:6826/3758826178 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1e1c0aed40 con 0x7f1e1c0ac1d0 2024-03-30T00:56:46.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.671+0000 7f1e3da00700 1 -- 172.21.15.116:0/2986855872 --> [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1e38068ff0 con 0x7f1e1c0ac1d0 2024-03-30T00:56:46.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.671+0000 7f1e34c00700 1 -- 172.21.15.116:0/2986855872 <== osd.9 v2:172.21.15.116:6826/3758826178 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1e38068ff0 con 0x7f1e1c0ac1d0 2024-03-30T00:56:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.672+0000 7f1e3da00700 1 -- 172.21.15.116:0/2986855872 >> [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178] conn(0x7f1e1c0ac1d0 msgr2=0x7f1e1c0ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.672+0000 7f1e3da00700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178] conn(0x7f1e1c0ac1d0 0x7f1e1c0ae670 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.672+0000 7f1e3da00700 1 -- 172.21.15.116:0/2986855872 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1e1c0a11b0 msgr2=0x7f1e1c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.672+0000 7f1e3da00700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1e1c0a11b0 0x7f1e1c0a3670 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1e3819aaf0 tx=0x7f1e24009260 comp rx=0 tx=0).stop 2024-03-30T00:56:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.672+0000 7f1e3da00700 1 -- 172.21.15.116:0/2986855872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e38101da0 msgr2=0x7f1e38199590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.672+0000 7f1e3da00700 1 --2- 172.21.15.116:0/2986855872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e38101da0 0x7f1e38199590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800e9f0 tx=0x7f1e2800ed00 comp rx=0 tx=0).stop 2024-03-30T00:56:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.672+0000 7f1e3d000700 1 -- 172.21.15.116:0/2986855872 reap_dead start 2024-03-30T00:56:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.672+0000 7f1e3da00700 1 -- 172.21.15.116:0/2986855872 shutdown_connections 2024-03-30T00:56:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.672+0000 7f1e3da00700 1 -- 172.21.15.116:0/2986855872 >> 172.21.15.116:0/2986855872 conn(0x7f1e380fb5f0 msgr2=0x7f1e380fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:46.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.672+0000 7f1e3da00700 1 -- 172.21.15.116:0/2986855872 shutdown_connections 2024-03-30T00:56:46.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:46.672+0000 7f1e3da00700 1 -- 172.21.15.116:0/2986855872 wait complete. 2024-03-30T00:56:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:46.748 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:46 smithi116 ceph-mon[38086]: pgmap v258: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 530 B/s wr, 6 op/s; 15 B/s, 1 keys/s, 0 objects/s recovering 2024-03-30T00:56:46.751 INFO:teuthology.orchestra.run.smithi116.stdout:133143986223 2024-03-30T00:56:46.751 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-03-30T00:56:46.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:46 smithi181 ceph-mon[47622]: pgmap v258: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 530 B/s wr, 6 op/s; 15 B/s, 1 keys/s, 0 objects/s recovering 2024-03-30T00:56:46.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:46 smithi130 ceph-mon[47626]: pgmap v258: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 530 B/s wr, 6 op/s; 15 B/s, 1 keys/s, 0 objects/s recovering 2024-03-30T00:56:47.043 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:47.526 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:47.757 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:47.762 INFO:teuthology.orchestra.run.smithi116.stdout:257698037787 2024-03-30T00:56:47.762 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.9 2024-03-30T00:56:48.444 INFO:teuthology.orchestra.run.smithi116.stdout:51539607609 2024-03-30T00:56:48.444 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-03-30T00:56:48.465 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:48.683 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:48 smithi116 ceph-mon[38086]: pgmap v259: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 480 B/s wr, 6 op/s; 14 B/s, 0 keys/s, 0 objects/s recovering 2024-03-30T00:56:48.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:48 smithi181 ceph-mon[47622]: pgmap v259: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 480 B/s wr, 6 op/s; 14 B/s, 0 keys/s, 0 objects/s recovering 2024-03-30T00:56:48.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:48 smithi130 ceph-mon[47626]: pgmap v259: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 480 B/s wr, 6 op/s; 14 B/s, 0 keys/s, 0 objects/s recovering 2024-03-30T00:56:48.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.771+0000 7f13ddc00700 1 -- 172.21.15.116:0/2698611199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d81017c0 msgr2=0x7f13d8105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:48.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.771+0000 7f13ddc00700 1 --2- 172.21.15.116:0/2698611199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d81017c0 0x7f13d8105810 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009990 tx=0x7f13c4009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:48.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.772+0000 7f13ddc00700 1 -- 172.21.15.116:0/2698611199 shutdown_connections 2024-03-30T00:56:48.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.772+0000 7f13ddc00700 1 --2- 172.21.15.116:0/2698611199 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13d8100e10 0x7f13d81011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:48.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.772+0000 7f13ddc00700 1 --2- 172.21.15.116:0/2698611199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13d8105ef0 0x7f13d810dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:48.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.772+0000 7f13ddc00700 1 --2- 172.21.15.116:0/2698611199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d81017c0 0x7f13d8105810 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:48.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.772+0000 7f13ddc00700 1 -- 172.21.15.116:0/2698611199 >> 172.21.15.116:0/2698611199 conn(0x7f13d80fc6e0 msgr2=0x7f13d80feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:48.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.772+0000 7f13ddc00700 1 -- 172.21.15.116:0/2698611199 shutdown_connections 2024-03-30T00:56:48.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.772+0000 7f13ddc00700 1 -- 172.21.15.116:0/2698611199 wait complete. 2024-03-30T00:56:48.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.773+0000 7f13ddc00700 1 Processor -- start 2024-03-30T00:56:48.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.773+0000 7f13ddc00700 1 -- start start 2024-03-30T00:56:48.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.773+0000 7f13ddc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13d8100e10 0x7f13d81970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:48.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.773+0000 7f13ddc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d81017c0 0x7f13d81975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:48.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.773+0000 7f13ddc00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13d8105ef0 0x7f13d819b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:48.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.773+0000 7f13ddc00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d819bfa0 con 0x7f13d81017c0 2024-03-30T00:56:48.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.773+0000 7f13ddc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d819c110 con 0x7f13d8100e10 2024-03-30T00:56:48.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.773+0000 7f13ddc00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d819c2d0 con 0x7f13d8105ef0 2024-03-30T00:56:48.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.773+0000 7f13d7e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13d8100e10 0x7f13d81970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:48.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.773+0000 7f13d7e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13d8100e10 0x7f13d81970a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:42202/0 (socket says 172.21.15.116:42202) 2024-03-30T00:56:48.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.773+0000 7f13d7e00700 1 -- 172.21.15.116:0/3460457247 learned_addr learned my addr 172.21.15.116:0/3460457247 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:48.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.774+0000 7f13dd200700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13d8105ef0 0x7f13d819b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:48.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.774+0000 7f13d7e00700 1 -- 172.21.15.116:0/3460457247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13d8105ef0 msgr2=0x7f13d819b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:48.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.774+0000 7f13d7e00700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13d8105ef0 0x7f13d819b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:48.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.774+0000 7f13d7400700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d81017c0 0x7f13d81975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:48.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.774+0000 7f13d7e00700 1 -- 172.21.15.116:0/3460457247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d81017c0 msgr2=0x7f13d81975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:48.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.774+0000 7f13d7e00700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d81017c0 0x7f13d81975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:48.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.774+0000 7f13d7e00700 1 -- 172.21.15.116:0/3460457247 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c40095f0 con 0x7f13d8100e10 2024-03-30T00:56:48.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.774+0000 7f13d7400700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d81017c0 0x7f13d81975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T00:56:48.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.774+0000 7f13d7e00700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13d8100e10 0x7f13d81970a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f13c800d950 tx=0x7f13c800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:48.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.774+0000 7f13d4c00700 1 -- 172.21.15.116:0/3460457247 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c8009940 con 0x7f13d8100e10 2024-03-30T00:56:48.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.775+0000 7f13ddc00700 1 -- 172.21.15.116:0/3460457247 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d819c5b0 con 0x7f13d8100e10 2024-03-30T00:56:48.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.775+0000 7f13d4c00700 1 -- 172.21.15.116:0/3460457247 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13c8010460 con 0x7f13d8100e10 2024-03-30T00:56:48.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.775+0000 7f13d4c00700 1 -- 172.21.15.116:0/3460457247 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c800f620 con 0x7f13d8100e10 2024-03-30T00:56:48.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.775+0000 7f13ddc00700 1 -- 172.21.15.116:0/3460457247 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d81a3e10 con 0x7f13d8100e10 2024-03-30T00:56:48.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.775+0000 7f13ddc00700 1 -- 172.21.15.116:0/3460457247 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f13d81022f0 con 0x7f13d8100e10 2024-03-30T00:56:48.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.778+0000 7f13d4c00700 1 -- 172.21.15.116:0/3460457247 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f13c80108d0 con 0x7f13d8100e10 2024-03-30T00:56:48.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.779+0000 7f13d4c00700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13bc0a11b0 0x7f13bc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:48.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.779+0000 7f13d4c00700 1 -- 172.21.15.116:0/3460457247 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f13c80cd140 con 0x7f13d8100e10 2024-03-30T00:56:48.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.779+0000 7f13d7400700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13bc0a11b0 0x7f13bc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:48.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.779+0000 7f13d4c00700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701] conn(0x7f13bc0ac1d0 0x7f13bc0ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:48.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.779+0000 7f13d4c00700 1 -- 172.21.15.116:0/3460457247 --> [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13bc0aed40 con 0x7f13bc0ac1d0 2024-03-30T00:56:48.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.779+0000 7f13d4c00700 1 -- 172.21.15.116:0/3460457247 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13c80cd4b0 con 0x7f13d8100e10 2024-03-30T00:56:48.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.780+0000 7f13dd200700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701] conn(0x7f13bc0ac1d0 0x7f13bc0ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:48.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.780+0000 7f13d7400700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13bc0a11b0 0x7f13bc0a3670 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f13c400b2e0 tx=0x7f13c4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:48.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.780+0000 7f13dd200700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701] conn(0x7f13bc0ac1d0 0x7f13bc0ae670 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:48.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.781+0000 7f13d4c00700 1 -- 172.21.15.116:0/3460457247 <== osd.7 v2:172.21.15.130:6816/2137401701 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f13bc0aed40 con 0x7f13bc0ac1d0 2024-03-30T00:56:48.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.798+0000 7f13ddc00700 1 -- 172.21.15.116:0/3460457247 --> [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f13d8110d70 con 0x7f13bc0ac1d0 2024-03-30T00:56:48.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.798+0000 7f13d4c00700 1 -- 172.21.15.116:0/3460457247 <== osd.7 v2:172.21.15.130:6816/2137401701 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f13d8110d70 con 0x7f13bc0ac1d0 2024-03-30T00:56:48.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.799+0000 7f13ddc00700 1 -- 172.21.15.116:0/3460457247 >> [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701] conn(0x7f13bc0ac1d0 msgr2=0x7f13bc0ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:48.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.799+0000 7f13ddc00700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701] conn(0x7f13bc0ac1d0 0x7f13bc0ae670 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:48.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.799+0000 7f13ddc00700 1 -- 172.21.15.116:0/3460457247 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13bc0a11b0 msgr2=0x7f13bc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:48.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.799+0000 7f13ddc00700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13bc0a11b0 0x7f13bc0a3670 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f13c400b2e0 tx=0x7f13c4005fb0 comp rx=0 tx=0).stop 2024-03-30T00:56:48.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.799+0000 7f13ddc00700 1 -- 172.21.15.116:0/3460457247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13d8100e10 msgr2=0x7f13d81970a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:48.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.799+0000 7f13ddc00700 1 --2- 172.21.15.116:0/3460457247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13d8100e10 0x7f13d81970a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f13c800d950 tx=0x7f13c800dc60 comp rx=0 tx=0).stop 2024-03-30T00:56:48.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.799+0000 7f13dd200700 1 -- 172.21.15.116:0/3460457247 reap_dead start 2024-03-30T00:56:48.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.799+0000 7f13ddc00700 1 -- 172.21.15.116:0/3460457247 shutdown_connections 2024-03-30T00:56:48.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.799+0000 7f13ddc00700 1 -- 172.21.15.116:0/3460457247 >> 172.21.15.116:0/3460457247 conn(0x7f13d80fc6e0 msgr2=0x7f13d80fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:48.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.799+0000 7f13ddc00700 1 -- 172.21.15.116:0/3460457247 shutdown_connections 2024-03-30T00:56:48.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:48.799+0000 7f13ddc00700 1 -- 172.21.15.116:0/3460457247 wait complete. 2024-03-30T00:56:49.219 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:49.455 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:49.614 INFO:teuthology.orchestra.run.smithi116.stdout:197568495651 2024-03-30T00:56:49.615 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-03-30T00:56:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.519+0000 7f40d3200700 1 -- 172.21.15.116:0/3574742785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40d4100e10 msgr2=0x7f40d41011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.519+0000 7f40d3200700 1 --2- 172.21.15.116:0/3574742785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40d4100e10 0x7f40d41011f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f40c400b0a0 tx=0x7f40c400b3b0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.520+0000 7f40d3200700 1 -- 172.21.15.116:0/3574742785 shutdown_connections 2024-03-30T00:56:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.520+0000 7f40d3200700 1 --2- 172.21.15.116:0/3574742785 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40d4105ef0 0x7f40d410dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.520+0000 7f40d3200700 1 --2- 172.21.15.116:0/3574742785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40d4100e10 0x7f40d41011f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.520+0000 7f40d3200700 1 --2- 172.21.15.116:0/3574742785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d41017c0 0x7f40d4105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.520+0000 7f40d3200700 1 -- 172.21.15.116:0/3574742785 >> 172.21.15.116:0/3574742785 conn(0x7f40d40fc6e0 msgr2=0x7f40d40feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.520+0000 7f40d3200700 1 -- 172.21.15.116:0/3574742785 shutdown_connections 2024-03-30T00:56:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.520+0000 7f40d3200700 1 -- 172.21.15.116:0/3574742785 wait complete. 2024-03-30T00:56:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.521+0000 7f40d3200700 1 Processor -- start 2024-03-30T00:56:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.521+0000 7f40d3200700 1 -- start start 2024-03-30T00:56:50.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.521+0000 7f40d3200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40d4100e10 0x7f40d4199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:50.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.521+0000 7f40d3200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d41017c0 0x7f40d4199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:50.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.522+0000 7f40d3200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40d4105ef0 0x7f40d419dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:50.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.522+0000 7f40d3200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d419e310 con 0x7f40d41017c0 2024-03-30T00:56:50.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.522+0000 7f40d3200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d419e480 con 0x7f40d4105ef0 2024-03-30T00:56:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.522+0000 7f40d3200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d419e640 con 0x7f40d4100e10 2024-03-30T00:56:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.522+0000 7f40d1e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40d4100e10 0x7f40d4199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.522+0000 7f40cbe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d41017c0 0x7f40d4199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.522+0000 7f40d1e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40d4100e10 0x7f40d4199410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:48706/0 (socket says 172.21.15.116:48706) 2024-03-30T00:56:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.522+0000 7f40d1e00700 1 -- 172.21.15.116:0/1100879257 learned_addr learned my addr 172.21.15.116:0/1100879257 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.522+0000 7f40d2800700 1 --2- 172.21.15.116:0/1100879257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40d4105ef0 0x7f40d419dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.522+0000 7f40d1e00700 1 -- 172.21.15.116:0/1100879257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40d4105ef0 msgr2=0x7f40d419dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.522+0000 7f40d1e00700 1 --2- 172.21.15.116:0/1100879257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40d4105ef0 0x7f40d419dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.522+0000 7f40d1e00700 1 -- 172.21.15.116:0/1100879257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d41017c0 msgr2=0x7f40d4199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.522+0000 7f40d1e00700 1 --2- 172.21.15.116:0/1100879257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d41017c0 0x7f40d4199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.522+0000 7f40d1e00700 1 -- 172.21.15.116:0/1100879257 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc009520 con 0x7f40d4100e10 2024-03-30T00:56:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.523+0000 7f40d1e00700 1 --2- 172.21.15.116:0/1100879257 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40d4100e10 0x7f40d4199410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f40c400b070 tx=0x7f40c4007990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.523+0000 7f40caa00700 1 -- 172.21.15.116:0/1100879257 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c400e040 con 0x7f40d4100e10 2024-03-30T00:56:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.523+0000 7f40caa00700 1 -- 172.21.15.116:0/1100879257 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40c400be40 con 0x7f40d4100e10 2024-03-30T00:56:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.523+0000 7f40d3200700 1 -- 172.21.15.116:0/1100879257 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c4009d00 con 0x7f40d4100e10 2024-03-30T00:56:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.523+0000 7f40caa00700 1 -- 172.21.15.116:0/1100879257 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c401b6c0 con 0x7f40d4100e10 2024-03-30T00:56:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.523+0000 7f40d3200700 1 -- 172.21.15.116:0/1100879257 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d41a5fb0 con 0x7f40d4100e10 2024-03-30T00:56:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.524+0000 7f40d3200700 1 -- 172.21.15.116:0/1100879257 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d41022f0 con 0x7f40d4100e10 2024-03-30T00:56:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.526+0000 7f40caa00700 1 -- 172.21.15.116:0/1100879257 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f40c401d470 con 0x7f40d4100e10 2024-03-30T00:56:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.527+0000 7f40caa00700 1 --2- 172.21.15.116:0/1100879257 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f40b80a11b0 0x7f40b80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.527+0000 7f40caa00700 1 -- 172.21.15.116:0/1100879257 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f40c401de40 con 0x7f40d4100e10 2024-03-30T00:56:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.527+0000 7f40cbe00700 1 --2- 172.21.15.116:0/1100879257 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f40b80a11b0 0x7f40b80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.528+0000 7f40cbe00700 1 --2- 172.21.15.116:0/1100879257 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f40b80a11b0 0x7f40b80a3670 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f40d419a970 tx=0x7f40bc0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.530+0000 7f40caa00700 1 -- 172.21.15.116:0/1100879257 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f40c4097080 con 0x7f40d4100e10 2024-03-30T00:56:50.554 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:50 smithi116 ceph-mon[38086]: pgmap v260: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s; 12 B/s, 0 keys/s, 0 objects/s recovering 2024-03-30T00:56:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.776+0000 7f40d3200700 1 -- 172.21.15.116:0/1100879257 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f40d419a7c0 con 0x7f40d4100e10 2024-03-30T00:56:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.776+0000 7f40caa00700 1 -- 172.21.15.116:0/1100879257 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f40c409a6a0 con 0x7f40d4100e10 2024-03-30T00:56:50.782 INFO:teuthology.orchestra.run.smithi116.stdout:180388626472 2024-03-30T00:56:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.779+0000 7f40d3200700 1 -- 172.21.15.116:0/1100879257 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f40b80a11b0 msgr2=0x7f40b80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.779+0000 7f40d3200700 1 --2- 172.21.15.116:0/1100879257 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f40b80a11b0 0x7f40b80a3670 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f40d419a970 tx=0x7f40bc0073d0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.779+0000 7f40d3200700 1 -- 172.21.15.116:0/1100879257 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40d4100e10 msgr2=0x7f40d4199410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.779+0000 7f40d3200700 1 --2- 172.21.15.116:0/1100879257 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40d4100e10 0x7f40d4199410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f40c400b070 tx=0x7f40c4007990 comp rx=0 tx=0).stop 2024-03-30T00:56:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.779+0000 7f40d3200700 1 -- 172.21.15.116:0/1100879257 shutdown_connections 2024-03-30T00:56:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.779+0000 7f40d3200700 1 --2- 172.21.15.116:0/1100879257 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f40b80a11b0 0x7f40b80a3670 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.779+0000 7f40d3200700 1 --2- 172.21.15.116:0/1100879257 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40d4100e10 0x7f40d4199410 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.779+0000 7f40d3200700 1 --2- 172.21.15.116:0/1100879257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40d4105ef0 0x7f40d419dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.779+0000 7f40d3200700 1 --2- 172.21.15.116:0/1100879257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d41017c0 0x7f40d4199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.779+0000 7f40d3200700 1 -- 172.21.15.116:0/1100879257 >> 172.21.15.116:0/1100879257 conn(0x7f40d40fc6e0 msgr2=0x7f40d40fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.779+0000 7f40d3200700 1 -- 172.21.15.116:0/1100879257 shutdown_connections 2024-03-30T00:56:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.780+0000 7f40d3200700 1 -- 172.21.15.116:0/1100879257 wait complete. 2024-03-30T00:56:50.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:50 smithi130 ceph-mon[47626]: pgmap v260: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s; 12 B/s, 0 keys/s, 0 objects/s recovering 2024-03-30T00:56:50.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.954+0000 7f3446e00700 1 -- 172.21.15.116:0/3266765181 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34400731e0 msgr2=0x7f34400735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:50.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.954+0000 7f3446e00700 1 --2- 172.21.15.116:0/3266765181 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34400731e0 0x7f34400735c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f34340098f0 tx=0x7f3434009c00 comp rx=0 tx=0).stop 2024-03-30T00:56:50.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.954+0000 7f3446e00700 1 -- 172.21.15.116:0/3266765181 shutdown_connections 2024-03-30T00:56:50.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.954+0000 7f3446e00700 1 --2- 172.21.15.116:0/3266765181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f344010d3f0 0x7f344010f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.954+0000 7f3446e00700 1 --2- 172.21.15.116:0/3266765181 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34400731e0 0x7f34400735c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.954+0000 7f3446e00700 1 --2- 172.21.15.116:0/3266765181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3440073b00 0x7f344010ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.954+0000 7f3446e00700 1 -- 172.21.15.116:0/3266765181 >> 172.21.15.116:0/3266765181 conn(0x7f34400fc6e0 msgr2=0x7f34400feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:50.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.954+0000 7f3446e00700 1 -- 172.21.15.116:0/3266765181 shutdown_connections 2024-03-30T00:56:50.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.954+0000 7f3446e00700 1 -- 172.21.15.116:0/3266765181 wait complete. 2024-03-30T00:56:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.955+0000 7f3446e00700 1 Processor -- start 2024-03-30T00:56:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.956+0000 7f3446e00700 1 -- start start 2024-03-30T00:56:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.956+0000 7f3446e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34400731e0 0x7f3440199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.956+0000 7f3446e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3440073b00 0x7f3440199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.956+0000 7f3446e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f344010d3f0 0x7f344019dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.956+0000 7f3446e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344019e200 con 0x7f3440073b00 2024-03-30T00:56:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.956+0000 7f3446e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344019e370 con 0x7f34400731e0 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.956+0000 7f3446e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344019e530 con 0x7f344010d3f0 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.956+0000 7f3445000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3440073b00 0x7f3440199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.956+0000 7f3445000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3440073b00 0x7f3440199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38134/0 (socket says 172.21.15.116:38134) 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.956+0000 7f3445a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34400731e0 0x7f3440199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.956+0000 7f3445000700 1 -- 172.21.15.116:0/2367814941 learned_addr learned my addr 172.21.15.116:0/2367814941 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.956+0000 7f3446400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f344010d3f0 0x7f344019dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.957+0000 7f3445a00700 1 -- 172.21.15.116:0/2367814941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f344010d3f0 msgr2=0x7f344019dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.957+0000 7f3445a00700 1 --2- 172.21.15.116:0/2367814941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f344010d3f0 0x7f344019dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.957+0000 7f3445a00700 1 -- 172.21.15.116:0/2367814941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3440073b00 msgr2=0x7f3440199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.957+0000 7f3445a00700 1 --2- 172.21.15.116:0/2367814941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3440073b00 0x7f3440199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.957+0000 7f3445a00700 1 -- 172.21.15.116:0/2367814941 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342c0095f0 con 0x7f34400731e0 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.957+0000 7f3445a00700 1 --2- 172.21.15.116:0/2367814941 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34400731e0 0x7f3440199300 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3434009fd0 tx=0x7f343400fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.957+0000 7f343e000700 1 -- 172.21.15.116:0/2367814941 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343401c070 con 0x7f34400731e0 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.957+0000 7f343e000700 1 -- 172.21.15.116:0/2367814941 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f343400baa0 con 0x7f34400731e0 2024-03-30T00:56:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.957+0000 7f3446e00700 1 -- 172.21.15.116:0/2367814941 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3434009520 con 0x7f34400731e0 2024-03-30T00:56:50.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.957+0000 7f343e000700 1 -- 172.21.15.116:0/2367814941 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34340038e0 con 0x7f34400731e0 2024-03-30T00:56:50.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.958+0000 7f3446e00700 1 -- 172.21.15.116:0/2367814941 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34401a5ea0 con 0x7f34400731e0 2024-03-30T00:56:50.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.958+0000 7f3446e00700 1 -- 172.21.15.116:0/2367814941 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3440108600 con 0x7f34400731e0 2024-03-30T00:56:50.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.961+0000 7f343e000700 1 -- 172.21.15.116:0/2367814941 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f3434020030 con 0x7f34400731e0 2024-03-30T00:56:50.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.962+0000 7f343e000700 1 --2- 172.21.15.116:0/2367814941 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f34280a11b0 0x7f34280a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:50.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.962+0000 7f343e000700 1 -- 172.21.15.116:0/2367814941 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f34340cddd0 con 0x7f34400731e0 2024-03-30T00:56:50.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.962+0000 7f3445000700 1 --2- 172.21.15.116:0/2367814941 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f34280a11b0 0x7f34280a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:50.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.963+0000 7f3445000700 1 --2- 172.21.15.116:0/2367814941 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f34280a11b0 0x7f34280a3670 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f344019a860 tx=0x7f342c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:50.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.964+0000 7f343e000700 1 -- 172.21.15.116:0/2367814941 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3434097120 con 0x7f34400731e0 2024-03-30T00:56:51.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:50 smithi181 ceph-mon[47622]: pgmap v260: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s; 12 B/s, 0 keys/s, 0 objects/s recovering 2024-03-30T00:56:51.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.189+0000 7f3446e00700 1 -- 172.21.15.116:0/2367814941 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f344019a610 con 0x7f34400731e0 2024-03-30T00:56:51.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.190+0000 7f343e000700 1 -- 172.21.15.116:0/2367814941 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f343409a740 con 0x7f34400731e0 2024-03-30T00:56:51.195 INFO:teuthology.orchestra.run.smithi116.stdout:150323855403 2024-03-30T00:56:51.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.192+0000 7f3446e00700 1 -- 172.21.15.116:0/2367814941 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f34280a11b0 msgr2=0x7f34280a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:51.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.192+0000 7f3446e00700 1 --2- 172.21.15.116:0/2367814941 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f34280a11b0 0x7f34280a3670 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f344019a860 tx=0x7f342c009310 comp rx=0 tx=0).stop 2024-03-30T00:56:51.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.192+0000 7f3446e00700 1 -- 172.21.15.116:0/2367814941 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34400731e0 msgr2=0x7f3440199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:51.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.192+0000 7f3446e00700 1 --2- 172.21.15.116:0/2367814941 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34400731e0 0x7f3440199300 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3434009fd0 tx=0x7f343400fa00 comp rx=0 tx=0).stop 2024-03-30T00:56:51.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.192+0000 7f3446e00700 1 -- 172.21.15.116:0/2367814941 shutdown_connections 2024-03-30T00:56:51.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.192+0000 7f3446e00700 1 --2- 172.21.15.116:0/2367814941 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f34280a11b0 0x7f34280a3670 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.192+0000 7f3446e00700 1 --2- 172.21.15.116:0/2367814941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f344010d3f0 0x7f344019dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.192+0000 7f3446e00700 1 --2- 172.21.15.116:0/2367814941 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34400731e0 0x7f3440199300 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.192+0000 7f3446e00700 1 --2- 172.21.15.116:0/2367814941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3440073b00 0x7f3440199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.192+0000 7f3446e00700 1 -- 172.21.15.116:0/2367814941 >> 172.21.15.116:0/2367814941 conn(0x7f34400fc6e0 msgr2=0x7f34400fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:51.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.192+0000 7f3446e00700 1 -- 172.21.15.116:0/2367814941 shutdown_connections 2024-03-30T00:56:51.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.192+0000 7f3446e00700 1 -- 172.21.15.116:0/2367814941 wait complete. 2024-03-30T00:56:51.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f717a600700 1 -- 172.21.15.116:0/2592275210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7174100e60 msgr2=0x7f71741012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:51.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f717a600700 1 --2- 172.21.15.116:0/2592275210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7174100e60 0x7f71741012e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7160009920 tx=0x7f7160009c30 comp rx=0 tx=0).stop 2024-03-30T00:56:51.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f717a600700 1 -- 172.21.15.116:0/2592275210 shutdown_connections 2024-03-30T00:56:51.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f717a600700 1 --2- 172.21.15.116:0/2592275210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7174106ef0 0x7f71741072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f717a600700 1 --2- 172.21.15.116:0/2592275210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7174101820 0x7f717410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f717a600700 1 --2- 172.21.15.116:0/2592275210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7174100e60 0x7f71741012e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f717a600700 1 -- 172.21.15.116:0/2592275210 >> 172.21.15.116:0/2592275210 conn(0x7f7174077fa0 msgr2=0x7f71740783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:51.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f717a600700 1 -- 172.21.15.116:0/2592275210 shutdown_connections 2024-03-30T00:56:51.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f717a600700 1 -- 172.21.15.116:0/2592275210 wait complete. 2024-03-30T00:56:51.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.141+0000 7f717a600700 1 Processor -- start 2024-03-30T00:56:51.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.141+0000 7f717a600700 1 -- start start 2024-03-30T00:56:51.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.142+0000 7f717a600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7174100e60 0x7f71741972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:51.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.142+0000 7f717a600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7174101820 0x7f7174197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:51.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.142+0000 7f717a600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7174106ef0 0x7f717419bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:51.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.142+0000 7f717a600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717419c1c0 con 0x7f7174106ef0 2024-03-30T00:56:51.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.142+0000 7f717a600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717419c330 con 0x7f7174101820 2024-03-30T00:56:51.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.142+0000 7f717a600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717419c4f0 con 0x7f7174100e60 2024-03-30T00:56:51.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.142+0000 7f7173e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7174106ef0 0x7f717419bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:51.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.142+0000 7f7172a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7174101820 0x7f7174197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:51.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.142+0000 7f7173400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7174100e60 0x7f71741972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:51.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.142+0000 7f7173e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7174106ef0 0x7f717419bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38156/0 (socket says 172.21.15.116:38156) 2024-03-30T00:56:51.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.142+0000 7f7173e00700 1 -- 172.21.15.116:0/4206037585 learned_addr learned my addr 172.21.15.116:0/4206037585 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:51.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.142+0000 7f7172a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7174101820 0x7f7174197800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:42270/0 (socket says 172.21.15.116:42270) 2024-03-30T00:56:51.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.143+0000 7f7173e00700 1 -- 172.21.15.116:0/4206037585 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7174100e60 msgr2=0x7f71741972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:51.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.143+0000 7f7173e00700 1 --2- 172.21.15.116:0/4206037585 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7174100e60 0x7f71741972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.143+0000 7f7173e00700 1 -- 172.21.15.116:0/4206037585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7174101820 msgr2=0x7f7174197800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:51.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.143+0000 7f7173e00700 1 --2- 172.21.15.116:0/4206037585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7174101820 0x7f7174197800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.143+0000 7f7173e00700 1 -- 172.21.15.116:0/4206037585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c009c40 con 0x7f7174106ef0 2024-03-30T00:56:51.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.143+0000 7f7173e00700 1 --2- 172.21.15.116:0/4206037585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7174106ef0 0x7f717419bbb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f716c00c0f0 tx=0x7f716c00c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:51.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.143+0000 7f716be00700 1 -- 172.21.15.116:0/4206037585 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716c003a00 con 0x7f7174106ef0 2024-03-30T00:56:51.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.143+0000 7f716be00700 1 -- 172.21.15.116:0/4206037585 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f716c00e030 con 0x7f7174106ef0 2024-03-30T00:56:51.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.143+0000 7f717a600700 1 -- 172.21.15.116:0/4206037585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7160009520 con 0x7f7174106ef0 2024-03-30T00:56:51.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.143+0000 7f716be00700 1 -- 172.21.15.116:0/4206037585 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716c004630 con 0x7f7174106ef0 2024-03-30T00:56:51.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.143+0000 7f717a600700 1 -- 172.21.15.116:0/4206037585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7174072fe0 con 0x7f7174106ef0 2024-03-30T00:56:51.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.144+0000 7f717a600700 1 -- 172.21.15.116:0/4206037585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71741092f0 con 0x7f7174106ef0 2024-03-30T00:56:51.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.149+0000 7f716be00700 1 -- 172.21.15.116:0/4206037585 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f716c004790 con 0x7f7174106ef0 2024-03-30T00:56:51.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.150+0000 7f716be00700 1 --2- 172.21.15.116:0/4206037585 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f71580a11b0 0x7f71580a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:51.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.151+0000 7f716be00700 1 -- 172.21.15.116:0/4206037585 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f716c0cca50 con 0x7f7174106ef0 2024-03-30T00:56:51.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.151+0000 7f7173400700 1 --2- 172.21.15.116:0/4206037585 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f71580a11b0 0x7f71580a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:51.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.151+0000 7f7173400700 1 --2- 172.21.15.116:0/4206037585 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f71580a11b0 0x7f71580a3670 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7164005fd0 tx=0x7f7164009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:51.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.155+0000 7f716be00700 1 -- 172.21.15.116:0/4206037585 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f716c095d10 con 0x7f7174106ef0 2024-03-30T00:56:51.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.402+0000 7f717a600700 1 -- 172.21.15.116:0/4206037585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f71741024f0 con 0x7f7174106ef0 2024-03-30T00:56:51.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.403+0000 7f716be00700 1 -- 172.21.15.116:0/4206037585 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f716c099330 con 0x7f7174106ef0 2024-03-30T00:56:51.409 INFO:teuthology.orchestra.run.smithi116.stdout:73014444087 2024-03-30T00:56:51.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.405+0000 7f717a600700 1 -- 172.21.15.116:0/4206037585 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f71580a11b0 msgr2=0x7f71580a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:51.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.405+0000 7f717a600700 1 --2- 172.21.15.116:0/4206037585 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f71580a11b0 0x7f71580a3670 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7164005fd0 tx=0x7f7164009310 comp rx=0 tx=0).stop 2024-03-30T00:56:51.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.405+0000 7f717a600700 1 -- 172.21.15.116:0/4206037585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7174106ef0 msgr2=0x7f717419bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:51.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.405+0000 7f717a600700 1 --2- 172.21.15.116:0/4206037585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7174106ef0 0x7f717419bbb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f716c00c0f0 tx=0x7f716c00c400 comp rx=0 tx=0).stop 2024-03-30T00:56:51.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.405+0000 7f717a600700 1 -- 172.21.15.116:0/4206037585 shutdown_connections 2024-03-30T00:56:51.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.405+0000 7f717a600700 1 --2- 172.21.15.116:0/4206037585 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f71580a11b0 0x7f71580a3670 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.405+0000 7f717a600700 1 --2- 172.21.15.116:0/4206037585 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7174100e60 0x7f71741972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.405+0000 7f717a600700 1 --2- 172.21.15.116:0/4206037585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7174106ef0 0x7f717419bbb0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.405+0000 7f717a600700 1 --2- 172.21.15.116:0/4206037585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7174101820 0x7f7174197800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.405+0000 7f717a600700 1 -- 172.21.15.116:0/4206037585 >> 172.21.15.116:0/4206037585 conn(0x7f7174077fa0 msgr2=0x7f71740ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:51.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.405+0000 7f717a600700 1 -- 172.21.15.116:0/4206037585 shutdown_connections 2024-03-30T00:56:51.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.406+0000 7f717a600700 1 -- 172.21.15.116:0/4206037585 wait complete. 2024-03-30T00:56:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.896+0000 7f98c1c00700 1 -- 172.21.15.116:0/2305507951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98bc106ef0 msgr2=0x7f98bc1072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.896+0000 7f98c1c00700 1 --2- 172.21.15.116:0/2305507951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98bc106ef0 0x7f98bc1072d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f98ac009990 tx=0x7f98ac009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.896+0000 7f98c1c00700 1 -- 172.21.15.116:0/2305507951 shutdown_connections 2024-03-30T00:56:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.896+0000 7f98c1c00700 1 --2- 172.21.15.116:0/2305507951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98bc106ef0 0x7f98bc1072d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.896+0000 7f98c1c00700 1 --2- 172.21.15.116:0/2305507951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98bc0ff890 0x7f98bc10dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.896+0000 7f98c1c00700 1 --2- 172.21.15.116:0/2305507951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98bc0feed0 0x7f98bc0ff350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.896+0000 7f98c1c00700 1 -- 172.21.15.116:0/2305507951 >> 172.21.15.116:0/2305507951 conn(0x7f98bc075e80 msgr2=0x7f98bc0fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.896+0000 7f98c1c00700 1 -- 172.21.15.116:0/2305507951 shutdown_connections 2024-03-30T00:56:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.897+0000 7f98c1c00700 1 -- 172.21.15.116:0/2305507951 wait complete. 2024-03-30T00:56:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.898+0000 7f98c1c00700 1 Processor -- start 2024-03-30T00:56:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.898+0000 7f98c1c00700 1 -- start start 2024-03-30T00:56:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.898+0000 7f98c1c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98bc0feed0 0x7f98bc199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.898+0000 7f98c1c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98bc0ff890 0x7f98bc199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.898+0000 7f98c1c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98bc106ef0 0x7f98bc19dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.898+0000 7f98c1c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc19e230 con 0x7f98bc106ef0 2024-03-30T00:56:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.898+0000 7f98c1c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc19e3a0 con 0x7f98bc0ff890 2024-03-30T00:56:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.898+0000 7f98c1c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc19e560 con 0x7f98bc0feed0 2024-03-30T00:56:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.898+0000 7f98baa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98bc0feed0 0x7f98bc199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.898+0000 7f98baa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98bc0feed0 0x7f98bc199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:48712/0 (socket says 172.21.15.116:48712) 2024-03-30T00:56:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.899+0000 7f98baa00700 1 -- 172.21.15.116:0/1347235112 learned_addr learned my addr 172.21.15.116:0/1347235112 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.899+0000 7f98bb400700 1 --2- 172.21.15.116:0/1347235112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98bc106ef0 0x7f98bc19dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.899+0000 7f98ba000700 1 --2- 172.21.15.116:0/1347235112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98bc0ff890 0x7f98bc199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.899+0000 7f98baa00700 1 -- 172.21.15.116:0/1347235112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98bc0ff890 msgr2=0x7f98bc199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.899+0000 7f98baa00700 1 --2- 172.21.15.116:0/1347235112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98bc0ff890 0x7f98bc199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.899+0000 7f98baa00700 1 -- 172.21.15.116:0/1347235112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98bc106ef0 msgr2=0x7f98bc19dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.899+0000 7f98baa00700 1 --2- 172.21.15.116:0/1347235112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98bc106ef0 0x7f98bc19dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.899+0000 7f98baa00700 1 -- 172.21.15.116:0/1347235112 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac0095f0 con 0x7f98bc0feed0 2024-03-30T00:56:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.899+0000 7f98baa00700 1 --2- 172.21.15.116:0/1347235112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98bc0feed0 0x7f98bc199330 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f98ac005720 tx=0x7f98ac005750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.899+0000 7f98b3400700 1 -- 172.21.15.116:0/1347235112 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac01d070 con 0x7f98bc0feed0 2024-03-30T00:56:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.900+0000 7f98c1c00700 1 -- 172.21.15.116:0/1347235112 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98bc19e7e0 con 0x7f98bc0feed0 2024-03-30T00:56:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.900+0000 7f98b3400700 1 -- 172.21.15.116:0/1347235112 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98ac022470 con 0x7f98bc0feed0 2024-03-30T00:56:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.900+0000 7f98b3400700 1 -- 172.21.15.116:0/1347235112 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac017650 con 0x7f98bc0feed0 2024-03-30T00:56:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.900+0000 7f98c1c00700 1 -- 172.21.15.116:0/1347235112 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98bc19eaa0 con 0x7f98bc0feed0 2024-03-30T00:56:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.901+0000 7f98c1c00700 1 -- 172.21.15.116:0/1347235112 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98bc1002a0 con 0x7f98bc0feed0 2024-03-30T00:56:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.903+0000 7f98b3400700 1 -- 172.21.15.116:0/1347235112 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f98ac0225e0 con 0x7f98bc0feed0 2024-03-30T00:56:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.904+0000 7f98b3400700 1 --2- 172.21.15.116:0/1347235112 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f98a00a1230 0x7f98a00a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.904+0000 7f98b3400700 1 -- 172.21.15.116:0/1347235112 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f98ac0ce060 con 0x7f98bc0feed0 2024-03-30T00:56:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.904+0000 7f98ba000700 1 --2- 172.21.15.116:0/1347235112 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f98a00a1230 0x7f98a00a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.905+0000 7f98ba000700 1 --2- 172.21.15.116:0/1347235112 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f98a00a1230 0x7f98a00a36f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f98bc19a890 tx=0x7f98a800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:50.907+0000 7f98b3400700 1 -- 172.21.15.116:0/1347235112 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f98ac0973b0 con 0x7f98bc0feed0 2024-03-30T00:56:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.135+0000 7f98c1c00700 1 -- 172.21.15.116:0/1347235112 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f98bc110cc0 con 0x7f98bc0feed0 2024-03-30T00:56:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.136+0000 7f98b3400700 1 -- 172.21.15.116:0/1347235112 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f98ac09a9d0 con 0x7f98bc0feed0 2024-03-30T00:56:51.700 INFO:teuthology.orchestra.run.smithi116.stdout:279172874264 2024-03-30T00:56:51.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f98c1c00700 1 -- 172.21.15.116:0/1347235112 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f98a00a1230 msgr2=0x7f98a00a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:51.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f98c1c00700 1 --2- 172.21.15.116:0/1347235112 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f98a00a1230 0x7f98a00a36f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f98bc19a890 tx=0x7f98a800b410 comp rx=0 tx=0).stop 2024-03-30T00:56:51.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f98c1c00700 1 -- 172.21.15.116:0/1347235112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98bc0feed0 msgr2=0x7f98bc199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:51.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f98c1c00700 1 --2- 172.21.15.116:0/1347235112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98bc0feed0 0x7f98bc199330 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f98ac005720 tx=0x7f98ac005750 comp rx=0 tx=0).stop 2024-03-30T00:56:51.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f98c1c00700 1 -- 172.21.15.116:0/1347235112 shutdown_connections 2024-03-30T00:56:51.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f98c1c00700 1 --2- 172.21.15.116:0/1347235112 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f98a00a1230 0x7f98a00a36f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f98c1c00700 1 --2- 172.21.15.116:0/1347235112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98bc0feed0 0x7f98bc199330 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f98c1c00700 1 --2- 172.21.15.116:0/1347235112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98bc106ef0 0x7f98bc19dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f98c1c00700 1 --2- 172.21.15.116:0/1347235112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98bc0ff890 0x7f98bc199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:51.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f98c1c00700 1 -- 172.21.15.116:0/1347235112 >> 172.21.15.116:0/1347235112 conn(0x7f98bc075e80 msgr2=0x7f98bc0fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:51.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f98c1c00700 1 -- 172.21.15.116:0/1347235112 shutdown_connections 2024-03-30T00:56:51.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:51.139+0000 7f98c1c00700 1 -- 172.21.15.116:0/1347235112 wait complete. 2024-03-30T00:56:51.797 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:51 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1100879257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T00:56:51.797 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:51 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1347235112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-30T00:56:51.797 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:51 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2367814941' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T00:56:51.797 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:51 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/4206037585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T00:56:51.802 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:51.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:51 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1100879257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T00:56:51.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:51 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1347235112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-30T00:56:51.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:51 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2367814941' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T00:56:51.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:51 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/4206037585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T00:56:52.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:51 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1100879257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T00:56:52.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:51 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1347235112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-30T00:56:52.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:51 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2367814941' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T00:56:52.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:51 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/4206037585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T00:56:52.225 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626471 got 180388626472 for osd.6 2024-03-30T00:56:52.225 DEBUG:teuthology.parallel:result is None 2024-03-30T00:56:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.385+0000 7f6fd5200700 1 -- 172.21.15.116:0/1132397307 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6fd0100e60 msgr2=0x7f6fd01012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.385+0000 7f6fd5200700 1 --2- 172.21.15.116:0/1132397307 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6fd0100e60 0x7f6fd01012e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0098f0 tx=0x7f6fbc009c00 comp rx=0 tx=0).stop 2024-03-30T00:56:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.385+0000 7f6fd5200700 1 -- 172.21.15.116:0/1132397307 shutdown_connections 2024-03-30T00:56:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.385+0000 7f6fd5200700 1 --2- 172.21.15.116:0/1132397307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fd0101820 0x7f6fd010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.385+0000 7f6fd5200700 1 --2- 172.21.15.116:0/1132397307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd0106ef0 0x7f6fd01072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.385+0000 7f6fd5200700 1 --2- 172.21.15.116:0/1132397307 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6fd0100e60 0x7f6fd01012e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.385+0000 7f6fd5200700 1 -- 172.21.15.116:0/1132397307 >> 172.21.15.116:0/1132397307 conn(0x7f6fd0077fa0 msgr2=0x7f6fd00783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.385+0000 7f6fd5200700 1 -- 172.21.15.116:0/1132397307 shutdown_connections 2024-03-30T00:56:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.385+0000 7f6fd5200700 1 -- 172.21.15.116:0/1132397307 wait complete. 2024-03-30T00:56:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.386+0000 7f6fd5200700 1 Processor -- start 2024-03-30T00:56:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.386+0000 7f6fd5200700 1 -- start start 2024-03-30T00:56:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.387+0000 7f6fd5200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd0100e60 0x7f6fd00730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.387+0000 7f6fd5200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6fd0101820 0x7f6fd00735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.387+0000 7f6fd5200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fd0106ef0 0x7f6fd0073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.387+0000 7f6fd5200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd01a56e0 con 0x7f6fd0100e60 2024-03-30T00:56:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.387+0000 7f6fd5200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd01a5850 con 0x7f6fd0106ef0 2024-03-30T00:56:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.387+0000 7f6fd5200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd01a5a10 con 0x7f6fd0101820 2024-03-30T00:56:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.387+0000 7f6fcde00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd0100e60 0x7f6fd00730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.387+0000 7f6fce800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fd0106ef0 0x7f6fd0073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.387+0000 7f6fcde00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd0100e60 0x7f6fd00730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38176/0 (socket says 172.21.15.116:38176) 2024-03-30T00:56:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.387+0000 7f6fcd400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6fd0101820 0x7f6fd00735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.387+0000 7f6fcde00700 1 -- 172.21.15.116:0/1592803630 learned_addr learned my addr 172.21.15.116:0/1592803630 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.388+0000 7f6fcd400700 1 -- 172.21.15.116:0/1592803630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fd0106ef0 msgr2=0x7f6fd0073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.388+0000 7f6fcd400700 1 --2- 172.21.15.116:0/1592803630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fd0106ef0 0x7f6fd0073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.388+0000 7f6fcd400700 1 -- 172.21.15.116:0/1592803630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd0100e60 msgr2=0x7f6fd00730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.388+0000 7f6fcd400700 1 --2- 172.21.15.116:0/1592803630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd0100e60 0x7f6fd00730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.388+0000 7f6fcd400700 1 -- 172.21.15.116:0/1592803630 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc00095f0 con 0x7f6fd0101820 2024-03-30T00:56:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.388+0000 7f6fce800700 1 --2- 172.21.15.116:0/1592803630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fd0106ef0 0x7f6fd0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.388+0000 7f6fcd400700 1 --2- 172.21.15.116:0/1592803630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6fd0101820 0x7f6fd00735f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009fd0 tx=0x7f6fbc00fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.388+0000 7f6fc6a00700 1 -- 172.21.15.116:0/1592803630 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc01c070 con 0x7f6fd0101820 2024-03-30T00:56:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.388+0000 7f6fc6a00700 1 -- 172.21.15.116:0/1592803630 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fbc00ba20 con 0x7f6fd0101820 2024-03-30T00:56:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.388+0000 7f6fd5200700 1 -- 172.21.15.116:0/1592803630 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fbc009520 con 0x7f6fd0101820 2024-03-30T00:56:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.388+0000 7f6fc6a00700 1 -- 172.21.15.116:0/1592803630 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0038e0 con 0x7f6fd0101820 2024-03-30T00:56:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.388+0000 7f6fd5200700 1 -- 172.21.15.116:0/1592803630 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd01a5f70 con 0x7f6fd0101820 2024-03-30T00:56:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.389+0000 7f6fd5200700 1 -- 172.21.15.116:0/1592803630 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd01092f0 con 0x7f6fd0101820 2024-03-30T00:56:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.391+0000 7f6fc6a00700 1 -- 172.21.15.116:0/1592803630 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f6fbc020030 con 0x7f6fd0101820 2024-03-30T00:56:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.393+0000 7f6fc6a00700 1 --2- 172.21.15.116:0/1592803630 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6fb40a1230 0x7f6fb40a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.393+0000 7f6fc6a00700 1 -- 172.21.15.116:0/1592803630 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6fbc0cdff0 con 0x7f6fd0101820 2024-03-30T00:56:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.393+0000 7f6fcde00700 1 --2- 172.21.15.116:0/1592803630 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6fb40a1230 0x7f6fb40a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.394+0000 7f6fcde00700 1 --2- 172.21.15.116:0/1592803630 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6fb40a1230 0x7f6fb40a36f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0005d10 tx=0x7f6fc0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.396+0000 7f6fc6a00700 1 -- 172.21.15.116:0/1592803630 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6fbc097380 con 0x7f6fd0101820 2024-03-30T00:56:52.577 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855402 got 150323855403 for osd.5 2024-03-30T00:56:52.577 DEBUG:teuthology.parallel:result is None 2024-03-30T00:56:52.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.633+0000 7f6fd5200700 1 -- 172.21.15.116:0/1592803630 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f6fd01024f0 con 0x7f6fd0101820 2024-03-30T00:56:52.655 INFO:teuthology.orchestra.run.smithi116.stdout:309237645334 2024-03-30T00:56:52.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.634+0000 7f6fc6a00700 1 -- 172.21.15.116:0/1592803630 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f6fbc09a9a0 con 0x7f6fd0101820 2024-03-30T00:56:52.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.637+0000 7f6fd5200700 1 -- 172.21.15.116:0/1592803630 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6fb40a1230 msgr2=0x7f6fb40a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:52.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.637+0000 7f6fd5200700 1 --2- 172.21.15.116:0/1592803630 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6fb40a1230 0x7f6fb40a36f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0005d10 tx=0x7f6fc0009310 comp rx=0 tx=0).stop 2024-03-30T00:56:52.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.637+0000 7f6fd5200700 1 -- 172.21.15.116:0/1592803630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6fd0101820 msgr2=0x7f6fd00735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:52.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.637+0000 7f6fd5200700 1 --2- 172.21.15.116:0/1592803630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6fd0101820 0x7f6fd00735f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009fd0 tx=0x7f6fbc00fa00 comp rx=0 tx=0).stop 2024-03-30T00:56:52.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.637+0000 7f6fd5200700 1 -- 172.21.15.116:0/1592803630 shutdown_connections 2024-03-30T00:56:52.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.637+0000 7f6fd5200700 1 --2- 172.21.15.116:0/1592803630 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6fb40a1230 0x7f6fb40a36f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:52.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.637+0000 7f6fd5200700 1 --2- 172.21.15.116:0/1592803630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fd0106ef0 0x7f6fd0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:52.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.637+0000 7f6fd5200700 1 --2- 172.21.15.116:0/1592803630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd0100e60 0x7f6fd00730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:52.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.637+0000 7f6fd5200700 1 --2- 172.21.15.116:0/1592803630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6fd0101820 0x7f6fd00735f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:52.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.637+0000 7f6fd5200700 1 -- 172.21.15.116:0/1592803630 >> 172.21.15.116:0/1592803630 conn(0x7f6fd0077fa0 msgr2=0x7f6fd00ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:52.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.637+0000 7f6fd5200700 1 -- 172.21.15.116:0/1592803630 shutdown_connections 2024-03-30T00:56:52.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:52.637+0000 7f6fd5200700 1 -- 172.21.15.116:0/1592803630 wait complete. 2024-03-30T00:56:52.665 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:52 smithi116 ceph-mon[38086]: pgmap v261: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-30T00:56:52.665 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:52 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:56:52.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:52 smithi130 ceph-mon[47626]: pgmap v261: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-30T00:56:52.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:52 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:56:53.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:52 smithi181 ceph-mon[47622]: pgmap v261: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-30T00:56:53.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:52 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:56:53.212 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645332 got 309237645334 for osd.11 2024-03-30T00:56:53.212 DEBUG:teuthology.parallel:result is None 2024-03-30T00:56:53.285 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:53.390 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:53.418 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874263 got 279172874264 for osd.10 2024-03-30T00:56:53.418 DEBUG:teuthology.parallel:result is None 2024-03-30T00:56:53.530 INFO:tasks.cephadm.ceph_manager.ceph:need seq 73014444085 got 73014444087 for osd.2 2024-03-30T00:56:53.531 DEBUG:teuthology.parallel:result is None 2024-03-30T00:56:53.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:53 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1592803630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-30T00:56:53.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:56:53.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:56:53.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:56:53.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:56:53.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:53 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1592803630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-30T00:56:53.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:56:53.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:56:53.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:56:53.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:56:54.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:53 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1592803630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-30T00:56:54.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:56:54.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:56:54.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:56:54.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:56:54.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.695+0000 7fef8ae00700 1 -- 172.21.15.116:0/1746082055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef84073520 msgr2=0x7fef840739a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:54.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.695+0000 7fef8ae00700 1 --2- 172.21.15.116:0/1746082055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef84073520 0x7fef840739a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef70009990 tx=0x7fef70009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.695+0000 7fef8ae00700 1 -- 172.21.15.116:0/1746082055 shutdown_connections 2024-03-30T00:56:54.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.695+0000 7fef8ae00700 1 --2- 172.21.15.116:0/1746082055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef84111a40 0x7fef84113e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.695+0000 7fef8ae00700 1 --2- 172.21.15.116:0/1746082055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84074de0 0x7fef84072fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.695+0000 7fef8ae00700 1 --2- 172.21.15.116:0/1746082055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef84073520 0x7fef840739a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.695+0000 7fef8ae00700 1 -- 172.21.15.116:0/1746082055 >> 172.21.15.116:0/1746082055 conn(0x7fef84077f10 msgr2=0x7fef84078320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:54.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.695+0000 7fef8ae00700 1 -- 172.21.15.116:0/1746082055 shutdown_connections 2024-03-30T00:56:54.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.695+0000 7fef8ae00700 1 -- 172.21.15.116:0/1746082055 wait complete. 2024-03-30T00:56:54.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.696+0000 7fef8ae00700 1 Processor -- start 2024-03-30T00:56:54.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.696+0000 7fef8ae00700 1 -- start start 2024-03-30T00:56:54.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.697+0000 7fef8ae00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84073520 0x7fef841a3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:54.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.697+0000 7fef8ae00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef84074de0 0x7fef841a3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:54.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.697+0000 7fef8ae00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef84111a40 0x7fef8419d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:54.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.697+0000 7fef8ae00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8419dd10 con 0x7fef84073520 2024-03-30T00:56:54.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.697+0000 7fef8ae00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8419de80 con 0x7fef84074de0 2024-03-30T00:56:54.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.697+0000 7fef8ae00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8419e040 con 0x7fef84111a40 2024-03-30T00:56:54.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.697+0000 7fef83e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84073520 0x7fef841a3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:54.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.697+0000 7fef83e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84073520 0x7fef841a3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38202/0 (socket says 172.21.15.116:38202) 2024-03-30T00:56:54.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.697+0000 7fef83400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef84074de0 0x7fef841a3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:54.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.697+0000 7fef83e00700 1 -- 172.21.15.116:0/3082889279 learned_addr learned my addr 172.21.15.116:0/3082889279 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:54.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.697+0000 7fef88e00700 1 --2- 172.21.15.116:0/3082889279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef84111a40 0x7fef8419d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:54.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.698+0000 7fef83e00700 1 -- 172.21.15.116:0/3082889279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef84111a40 msgr2=0x7fef8419d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:54.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.698+0000 7fef83e00700 1 --2- 172.21.15.116:0/3082889279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef84111a40 0x7fef8419d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.698+0000 7fef83e00700 1 -- 172.21.15.116:0/3082889279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef84074de0 msgr2=0x7fef841a3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:54.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.698+0000 7fef83e00700 1 --2- 172.21.15.116:0/3082889279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef84074de0 0x7fef841a3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.698+0000 7fef83e00700 1 -- 172.21.15.116:0/3082889279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef700095f0 con 0x7fef84073520 2024-03-30T00:56:54.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.698+0000 7fef83400700 1 --2- 172.21.15.116:0/3082889279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef84074de0 0x7fef841a3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:54.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.698+0000 7fef83e00700 1 --2- 172.21.15.116:0/3082889279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84073520 0x7fef841a3560 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fef7400e9a0 tx=0x7fef7400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:54.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.698+0000 7fef80c00700 1 -- 172.21.15.116:0/3082889279 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef74009790 con 0x7fef84073520 2024-03-30T00:56:54.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.698+0000 7fef8ae00700 1 -- 172.21.15.116:0/3082889279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef8419e320 con 0x7fef84073520 2024-03-30T00:56:54.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.698+0000 7fef8ae00700 1 -- 172.21.15.116:0/3082889279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef841aa550 con 0x7fef84073520 2024-03-30T00:56:54.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.699+0000 7fef80c00700 1 -- 172.21.15.116:0/3082889279 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef74010430 con 0x7fef84073520 2024-03-30T00:56:54.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.699+0000 7fef80c00700 1 -- 172.21.15.116:0/3082889279 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef74018680 con 0x7fef84073520 2024-03-30T00:56:54.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.700+0000 7fef8ae00700 1 -- 172.21.15.116:0/3082889279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef640052f0 con 0x7fef84073520 2024-03-30T00:56:54.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.702+0000 7fef80c00700 1 -- 172.21.15.116:0/3082889279 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fef7400ca10 con 0x7fef84073520 2024-03-30T00:56:54.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.703+0000 7fef80c00700 1 --2- 172.21.15.116:0/3082889279 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fef680a1230 0x7fef680a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:54.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.703+0000 7fef80c00700 1 -- 172.21.15.116:0/3082889279 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fef74014070 con 0x7fef84073520 2024-03-30T00:56:54.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.704+0000 7fef83400700 1 --2- 172.21.15.116:0/3082889279 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fef680a1230 0x7fef680a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:54.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.705+0000 7fef83400700 1 --2- 172.21.15.116:0/3082889279 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fef680a1230 0x7fef680a36f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fef840786c0 tx=0x7fef70005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:54.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.709+0000 7fef80c00700 1 -- 172.21.15.116:0/3082889279 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fef740529f0 con 0x7fef84073520 2024-03-30T00:56:54.222 INFO:teuthology.orchestra.run.smithi116.stdout:103079215156 2024-03-30T00:56:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.952+0000 7fef8ae00700 1 -- 172.21.15.116:0/3082889279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fef64005160 con 0x7fef84073520 2024-03-30T00:56:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.953+0000 7fef80c00700 1 -- 172.21.15.116:0/3082889279 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fef74099ac0 con 0x7fef84073520 2024-03-30T00:56:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.955+0000 7fef8ae00700 1 -- 172.21.15.116:0/3082889279 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fef680a1230 msgr2=0x7fef680a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.955+0000 7fef8ae00700 1 --2- 172.21.15.116:0/3082889279 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fef680a1230 0x7fef680a36f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fef840786c0 tx=0x7fef70005fd0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.955+0000 7fef8ae00700 1 -- 172.21.15.116:0/3082889279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84073520 msgr2=0x7fef841a3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.955+0000 7fef8ae00700 1 --2- 172.21.15.116:0/3082889279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84073520 0x7fef841a3560 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fef7400e9a0 tx=0x7fef7400ecb0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.955+0000 7fef8ae00700 1 -- 172.21.15.116:0/3082889279 shutdown_connections 2024-03-30T00:56:54.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.955+0000 7fef8ae00700 1 --2- 172.21.15.116:0/3082889279 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fef680a1230 0x7fef680a36f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.955+0000 7fef8ae00700 1 --2- 172.21.15.116:0/3082889279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef84111a40 0x7fef8419d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.955+0000 7fef8ae00700 1 --2- 172.21.15.116:0/3082889279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84073520 0x7fef841a3560 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.955+0000 7fef8ae00700 1 --2- 172.21.15.116:0/3082889279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef84074de0 0x7fef841a3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.955+0000 7fef8ae00700 1 -- 172.21.15.116:0/3082889279 >> 172.21.15.116:0/3082889279 conn(0x7fef84077f10 msgr2=0x7fef84110800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:54.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.956+0000 7fef8ae00700 1 -- 172.21.15.116:0/3082889279 shutdown_connections 2024-03-30T00:56:54.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:53.956+0000 7fef8ae00700 1 -- 172.21.15.116:0/3082889279 wait complete. 2024-03-30T00:56:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.475+0000 7fa351c00700 1 -- 172.21.15.116:0/3710588883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa34c074d30 msgr2=0x7fa34c072fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.475+0000 7fa351c00700 1 --2- 172.21.15.116:0/3710588883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa34c074d30 0x7fa34c072fe0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa33c009990 tx=0x7fa33c009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.475+0000 7fa351c00700 1 -- 172.21.15.116:0/3710588883 shutdown_connections 2024-03-30T00:56:54.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.475+0000 7fa351c00700 1 --2- 172.21.15.116:0/3710588883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa34c111b30 0x7fa34c113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.475+0000 7fa351c00700 1 --2- 172.21.15.116:0/3710588883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa34c074d30 0x7fa34c072fe0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.475+0000 7fa351c00700 1 --2- 172.21.15.116:0/3710588883 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa34c073520 0x7fa34c0739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.475+0000 7fa351c00700 1 -- 172.21.15.116:0/3710588883 >> 172.21.15.116:0/3710588883 conn(0x7fa34c077fa0 msgr2=0x7fa34c0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:54.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.475+0000 7fa351c00700 1 -- 172.21.15.116:0/3710588883 shutdown_connections 2024-03-30T00:56:54.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.476+0000 7fa351c00700 1 -- 172.21.15.116:0/3710588883 wait complete. 2024-03-30T00:56:54.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.476+0000 7fa351c00700 1 Processor -- start 2024-03-30T00:56:54.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.477+0000 7fa351c00700 1 -- start start 2024-03-30T00:56:54.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.477+0000 7fa351c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa34c073520 0x7fa34c1a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:54.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.477+0000 7fa351c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa34c074d30 0x7fa34c1a3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:54.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.477+0000 7fa351c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa34c111b30 0x7fa34c19d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:54.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.477+0000 7fa351c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c19ddc0 con 0x7fa34c074d30 2024-03-30T00:56:54.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.477+0000 7fa351c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c19df30 con 0x7fa34c073520 2024-03-30T00:56:54.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.477+0000 7fa351c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c19e0f0 con 0x7fa34c111b30 2024-03-30T00:56:54.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.477+0000 7fa34a000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa34c074d30 0x7fa34c1a3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:54.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.477+0000 7fa34a000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa34c074d30 0x7fa34c1a3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38208/0 (socket says 172.21.15.116:38208) 2024-03-30T00:56:54.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.477+0000 7fa34a000700 1 -- 172.21.15.116:0/345454761 learned_addr learned my addr 172.21.15.116:0/345454761 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:54.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.477+0000 7fa34aa00700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa34c073520 0x7fa34c1a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:54.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.477+0000 7fa34b400700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa34c111b30 0x7fa34c19d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.478+0000 7fa34a000700 1 -- 172.21.15.116:0/345454761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa34c111b30 msgr2=0x7fa34c19d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.478+0000 7fa34a000700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa34c111b30 0x7fa34c19d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.478+0000 7fa34a000700 1 -- 172.21.15.116:0/345454761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa34c073520 msgr2=0x7fa34c1a35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.478+0000 7fa34a000700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa34c073520 0x7fa34c1a35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.478+0000 7fa34a000700 1 -- 172.21.15.116:0/345454761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c0095f0 con 0x7fa34c074d30 2024-03-30T00:56:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.478+0000 7fa34b400700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa34c111b30 0x7fa34c19d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.478+0000 7fa34a000700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa34c074d30 0x7fa34c1a3b10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa33800e9a0 tx=0x7fa33800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.478+0000 7fa343400700 1 -- 172.21.15.116:0/345454761 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338009790 con 0x7fa34c074d30 2024-03-30T00:56:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.478+0000 7fa343400700 1 -- 172.21.15.116:0/345454761 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa338010430 con 0x7fa34c074d30 2024-03-30T00:56:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.478+0000 7fa351c00700 1 -- 172.21.15.116:0/345454761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34c19e3d0 con 0x7fa34c074d30 2024-03-30T00:56:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.479+0000 7fa343400700 1 -- 172.21.15.116:0/345454761 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338018680 con 0x7fa34c074d30 2024-03-30T00:56:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.479+0000 7fa351c00700 1 -- 172.21.15.116:0/345454761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34c1aa5c0 con 0x7fa34c074d30 2024-03-30T00:56:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.480+0000 7fa351c00700 1 -- 172.21.15.116:0/345454761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34c074670 con 0x7fa34c074d30 2024-03-30T00:56:54.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.482+0000 7fa343400700 1 -- 172.21.15.116:0/345454761 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fa338010880 con 0x7fa34c074d30 2024-03-30T00:56:54.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.485+0000 7fa343400700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3300a11b0 0x7fa3300a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:54.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.485+0000 7fa343400700 1 -- 172.21.15.116:0/345454761 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa338014070 con 0x7fa34c074d30 2024-03-30T00:56:54.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.485+0000 7fa34aa00700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3300a11b0 0x7fa3300a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:54.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.486+0000 7fa34aa00700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3300a11b0 0x7fa3300a3670 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00b2e0 tx=0x7fa33c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:54.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.486+0000 7fa343400700 1 -- 172.21.15.116:0/345454761 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa338096460 con 0x7fa34c074d30 2024-03-30T00:56:54.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.704+0000 7fa351c00700 1 -- 172.21.15.116:0/345454761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa34c19ec00 con 0x7fa34c074d30 2024-03-30T00:56:54.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.704+0000 7fa343400700 1 -- 172.21.15.116:0/345454761 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa338026090 con 0x7fa34c074d30 2024-03-30T00:56:54.711 INFO:teuthology.orchestra.run.smithi116.stdout:133143986224 2024-03-30T00:56:54.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.708+0000 7fa351c00700 1 -- 172.21.15.116:0/345454761 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3300a11b0 msgr2=0x7fa3300a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:54.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.708+0000 7fa351c00700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3300a11b0 0x7fa3300a3670 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00b2e0 tx=0x7fa33c005fd0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.708+0000 7fa351c00700 1 -- 172.21.15.116:0/345454761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa34c074d30 msgr2=0x7fa34c1a3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:54.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.708+0000 7fa351c00700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa34c074d30 0x7fa34c1a3b10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa33800e9a0 tx=0x7fa33800ecb0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.709+0000 7fa351c00700 1 -- 172.21.15.116:0/345454761 shutdown_connections 2024-03-30T00:56:54.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.709+0000 7fa351c00700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3300a11b0 0x7fa3300a3670 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.709+0000 7fa351c00700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa34c111b30 0x7fa34c19d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.709+0000 7fa351c00700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa34c073520 0x7fa34c1a35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.709+0000 7fa351c00700 1 --2- 172.21.15.116:0/345454761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa34c074d30 0x7fa34c1a3b10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:54.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.709+0000 7fa351c00700 1 -- 172.21.15.116:0/345454761 >> 172.21.15.116:0/345454761 conn(0x7fa34c077fa0 msgr2=0x7fa34c1107a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:54.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.709+0000 7fa351c00700 1 -- 172.21.15.116:0/345454761 shutdown_connections 2024-03-30T00:56:54.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:54.709+0000 7fa351c00700 1 -- 172.21.15.116:0/345454761 wait complete. 2024-03-30T00:56:54.739 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:54 smithi116 ceph-mon[38086]: pgmap v262: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-30T00:56:54.739 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:54 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3082889279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T00:56:54.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:54 smithi130 ceph-mon[47626]: pgmap v262: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-30T00:56:54.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:54 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3082889279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T00:56:55.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:54 smithi181 ceph-mon[47622]: pgmap v262: 129 pgs: 129 active+clean; 639 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-30T00:56:55.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:54 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3082889279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T00:56:55.036 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.295+0000 7f5c42c00700 1 -- 172.21.15.116:0/3401245736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c101820 msgr2=0x7f5c3c10dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.295+0000 7f5c42c00700 1 --2- 172.21.15.116:0/3401245736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c101820 0x7f5c3c10dc10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5c3400b0d0 tx=0x7f5c3400b3e0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.295+0000 7f5c42c00700 1 -- 172.21.15.116:0/3401245736 shutdown_connections 2024-03-30T00:56:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.295+0000 7f5c42c00700 1 --2- 172.21.15.116:0/3401245736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c101820 0x7f5c3c10dc10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.295+0000 7f5c42c00700 1 --2- 172.21.15.116:0/3401245736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c3c106ef0 0x7f5c3c1072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.295+0000 7f5c42c00700 1 --2- 172.21.15.116:0/3401245736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c3c100e60 0x7f5c3c1012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.295+0000 7f5c42c00700 1 -- 172.21.15.116:0/3401245736 >> 172.21.15.116:0/3401245736 conn(0x7f5c3c077fa0 msgr2=0x7f5c3c0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.295+0000 7f5c42c00700 1 -- 172.21.15.116:0/3401245736 shutdown_connections 2024-03-30T00:56:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.295+0000 7f5c42c00700 1 -- 172.21.15.116:0/3401245736 wait complete. 2024-03-30T00:56:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.296+0000 7f5c42c00700 1 Processor -- start 2024-03-30T00:56:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.296+0000 7f5c42c00700 1 -- start start 2024-03-30T00:56:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.296+0000 7f5c42c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c100e60 0x7f5c3c197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c42c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c3c101820 0x7f5c3c1978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c42c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c3c106ef0 0x7f5c3c19bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c42c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c19c280 con 0x7f5c3c100e60 2024-03-30T00:56:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c42c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c19c3f0 con 0x7f5c3c101820 2024-03-30T00:56:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c42c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c19c5b0 con 0x7f5c3c106ef0 2024-03-30T00:56:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c3be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c100e60 0x7f5c3c197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c3b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c3c101820 0x7f5c3c1978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c3be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c100e60 0x7f5c3c197380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59160/0 (socket says 172.21.15.116:59160) 2024-03-30T00:56:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c3be00700 1 -- 172.21.15.116:0/4145370428 learned_addr learned my addr 172.21.15.116:0/4145370428 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c40c00700 1 --2- 172.21.15.116:0/4145370428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c3c106ef0 0x7f5c3c19bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c3b400700 1 -- 172.21.15.116:0/4145370428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c3c106ef0 msgr2=0x7f5c3c19bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:55.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c3b400700 1 --2- 172.21.15.116:0/4145370428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c3c106ef0 0x7f5c3c19bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c3b400700 1 -- 172.21.15.116:0/4145370428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c100e60 msgr2=0x7f5c3c197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:55.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c3b400700 1 --2- 172.21.15.116:0/4145370428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c100e60 0x7f5c3c197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c3b400700 1 -- 172.21.15.116:0/4145370428 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c34009d00 con 0x7f5c3c101820 2024-03-30T00:56:55.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.297+0000 7f5c3be00700 1 --2- 172.21.15.116:0/4145370428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c100e60 0x7f5c3c197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:55.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.298+0000 7f5c3b400700 1 --2- 172.21.15.116:0/4145370428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c3c101820 0x7f5c3c1978c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800d950 tx=0x7f5c2800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:55.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.298+0000 7f5c38c00700 1 -- 172.21.15.116:0/4145370428 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c28009930 con 0x7f5c3c101820 2024-03-30T00:56:55.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.298+0000 7f5c42c00700 1 -- 172.21.15.116:0/4145370428 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3c19c890 con 0x7f5c3c101820 2024-03-30T00:56:55.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.298+0000 7f5c38c00700 1 -- 172.21.15.116:0/4145370428 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c28010460 con 0x7f5c3c101820 2024-03-30T00:56:55.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.298+0000 7f5c38c00700 1 -- 172.21.15.116:0/4145370428 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2800b5d0 con 0x7f5c3c101820 2024-03-30T00:56:55.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.298+0000 7f5c42c00700 1 -- 172.21.15.116:0/4145370428 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3c0731b0 con 0x7f5c3c101820 2024-03-30T00:56:55.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.299+0000 7f5c42c00700 1 -- 172.21.15.116:0/4145370428 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3c1092f0 con 0x7f5c3c101820 2024-03-30T00:56:55.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.301+0000 7f5c38c00700 1 -- 172.21.15.116:0/4145370428 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f5c280107d0 con 0x7f5c3c101820 2024-03-30T00:56:55.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.303+0000 7f5c38c00700 1 --2- 172.21.15.116:0/4145370428 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5c200a1230 0x7f5c200a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:55.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.303+0000 7f5c38c00700 1 -- 172.21.15.116:0/4145370428 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5c280cc8f0 con 0x7f5c3c101820 2024-03-30T00:56:55.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.303+0000 7f5c3be00700 1 --2- 172.21.15.116:0/4145370428 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5c200a1230 0x7f5c200a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:55.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.304+0000 7f5c3be00700 1 --2- 172.21.15.116:0/4145370428 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5c200a1230 0x7f5c200a36f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c005950 tx=0x7f5c2c00b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:55.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.305+0000 7f5c38c00700 1 -- 172.21.15.116:0/4145370428 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5c28095c30 con 0x7f5c3c101820 2024-03-30T00:56:55.363 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986223 got 133143986224 for osd.4 2024-03-30T00:56:55.364 DEBUG:teuthology.parallel:result is None 2024-03-30T00:56:55.475 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:56:55.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.542+0000 7f5c42c00700 1 -- 172.21.15.116:0/4145370428 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f5c3c1024f0 con 0x7f5c3c101820 2024-03-30T00:56:55.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.543+0000 7f5c38c00700 1 -- 172.21.15.116:0/4145370428 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5c28021070 con 0x7f5c3c101820 2024-03-30T00:56:55.549 INFO:teuthology.orchestra.run.smithi116.stdout:34359738431 2024-03-30T00:56:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.545+0000 7f5c42c00700 1 -- 172.21.15.116:0/4145370428 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5c200a1230 msgr2=0x7f5c200a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.545+0000 7f5c42c00700 1 --2- 172.21.15.116:0/4145370428 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5c200a1230 0x7f5c200a36f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c005950 tx=0x7f5c2c00b500 comp rx=0 tx=0).stop 2024-03-30T00:56:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.545+0000 7f5c42c00700 1 -- 172.21.15.116:0/4145370428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c3c101820 msgr2=0x7f5c3c1978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.545+0000 7f5c42c00700 1 --2- 172.21.15.116:0/4145370428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c3c101820 0x7f5c3c1978c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800d950 tx=0x7f5c2800dc60 comp rx=0 tx=0).stop 2024-03-30T00:56:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.545+0000 7f5c42c00700 1 -- 172.21.15.116:0/4145370428 shutdown_connections 2024-03-30T00:56:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.545+0000 7f5c42c00700 1 --2- 172.21.15.116:0/4145370428 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5c200a1230 0x7f5c200a36f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.545+0000 7f5c42c00700 1 --2- 172.21.15.116:0/4145370428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c3c106ef0 0x7f5c3c19bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.545+0000 7f5c42c00700 1 --2- 172.21.15.116:0/4145370428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c100e60 0x7f5c3c197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.545+0000 7f5c42c00700 1 --2- 172.21.15.116:0/4145370428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c3c101820 0x7f5c3c1978c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.545+0000 7f5c42c00700 1 -- 172.21.15.116:0/4145370428 >> 172.21.15.116:0/4145370428 conn(0x7f5c3c077fa0 msgr2=0x7f5c3c0ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.545+0000 7f5c42c00700 1 -- 172.21.15.116:0/4145370428 shutdown_connections 2024-03-30T00:56:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.545+0000 7f5c42c00700 1 -- 172.21.15.116:0/4145370428 wait complete. 2024-03-30T00:56:55.597 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:55 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/345454761' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T00:56:55.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.797+0000 7f6b35200700 1 -- 172.21.15.116:0/3367265677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b30101820 msgr2=0x7f6b3010dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:55.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.797+0000 7f6b35200700 1 --2- 172.21.15.116:0/3367265677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b30101820 0x7f6b3010dc10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800b0d0 tx=0x7f6b2800b3e0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.797+0000 7f6b35200700 1 -- 172.21.15.116:0/3367265677 shutdown_connections 2024-03-30T00:56:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.797+0000 7f6b35200700 1 --2- 172.21.15.116:0/3367265677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b30106ef0 0x7f6b301072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.797+0000 7f6b35200700 1 --2- 172.21.15.116:0/3367265677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b30101820 0x7f6b3010dc10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.797+0000 7f6b35200700 1 --2- 172.21.15.116:0/3367265677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b30100e60 0x7f6b301012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.797+0000 7f6b35200700 1 -- 172.21.15.116:0/3367265677 >> 172.21.15.116:0/3367265677 conn(0x7f6b30077fa0 msgr2=0x7f6b300783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.798+0000 7f6b35200700 1 -- 172.21.15.116:0/3367265677 shutdown_connections 2024-03-30T00:56:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.798+0000 7f6b35200700 1 -- 172.21.15.116:0/3367265677 wait complete. 2024-03-30T00:56:55.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.798+0000 7f6b35200700 1 Processor -- start 2024-03-30T00:56:55.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.799+0000 7f6b35200700 1 -- start start 2024-03-30T00:56:55.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.799+0000 7f6b35200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b30100e60 0x7f6b301972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:55.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.799+0000 7f6b35200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b30101820 0x7f6b30197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:55.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.799+0000 7f6b35200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b30106ef0 0x7f6b3019bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:55.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.799+0000 7f6b35200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3019c1c0 con 0x7f6b30106ef0 2024-03-30T00:56:55.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.799+0000 7f6b35200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3019c330 con 0x7f6b30100e60 2024-03-30T00:56:55.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.799+0000 7f6b35200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3019c4f0 con 0x7f6b30101820 2024-03-30T00:56:55.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.799+0000 7f6b2e800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b30106ef0 0x7f6b3019bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:55.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.799+0000 7f6b2de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b30100e60 0x7f6b301972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:55.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.799+0000 7f6b2d400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b30101820 0x7f6b30197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:55.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.799+0000 7f6b2d400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b30101820 0x7f6b30197800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:43076/0 (socket says 172.21.15.116:43076) 2024-03-30T00:56:55.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.799+0000 7f6b2e800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b30106ef0 0x7f6b3019bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59186/0 (socket says 172.21.15.116:59186) 2024-03-30T00:56:55.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.799+0000 7f6b2d400700 1 -- 172.21.15.116:0/2737292269 learned_addr learned my addr 172.21.15.116:0/2737292269 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:55.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.799+0000 7f6b2de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b30100e60 0x7f6b301972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:35582/0 (socket says 172.21.15.116:35582) 2024-03-30T00:56:55.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.800+0000 7f6b2e800700 1 -- 172.21.15.116:0/2737292269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b30101820 msgr2=0x7f6b30197800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:55.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.800+0000 7f6b2e800700 1 --2- 172.21.15.116:0/2737292269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b30101820 0x7f6b30197800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.800+0000 7f6b2e800700 1 -- 172.21.15.116:0/2737292269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b30100e60 msgr2=0x7f6b301972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:55.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.800+0000 7f6b2e800700 1 --2- 172.21.15.116:0/2737292269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b30100e60 0x7f6b301972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:55.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.800+0000 7f6b2e800700 1 -- 172.21.15.116:0/2737292269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b28009d00 con 0x7f6b30106ef0 2024-03-30T00:56:55.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.800+0000 7f6b2de00700 1 --2- 172.21.15.116:0/2737292269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b30100e60 0x7f6b301972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:55.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.800+0000 7f6b2d400700 1 --2- 172.21.15.116:0/2737292269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b30101820 0x7f6b30197800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:55.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.800+0000 7f6b2e800700 1 --2- 172.21.15.116:0/2737292269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b30106ef0 0x7f6b3019bbb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800b770 tx=0x7f6b2800bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:55.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.800+0000 7f6b26a00700 1 -- 172.21.15.116:0/2737292269 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2800e040 con 0x7f6b30106ef0 2024-03-30T00:56:55.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.800+0000 7f6b26a00700 1 -- 172.21.15.116:0/2737292269 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b2801b460 con 0x7f6b30106ef0 2024-03-30T00:56:55.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.800+0000 7f6b35200700 1 -- 172.21.15.116:0/2737292269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3019c770 con 0x7f6b30106ef0 2024-03-30T00:56:55.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.800+0000 7f6b26a00700 1 -- 172.21.15.116:0/2737292269 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b28005290 con 0x7f6b30106ef0 2024-03-30T00:56:55.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.801+0000 7f6b35200700 1 -- 172.21.15.116:0/2737292269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b300731b0 con 0x7f6b30106ef0 2024-03-30T00:56:55.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.801+0000 7f6b35200700 1 -- 172.21.15.116:0/2737292269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b301092f0 con 0x7f6b30106ef0 2024-03-30T00:56:55.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.803+0000 7f6b26a00700 1 -- 172.21.15.116:0/2737292269 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f6b2801b5d0 con 0x7f6b30106ef0 2024-03-30T00:56:55.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.805+0000 7f6b26a00700 1 --2- 172.21.15.116:0/2737292269 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6b140a1230 0x7f6b140a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:55.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.805+0000 7f6b26a00700 1 -- 172.21.15.116:0/2737292269 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6b280cdea0 con 0x7f6b30106ef0 2024-03-30T00:56:55.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.805+0000 7f6b2de00700 1 --2- 172.21.15.116:0/2737292269 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6b140a1230 0x7f6b140a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:55.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.806+0000 7f6b2de00700 1 --2- 172.21.15.116:0/2737292269 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6b140a1230 0x7f6b140a36f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6b20011490 tx=0x7f6b20009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:55.807+0000 7f6b26a00700 1 -- 172.21.15.116:0/2737292269 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6b280971f0 con 0x7f6b30106ef0 2024-03-30T00:56:55.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:55 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/345454761' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T00:56:55.930 INFO:tasks.cephadm.ceph_manager.ceph:need seq 103079215154 got 103079215156 for osd.3 2024-03-30T00:56:55.930 DEBUG:teuthology.parallel:result is None 2024-03-30T00:56:56.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:55 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/345454761' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T00:56:56.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.032+0000 7f6b35200700 1 -- 172.21.15.116:0/2737292269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f6b301024f0 con 0x7f6b30106ef0 2024-03-30T00:56:56.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.032+0000 7f6b26a00700 1 -- 172.21.15.116:0/2737292269 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6b2809a810 con 0x7f6b30106ef0 2024-03-30T00:56:56.038 INFO:teuthology.orchestra.run.smithi116.stdout:227633266721 2024-03-30T00:56:56.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.035+0000 7f6b35200700 1 -- 172.21.15.116:0/2737292269 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6b140a1230 msgr2=0x7f6b140a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.035+0000 7f6b35200700 1 --2- 172.21.15.116:0/2737292269 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6b140a1230 0x7f6b140a36f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6b20011490 tx=0x7f6b20009260 comp rx=0 tx=0).stop 2024-03-30T00:56:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.035+0000 7f6b35200700 1 -- 172.21.15.116:0/2737292269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b30106ef0 msgr2=0x7f6b3019bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.035+0000 7f6b35200700 1 --2- 172.21.15.116:0/2737292269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b30106ef0 0x7f6b3019bbb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800b770 tx=0x7f6b2800bcd0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.035+0000 7f6b35200700 1 -- 172.21.15.116:0/2737292269 shutdown_connections 2024-03-30T00:56:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.035+0000 7f6b35200700 1 --2- 172.21.15.116:0/2737292269 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6b140a1230 0x7f6b140a36f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.035+0000 7f6b35200700 1 --2- 172.21.15.116:0/2737292269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b30106ef0 0x7f6b3019bbb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.035+0000 7f6b35200700 1 --2- 172.21.15.116:0/2737292269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b30100e60 0x7f6b301972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.035+0000 7f6b35200700 1 --2- 172.21.15.116:0/2737292269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b30101820 0x7f6b30197800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.035+0000 7f6b35200700 1 -- 172.21.15.116:0/2737292269 >> 172.21.15.116:0/2737292269 conn(0x7f6b30077fa0 msgr2=0x7f6b300ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.035+0000 7f6b35200700 1 -- 172.21.15.116:0/2737292269 shutdown_connections 2024-03-30T00:56:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.035+0000 7f6b35200700 1 -- 172.21.15.116:0/2737292269 wait complete. 2024-03-30T00:56:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.360+0000 7fb296400700 1 -- 172.21.15.116:0/2443376146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb290073520 msgr2=0x7fb2900739a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.360+0000 7fb296400700 1 --2- 172.21.15.116:0/2443376146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb290073520 0x7fb2900739a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb27c009990 tx=0x7fb27c009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.360+0000 7fb296400700 1 -- 172.21.15.116:0/2443376146 shutdown_connections 2024-03-30T00:56:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.360+0000 7fb296400700 1 --2- 172.21.15.116:0/2443376146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb290074d30 0x7fb290072fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.360+0000 7fb296400700 1 --2- 172.21.15.116:0/2443376146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb290111b30 0x7fb290113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.360+0000 7fb296400700 1 --2- 172.21.15.116:0/2443376146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb290073520 0x7fb2900739a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.360+0000 7fb296400700 1 -- 172.21.15.116:0/2443376146 >> 172.21.15.116:0/2443376146 conn(0x7fb290077fa0 msgr2=0x7fb2900783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.360+0000 7fb296400700 1 -- 172.21.15.116:0/2443376146 shutdown_connections 2024-03-30T00:56:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.360+0000 7fb296400700 1 -- 172.21.15.116:0/2443376146 wait complete. 2024-03-30T00:56:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.361+0000 7fb296400700 1 Processor -- start 2024-03-30T00:56:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.361+0000 7fb296400700 1 -- start start 2024-03-30T00:56:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.362+0000 7fb296400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb290073520 0x7fb2901a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.362+0000 7fb296400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb290074d30 0x7fb2901a3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.362+0000 7fb296400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb290111b30 0x7fb29019d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.362+0000 7fb296400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29019dd80 con 0x7fb290073520 2024-03-30T00:56:56.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.362+0000 7fb296400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29019def0 con 0x7fb290074d30 2024-03-30T00:56:56.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.362+0000 7fb296400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29019e0b0 con 0x7fb290111b30 2024-03-30T00:56:56.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.362+0000 7fb28f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb290073520 0x7fb2901a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:56.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.362+0000 7fb28f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb290073520 0x7fb2901a3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59204/0 (socket says 172.21.15.116:59204) 2024-03-30T00:56:56.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.362+0000 7fb28f400700 1 -- 172.21.15.116:0/2308861 learned_addr learned my addr 172.21.15.116:0/2308861 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:56.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.362+0000 7fb28fe00700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb290111b30 0x7fb29019d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:56.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.362+0000 7fb28ea00700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb290074d30 0x7fb2901a3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:56.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.363+0000 7fb28f400700 1 -- 172.21.15.116:0/2308861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb290111b30 msgr2=0x7fb29019d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:56.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.363+0000 7fb28f400700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb290111b30 0x7fb29019d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.363+0000 7fb28f400700 1 -- 172.21.15.116:0/2308861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb290074d30 msgr2=0x7fb2901a3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:56.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.363+0000 7fb28f400700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb290074d30 0x7fb2901a3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.363+0000 7fb28f400700 1 -- 172.21.15.116:0/2308861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb288009c40 con 0x7fb290073520 2024-03-30T00:56:56.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.363+0000 7fb28ea00700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb290074d30 0x7fb2901a3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:56:56.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.363+0000 7fb28f400700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb290073520 0x7fb2901a3590 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb28000e9a0 tx=0x7fb28000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:56.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.363+0000 7fb287e00700 1 -- 172.21.15.116:0/2308861 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb280009790 con 0x7fb290073520 2024-03-30T00:56:56.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.364+0000 7fb287e00700 1 -- 172.21.15.116:0/2308861 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb280010430 con 0x7fb290073520 2024-03-30T00:56:56.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.364+0000 7fb296400700 1 -- 172.21.15.116:0/2308861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27c0095f0 con 0x7fb290073520 2024-03-30T00:56:56.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.364+0000 7fb287e00700 1 -- 172.21.15.116:0/2308861 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb280018680 con 0x7fb290073520 2024-03-30T00:56:56.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.364+0000 7fb296400700 1 -- 172.21.15.116:0/2308861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2901aa3b0 con 0x7fb290073520 2024-03-30T00:56:56.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.365+0000 7fb296400700 1 -- 172.21.15.116:0/2308861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb290074670 con 0x7fb290073520 2024-03-30T00:56:56.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.367+0000 7fb287e00700 1 -- 172.21.15.116:0/2308861 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fb2800105a0 con 0x7fb290073520 2024-03-30T00:56:56.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.368+0000 7fb287e00700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb2740a11b0 0x7fb2740a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:56.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.368+0000 7fb287e00700 1 -- 172.21.15.116:0/2308861 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb280014070 con 0x7fb290073520 2024-03-30T00:56:56.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.368+0000 7fb28ea00700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb2740a11b0 0x7fb2740a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:56.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.369+0000 7fb28ea00700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb2740a11b0 0x7fb2740a3670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb27c0098e0 tx=0x7fb27c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:56.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.375+0000 7fb287e00700 1 -- 172.21.15.116:0/2308861 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb2800529f0 con 0x7fb290073520 2024-03-30T00:56:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.599+0000 7fb296400700 1 -- 172.21.15.116:0/2308861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fb29019ebc0 con 0x7fb290073520 2024-03-30T00:56:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.599+0000 7fb287e00700 1 -- 172.21.15.116:0/2308861 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb280096250 con 0x7fb290073520 2024-03-30T00:56:56.605 INFO:teuthology.orchestra.run.smithi116.stdout:257698037788 2024-03-30T00:56:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.602+0000 7fb296400700 1 -- 172.21.15.116:0/2308861 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb2740a11b0 msgr2=0x7fb2740a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.602+0000 7fb296400700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb2740a11b0 0x7fb2740a3670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb27c0098e0 tx=0x7fb27c005960 comp rx=0 tx=0).stop 2024-03-30T00:56:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.602+0000 7fb296400700 1 -- 172.21.15.116:0/2308861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb290073520 msgr2=0x7fb2901a3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.602+0000 7fb296400700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb290073520 0x7fb2901a3590 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb28000e9a0 tx=0x7fb28000ecb0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.602+0000 7fb296400700 1 -- 172.21.15.116:0/2308861 shutdown_connections 2024-03-30T00:56:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.602+0000 7fb296400700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb2740a11b0 0x7fb2740a3670 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.602+0000 7fb296400700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb290111b30 0x7fb29019d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.602+0000 7fb296400700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb290073520 0x7fb2901a3590 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.602+0000 7fb296400700 1 --2- 172.21.15.116:0/2308861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb290074d30 0x7fb2901a3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.602+0000 7fb296400700 1 -- 172.21.15.116:0/2308861 >> 172.21.15.116:0/2308861 conn(0x7fb290077fa0 msgr2=0x7fb2901107a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.602+0000 7fb296400700 1 -- 172.21.15.116:0/2308861 shutdown_connections 2024-03-30T00:56:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:56.602+0000 7fb296400700 1 -- 172.21.15.116:0/2308861 wait complete. 2024-03-30T00:56:56.629 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:56 smithi116 ceph-mon[38086]: pgmap v263: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-30T00:56:56.630 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:56 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/4145370428' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T00:56:56.630 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:56 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2737292269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-30T00:56:56.717 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266719 got 227633266721 for osd.8 2024-03-30T00:56:56.718 DEBUG:teuthology.parallel:result is None 2024-03-30T00:56:56.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:56 smithi130 ceph-mon[47626]: pgmap v263: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-30T00:56:56.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:56 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/4145370428' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T00:56:56.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:56 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2737292269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-30T00:56:57.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:56 smithi181 ceph-mon[47622]: pgmap v263: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-30T00:56:57.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:56 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/4145370428' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T00:56:57.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:56 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2737292269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-30T00:56:57.380 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738429 got 34359738431 for osd.0 2024-03-30T00:56:57.380 DEBUG:teuthology.parallel:result is None 2024-03-30T00:56:57.614 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037787 got 257698037788 for osd.9 2024-03-30T00:56:57.614 DEBUG:teuthology.parallel:result is None 2024-03-30T00:56:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.698+0000 7eff4a800700 1 -- 172.21.15.116:0/1241134438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff441017c0 msgr2=0x7eff4410dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.698+0000 7eff4a800700 1 --2- 172.21.15.116:0/1241134438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff441017c0 0x7eff4410dc60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7eff3c009f90 tx=0x7eff3c00b330 comp rx=0 tx=0).stop 2024-03-30T00:56:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.698+0000 7eff4a800700 1 -- 172.21.15.116:0/1241134438 shutdown_connections 2024-03-30T00:56:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.698+0000 7eff4a800700 1 --2- 172.21.15.116:0/1241134438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff441017c0 0x7eff4410dc60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.698+0000 7eff4a800700 1 --2- 172.21.15.116:0/1241134438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff44106f40 0x7eff44107320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.698+0000 7eff4a800700 1 --2- 172.21.15.116:0/1241134438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff44100e00 0x7eff44101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.698+0000 7eff4a800700 1 -- 172.21.15.116:0/1241134438 >> 172.21.15.116:0/1241134438 conn(0x7eff44077f10 msgr2=0x7eff44078320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.698+0000 7eff4a800700 1 -- 172.21.15.116:0/1241134438 shutdown_connections 2024-03-30T00:56:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.698+0000 7eff4a800700 1 -- 172.21.15.116:0/1241134438 wait complete. 2024-03-30T00:56:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.700+0000 7eff4a800700 1 Processor -- start 2024-03-30T00:56:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.700+0000 7eff4a800700 1 -- start start 2024-03-30T00:56:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.700+0000 7eff4a800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff44100e00 0x7eff440730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff4a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff441017c0 0x7eff440735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff4a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff44106f40 0x7eff44073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff4a800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff441a56b0 con 0x7eff441017c0 2024-03-30T00:56:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff4a800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff441a5820 con 0x7eff44106f40 2024-03-30T00:56:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff4a800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff441a59e0 con 0x7eff44100e00 2024-03-30T00:56:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff43400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff44100e00 0x7eff440730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff42a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff441017c0 0x7eff440735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff43400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff44100e00 0x7eff440730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:43092/0 (socket says 172.21.15.116:43092) 2024-03-30T00:56:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff43400700 1 -- 172.21.15.116:0/1556562186 learned_addr learned my addr 172.21.15.116:0/1556562186 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff43e00700 1 --2- 172.21.15.116:0/1556562186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff44106f40 0x7eff44073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff42a00700 1 -- 172.21.15.116:0/1556562186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff44100e00 msgr2=0x7eff440730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff42a00700 1 --2- 172.21.15.116:0/1556562186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff44100e00 0x7eff440730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff42a00700 1 -- 172.21.15.116:0/1556562186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff44106f40 msgr2=0x7eff44073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff42a00700 1 --2- 172.21.15.116:0/1556562186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff44106f40 0x7eff44073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.701+0000 7eff42a00700 1 -- 172.21.15.116:0/1556562186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c009c40 con 0x7eff441017c0 2024-03-30T00:56:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.702+0000 7eff42a00700 1 --2- 172.21.15.116:0/1556562186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff441017c0 0x7eff440735f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7eff3000e9f0 tx=0x7eff3000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.702+0000 7eff3be00700 1 -- 172.21.15.116:0/1556562186 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3000c9c0 con 0x7eff441017c0 2024-03-30T00:56:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.702+0000 7eff4a800700 1 -- 172.21.15.116:0/1556562186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff441a5cc0 con 0x7eff441017c0 2024-03-30T00:56:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.702+0000 7eff3be00700 1 -- 172.21.15.116:0/1556562186 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff30010430 con 0x7eff441017c0 2024-03-30T00:56:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.702+0000 7eff3be00700 1 -- 172.21.15.116:0/1556562186 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30018680 con 0x7eff441017c0 2024-03-30T00:56:57.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.702+0000 7eff4a800700 1 -- 172.21.15.116:0/1556562186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff441a61c0 con 0x7eff441017c0 2024-03-30T00:56:57.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.703+0000 7eff4a800700 1 -- 172.21.15.116:0/1556562186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff44109260 con 0x7eff441017c0 2024-03-30T00:56:57.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.706+0000 7eff3be00700 1 -- 172.21.15.116:0/1556562186 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7eff30010860 con 0x7eff441017c0 2024-03-30T00:56:57.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.707+0000 7eff3be00700 1 --2- 172.21.15.116:0/1556562186 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7eff280a1230 0x7eff280a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:57.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.707+0000 7eff3be00700 1 -- 172.21.15.116:0/1556562186 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7eff30014070 con 0x7eff441017c0 2024-03-30T00:56:57.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.708+0000 7eff43400700 1 --2- 172.21.15.116:0/1556562186 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7eff280a1230 0x7eff280a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:57.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.708+0000 7eff43400700 1 --2- 172.21.15.116:0/1556562186 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7eff280a1230 0x7eff280a36f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7eff34005950 tx=0x7eff34009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:57.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.711+0000 7eff3be00700 1 -- 172.21.15.116:0/1556562186 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7eff300529f0 con 0x7eff441017c0 2024-03-30T00:56:57.747 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:57 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2308861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-30T00:56:57.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:57 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2308861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-30T00:56:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.929+0000 7eff4a800700 1 -- 172.21.15.116:0/1556562186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7eff441024f0 con 0x7eff441017c0 2024-03-30T00:56:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.930+0000 7eff3be00700 1 -- 172.21.15.116:0/1556562186 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7eff300960f0 con 0x7eff441017c0 2024-03-30T00:56:57.936 INFO:teuthology.orchestra.run.smithi116.stdout:197568495653 2024-03-30T00:56:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.932+0000 7eff4a800700 1 -- 172.21.15.116:0/1556562186 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7eff280a1230 msgr2=0x7eff280a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.932+0000 7eff4a800700 1 --2- 172.21.15.116:0/1556562186 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7eff280a1230 0x7eff280a36f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7eff34005950 tx=0x7eff34009310 comp rx=0 tx=0).stop 2024-03-30T00:56:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.932+0000 7eff4a800700 1 -- 172.21.15.116:0/1556562186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff441017c0 msgr2=0x7eff440735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.932+0000 7eff4a800700 1 --2- 172.21.15.116:0/1556562186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff441017c0 0x7eff440735f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7eff3000e9f0 tx=0x7eff3000ed00 comp rx=0 tx=0).stop 2024-03-30T00:56:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.932+0000 7eff4a800700 1 -- 172.21.15.116:0/1556562186 shutdown_connections 2024-03-30T00:56:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.932+0000 7eff4a800700 1 --2- 172.21.15.116:0/1556562186 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7eff280a1230 0x7eff280a36f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.932+0000 7eff4a800700 1 --2- 172.21.15.116:0/1556562186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff44100e00 0x7eff440730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.932+0000 7eff4a800700 1 --2- 172.21.15.116:0/1556562186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff44106f40 0x7eff44073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.932+0000 7eff4a800700 1 --2- 172.21.15.116:0/1556562186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff441017c0 0x7eff440735f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.932+0000 7eff4a800700 1 -- 172.21.15.116:0/1556562186 >> 172.21.15.116:0/1556562186 conn(0x7eff44077f10 msgr2=0x7eff440ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.933+0000 7eff4a800700 1 -- 172.21.15.116:0/1556562186 shutdown_connections 2024-03-30T00:56:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.933+0000 7eff4a800700 1 -- 172.21.15.116:0/1556562186 wait complete. 2024-03-30T00:56:58.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:57 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2308861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-30T00:56:58.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.996+0000 7fdefd800700 1 -- 172.21.15.116:0/4071521254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdef8073520 msgr2=0x7fdef80739a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:58.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.996+0000 7fdefd800700 1 --2- 172.21.15.116:0/4071521254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdef8073520 0x7fdef80739a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009990 tx=0x7fdee4009ca0 comp rx=0 tx=0).stop 2024-03-30T00:56:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.997+0000 7fdefd800700 1 -- 172.21.15.116:0/4071521254 shutdown_connections 2024-03-30T00:56:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.997+0000 7fdefd800700 1 --2- 172.21.15.116:0/4071521254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdef8111b30 0x7fdef8113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.997+0000 7fdefd800700 1 --2- 172.21.15.116:0/4071521254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef8074d30 0x7fdef8072fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.997+0000 7fdefd800700 1 --2- 172.21.15.116:0/4071521254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdef8073520 0x7fdef80739a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.997+0000 7fdefd800700 1 -- 172.21.15.116:0/4071521254 >> 172.21.15.116:0/4071521254 conn(0x7fdef8077fa0 msgr2=0x7fdef80783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.997+0000 7fdefd800700 1 -- 172.21.15.116:0/4071521254 shutdown_connections 2024-03-30T00:56:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.997+0000 7fdefd800700 1 -- 172.21.15.116:0/4071521254 wait complete. 2024-03-30T00:56:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.998+0000 7fdefd800700 1 Processor -- start 2024-03-30T00:56:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.998+0000 7fdefd800700 1 -- start start 2024-03-30T00:56:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.998+0000 7fdefd800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdef8073520 0x7fdef81a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.998+0000 7fdefd800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef8074d30 0x7fdef81a3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:58.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.998+0000 7fdefd800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdef8111b30 0x7fdef819d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:58.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.998+0000 7fdefd800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef819dd90 con 0x7fdef8074d30 2024-03-30T00:56:58.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.998+0000 7fdefd800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef819df00 con 0x7fdef8111b30 2024-03-30T00:56:58.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.998+0000 7fdefd800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef819e0c0 con 0x7fdef8073520 2024-03-30T00:56:58.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.999+0000 7fdef6000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef8074d30 0x7fdef81a3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:58.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.999+0000 7fdef6a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdef8073520 0x7fdef81a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:58.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.999+0000 7fdef6000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef8074d30 0x7fdef81a3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59250/0 (socket says 172.21.15.116:59250) 2024-03-30T00:56:58.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.999+0000 7fdef6a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdef8073520 0x7fdef81a35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:43106/0 (socket says 172.21.15.116:43106) 2024-03-30T00:56:58.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.999+0000 7fdef6000700 1 -- 172.21.15.116:0/1217628346 learned_addr learned my addr 172.21.15.116:0/1217628346 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:56:58.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:57.999+0000 7fdef7400700 1 --2- 172.21.15.116:0/1217628346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdef8111b30 0x7fdef819d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:58.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.000+0000 7fdef6a00700 1 -- 172.21.15.116:0/1217628346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdef8111b30 msgr2=0x7fdef819d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:58.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.000+0000 7fdef6a00700 1 --2- 172.21.15.116:0/1217628346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdef8111b30 0x7fdef819d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:58.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.000+0000 7fdef6a00700 1 -- 172.21.15.116:0/1217628346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef8074d30 msgr2=0x7fdef81a3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:58.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.000+0000 7fdef6a00700 1 --2- 172.21.15.116:0/1217628346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef8074d30 0x7fdef81a3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:58.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.000+0000 7fdef6a00700 1 -- 172.21.15.116:0/1217628346 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee40095f0 con 0x7fdef8073520 2024-03-30T00:56:58.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.000+0000 7fdef6a00700 1 --2- 172.21.15.116:0/1217628346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdef8073520 0x7fdef81a35a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdee800e9f0 tx=0x7fdee800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:58.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.000+0000 7fdeef400700 1 -- 172.21.15.116:0/1217628346 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee800eee0 con 0x7fdef8073520 2024-03-30T00:56:58.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.000+0000 7fdeef400700 1 -- 172.21.15.116:0/1217628346 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdee800cbd0 con 0x7fdef8073520 2024-03-30T00:56:58.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.000+0000 7fdefd800700 1 -- 172.21.15.116:0/1217628346 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef819e3a0 con 0x7fdef8073520 2024-03-30T00:56:58.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.000+0000 7fdeef400700 1 -- 172.21.15.116:0/1217628346 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8018680 con 0x7fdef8073520 2024-03-30T00:56:58.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.001+0000 7fdefd800700 1 -- 172.21.15.116:0/1217628346 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef81aa590 con 0x7fdef8073520 2024-03-30T00:56:58.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.002+0000 7fdefd800700 1 -- 172.21.15.116:0/1217628346 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded80052f0 con 0x7fdef8073520 2024-03-30T00:56:58.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.005+0000 7fdeef400700 1 -- 172.21.15.116:0/1217628346 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fdee8010430 con 0x7fdef8073520 2024-03-30T00:56:58.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.007+0000 7fdeef400700 1 --2- 172.21.15.116:0/1217628346 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdedc0a1230 0x7fdedc0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:56:58.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.007+0000 7fdeef400700 1 -- 172.21.15.116:0/1217628346 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdee8014070 con 0x7fdef8073520 2024-03-30T00:56:58.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.007+0000 7fdef6000700 1 --2- 172.21.15.116:0/1217628346 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdedc0a1230 0x7fdedc0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:56:58.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.008+0000 7fdef6000700 1 --2- 172.21.15.116:0/1217628346 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdedc0a1230 0x7fdedc0a36f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdef819edf0 tx=0x7fdee400b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:56:58.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.011+0000 7fdeef400700 1 -- 172.21.15.116:0/1217628346 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdee80529f0 con 0x7fdef8073520 2024-03-30T00:56:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.230+0000 7fdefd800700 1 -- 172.21.15.116:0/1217628346 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fded8005160 con 0x7fdef8073520 2024-03-30T00:56:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.231+0000 7fdeef400700 1 -- 172.21.15.116:0/1217628346 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdee8099af0 con 0x7fdef8073520 2024-03-30T00:56:58.236 INFO:teuthology.orchestra.run.smithi116.stdout:51539607611 2024-03-30T00:56:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.233+0000 7fdefd800700 1 -- 172.21.15.116:0/1217628346 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdedc0a1230 msgr2=0x7fdedc0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.233+0000 7fdefd800700 1 --2- 172.21.15.116:0/1217628346 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdedc0a1230 0x7fdedc0a36f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdef819edf0 tx=0x7fdee400b280 comp rx=0 tx=0).stop 2024-03-30T00:56:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.233+0000 7fdefd800700 1 -- 172.21.15.116:0/1217628346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdef8073520 msgr2=0x7fdef81a35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:56:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.233+0000 7fdefd800700 1 --2- 172.21.15.116:0/1217628346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdef8073520 0x7fdef81a35a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdee800e9f0 tx=0x7fdee800ed00 comp rx=0 tx=0).stop 2024-03-30T00:56:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.233+0000 7fdefd800700 1 -- 172.21.15.116:0/1217628346 shutdown_connections 2024-03-30T00:56:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.233+0000 7fdefd800700 1 --2- 172.21.15.116:0/1217628346 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdedc0a1230 0x7fdedc0a36f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.233+0000 7fdefd800700 1 --2- 172.21.15.116:0/1217628346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdef8073520 0x7fdef81a35a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.233+0000 7fdefd800700 1 --2- 172.21.15.116:0/1217628346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdef8111b30 0x7fdef819d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.233+0000 7fdefd800700 1 --2- 172.21.15.116:0/1217628346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef8074d30 0x7fdef81a3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:56:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.233+0000 7fdefd800700 1 -- 172.21.15.116:0/1217628346 >> 172.21.15.116:0/1217628346 conn(0x7fdef8077fa0 msgr2=0x7fdef81107a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:56:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.233+0000 7fdefd800700 1 -- 172.21.15.116:0/1217628346 shutdown_connections 2024-03-30T00:56:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:56:58.233+0000 7fdefd800700 1 -- 172.21.15.116:0/1217628346 wait complete. 2024-03-30T00:56:58.418 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495651 got 197568495653 for osd.7 2024-03-30T00:56:58.418 DEBUG:teuthology.parallel:result is None 2024-03-30T00:56:58.727 INFO:tasks.cephadm.ceph_manager.ceph:need seq 51539607609 got 51539607611 for osd.1 2024-03-30T00:56:58.727 DEBUG:teuthology.parallel:result is None 2024-03-30T00:56:58.727 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-30T00:56:58.727 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T00:56:58.791 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:58 smithi116 ceph-mon[38086]: pgmap v264: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T00:56:58.791 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:58 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1556562186' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T00:56:58.791 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:56:58 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1217628346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T00:56:58.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:58 smithi130 ceph-mon[47626]: pgmap v264: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T00:56:58.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:58 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1556562186' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T00:56:58.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:56:58 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1217628346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T00:56:59.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:58 smithi181 ceph-mon[47622]: pgmap v264: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T00:56:59.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:58 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1556562186' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T00:56:59.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:56:58 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1217628346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T00:56:59.111 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:57:00.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.371+0000 7f15ca200700 1 -- 172.21.15.116:0/3857065648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15c4101820 msgr2=0x7f15c410dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:00.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.371+0000 7f15ca200700 1 --2- 172.21.15.116:0/3857065648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15c4101820 0x7f15c410dc10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00b0d0 tx=0x7f15bc00b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:00.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.371+0000 7f15ca200700 1 -- 172.21.15.116:0/3857065648 shutdown_connections 2024-03-30T00:57:00.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.371+0000 7f15ca200700 1 --2- 172.21.15.116:0/3857065648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15c4106ef0 0x7f15c41072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:00.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.371+0000 7f15ca200700 1 --2- 172.21.15.116:0/3857065648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15c4101820 0x7f15c410dc10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:00.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.371+0000 7f15ca200700 1 --2- 172.21.15.116:0/3857065648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15c4100e60 0x7f15c41012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:00.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.371+0000 7f15ca200700 1 -- 172.21.15.116:0/3857065648 >> 172.21.15.116:0/3857065648 conn(0x7f15c4077fa0 msgr2=0x7f15c40783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:00.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.371+0000 7f15ca200700 1 -- 172.21.15.116:0/3857065648 shutdown_connections 2024-03-30T00:57:00.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.371+0000 7f15ca200700 1 -- 172.21.15.116:0/3857065648 wait complete. 2024-03-30T00:57:00.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.372+0000 7f15ca200700 1 Processor -- start 2024-03-30T00:57:00.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.372+0000 7f15ca200700 1 -- start start 2024-03-30T00:57:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15ca200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15c4100e60 0x7f15c41972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15ca200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15c4101820 0x7f15c4197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15ca200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15c4106ef0 0x7f15c419bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15ca200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c419c1c0 con 0x7f15c4106ef0 2024-03-30T00:57:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15ca200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c419c330 con 0x7f15c4100e60 2024-03-30T00:57:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15ca200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c419c4f0 con 0x7f15c4101820 2024-03-30T00:57:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15c3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15c4106ef0 0x7f15c419bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15c3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15c4106ef0 0x7f15c419bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59270/0 (socket says 172.21.15.116:59270) 2024-03-30T00:57:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15c2a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15c4101820 0x7f15c4197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15c3e00700 1 -- 172.21.15.116:0/752381170 learned_addr learned my addr 172.21.15.116:0/752381170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15c3400700 1 --2- 172.21.15.116:0/752381170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15c4100e60 0x7f15c41972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15c2a00700 1 -- 172.21.15.116:0/752381170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15c4100e60 msgr2=0x7f15c41972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15c2a00700 1 --2- 172.21.15.116:0/752381170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15c4100e60 0x7f15c41972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15c2a00700 1 -- 172.21.15.116:0/752381170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15c4106ef0 msgr2=0x7f15c419bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15c2a00700 1 --2- 172.21.15.116:0/752381170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15c4106ef0 0x7f15c419bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15c2a00700 1 -- 172.21.15.116:0/752381170 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15bc009d00 con 0x7f15c4101820 2024-03-30T00:57:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.373+0000 7f15c3e00700 1 --2- 172.21.15.116:0/752381170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15c4106ef0 0x7f15c419bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.374+0000 7f15c2a00700 1 --2- 172.21.15.116:0/752381170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15c4101820 0x7f15c4197800 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f15b000e9f0 tx=0x7f15b000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.374+0000 7f15bbe00700 1 -- 172.21.15.116:0/752381170 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b000c9c0 con 0x7f15c4101820 2024-03-30T00:57:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.374+0000 7f15bbe00700 1 -- 172.21.15.116:0/752381170 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15b0010430 con 0x7f15c4101820 2024-03-30T00:57:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.374+0000 7f15ca200700 1 -- 172.21.15.116:0/752381170 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c419c7d0 con 0x7f15c4101820 2024-03-30T00:57:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.374+0000 7f15bbe00700 1 -- 172.21.15.116:0/752381170 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b0018680 con 0x7f15c4101820 2024-03-30T00:57:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.374+0000 7f15ca200700 1 -- 172.21.15.116:0/752381170 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c41a4030 con 0x7f15c4101820 2024-03-30T00:57:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.375+0000 7f15ca200700 1 -- 172.21.15.116:0/752381170 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c41092f0 con 0x7f15c4101820 2024-03-30T00:57:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.377+0000 7f15bbe00700 1 -- 172.21.15.116:0/752381170 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f15b0010860 con 0x7f15c4101820 2024-03-30T00:57:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.378+0000 7f15bbe00700 1 --2- 172.21.15.116:0/752381170 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f15a80a1230 0x7f15a80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.378+0000 7f15bbe00700 1 -- 172.21.15.116:0/752381170 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f15b0014070 con 0x7f15c4101820 2024-03-30T00:57:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.379+0000 7f15c3400700 1 --2- 172.21.15.116:0/752381170 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f15a80a1230 0x7f15a80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.379+0000 7f15c3400700 1 --2- 172.21.15.116:0/752381170 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f15a80a1230 0x7f15a80a36f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009c60 tx=0x7f15b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:00.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.383+0000 7f15bbe00700 1 -- 172.21.15.116:0/752381170 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f15b00529f0 con 0x7f15c4101820 2024-03-30T00:57:00.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:00 smithi116 ceph-mon[38086]: pgmap v265: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T00:57:00.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:00 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:57:00.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:00 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:57:00.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:00 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:57:00.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:00 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:57:00.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:00 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:57:00.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.582+0000 7f15ca200700 1 -- 172.21.15.116:0/752381170 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f15c41985d0 con 0x7f15a80a1230 2024-03-30T00:57:00.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:00 smithi130 ceph-mon[47626]: pgmap v265: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T00:57:00.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:00 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:57:00.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:00 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:57:00.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:00 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:57:00.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:00 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:57:00.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:00 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:57:00.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.595+0000 7f15bbe00700 1 -- 172.21.15.116:0/752381170 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426261 (secure 0 0 0) 0x7f15c41985d0 con 0x7f15a80a1230 2024-03-30T00:57:00.608 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:00.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.601+0000 7f15ca200700 1 -- 172.21.15.116:0/752381170 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f15a80a1230 msgr2=0x7f15a80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:00.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.601+0000 7f15ca200700 1 --2- 172.21.15.116:0/752381170 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f15a80a1230 0x7f15a80a36f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009c60 tx=0x7f15b4009260 comp rx=0 tx=0).stop 2024-03-30T00:57:00.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.601+0000 7f15ca200700 1 -- 172.21.15.116:0/752381170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15c4101820 msgr2=0x7f15c4197800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:00.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.601+0000 7f15ca200700 1 --2- 172.21.15.116:0/752381170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15c4101820 0x7f15c4197800 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f15b000e9f0 tx=0x7f15b000ed00 comp rx=0 tx=0).stop 2024-03-30T00:57:00.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.601+0000 7f15ca200700 1 -- 172.21.15.116:0/752381170 shutdown_connections 2024-03-30T00:57:00.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.601+0000 7f15ca200700 1 --2- 172.21.15.116:0/752381170 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f15a80a1230 0x7f15a80a36f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:00.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.601+0000 7f15ca200700 1 --2- 172.21.15.116:0/752381170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15c4106ef0 0x7f15c419bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:00.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.601+0000 7f15ca200700 1 --2- 172.21.15.116:0/752381170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15c4100e60 0x7f15c41972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:00.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.601+0000 7f15ca200700 1 --2- 172.21.15.116:0/752381170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15c4101820 0x7f15c4197800 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:00.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.601+0000 7f15ca200700 1 -- 172.21.15.116:0/752381170 >> 172.21.15.116:0/752381170 conn(0x7f15c4077fa0 msgr2=0x7f15c40ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:00.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.601+0000 7f15ca200700 1 -- 172.21.15.116:0/752381170 shutdown_connections 2024-03-30T00:57:00.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:00.601+0000 7f15ca200700 1 -- 172.21.15.116:0/752381170 wait complete. 2024-03-30T00:57:00.613 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-30T00:57:00.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:00 smithi181 ceph-mon[47622]: pgmap v265: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T00:57:00.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:00 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:57:00.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:00 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:57:00.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:00 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:57:00.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:00 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:57:00.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:00 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:57:01.002 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-03-30T00:56:59.117390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655934,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":357,"num_read_kb":292,"num_write":339,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180887,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":277,"ondisk_log_size":277,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":351736,"kb_used_data":11104,"kb_used_omap":18,"kb_used_meta":340461,"kb_avail":1124590088,"statfs":{"total":1151940427776,"available":1151580250112,"internally_reserved":0,"allocated":11370496,"data_stored":7488924,"data_compressed":320673,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19128,"internal_metadata":348632392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1652,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":32,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003409"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460643+0000","last_change":"2024-03-30T00:55:51.809659+0000","last_active":"2024-03-30T00:56:35.460643+0000","last_peered":"2024-03-30T00:56:35.460643+0000","last_clean":"2024-03-30T00:56:35.460643+0000","last_became_active":"2024-03-30T00:55:51.809337+0000","last_became_peered":"2024-03-30T00:55:51.809337+0000","last_unstale":"2024-03-30T00:56:35.460643+0000","last_undegraded":"2024-03-30T00:56:35.460643+0000","last_fullsized":"2024-03-30T00:56:35.460643+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:20:50.640033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495951+0000","last_change":"2024-03-30T00:55:51.813778+0000","last_active":"2024-03-30T00:56:35.495951+0000","last_peered":"2024-03-30T00:56:35.495951+0000","last_clean":"2024-03-30T00:56:35.495951+0000","last_became_active":"2024-03-30T00:55:51.810347+0000","last_became_peered":"2024-03-30T00:55:51.810347+0000","last_unstale":"2024-03-30T00:56:35.495951+0000","last_undegraded":"2024-03-30T00:56:35.495951+0000","last_fullsized":"2024-03-30T00:56:35.495951+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:49:07.031477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495722+0000","last_change":"2024-03-30T00:55:51.817647+0000","last_active":"2024-03-30T00:56:35.495722+0000","last_peered":"2024-03-30T00:56:35.495722+0000","last_clean":"2024-03-30T00:56:35.495722+0000","last_became_active":"2024-03-30T00:55:51.816495+0000","last_became_peered":"2024-03-30T00:55:51.816495+0000","last_unstale":"2024-03-30T00:56:35.495722+0000","last_undegraded":"2024-03-30T00:56:35.495722+0000","last_fullsized":"2024-03-30T00:56:35.495722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:59:42.965346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463453+0000","last_change":"2024-03-30T00:55:51.817581+0000","last_active":"2024-03-30T00:56:35.463453+0000","last_peered":"2024-03-30T00:56:35.463453+0000","last_clean":"2024-03-30T00:56:35.463453+0000","last_became_active":"2024-03-30T00:55:51.816223+0000","last_became_peered":"2024-03-30T00:55:51.816223+0000","last_unstale":"2024-03-30T00:56:35.463453+0000","last_undegraded":"2024-03-30T00:56:35.463453+0000","last_fullsized":"2024-03-30T00:56:35.463453+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:48:32.203975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462897+0000","last_change":"2024-03-30T00:55:51.811707+0000","last_active":"2024-03-30T00:56:35.462897+0000","last_peered":"2024-03-30T00:56:35.462897+0000","last_clean":"2024-03-30T00:56:35.462897+0000","last_became_active":"2024-03-30T00:55:51.811366+0000","last_became_peered":"2024-03-30T00:55:51.811366+0000","last_unstale":"2024-03-30T00:56:35.462897+0000","last_undegraded":"2024-03-30T00:56:35.462897+0000","last_fullsized":"2024-03-30T00:56:35.462897+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:20.438505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467445+0000","last_change":"2024-03-30T00:55:51.826849+0000","last_active":"2024-03-30T00:56:35.467445+0000","last_peered":"2024-03-30T00:56:35.467445+0000","last_clean":"2024-03-30T00:56:35.467445+0000","last_became_active":"2024-03-30T00:55:51.826520+0000","last_became_peered":"2024-03-30T00:55:51.826520+0000","last_unstale":"2024-03-30T00:56:35.467445+0000","last_undegraded":"2024-03-30T00:56:35.467445+0000","last_fullsized":"2024-03-30T00:56:35.467445+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:54:03.423725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466080+0000","last_change":"2024-03-30T00:55:51.812880+0000","last_active":"2024-03-30T00:56:35.466080+0000","last_peered":"2024-03-30T00:56:35.466080+0000","last_clean":"2024-03-30T00:56:35.466080+0000","last_became_active":"2024-03-30T00:55:51.812572+0000","last_became_peered":"2024-03-30T00:55:51.812572+0000","last_unstale":"2024-03-30T00:56:35.466080+0000","last_undegraded":"2024-03-30T00:56:35.466080+0000","last_fullsized":"2024-03-30T00:56:35.466080+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:44:49.670641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470643+0000","last_change":"2024-03-30T00:55:51.815480+0000","last_active":"2024-03-30T00:56:35.470643+0000","last_peered":"2024-03-30T00:56:35.470643+0000","last_clean":"2024-03-30T00:56:35.470643+0000","last_became_active":"2024-03-30T00:55:51.814052+0000","last_became_peered":"2024-03-30T00:55:51.814052+0000","last_unstale":"2024-03-30T00:56:35.470643+0000","last_undegraded":"2024-03-30T00:56:35.470643+0000","last_fullsized":"2024-03-30T00:56:35.470643+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:27:49.930928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468858+0000","last_change":"2024-03-30T00:55:51.814127+0000","last_active":"2024-03-30T00:56:35.468858+0000","last_peered":"2024-03-30T00:56:35.468858+0000","last_clean":"2024-03-30T00:56:35.468858+0000","last_became_active":"2024-03-30T00:55:51.813820+0000","last_became_peered":"2024-03-30T00:55:51.813820+0000","last_unstale":"2024-03-30T00:56:35.468858+0000","last_undegraded":"2024-03-30T00:56:35.468858+0000","last_fullsized":"2024-03-30T00:56:35.468858+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:21:31.605935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468423+0000","last_change":"2024-03-30T00:55:51.815198+0000","last_active":"2024-03-30T00:56:35.468423+0000","last_peered":"2024-03-30T00:56:35.468423+0000","last_clean":"2024-03-30T00:56:35.468423+0000","last_became_active":"2024-03-30T00:55:51.814901+0000","last_became_peered":"2024-03-30T00:55:51.814901+0000","last_unstale":"2024-03-30T00:56:35.468423+0000","last_undegraded":"2024-03-30T00:56:35.468423+0000","last_fullsized":"2024-03-30T00:56:35.468423+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:45:51.224346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466742+0000","last_change":"2024-03-30T00:55:51.809884+0000","last_active":"2024-03-30T00:56:35.466742+0000","last_peered":"2024-03-30T00:56:35.466742+0000","last_clean":"2024-03-30T00:56:35.466742+0000","last_became_active":"2024-03-30T00:55:51.809576+0000","last_became_peered":"2024-03-30T00:55:51.809576+0000","last_unstale":"2024-03-30T00:56:35.466742+0000","last_undegraded":"2024-03-30T00:56:35.466742+0000","last_fullsized":"2024-03-30T00:56:35.466742+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:39:51.767744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463361+0000","last_change":"2024-03-30T00:55:51.817384+0000","last_active":"2024-03-30T00:56:35.463361+0000","last_peered":"2024-03-30T00:56:35.463361+0000","last_clean":"2024-03-30T00:56:35.463361+0000","last_became_active":"2024-03-30T00:55:51.816564+0000","last_became_peered":"2024-03-30T00:55:51.816564+0000","last_unstale":"2024-03-30T00:56:35.463361+0000","last_undegraded":"2024-03-30T00:56:35.463361+0000","last_fullsized":"2024-03-30T00:56:35.463361+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:47:07.195650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.475257+0000","last_change":"2024-03-30T00:55:51.814310+0000","last_active":"2024-03-30T00:56:35.475257+0000","last_peered":"2024-03-30T00:56:35.475257+0000","last_clean":"2024-03-30T00:56:35.475257+0000","last_became_active":"2024-03-30T00:55:51.813968+0000","last_became_peered":"2024-03-30T00:55:51.813968+0000","last_unstale":"2024-03-30T00:56:35.475257+0000","last_undegraded":"2024-03-30T00:56:35.475257+0000","last_fullsized":"2024-03-30T00:56:35.475257+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:59:55.955502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495223+0000","last_change":"2024-03-30T00:55:51.815655+0000","last_active":"2024-03-30T00:56:35.495223+0000","last_peered":"2024-03-30T00:56:35.495223+0000","last_clean":"2024-03-30T00:56:35.495223+0000","last_became_active":"2024-03-30T00:55:51.814099+0000","last_became_peered":"2024-03-30T00:55:51.814099+0000","last_unstale":"2024-03-30T00:56:35.495223+0000","last_undegraded":"2024-03-30T00:56:35.495223+0000","last_fullsized":"2024-03-30T00:56:35.495223+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:42.958153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.472491+0000","last_change":"2024-03-30T00:56:36.472491+0000","last_active":"2024-03-30T00:56:36.472491+0000","last_peered":"2024-03-30T00:56:36.472491+0000","last_clean":"2024-03-30T00:56:36.472491+0000","last_became_active":"2024-03-30T00:56:36.472037+0000","last_became_peered":"2024-03-30T00:56:36.472037+0000","last_unstale":"2024-03-30T00:56:36.472491+0000","last_undegraded":"2024-03-30T00:56:36.472491+0000","last_fullsized":"2024-03-30T00:56:36.472491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:52:08.402775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467785+0000","last_change":"2024-03-30T00:55:51.822278+0000","last_active":"2024-03-30T00:56:35.467785+0000","last_peered":"2024-03-30T00:56:35.467785+0000","last_clean":"2024-03-30T00:56:35.467785+0000","last_became_active":"2024-03-30T00:55:51.821996+0000","last_became_peered":"2024-03-30T00:55:51.821996+0000","last_unstale":"2024-03-30T00:56:35.467785+0000","last_undegraded":"2024-03-30T00:56:35.467785+0000","last_fullsized":"2024-03-30T00:56:35.467785+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:37:45.841853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460610+0000","last_change":"2024-03-30T00:55:51.809547+0000","last_active":"2024-03-30T00:56:35.460610+0000","last_peered":"2024-03-30T00:56:35.460610+0000","last_clean":"2024-03-30T00:56:35.460610+0000","last_became_active":"2024-03-30T00:55:51.809314+0000","last_became_peered":"2024-03-30T00:55:51.809314+0000","last_unstale":"2024-03-30T00:56:35.460610+0000","last_undegraded":"2024-03-30T00:56:35.460610+0000","last_fullsized":"2024-03-30T00:56:35.460610+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:39:31.661362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.488652+0000","last_change":"2024-03-30T00:55:51.810857+0000","last_active":"2024-03-30T00:56:35.488652+0000","last_peered":"2024-03-30T00:56:35.488652+0000","last_clean":"2024-03-30T00:56:35.488652+0000","last_became_active":"2024-03-30T00:55:51.809420+0000","last_became_peered":"2024-03-30T00:55:51.809420+0000","last_unstale":"2024-03-30T00:56:35.488652+0000","last_undegraded":"2024-03-30T00:56:35.488652+0000","last_fullsized":"2024-03-30T00:56:35.488652+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:36:48.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.471006+0000","last_change":"2024-03-30T00:55:51.814693+0000","last_active":"2024-03-30T00:56:35.471006+0000","last_peered":"2024-03-30T00:56:35.471006+0000","last_clean":"2024-03-30T00:56:35.471006+0000","last_became_active":"2024-03-30T00:55:51.814359+0000","last_became_peered":"2024-03-30T00:55:51.814359+0000","last_unstale":"2024-03-30T00:56:35.471006+0000","last_undegraded":"2024-03-30T00:56:35.471006+0000","last_fullsized":"2024-03-30T00:56:35.471006+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:20:50.856313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470709+0000","last_change":"2024-03-30T00:55:55.839089+0000","last_active":"2024-03-30T00:56:35.470709+0000","last_peered":"2024-03-30T00:56:35.470709+0000","last_clean":"2024-03-30T00:56:35.470709+0000","last_became_active":"2024-03-30T00:55:55.838705+0000","last_became_peered":"2024-03-30T00:55:55.838705+0000","last_unstale":"2024-03-30T00:56:35.470709+0000","last_undegraded":"2024-03-30T00:56:35.470709+0000","last_fullsized":"2024-03-30T00:56:35.470709+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:33:56.010272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'9","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.487878+0000","last_change":"2024-03-30T00:55:51.298577+0000","last_active":"2024-03-30T00:56:35.487878+0000","last_peered":"2024-03-30T00:56:35.487878+0000","last_clean":"2024-03-30T00:56:35.487878+0000","last_became_active":"2024-03-30T00:55:51.298269+0000","last_became_peered":"2024-03-30T00:55:51.298269+0000","last_unstale":"2024-03-30T00:56:35.487878+0000","last_undegraded":"2024-03-30T00:56:35.487878+0000","last_fullsized":"2024-03-30T00:56:35.487878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:10:04.998621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470406+0000","last_change":"2024-03-30T00:55:51.818517+0000","last_active":"2024-03-30T00:56:35.470406+0000","last_peered":"2024-03-30T00:56:35.470406+0000","last_clean":"2024-03-30T00:56:35.470406+0000","last_became_active":"2024-03-30T00:55:51.818174+0000","last_became_peered":"2024-03-30T00:55:51.818174+0000","last_unstale":"2024-03-30T00:56:35.470406+0000","last_undegraded":"2024-03-30T00:56:35.470406+0000","last_fullsized":"2024-03-30T00:56:35.470406+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:03:01.604135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.623912+0000","last_change":"2024-03-30T00:56:36.623912+0000","last_active":"2024-03-30T00:56:36.623912+0000","last_peered":"2024-03-30T00:56:36.623912+0000","last_clean":"2024-03-30T00:56:36.623912+0000","last_became_active":"2024-03-30T00:56:36.623266+0000","last_became_peered":"2024-03-30T00:56:36.623266+0000","last_unstale":"2024-03-30T00:56:36.623912+0000","last_undegraded":"2024-03-30T00:56:36.623912+0000","last_fullsized":"2024-03-30T00:56:36.623912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:46:47.166079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.471579+0000","last_change":"2024-03-30T00:55:51.312686+0000","last_active":"2024-03-30T00:56:35.471579+0000","last_peered":"2024-03-30T00:56:35.471579+0000","last_clean":"2024-03-30T00:56:35.471579+0000","last_became_active":"2024-03-30T00:55:51.312296+0000","last_became_peered":"2024-03-30T00:55:51.312296+0000","last_unstale":"2024-03-30T00:56:35.471579+0000","last_undegraded":"2024-03-30T00:56:35.471579+0000","last_fullsized":"2024-03-30T00:56:35.471579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:28:58.398577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495565+0000","last_change":"2024-03-30T00:55:51.812197+0000","last_active":"2024-03-30T00:56:35.495565+0000","last_peered":"2024-03-30T00:56:35.495565+0000","last_clean":"2024-03-30T00:56:35.495565+0000","last_became_active":"2024-03-30T00:55:51.811892+0000","last_became_peered":"2024-03-30T00:55:51.811892+0000","last_unstale":"2024-03-30T00:56:35.495565+0000","last_undegraded":"2024-03-30T00:56:35.495565+0000","last_fullsized":"2024-03-30T00:56:35.495565+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:06:53.459760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462318+0000","last_change":"2024-03-30T00:55:55.839015+0000","last_active":"2024-03-30T00:56:35.462318+0000","last_peered":"2024-03-30T00:56:35.462318+0000","last_clean":"2024-03-30T00:56:35.462318+0000","last_became_active":"2024-03-30T00:55:55.838608+0000","last_became_peered":"2024-03-30T00:55:55.838608+0000","last_unstale":"2024-03-30T00:56:35.462318+0000","last_undegraded":"2024-03-30T00:56:35.462318+0000","last_fullsized":"2024-03-30T00:56:35.462318+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:24:23.804754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'4","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463392+0000","last_change":"2024-03-30T00:55:51.304199+0000","last_active":"2024-03-30T00:56:35.463392+0000","last_peered":"2024-03-30T00:56:35.463392+0000","last_clean":"2024-03-30T00:56:35.463392+0000","last_became_active":"2024-03-30T00:55:51.303837+0000","last_became_peered":"2024-03-30T00:55:51.303837+0000","last_unstale":"2024-03-30T00:56:35.463392+0000","last_undegraded":"2024-03-30T00:56:35.463392+0000","last_fullsized":"2024-03-30T00:56:35.463392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:19.154957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.469403+0000","last_change":"2024-03-30T00:55:51.817984+0000","last_active":"2024-03-30T00:56:35.469403+0000","last_peered":"2024-03-30T00:56:35.469403+0000","last_clean":"2024-03-30T00:56:35.469403+0000","last_became_active":"2024-03-30T00:55:51.817767+0000","last_became_peered":"2024-03-30T00:55:51.817767+0000","last_unstale":"2024-03-30T00:56:35.469403+0000","last_undegraded":"2024-03-30T00:56:35.469403+0000","last_fullsized":"2024-03-30T00:56:35.469403+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:12:48.558024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.623581+0000","last_change":"2024-03-30T00:56:36.623581+0000","last_active":"2024-03-30T00:56:36.623581+0000","last_peered":"2024-03-30T00:56:36.623581+0000","last_clean":"2024-03-30T00:56:36.623581+0000","last_became_active":"2024-03-30T00:56:36.623168+0000","last_became_peered":"2024-03-30T00:56:36.623168+0000","last_unstale":"2024-03-30T00:56:36.623581+0000","last_undegraded":"2024-03-30T00:56:36.623581+0000","last_fullsized":"2024-03-30T00:56:36.623581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:41.448935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"76'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.480921+0000","last_change":"2024-03-30T00:56:36.480478+0000","last_active":"2024-03-30T00:56:36.480921+0000","last_peered":"2024-03-30T00:56:36.480921+0000","last_clean":"2024-03-30T00:56:36.480921+0000","last_became_active":"2024-03-30T00:56:36.473297+0000","last_became_peered":"2024-03-30T00:56:36.473297+0000","last_unstale":"2024-03-30T00:56:36.480921+0000","last_undegraded":"2024-03-30T00:56:36.480921+0000","last_fullsized":"2024-03-30T00:56:36.480921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:37:08.090281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466888+0000","last_change":"2024-03-30T00:55:51.827985+0000","last_active":"2024-03-30T00:56:35.466888+0000","last_peered":"2024-03-30T00:56:35.466888+0000","last_clean":"2024-03-30T00:56:35.466888+0000","last_became_active":"2024-03-30T00:55:51.827678+0000","last_became_peered":"2024-03-30T00:55:51.827678+0000","last_unstale":"2024-03-30T00:56:35.466888+0000","last_undegraded":"2024-03-30T00:56:35.466888+0000","last_fullsized":"2024-03-30T00:56:35.466888+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:47.088009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470870+0000","last_change":"2024-03-30T00:55:55.827267+0000","last_active":"2024-03-30T00:56:35.470870+0000","last_peered":"2024-03-30T00:56:35.470870+0000","last_clean":"2024-03-30T00:56:35.470870+0000","last_became_active":"2024-03-30T00:55:55.826821+0000","last_became_peered":"2024-03-30T00:55:55.826821+0000","last_unstale":"2024-03-30T00:56:35.470870+0000","last_undegraded":"2024-03-30T00:56:35.470870+0000","last_fullsized":"2024-03-30T00:56:35.470870+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:15:49.084359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":59,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:53.200545+0000","last_change":"2024-03-30T00:55:51.305177+0000","last_active":"2024-03-30T00:56:53.200545+0000","last_peered":"2024-03-30T00:56:53.200545+0000","last_clean":"2024-03-30T00:56:53.200545+0000","last_became_active":"2024-03-30T00:55:51.304813+0000","last_became_peered":"2024-03-30T00:55:51.304813+0000","last_unstale":"2024-03-30T00:56:53.200545+0000","last_undegraded":"2024-03-30T00:56:53.200545+0000","last_fullsized":"2024-03-30T00:56:53.200545+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:05:20.578897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463357+0000","last_change":"2024-03-30T00:55:51.812714+0000","last_active":"2024-03-30T00:56:35.463357+0000","last_peered":"2024-03-30T00:56:35.463357+0000","last_clean":"2024-03-30T00:56:35.463357+0000","last_became_active":"2024-03-30T00:55:51.812406+0000","last_became_peered":"2024-03-30T00:55:51.812406+0000","last_unstale":"2024-03-30T00:56:35.463357+0000","last_undegraded":"2024-03-30T00:56:35.463357+0000","last_fullsized":"2024-03-30T00:56:35.463357+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:47:01.691412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470799+0000","last_change":"2024-03-30T00:55:55.831994+0000","last_active":"2024-03-30T00:56:35.470799+0000","last_peered":"2024-03-30T00:56:35.470799+0000","last_clean":"2024-03-30T00:56:35.470799+0000","last_became_active":"2024-03-30T00:55:55.831537+0000","last_became_peered":"2024-03-30T00:55:55.831537+0000","last_unstale":"2024-03-30T00:56:35.470799+0000","last_undegraded":"2024-03-30T00:56:35.470799+0000","last_fullsized":"2024-03-30T00:56:35.470799+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:44:47.240981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467196+0000","last_change":"2024-03-30T00:55:51.303468+0000","last_active":"2024-03-30T00:56:35.467196+0000","last_peered":"2024-03-30T00:56:35.467196+0000","last_clean":"2024-03-30T00:56:35.467196+0000","last_became_active":"2024-03-30T00:55:51.303068+0000","last_became_peered":"2024-03-30T00:55:51.303068+0000","last_unstale":"2024-03-30T00:56:35.467196+0000","last_undegraded":"2024-03-30T00:56:35.467196+0000","last_fullsized":"2024-03-30T00:56:35.467196+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:45:25.278196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466931+0000","last_change":"2024-03-30T00:55:51.825099+0000","last_active":"2024-03-30T00:56:35.466931+0000","last_peered":"2024-03-30T00:56:35.466931+0000","last_clean":"2024-03-30T00:56:35.466931+0000","last_became_active":"2024-03-30T00:55:51.824717+0000","last_became_peered":"2024-03-30T00:55:51.824717+0000","last_unstale":"2024-03-30T00:56:35.466931+0000","last_undegraded":"2024-03-30T00:56:35.466931+0000","last_fullsized":"2024-03-30T00:56:35.466931+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:49:31.542430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460499+0000","last_change":"2024-03-30T00:55:55.826106+0000","last_active":"2024-03-30T00:56:35.460499+0000","last_peered":"2024-03-30T00:56:35.460499+0000","last_clean":"2024-03-30T00:56:35.460499+0000","last_became_active":"2024-03-30T00:55:55.825597+0000","last_became_peered":"2024-03-30T00:55:55.825597+0000","last_unstale":"2024-03-30T00:56:35.460499+0000","last_undegraded":"2024-03-30T00:56:35.460499+0000","last_fullsized":"2024-03-30T00:56:35.460499+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:00:18.991732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.473981+0000","last_change":"2024-03-30T00:55:51.294090+0000","last_active":"2024-03-30T00:56:35.473981+0000","last_peered":"2024-03-30T00:56:35.473981+0000","last_clean":"2024-03-30T00:56:35.473981+0000","last_became_active":"2024-03-30T00:55:51.293668+0000","last_became_peered":"2024-03-30T00:55:51.293668+0000","last_unstale":"2024-03-30T00:56:35.473981+0000","last_undegraded":"2024-03-30T00:56:35.473981+0000","last_fullsized":"2024-03-30T00:56:35.473981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:33:36.661760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470557+0000","last_change":"2024-03-30T00:55:51.823428+0000","last_active":"2024-03-30T00:56:35.470557+0000","last_peered":"2024-03-30T00:56:35.470557+0000","last_clean":"2024-03-30T00:56:35.470557+0000","last_became_active":"2024-03-30T00:55:51.822222+0000","last_became_peered":"2024-03-30T00:55:51.822222+0000","last_unstale":"2024-03-30T00:56:35.470557+0000","last_undegraded":"2024-03-30T00:56:35.470557+0000","last_fullsized":"2024-03-30T00:56:35.470557+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:05:34.798940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495876+0000","last_change":"2024-03-30T00:55:55.828115+0000","last_active":"2024-03-30T00:56:35.495876+0000","last_peered":"2024-03-30T00:56:35.495876+0000","last_clean":"2024-03-30T00:56:35.495876+0000","last_became_active":"2024-03-30T00:55:55.827681+0000","last_became_peered":"2024-03-30T00:55:55.827681+0000","last_unstale":"2024-03-30T00:56:35.495876+0000","last_undegraded":"2024-03-30T00:56:35.495876+0000","last_fullsized":"2024-03-30T00:56:35.495876+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:18:25.751350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'1","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466298+0000","last_change":"2024-03-30T00:55:51.305163+0000","last_active":"2024-03-30T00:56:35.466298+0000","last_peered":"2024-03-30T00:56:35.466298+0000","last_clean":"2024-03-30T00:56:35.466298+0000","last_became_active":"2024-03-30T00:55:51.304860+0000","last_became_peered":"2024-03-30T00:55:51.304860+0000","last_unstale":"2024-03-30T00:56:35.466298+0000","last_undegraded":"2024-03-30T00:56:35.466298+0000","last_fullsized":"2024-03-30T00:56:35.466298+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:44:41.850998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.483962+0000","last_change":"2024-03-30T00:55:51.819401+0000","last_active":"2024-03-30T00:56:35.483962+0000","last_peered":"2024-03-30T00:56:35.483962+0000","last_clean":"2024-03-30T00:56:35.483962+0000","last_became_active":"2024-03-30T00:55:51.819021+0000","last_became_peered":"2024-03-30T00:55:51.819021+0000","last_unstale":"2024-03-30T00:56:35.483962+0000","last_undegraded":"2024-03-30T00:56:35.483962+0000","last_fullsized":"2024-03-30T00:56:35.483962+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:01:45.195719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.469445+0000","last_change":"2024-03-30T00:55:55.836806+0000","last_active":"2024-03-30T00:56:35.469445+0000","last_peered":"2024-03-30T00:56:35.469445+0000","last_clean":"2024-03-30T00:56:35.469445+0000","last_became_active":"2024-03-30T00:55:55.836403+0000","last_became_peered":"2024-03-30T00:55:55.836403+0000","last_unstale":"2024-03-30T00:56:35.469445+0000","last_undegraded":"2024-03-30T00:56:35.469445+0000","last_fullsized":"2024-03-30T00:56:35.469445+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:17:13.913675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'3","reported_seq":53,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:53.200223+0000","last_change":"2024-03-30T00:55:51.300833+0000","last_active":"2024-03-30T00:56:53.200223+0000","last_peered":"2024-03-30T00:56:53.200223+0000","last_clean":"2024-03-30T00:56:53.200223+0000","last_became_active":"2024-03-30T00:55:51.300473+0000","last_became_peered":"2024-03-30T00:55:51.300473+0000","last_unstale":"2024-03-30T00:56:53.200223+0000","last_undegraded":"2024-03-30T00:56:53.200223+0000","last_fullsized":"2024-03-30T00:56:53.200223+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:00:39.211981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468719+0000","last_change":"2024-03-30T00:55:51.810781+0000","last_active":"2024-03-30T00:56:35.468719+0000","last_peered":"2024-03-30T00:56:35.468719+0000","last_clean":"2024-03-30T00:56:35.468719+0000","last_became_active":"2024-03-30T00:55:51.810215+0000","last_became_peered":"2024-03-30T00:55:51.810215+0000","last_unstale":"2024-03-30T00:56:35.468719+0000","last_undegraded":"2024-03-30T00:56:35.468719+0000","last_fullsized":"2024-03-30T00:56:35.468719+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:53:50.283209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.484181+0000","last_change":"2024-03-30T00:55:55.834505+0000","last_active":"2024-03-30T00:56:35.484181+0000","last_peered":"2024-03-30T00:56:35.484181+0000","last_clean":"2024-03-30T00:56:35.484181+0000","last_became_active":"2024-03-30T00:55:55.829319+0000","last_became_peered":"2024-03-30T00:55:55.829319+0000","last_unstale":"2024-03-30T00:56:35.484181+0000","last_undegraded":"2024-03-30T00:56:35.484181+0000","last_fullsized":"2024-03-30T00:56:35.484181+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:51:37.595523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466067+0000","last_change":"2024-03-30T00:55:51.302070+0000","last_active":"2024-03-30T00:56:35.466067+0000","last_peered":"2024-03-30T00:56:35.466067+0000","last_clean":"2024-03-30T00:56:35.466067+0000","last_became_active":"2024-03-30T00:55:51.298940+0000","last_became_peered":"2024-03-30T00:55:51.298940+0000","last_unstale":"2024-03-30T00:56:35.466067+0000","last_undegraded":"2024-03-30T00:56:35.466067+0000","last_fullsized":"2024-03-30T00:56:35.466067+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:07:47.915506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495434+0000","last_change":"2024-03-30T00:55:51.819918+0000","last_active":"2024-03-30T00:56:35.495434+0000","last_peered":"2024-03-30T00:56:35.495434+0000","last_clean":"2024-03-30T00:56:35.495434+0000","last_became_active":"2024-03-30T00:55:51.819493+0000","last_became_peered":"2024-03-30T00:55:51.819493+0000","last_unstale":"2024-03-30T00:56:35.495434+0000","last_undegraded":"2024-03-30T00:56:35.495434+0000","last_fullsized":"2024-03-30T00:56:35.495434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:43:43.067031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462710+0000","last_change":"2024-03-30T00:55:55.827176+0000","last_active":"2024-03-30T00:56:35.462710+0000","last_peered":"2024-03-30T00:56:35.462710+0000","last_clean":"2024-03-30T00:56:35.462710+0000","last_became_active":"2024-03-30T00:55:55.826772+0000","last_became_peered":"2024-03-30T00:55:55.826772+0000","last_unstale":"2024-03-30T00:56:35.462710+0000","last_undegraded":"2024-03-30T00:56:35.462710+0000","last_fullsized":"2024-03-30T00:56:35.462710+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:02:04.512050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'4","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467237+0000","last_change":"2024-03-30T00:55:51.306688+0000","last_active":"2024-03-30T00:56:35.467237+0000","last_peered":"2024-03-30T00:56:35.467237+0000","last_clean":"2024-03-30T00:56:35.467237+0000","last_became_active":"2024-03-30T00:55:51.306333+0000","last_became_peered":"2024-03-30T00:55:51.306333+0000","last_unstale":"2024-03-30T00:56:35.467237+0000","last_undegraded":"2024-03-30T00:56:35.467237+0000","last_fullsized":"2024-03-30T00:56:35.467237+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:21:13.719947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.469497+0000","last_change":"2024-03-30T00:55:51.824879+0000","last_active":"2024-03-30T00:56:35.469497+0000","last_peered":"2024-03-30T00:56:35.469497+0000","last_clean":"2024-03-30T00:56:35.469497+0000","last_became_active":"2024-03-30T00:55:51.824566+0000","last_became_peered":"2024-03-30T00:55:51.824566+0000","last_unstale":"2024-03-30T00:56:35.469497+0000","last_undegraded":"2024-03-30T00:56:35.469497+0000","last_fullsized":"2024-03-30T00:56:35.469497+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:04:28.505693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.484132+0000","last_change":"2024-03-30T00:55:55.826387+0000","last_active":"2024-03-30T00:56:35.484132+0000","last_peered":"2024-03-30T00:56:35.484132+0000","last_clean":"2024-03-30T00:56:35.484132+0000","last_became_active":"2024-03-30T00:55:55.825941+0000","last_became_peered":"2024-03-30T00:55:55.825941+0000","last_unstale":"2024-03-30T00:56:35.484132+0000","last_undegraded":"2024-03-30T00:56:35.484132+0000","last_fullsized":"2024-03-30T00:56:35.484132+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:34:43.432712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"83'33","reported_seq":124,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:54.201521+0000","last_change":"2024-03-30T00:55:51.304592+0000","last_active":"2024-03-30T00:56:54.201521+0000","last_peered":"2024-03-30T00:56:54.201521+0000","last_clean":"2024-03-30T00:56:54.201521+0000","last_became_active":"2024-03-30T00:55:51.303931+0000","last_became_peered":"2024-03-30T00:55:51.303931+0000","last_unstale":"2024-03-30T00:56:54.201521+0000","last_undegraded":"2024-03-30T00:56:54.201521+0000","last_fullsized":"2024-03-30T00:56:54.201521+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:53:40.976289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61032,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":74,"num_read_kb":96,"num_write":34,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463233+0000","last_change":"2024-03-30T00:55:51.815196+0000","last_active":"2024-03-30T00:56:35.463233+0000","last_peered":"2024-03-30T00:56:35.463233+0000","last_clean":"2024-03-30T00:56:35.463233+0000","last_became_active":"2024-03-30T00:55:51.813376+0000","last_became_peered":"2024-03-30T00:55:51.813376+0000","last_unstale":"2024-03-30T00:56:35.463233+0000","last_undegraded":"2024-03-30T00:56:35.463233+0000","last_fullsized":"2024-03-30T00:56:35.463233+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:40:26.350248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467447+0000","last_change":"2024-03-30T00:55:55.836906+0000","last_active":"2024-03-30T00:56:35.467447+0000","last_peered":"2024-03-30T00:56:35.467447+0000","last_clean":"2024-03-30T00:56:35.467447+0000","last_became_active":"2024-03-30T00:55:55.836539+0000","last_became_peered":"2024-03-30T00:55:55.836539+0000","last_unstale":"2024-03-30T00:56:35.467447+0000","last_undegraded":"2024-03-30T00:56:35.467447+0000","last_fullsized":"2024-03-30T00:56:35.467447+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:21:39.320192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'6","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.478499+0000","last_change":"2024-03-30T00:56:36.478132+0000","last_active":"2024-03-30T00:56:36.478499+0000","last_peered":"2024-03-30T00:56:36.478499+0000","last_clean":"2024-03-30T00:56:36.478499+0000","last_became_active":"2024-03-30T00:56:36.467449+0000","last_became_peered":"2024-03-30T00:56:36.467449+0000","last_unstale":"2024-03-30T00:56:36.478499+0000","last_undegraded":"2024-03-30T00:56:36.478499+0000","last_fullsized":"2024-03-30T00:56:36.478499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:27:51.986112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.474554+0000","last_change":"2024-03-30T00:55:51.819825+0000","last_active":"2024-03-30T00:56:35.474554+0000","last_peered":"2024-03-30T00:56:35.474554+0000","last_clean":"2024-03-30T00:56:35.474554+0000","last_became_active":"2024-03-30T00:55:51.819481+0000","last_became_peered":"2024-03-30T00:55:51.819481+0000","last_unstale":"2024-03-30T00:56:35.474554+0000","last_undegraded":"2024-03-30T00:56:35.474554+0000","last_fullsized":"2024-03-30T00:56:35.474554+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:10:17.138541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460246+0000","last_change":"2024-03-30T00:55:55.832675+0000","last_active":"2024-03-30T00:56:35.460246+0000","last_peered":"2024-03-30T00:56:35.460246+0000","last_clean":"2024-03-30T00:56:35.460246+0000","last_became_active":"2024-03-30T00:55:55.832325+0000","last_became_peered":"2024-03-30T00:55:55.832325+0000","last_unstale":"2024-03-30T00:56:35.460246+0000","last_undegraded":"2024-03-30T00:56:35.460246+0000","last_fullsized":"2024-03-30T00:56:35.460246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:38:46.765473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467208+0000","last_change":"2024-03-30T00:55:51.313700+0000","last_active":"2024-03-30T00:56:35.467208+0000","last_peered":"2024-03-30T00:56:35.467208+0000","last_clean":"2024-03-30T00:56:35.467208+0000","last_became_active":"2024-03-30T00:55:51.313355+0000","last_became_peered":"2024-03-30T00:55:51.313355+0000","last_unstale":"2024-03-30T00:56:35.467208+0000","last_undegraded":"2024-03-30T00:56:35.467208+0000","last_fullsized":"2024-03-30T00:56:35.467208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:43:04.461398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467209+0000","last_change":"2024-03-30T00:55:51.819939+0000","last_active":"2024-03-30T00:56:35.467209+0000","last_peered":"2024-03-30T00:56:35.467209+0000","last_clean":"2024-03-30T00:56:35.467209+0000","last_became_active":"2024-03-30T00:55:51.819614+0000","last_became_peered":"2024-03-30T00:55:51.819614+0000","last_unstale":"2024-03-30T00:56:35.467209+0000","last_undegraded":"2024-03-30T00:56:35.467209+0000","last_fullsized":"2024-03-30T00:56:35.467209+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:20:37.362779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.469703+0000","last_change":"2024-03-30T00:56:36.469703+0000","last_active":"2024-03-30T00:56:36.469703+0000","last_peered":"2024-03-30T00:56:36.469703+0000","last_clean":"2024-03-30T00:56:36.469703+0000","last_became_active":"2024-03-30T00:56:36.469195+0000","last_became_peered":"2024-03-30T00:56:36.469195+0000","last_unstale":"2024-03-30T00:56:36.469703+0000","last_undegraded":"2024-03-30T00:56:36.469703+0000","last_fullsized":"2024-03-30T00:56:36.469703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:10:10.421447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"81'4","reported_seq":61,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:57.206035+0000","last_change":"2024-03-30T00:55:51.299246+0000","last_active":"2024-03-30T00:56:57.206035+0000","last_peered":"2024-03-30T00:56:57.206035+0000","last_clean":"2024-03-30T00:56:57.206035+0000","last_became_active":"2024-03-30T00:55:51.298924+0000","last_became_peered":"2024-03-30T00:55:51.298924+0000","last_unstale":"2024-03-30T00:56:57.206035+0000","last_undegraded":"2024-03-30T00:56:57.206035+0000","last_fullsized":"2024-03-30T00:56:57.206035+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:50:05.897612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466900+0000","last_change":"2024-03-30T00:55:51.813804+0000","last_active":"2024-03-30T00:56:35.466900+0000","last_peered":"2024-03-30T00:56:35.466900+0000","last_clean":"2024-03-30T00:56:35.466900+0000","last_became_active":"2024-03-30T00:55:51.813500+0000","last_became_peered":"2024-03-30T00:55:51.813500+0000","last_unstale":"2024-03-30T00:56:35.466900+0000","last_undegraded":"2024-03-30T00:56:35.466900+0000","last_fullsized":"2024-03-30T00:56:35.466900+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:42:21.614900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.477003+0000","last_change":"2024-03-30T00:55:55.837737+0000","last_active":"2024-03-30T00:56:35.477003+0000","last_peered":"2024-03-30T00:56:35.477003+0000","last_clean":"2024-03-30T00:56:35.477003+0000","last_became_active":"2024-03-30T00:55:55.837312+0000","last_became_peered":"2024-03-30T00:55:55.837312+0000","last_unstale":"2024-03-30T00:56:35.477003+0000","last_undegraded":"2024-03-30T00:56:35.477003+0000","last_fullsized":"2024-03-30T00:56:35.477003+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:11:46.139979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'15","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462932+0000","last_change":"2024-03-30T00:55:51.306398+0000","last_active":"2024-03-30T00:56:35.462932+0000","last_peered":"2024-03-30T00:56:35.462932+0000","last_clean":"2024-03-30T00:56:35.462932+0000","last_became_active":"2024-03-30T00:55:51.305878+0000","last_became_peered":"2024-03-30T00:55:51.305878+0000","last_unstale":"2024-03-30T00:56:35.462932+0000","last_undegraded":"2024-03-30T00:56:35.462932+0000","last_fullsized":"2024-03-30T00:56:35.462932+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:20:32.742423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466290+0000","last_change":"2024-03-30T00:55:51.824294+0000","last_active":"2024-03-30T00:56:35.466290+0000","last_peered":"2024-03-30T00:56:35.466290+0000","last_clean":"2024-03-30T00:56:35.466290+0000","last_became_active":"2024-03-30T00:55:51.823957+0000","last_became_peered":"2024-03-30T00:55:51.823957+0000","last_unstale":"2024-03-30T00:56:35.466290+0000","last_undegraded":"2024-03-30T00:56:35.466290+0000","last_fullsized":"2024-03-30T00:56:35.466290+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:23:16.873140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466253+0000","last_change":"2024-03-30T00:55:55.837023+0000","last_active":"2024-03-30T00:56:35.466253+0000","last_peered":"2024-03-30T00:56:35.466253+0000","last_clean":"2024-03-30T00:56:35.466253+0000","last_became_active":"2024-03-30T00:55:55.836541+0000","last_became_peered":"2024-03-30T00:55:55.836541+0000","last_unstale":"2024-03-30T00:56:35.466253+0000","last_undegraded":"2024-03-30T00:56:35.466253+0000","last_fullsized":"2024-03-30T00:56:35.466253+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:58:40.323953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463352+0000","last_change":"2024-03-30T00:55:51.300883+0000","last_active":"2024-03-30T00:56:35.463352+0000","last_peered":"2024-03-30T00:56:35.463352+0000","last_clean":"2024-03-30T00:56:35.463352+0000","last_became_active":"2024-03-30T00:55:51.300531+0000","last_became_peered":"2024-03-30T00:55:51.300531+0000","last_unstale":"2024-03-30T00:56:35.463352+0000","last_undegraded":"2024-03-30T00:56:35.463352+0000","last_fullsized":"2024-03-30T00:56:35.463352+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:38:38.874075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460541+0000","last_change":"2024-03-30T00:55:51.822970+0000","last_active":"2024-03-30T00:56:35.460541+0000","last_peered":"2024-03-30T00:56:35.460541+0000","last_clean":"2024-03-30T00:56:35.460541+0000","last_became_active":"2024-03-30T00:55:51.822653+0000","last_became_peered":"2024-03-30T00:55:51.822653+0000","last_unstale":"2024-03-30T00:56:35.460541+0000","last_undegraded":"2024-03-30T00:56:35.460541+0000","last_fullsized":"2024-03-30T00:56:35.460541+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:59:26.306672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463256+0000","last_change":"2024-03-30T00:55:55.856605+0000","last_active":"2024-03-30T00:56:35.463256+0000","last_peered":"2024-03-30T00:56:35.463256+0000","last_clean":"2024-03-30T00:56:35.463256+0000","last_became_active":"2024-03-30T00:55:55.856061+0000","last_became_peered":"2024-03-30T00:55:55.856061+0000","last_unstale":"2024-03-30T00:56:35.463256+0000","last_undegraded":"2024-03-30T00:56:35.463256+0000","last_fullsized":"2024-03-30T00:56:35.463256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:58:41.010206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.488831+0000","last_change":"2024-03-30T00:55:51.301961+0000","last_active":"2024-03-30T00:56:35.488831+0000","last_peered":"2024-03-30T00:56:35.488831+0000","last_clean":"2024-03-30T00:56:35.488831+0000","last_became_active":"2024-03-30T00:55:51.301618+0000","last_became_peered":"2024-03-30T00:55:51.301618+0000","last_unstale":"2024-03-30T00:56:35.488831+0000","last_undegraded":"2024-03-30T00:56:35.488831+0000","last_fullsized":"2024-03-30T00:56:35.488831+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:38:46.681490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467943+0000","last_change":"2024-03-30T00:55:51.821331+0000","last_active":"2024-03-30T00:56:35.467943+0000","last_peered":"2024-03-30T00:56:35.467943+0000","last_clean":"2024-03-30T00:56:35.467943+0000","last_became_active":"2024-03-30T00:55:51.820988+0000","last_became_peered":"2024-03-30T00:55:51.820988+0000","last_unstale":"2024-03-30T00:56:35.467943+0000","last_undegraded":"2024-03-30T00:56:35.467943+0000","last_fullsized":"2024-03-30T00:56:35.467943+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:25:20.150789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.475451+0000","last_change":"2024-03-30T00:55:55.826251+0000","last_active":"2024-03-30T00:56:35.475451+0000","last_peered":"2024-03-30T00:56:35.475451+0000","last_clean":"2024-03-30T00:56:35.475451+0000","last_became_active":"2024-03-30T00:55:55.822695+0000","last_became_peered":"2024-03-30T00:55:55.822695+0000","last_unstale":"2024-03-30T00:56:35.475451+0000","last_undegraded":"2024-03-30T00:56:35.475451+0000","last_fullsized":"2024-03-30T00:56:35.475451+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:32:19.011454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466393+0000","last_change":"2024-03-30T00:55:51.309412+0000","last_active":"2024-03-30T00:56:35.466393+0000","last_peered":"2024-03-30T00:56:35.466393+0000","last_clean":"2024-03-30T00:56:35.466393+0000","last_became_active":"2024-03-30T00:55:51.309028+0000","last_became_peered":"2024-03-30T00:55:51.309028+0000","last_unstale":"2024-03-30T00:56:35.466393+0000","last_undegraded":"2024-03-30T00:56:35.466393+0000","last_fullsized":"2024-03-30T00:56:35.466393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:00:49.022772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.467986+0000","last_change":"2024-03-30T00:56:36.467986+0000","last_active":"2024-03-30T00:56:36.467986+0000","last_peered":"2024-03-30T00:56:36.467986+0000","last_clean":"2024-03-30T00:56:36.467986+0000","last_became_active":"2024-03-30T00:56:36.467601+0000","last_became_peered":"2024-03-30T00:56:36.467601+0000","last_unstale":"2024-03-30T00:56:36.467986+0000","last_undegraded":"2024-03-30T00:56:36.467986+0000","last_fullsized":"2024-03-30T00:56:36.467986+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:37:32.979094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.479139+0000","last_change":"2024-03-30T00:55:55.856377+0000","last_active":"2024-03-30T00:56:35.479139+0000","last_peered":"2024-03-30T00:56:35.479139+0000","last_clean":"2024-03-30T00:56:35.479139+0000","last_became_active":"2024-03-30T00:55:55.855986+0000","last_became_peered":"2024-03-30T00:55:55.855986+0000","last_unstale":"2024-03-30T00:56:35.479139+0000","last_undegraded":"2024-03-30T00:56:35.479139+0000","last_fullsized":"2024-03-30T00:56:35.479139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:45:47.303446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.466031+0000","last_change":"2024-03-30T00:56:36.464863+0000","last_active":"2024-03-30T00:56:36.466031+0000","last_peered":"2024-03-30T00:56:36.466031+0000","last_clean":"2024-03-30T00:56:36.466031+0000","last_became_active":"2024-03-30T00:56:36.464323+0000","last_became_peered":"2024-03-30T00:56:36.464323+0000","last_unstale":"2024-03-30T00:56:36.466031+0000","last_undegraded":"2024-03-30T00:56:36.466031+0000","last_fullsized":"2024-03-30T00:56:36.466031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:43:10.899726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"81'4","reported_seq":56,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:54.201715+0000","last_change":"2024-03-30T00:55:51.314834+0000","last_active":"2024-03-30T00:56:54.201715+0000","last_peered":"2024-03-30T00:56:54.201715+0000","last_clean":"2024-03-30T00:56:54.201715+0000","last_became_active":"2024-03-30T00:55:51.314429+0000","last_became_peered":"2024-03-30T00:55:51.314429+0000","last_unstale":"2024-03-30T00:56:54.201715+0000","last_undegraded":"2024-03-30T00:56:54.201715+0000","last_fullsized":"2024-03-30T00:56:54.201715+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:17:13.987180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466483+0000","last_change":"2024-03-30T00:55:55.840721+0000","last_active":"2024-03-30T00:56:35.466483+0000","last_peered":"2024-03-30T00:56:35.466483+0000","last_clean":"2024-03-30T00:56:35.466483+0000","last_became_active":"2024-03-30T00:55:55.840269+0000","last_became_peered":"2024-03-30T00:55:55.840269+0000","last_unstale":"2024-03-30T00:56:35.466483+0000","last_undegraded":"2024-03-30T00:56:35.466483+0000","last_fullsized":"2024-03-30T00:56:35.466483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:28:55.366411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.473307+0000","last_change":"2024-03-30T00:55:51.812609+0000","last_active":"2024-03-30T00:56:35.473307+0000","last_peered":"2024-03-30T00:56:35.473307+0000","last_clean":"2024-03-30T00:56:35.473307+0000","last_became_active":"2024-03-30T00:55:51.812325+0000","last_became_peered":"2024-03-30T00:55:51.812325+0000","last_unstale":"2024-03-30T00:56:35.473307+0000","last_undegraded":"2024-03-30T00:56:35.473307+0000","last_fullsized":"2024-03-30T00:56:35.473307+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:22:22.074405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'3","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463363+0000","last_change":"2024-03-30T00:55:51.296173+0000","last_active":"2024-03-30T00:56:35.463363+0000","last_peered":"2024-03-30T00:56:35.463363+0000","last_clean":"2024-03-30T00:56:35.463363+0000","last_became_active":"2024-03-30T00:55:51.295331+0000","last_became_peered":"2024-03-30T00:55:51.295331+0000","last_unstale":"2024-03-30T00:56:35.463363+0000","last_undegraded":"2024-03-30T00:56:35.463363+0000","last_fullsized":"2024-03-30T00:56:35.463363+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:51:58.343765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":326,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466805+0000","last_change":"2024-03-30T00:55:02.487656+0000","last_active":"2024-03-30T00:56:35.466805+0000","last_peered":"2024-03-30T00:56:35.466805+0000","last_clean":"2024-03-30T00:56:35.466805+0000","last_became_active":"2024-03-30T00:55:02.487263+0000","last_became_peered":"2024-03-30T00:55:02.487263+0000","last_unstale":"2024-03-30T00:56:35.466805+0000","last_undegraded":"2024-03-30T00:56:35.466805+0000","last_fullsized":"2024-03-30T00:56:35.466805+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:52:29.691461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:52:29.691461+0000","last_clean_scrub_stamp":"2024-03-30T00:52:29.691461+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:19:20.825038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466944+0000","last_change":"2024-03-30T00:55:55.856487+0000","last_active":"2024-03-30T00:56:35.466944+0000","last_peered":"2024-03-30T00:56:35.466944+0000","last_clean":"2024-03-30T00:56:35.466944+0000","last_became_active":"2024-03-30T00:55:55.856106+0000","last_became_peered":"2024-03-30T00:55:55.856106+0000","last_unstale":"2024-03-30T00:56:35.466944+0000","last_undegraded":"2024-03-30T00:56:35.466944+0000","last_fullsized":"2024-03-30T00:56:35.466944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:14.504218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.473794+0000","last_change":"2024-03-30T00:55:51.821519+0000","last_active":"2024-03-30T00:56:35.473794+0000","last_peered":"2024-03-30T00:56:35.473794+0000","last_clean":"2024-03-30T00:56:35.473794+0000","last_became_active":"2024-03-30T00:55:51.821095+0000","last_became_peered":"2024-03-30T00:55:51.821095+0000","last_unstale":"2024-03-30T00:56:35.473794+0000","last_undegraded":"2024-03-30T00:56:35.473794+0000","last_fullsized":"2024-03-30T00:56:35.473794+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:50:22.297450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'3","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466994+0000","last_change":"2024-03-30T00:55:51.303047+0000","last_active":"2024-03-30T00:56:35.466994+0000","last_peered":"2024-03-30T00:56:35.466994+0000","last_clean":"2024-03-30T00:56:35.466994+0000","last_became_active":"2024-03-30T00:55:51.302712+0000","last_became_peered":"2024-03-30T00:55:51.302712+0000","last_unstale":"2024-03-30T00:56:35.466994+0000","last_undegraded":"2024-03-30T00:56:35.466994+0000","last_fullsized":"2024-03-30T00:56:35.466994+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:07:59.451674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460262+0000","last_change":"2024-03-30T00:55:55.823692+0000","last_active":"2024-03-30T00:56:35.460262+0000","last_peered":"2024-03-30T00:56:35.460262+0000","last_clean":"2024-03-30T00:56:35.460262+0000","last_became_active":"2024-03-30T00:55:55.823188+0000","last_became_peered":"2024-03-30T00:55:55.823188+0000","last_unstale":"2024-03-30T00:56:35.460262+0000","last_undegraded":"2024-03-30T00:56:35.460262+0000","last_fullsized":"2024-03-30T00:56:35.460262+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:32:32.089166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'1","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463201+0000","last_change":"2024-03-30T00:55:51.306935+0000","last_active":"2024-03-30T00:56:35.463201+0000","last_peered":"2024-03-30T00:56:35.463201+0000","last_clean":"2024-03-30T00:56:35.463201+0000","last_became_active":"2024-03-30T00:55:51.306595+0000","last_became_peered":"2024-03-30T00:55:51.306595+0000","last_unstale":"2024-03-30T00:56:35.463201+0000","last_undegraded":"2024-03-30T00:56:35.463201+0000","last_fullsized":"2024-03-30T00:56:35.463201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:50:30.396762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.475643+0000","last_change":"2024-03-30T00:55:51.807084+0000","last_active":"2024-03-30T00:56:35.475643+0000","last_peered":"2024-03-30T00:56:35.475643+0000","last_clean":"2024-03-30T00:56:35.475643+0000","last_became_active":"2024-03-30T00:55:51.806505+0000","last_became_peered":"2024-03-30T00:55:51.806505+0000","last_unstale":"2024-03-30T00:56:35.475643+0000","last_undegraded":"2024-03-30T00:56:35.475643+0000","last_fullsized":"2024-03-30T00:56:35.475643+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:38:44.759863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470455+0000","last_change":"2024-03-30T00:55:55.856457+0000","last_active":"2024-03-30T00:56:35.470455+0000","last_peered":"2024-03-30T00:56:35.470455+0000","last_clean":"2024-03-30T00:56:35.470455+0000","last_became_active":"2024-03-30T00:55:55.855982+0000","last_became_peered":"2024-03-30T00:55:55.855982+0000","last_unstale":"2024-03-30T00:56:35.470455+0000","last_undegraded":"2024-03-30T00:56:35.470455+0000","last_fullsized":"2024-03-30T00:56:35.470455+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:18:25.154634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"81'5","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.488541+0000","last_change":"2024-03-30T00:55:51.312581+0000","last_active":"2024-03-30T00:56:35.488541+0000","last_peered":"2024-03-30T00:56:35.488541+0000","last_clean":"2024-03-30T00:56:35.488541+0000","last_became_active":"2024-03-30T00:55:51.312225+0000","last_became_peered":"2024-03-30T00:55:51.312225+0000","last_unstale":"2024-03-30T00:56:35.488541+0000","last_undegraded":"2024-03-30T00:56:35.488541+0000","last_fullsized":"2024-03-30T00:56:35.488541+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:27:12.615250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468012+0000","last_change":"2024-03-30T00:55:51.821527+0000","last_active":"2024-03-30T00:56:35.468012+0000","last_peered":"2024-03-30T00:56:35.468012+0000","last_clean":"2024-03-30T00:56:35.468012+0000","last_became_active":"2024-03-30T00:55:51.820689+0000","last_became_peered":"2024-03-30T00:55:51.820689+0000","last_unstale":"2024-03-30T00:56:35.468012+0000","last_undegraded":"2024-03-30T00:56:35.468012+0000","last_fullsized":"2024-03-30T00:56:35.468012+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:14:35.735985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470468+0000","last_change":"2024-03-30T00:55:55.824307+0000","last_active":"2024-03-30T00:56:35.470468+0000","last_peered":"2024-03-30T00:56:35.470468+0000","last_clean":"2024-03-30T00:56:35.470468+0000","last_became_active":"2024-03-30T00:55:55.823900+0000","last_became_peered":"2024-03-30T00:55:55.823900+0000","last_unstale":"2024-03-30T00:56:35.470468+0000","last_undegraded":"2024-03-30T00:56:35.470468+0000","last_fullsized":"2024-03-30T00:56:35.470468+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:44:22.139120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'4","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463230+0000","last_change":"2024-03-30T00:55:51.296965+0000","last_active":"2024-03-30T00:56:35.463230+0000","last_peered":"2024-03-30T00:56:35.463230+0000","last_clean":"2024-03-30T00:56:35.463230+0000","last_became_active":"2024-03-30T00:55:51.296645+0000","last_became_peered":"2024-03-30T00:55:51.296645+0000","last_unstale":"2024-03-30T00:56:35.463230+0000","last_undegraded":"2024-03-30T00:56:35.463230+0000","last_fullsized":"2024-03-30T00:56:35.463230+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:53:46.519972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.496024+0000","last_change":"2024-03-30T00:55:51.826282+0000","last_active":"2024-03-30T00:56:35.496024+0000","last_peered":"2024-03-30T00:56:35.496024+0000","last_clean":"2024-03-30T00:56:35.496024+0000","last_became_active":"2024-03-30T00:55:51.823910+0000","last_became_peered":"2024-03-30T00:55:51.823910+0000","last_unstale":"2024-03-30T00:56:35.496024+0000","last_undegraded":"2024-03-30T00:56:35.496024+0000","last_fullsized":"2024-03-30T00:56:35.496024+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:47.454280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462851+0000","last_change":"2024-03-30T00:55:55.856750+0000","last_active":"2024-03-30T00:56:35.462851+0000","last_peered":"2024-03-30T00:56:35.462851+0000","last_clean":"2024-03-30T00:56:35.462851+0000","last_became_active":"2024-03-30T00:55:55.856288+0000","last_became_peered":"2024-03-30T00:55:55.856288+0000","last_unstale":"2024-03-30T00:56:35.462851+0000","last_undegraded":"2024-03-30T00:56:35.462851+0000","last_fullsized":"2024-03-30T00:56:35.462851+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:43:28.085910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467887+0000","last_change":"2024-03-30T00:55:51.306357+0000","last_active":"2024-03-30T00:56:35.467887+0000","last_peered":"2024-03-30T00:56:35.467887+0000","last_clean":"2024-03-30T00:56:35.467887+0000","last_became_active":"2024-03-30T00:55:51.305815+0000","last_became_peered":"2024-03-30T00:55:51.305815+0000","last_unstale":"2024-03-30T00:56:35.467887+0000","last_undegraded":"2024-03-30T00:56:35.467887+0000","last_fullsized":"2024-03-30T00:56:35.467887+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:16:21.753446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462871+0000","last_change":"2024-03-30T00:55:51.810686+0000","last_active":"2024-03-30T00:56:35.462871+0000","last_peered":"2024-03-30T00:56:35.462871+0000","last_clean":"2024-03-30T00:56:35.462871+0000","last_became_active":"2024-03-30T00:55:51.810331+0000","last_became_peered":"2024-03-30T00:55:51.810331+0000","last_unstale":"2024-03-30T00:56:35.462871+0000","last_undegraded":"2024-03-30T00:56:35.462871+0000","last_fullsized":"2024-03-30T00:56:35.462871+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:24:13.750905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.487787+0000","last_change":"2024-03-30T00:55:55.842894+0000","last_active":"2024-03-30T00:56:35.487787+0000","last_peered":"2024-03-30T00:56:35.487787+0000","last_clean":"2024-03-30T00:56:35.487787+0000","last_became_active":"2024-03-30T00:55:55.842118+0000","last_became_peered":"2024-03-30T00:55:55.842118+0000","last_unstale":"2024-03-30T00:56:35.487787+0000","last_undegraded":"2024-03-30T00:56:35.487787+0000","last_fullsized":"2024-03-30T00:56:35.487787+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:39.246228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'3","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470956+0000","last_change":"2024-03-30T00:55:51.313188+0000","last_active":"2024-03-30T00:56:35.470956+0000","last_peered":"2024-03-30T00:56:35.470956+0000","last_clean":"2024-03-30T00:56:35.470956+0000","last_became_active":"2024-03-30T00:55:51.312879+0000","last_became_peered":"2024-03-30T00:55:51.312879+0000","last_unstale":"2024-03-30T00:56:35.470956+0000","last_undegraded":"2024-03-30T00:56:35.470956+0000","last_fullsized":"2024-03-30T00:56:35.470956+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:41:23.420427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460160+0000","last_change":"2024-03-30T00:55:51.808998+0000","last_active":"2024-03-30T00:56:35.460160+0000","last_peered":"2024-03-30T00:56:35.460160+0000","last_clean":"2024-03-30T00:56:35.460160+0000","last_became_active":"2024-03-30T00:55:51.808636+0000","last_became_peered":"2024-03-30T00:55:51.808636+0000","last_unstale":"2024-03-30T00:56:35.460160+0000","last_undegraded":"2024-03-30T00:56:35.460160+0000","last_fullsized":"2024-03-30T00:56:35.460160+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:02.379294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.480854+0000","last_change":"2024-03-30T00:56:36.480854+0000","last_active":"2024-03-30T00:56:36.480854+0000","last_peered":"2024-03-30T00:56:36.480854+0000","last_clean":"2024-03-30T00:56:36.480854+0000","last_became_active":"2024-03-30T00:56:36.467064+0000","last_became_peered":"2024-03-30T00:56:36.467064+0000","last_unstale":"2024-03-30T00:56:36.480854+0000","last_undegraded":"2024-03-30T00:56:36.480854+0000","last_fullsized":"2024-03-30T00:56:36.480854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:12:55.154544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470669+0000","last_change":"2024-03-30T00:55:51.313708+0000","last_active":"2024-03-30T00:56:35.470669+0000","last_peered":"2024-03-30T00:56:35.470669+0000","last_clean":"2024-03-30T00:56:35.470669+0000","last_became_active":"2024-03-30T00:55:51.313362+0000","last_became_peered":"2024-03-30T00:55:51.313362+0000","last_unstale":"2024-03-30T00:56:35.470669+0000","last_undegraded":"2024-03-30T00:56:35.470669+0000","last_fullsized":"2024-03-30T00:56:35.470669+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:03:17.859640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.471657+0000","last_change":"2024-03-30T00:55:51.819638+0000","last_active":"2024-03-30T00:56:35.471657+0000","last_peered":"2024-03-30T00:56:35.471657+0000","last_clean":"2024-03-30T00:56:35.471657+0000","last_became_active":"2024-03-30T00:55:51.819282+0000","last_became_peered":"2024-03-30T00:55:51.819282+0000","last_unstale":"2024-03-30T00:56:35.471657+0000","last_undegraded":"2024-03-30T00:56:35.471657+0000","last_fullsized":"2024-03-30T00:56:35.471657+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:13:39.249550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462909+0000","last_change":"2024-03-30T00:55:55.856498+0000","last_active":"2024-03-30T00:56:35.462909+0000","last_peered":"2024-03-30T00:56:35.462909+0000","last_clean":"2024-03-30T00:56:35.462909+0000","last_became_active":"2024-03-30T00:55:55.856015+0000","last_became_peered":"2024-03-30T00:55:55.856015+0000","last_unstale":"2024-03-30T00:56:35.462909+0000","last_undegraded":"2024-03-30T00:56:35.462909+0000","last_fullsized":"2024-03-30T00:56:35.462909+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:14:00.248812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462909+0000","last_change":"2024-03-30T00:55:51.302995+0000","last_active":"2024-03-30T00:56:35.462909+0000","last_peered":"2024-03-30T00:56:35.462909+0000","last_clean":"2024-03-30T00:56:35.462909+0000","last_became_active":"2024-03-30T00:55:51.302650+0000","last_became_peered":"2024-03-30T00:55:51.302650+0000","last_unstale":"2024-03-30T00:56:35.462909+0000","last_undegraded":"2024-03-30T00:56:35.462909+0000","last_fullsized":"2024-03-30T00:56:35.462909+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:17:32.556550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466402+0000","last_change":"2024-03-30T00:55:51.808969+0000","last_active":"2024-03-30T00:56:35.466402+0000","last_peered":"2024-03-30T00:56:35.466402+0000","last_clean":"2024-03-30T00:56:35.466402+0000","last_became_active":"2024-03-30T00:55:51.808633+0000","last_became_peered":"2024-03-30T00:55:51.808633+0000","last_unstale":"2024-03-30T00:56:35.466402+0000","last_undegraded":"2024-03-30T00:56:35.466402+0000","last_fullsized":"2024-03-30T00:56:35.466402+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:16:46.128066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468267+0000","last_change":"2024-03-30T00:55:55.824964+0000","last_active":"2024-03-30T00:56:35.468267+0000","last_peered":"2024-03-30T00:56:35.468267+0000","last_clean":"2024-03-30T00:56:35.468267+0000","last_became_active":"2024-03-30T00:55:55.824559+0000","last_became_peered":"2024-03-30T00:55:55.824559+0000","last_unstale":"2024-03-30T00:56:35.468267+0000","last_undegraded":"2024-03-30T00:56:35.468267+0000","last_fullsized":"2024-03-30T00:56:35.468267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:19:24.746963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'3","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470912+0000","last_change":"2024-03-30T00:55:51.301602+0000","last_active":"2024-03-30T00:56:35.470912+0000","last_peered":"2024-03-30T00:56:35.470912+0000","last_clean":"2024-03-30T00:56:35.470912+0000","last_became_active":"2024-03-30T00:55:51.301221+0000","last_became_peered":"2024-03-30T00:55:51.301221+0000","last_unstale":"2024-03-30T00:56:35.470912+0000","last_undegraded":"2024-03-30T00:56:35.470912+0000","last_fullsized":"2024-03-30T00:56:35.470912+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:37:36.727603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466837+0000","last_change":"2024-03-30T00:55:51.810197+0000","last_active":"2024-03-30T00:56:35.466837+0000","last_peered":"2024-03-30T00:56:35.466837+0000","last_clean":"2024-03-30T00:56:35.466837+0000","last_became_active":"2024-03-30T00:55:51.809193+0000","last_became_peered":"2024-03-30T00:55:51.809193+0000","last_unstale":"2024-03-30T00:56:35.466837+0000","last_undegraded":"2024-03-30T00:56:35.466837+0000","last_fullsized":"2024-03-30T00:56:35.466837+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:52:41.078425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.494873+0000","last_change":"2024-03-30T00:55:55.830549+0000","last_active":"2024-03-30T00:56:35.494873+0000","last_peered":"2024-03-30T00:56:35.494873+0000","last_clean":"2024-03-30T00:56:35.494873+0000","last_became_active":"2024-03-30T00:55:55.827966+0000","last_became_peered":"2024-03-30T00:55:55.827966+0000","last_unstale":"2024-03-30T00:56:35.494873+0000","last_undegraded":"2024-03-30T00:56:35.494873+0000","last_fullsized":"2024-03-30T00:56:35.494873+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:30:36.531960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"76'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467092+0000","last_change":"2024-03-30T00:55:51.311998+0000","last_active":"2024-03-30T00:56:35.467092+0000","last_peered":"2024-03-30T00:56:35.467092+0000","last_clean":"2024-03-30T00:56:35.467092+0000","last_became_active":"2024-03-30T00:55:51.311627+0000","last_became_peered":"2024-03-30T00:55:51.311627+0000","last_unstale":"2024-03-30T00:56:35.467092+0000","last_undegraded":"2024-03-30T00:56:35.467092+0000","last_fullsized":"2024-03-30T00:56:35.467092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:23:06.939919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466545+0000","last_change":"2024-03-30T00:55:51.819457+0000","last_active":"2024-03-30T00:56:35.466545+0000","last_peered":"2024-03-30T00:56:35.466545+0000","last_clean":"2024-03-30T00:56:35.466545+0000","last_became_active":"2024-03-30T00:55:51.819099+0000","last_became_peered":"2024-03-30T00:55:51.819099+0000","last_unstale":"2024-03-30T00:56:35.466545+0000","last_undegraded":"2024-03-30T00:56:35.466545+0000","last_fullsized":"2024-03-30T00:56:35.466545+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:31:57.363328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466097+0000","last_change":"2024-03-30T00:55:55.824882+0000","last_active":"2024-03-30T00:56:35.466097+0000","last_peered":"2024-03-30T00:56:35.466097+0000","last_clean":"2024-03-30T00:56:35.466097+0000","last_became_active":"2024-03-30T00:55:55.824493+0000","last_became_peered":"2024-03-30T00:55:55.824493+0000","last_unstale":"2024-03-30T00:56:35.466097+0000","last_undegraded":"2024-03-30T00:56:35.466097+0000","last_fullsized":"2024-03-30T00:56:35.466097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:24:48.825776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"76'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.476919+0000","last_change":"2024-03-30T00:55:51.311787+0000","last_active":"2024-03-30T00:56:35.476919+0000","last_peered":"2024-03-30T00:56:35.476919+0000","last_clean":"2024-03-30T00:56:35.476919+0000","last_became_active":"2024-03-30T00:55:51.311402+0000","last_became_peered":"2024-03-30T00:55:51.311402+0000","last_unstale":"2024-03-30T00:56:35.476919+0000","last_undegraded":"2024-03-30T00:56:35.476919+0000","last_fullsized":"2024-03-30T00:56:35.476919+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:37:13.960453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467729+0000","last_change":"2024-03-30T00:55:51.809742+0000","last_active":"2024-03-30T00:56:35.467729+0000","last_peered":"2024-03-30T00:56:35.467729+0000","last_clean":"2024-03-30T00:56:35.467729+0000","last_became_active":"2024-03-30T00:55:51.809396+0000","last_became_peered":"2024-03-30T00:55:51.809396+0000","last_unstale":"2024-03-30T00:56:35.467729+0000","last_undegraded":"2024-03-30T00:56:35.467729+0000","last_fullsized":"2024-03-30T00:56:35.467729+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:22.535472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460238+0000","last_change":"2024-03-30T00:55:51.814429+0000","last_active":"2024-03-30T00:56:35.460238+0000","last_peered":"2024-03-30T00:56:35.460238+0000","last_clean":"2024-03-30T00:56:35.460238+0000","last_became_active":"2024-03-30T00:55:51.814044+0000","last_became_peered":"2024-03-30T00:55:51.814044+0000","last_unstale":"2024-03-30T00:56:35.460238+0000","last_undegraded":"2024-03-30T00:56:35.460238+0000","last_fullsized":"2024-03-30T00:56:35.460238+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:24:58.830738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470585+0000","last_change":"2024-03-30T00:55:51.828977+0000","last_active":"2024-03-30T00:56:35.470585+0000","last_peered":"2024-03-30T00:56:35.470585+0000","last_clean":"2024-03-30T00:56:35.470585+0000","last_became_active":"2024-03-30T00:55:51.828656+0000","last_became_peered":"2024-03-30T00:55:51.828656+0000","last_unstale":"2024-03-30T00:56:35.470585+0000","last_undegraded":"2024-03-30T00:56:35.470585+0000","last_fullsized":"2024-03-30T00:56:35.470585+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:10:23.463770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470569+0000","last_change":"2024-03-30T00:55:51.821511+0000","last_active":"2024-03-30T00:56:35.470569+0000","last_peered":"2024-03-30T00:56:35.470569+0000","last_clean":"2024-03-30T00:56:35.470569+0000","last_became_active":"2024-03-30T00:55:51.820876+0000","last_became_peered":"2024-03-30T00:55:51.820876+0000","last_unstale":"2024-03-30T00:56:35.470569+0000","last_undegraded":"2024-03-30T00:56:35.470569+0000","last_fullsized":"2024-03-30T00:56:35.470569+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:14:48.299376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495025+0000","last_change":"2024-03-30T00:55:51.823867+0000","last_active":"2024-03-30T00:56:35.495025+0000","last_peered":"2024-03-30T00:56:35.495025+0000","last_clean":"2024-03-30T00:56:35.495025+0000","last_became_active":"2024-03-30T00:55:51.820230+0000","last_became_peered":"2024-03-30T00:55:51.820230+0000","last_unstale":"2024-03-30T00:56:35.495025+0000","last_undegraded":"2024-03-30T00:56:35.495025+0000","last_fullsized":"2024-03-30T00:56:35.495025+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:38:51.257855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.475716+0000","last_change":"2024-03-30T00:55:51.821413+0000","last_active":"2024-03-30T00:56:35.475716+0000","last_peered":"2024-03-30T00:56:35.475716+0000","last_clean":"2024-03-30T00:56:35.475716+0000","last_became_active":"2024-03-30T00:55:51.820833+0000","last_became_peered":"2024-03-30T00:55:51.820833+0000","last_unstale":"2024-03-30T00:56:35.475716+0000","last_undegraded":"2024-03-30T00:56:35.475716+0000","last_fullsized":"2024-03-30T00:56:35.475716+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:34:11.969849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460621+0000","last_change":"2024-03-30T00:55:51.813674+0000","last_active":"2024-03-30T00:56:35.460621+0000","last_peered":"2024-03-30T00:56:35.460621+0000","last_clean":"2024-03-30T00:56:35.460621+0000","last_became_active":"2024-03-30T00:55:51.813341+0000","last_became_peered":"2024-03-30T00:55:51.813341+0000","last_unstale":"2024-03-30T00:56:35.460621+0000","last_undegraded":"2024-03-30T00:56:35.460621+0000","last_fullsized":"2024-03-30T00:56:35.460621+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:54:17.285997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.471317+0000","last_change":"2024-03-30T00:55:51.812472+0000","last_active":"2024-03-30T00:56:35.471317+0000","last_peered":"2024-03-30T00:56:35.471317+0000","last_clean":"2024-03-30T00:56:35.471317+0000","last_became_active":"2024-03-30T00:55:51.812061+0000","last_became_peered":"2024-03-30T00:55:51.812061+0000","last_unstale":"2024-03-30T00:56:35.471317+0000","last_undegraded":"2024-03-30T00:56:35.471317+0000","last_fullsized":"2024-03-30T00:56:35.471317+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:46.550201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468639+0000","last_change":"2024-03-30T00:55:51.829179+0000","last_active":"2024-03-30T00:56:35.468639+0000","last_peered":"2024-03-30T00:56:35.468639+0000","last_clean":"2024-03-30T00:56:35.468639+0000","last_became_active":"2024-03-30T00:55:51.828805+0000","last_became_peered":"2024-03-30T00:55:51.828805+0000","last_unstale":"2024-03-30T00:56:35.468639+0000","last_undegraded":"2024-03-30T00:56:35.468639+0000","last_fullsized":"2024-03-30T00:56:35.468639+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:23.576125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466369+0000","last_change":"2024-03-30T00:55:51.823170+0000","last_active":"2024-03-30T00:56:35.466369+0000","last_peered":"2024-03-30T00:56:35.466369+0000","last_clean":"2024-03-30T00:56:35.466369+0000","last_became_active":"2024-03-30T00:55:51.822785+0000","last_became_peered":"2024-03-30T00:55:51.822785+0000","last_unstale":"2024-03-30T00:56:35.466369+0000","last_undegraded":"2024-03-30T00:56:35.466369+0000","last_fullsized":"2024-03-30T00:56:35.466369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:34:43.393540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.469874+0000","last_change":"2024-03-30T00:55:51.817160+0000","last_active":"2024-03-30T00:56:35.469874+0000","last_peered":"2024-03-30T00:56:35.469874+0000","last_clean":"2024-03-30T00:56:35.469874+0000","last_became_active":"2024-03-30T00:55:51.816881+0000","last_became_peered":"2024-03-30T00:55:51.816881+0000","last_unstale":"2024-03-30T00:56:35.469874+0000","last_undegraded":"2024-03-30T00:56:35.469874+0000","last_fullsized":"2024-03-30T00:56:35.469874+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:09:05.211242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468766+0000","last_change":"2024-03-30T00:55:51.817670+0000","last_active":"2024-03-30T00:56:35.468766+0000","last_peered":"2024-03-30T00:56:35.468766+0000","last_clean":"2024-03-30T00:56:35.468766+0000","last_became_active":"2024-03-30T00:55:51.817380+0000","last_became_peered":"2024-03-30T00:55:51.817380+0000","last_unstale":"2024-03-30T00:56:35.468766+0000","last_undegraded":"2024-03-30T00:56:35.468766+0000","last_fullsized":"2024-03-30T00:56:35.468766+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:24.582557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.475181+0000","last_change":"2024-03-30T00:55:51.813071+0000","last_active":"2024-03-30T00:56:35.475181+0000","last_peered":"2024-03-30T00:56:35.475181+0000","last_clean":"2024-03-30T00:56:35.475181+0000","last_became_active":"2024-03-30T00:55:51.811681+0000","last_became_peered":"2024-03-30T00:55:51.811681+0000","last_unstale":"2024-03-30T00:56:35.475181+0000","last_undegraded":"2024-03-30T00:56:35.475181+0000","last_fullsized":"2024-03-30T00:56:35.475181+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:10:53.840055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.472592+0000","last_change":"2024-03-30T00:56:36.472592+0000","last_active":"2024-03-30T00:56:36.472592+0000","last_peered":"2024-03-30T00:56:36.472592+0000","last_clean":"2024-03-30T00:56:36.472592+0000","last_became_active":"2024-03-30T00:56:36.472173+0000","last_became_peered":"2024-03-30T00:56:36.472173+0000","last_unstale":"2024-03-30T00:56:36.472592+0000","last_undegraded":"2024-03-30T00:56:36.472592+0000","last_fullsized":"2024-03-30T00:56:36.472592+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:22:27.915984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":212960,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65332,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":231,"num_read_kb":183,"num_write":156,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":195996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138,"ondisk_log_size":138,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645336,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":913408,"data_stored":462418,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.376}]},{"osd":1,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.40000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":4,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.505},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.57699999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.62},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54800000000000004}]},{"osd":6,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.71799999999999997}]},{"osd":7,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.68999999999999995}]},{"osd":8,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.749},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.67700000000000005}]},{"osd":9,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.70099999999999996}]},{"osd":10,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.76900000000000002}]}]},{"osd":10,"up_from":65,"seq":279172874266,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1114794,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.49099999999999999}]},{"osd":2,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.628}]},{"osd":3,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.66900000000000004}]},{"osd":4,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.58599999999999997}]},{"osd":5,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.76100000000000001}]},{"osd":6,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.78100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":8,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.69099999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.748},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.72199999999999998}]},{"osd":11,"last update":"Sat Mar 30 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.83399999999999996}]}]},{"osd":9,"up_from":60,"seq":257698037789,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":925696,"data_stored":462620,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.628,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.44600000000000001}]},{"osd":1,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.94199999999999995,"5min":1.274,"15min":1.274},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.73399999999999999,"15min":0.74299999999999999},"min":{"1min":0.51600000000000001,"5min":0.504,"15min":0.504},"max":{"1min":0.90600000000000003,"5min":1.411,"15min":1.411},"last":0.622}]},{"osd":2,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.73399999999999999,"15min":0.74299999999999999},"min":{"1min":0.55400000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.84699999999999998,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.501,"5min":0.435,"15min":0.435},"max":{"1min":0.83999999999999997,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.68799999999999994}]},{"osd":3,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.105,"5min":1.391,"15min":1.391},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.73999999999999999,"15min":0.75900000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95899999999999996,"5min":1.302,"15min":1.302},"last":0.497}]},{"osd":4,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69199999999999995,"15min":0.69799999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.88700000000000001,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.81100000000000005,"15min":0.82299999999999995},"min":{"1min":0.56200000000000006,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98299999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.60999999999999999}]},{"osd":5,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.71899999999999997,"15min":0.72899999999999998},"min":{"1min":0.45800000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.88300000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.83799999999999997,"15min":0.85799999999999998},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0129999999999999,"5min":1.853,"15min":1.853},"last":0.71799999999999997}]},{"osd":6,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.623}]},{"osd":7,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.86399999999999999,"15min":0.88},"min":{"1min":0.54100000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.94099999999999995,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.83299999999999996,"15min":0.84399999999999997},"min":{"1min":0.58699999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94599999999999995,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.59899999999999998}]},{"osd":8,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.90100000000000002,"15min":0.92000000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.98099999999999998,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.90000000000000002,"15min":0.92200000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0720000000000001,"5min":1.968,"15min":1.968},"last":0.69599999999999995}]},{"osd":10,"last update":"Sat Mar 30 00:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.47599999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":8,"up_from":53,"seq":227633266722,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":462209,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.41599999999999998}]},{"osd":1,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57099999999999995,"15min":0.56599999999999995},"min":{"1min":0.38600000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.46400000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.57799999999999996}]},{"osd":2,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.41999999999999998},"min":{"1min":0.29199999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.45100000000000001,"15min":0.44500000000000001},"min":{"1min":0.316,"5min":0.312,"15min":0.312},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.63300000000000001}]},{"osd":3,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.70899999999999996,"15min":0.73099999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74199999999999999,"5min":2.883,"15min":2.883},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58099999999999996,"15min":0.57699999999999996},"min":{"1min":0.436,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.502}]},{"osd":4,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.60699999999999998,"15min":0.59899999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.61599999999999999}]},{"osd":5,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.496,"15min":0.48899999999999999},"min":{"1min":0.35199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.34599999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.64700000000000002}]},{"osd":6,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.53200000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.876,"5min":2.8940000000000001,"15min":2.8940000000000001},"last":0.751}]},{"osd":7,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.73699999999999999}]},{"osd":9,"last update":"Sat Mar 30 00:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.70999999999999996,"15min":0.69899999999999995},"min":{"1min":0.58999999999999997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93500000000000005,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.74299999999999999,"15min":0.72599999999999998},"min":{"1min":0.68600000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.68000000000000005}]},{"osd":10,"last update":"Sat Mar 30 00:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.75900000000000001}]},{"osd":11,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.71799999999999997}]}]},{"osd":7,"up_from":46,"seq":197568495653,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":1058417,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60199999999999998}]},{"osd":1,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.45200000000000001,"15min":0.44800000000000001},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.60599999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.44700000000000001,"15min":0.441},"min":{"1min":0.32500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68799999999999994,"5min":0.747,"15min":0.747},"last":0.45400000000000001}]},{"osd":2,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.46700000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.57399999999999995,"15min":0.56100000000000005},"min":{"1min":0.51800000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.40400000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.89600000000000002,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61599999999999999,"15min":0.60999999999999999},"min":{"1min":0.53200000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89400000000000002,"5min":0.997,"15min":0.997},"last":0.51100000000000001}]},{"osd":4,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.36599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.39200000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.41399999999999998}]},{"osd":5,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.629,"15min":0.61899999999999999},"min":{"1min":0.505,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.63200000000000001,"15min":0.62},"min":{"1min":0.58099999999999996,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86199999999999999,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.65200000000000002}]},{"osd":6,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.58799999999999997,"5min":0.372,"15min":0.372},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.745,"15min":0.72099999999999997},"min":{"1min":0.55300000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":4.117,"5min":4.117,"15min":4.117},"last":0.59299999999999997}]},{"osd":8,"last update":"Sat Mar 30 00:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.56299999999999994,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95199999999999996,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.755,"15min":0.752},"min":{"1min":0.53400000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.71399999999999997}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.74399999999999999,"15min":0.73999999999999999},"min":{"1min":0.60299999999999998,"5min":0.495,"15min":0.495},"max":{"1min":0.94199999999999995,"5min":1.018,"15min":1.018},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.91400000000000003,"15min":0.91900000000000004},"min":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":3.8839999999999999,"5min":4.0519999999999996,"15min":4.0519999999999996},"last":0.625}]},{"osd":10,"last update":"Sat Mar 30 00:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.61599999999999999}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.75900000000000001}]}]},{"osd":6,"up_from":42,"seq":180388626474,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":905216,"data_stored":457796,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.308,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.499,"15min":0.48699999999999999},"min":{"1min":0.373,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.47399999999999998}]},{"osd":1,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.66300000000000003,"15min":0.63500000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60599999999999998,"15min":0.59599999999999997},"min":{"1min":0.39300000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.50600000000000001}]},{"osd":2,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70899999999999996,"15min":0.68300000000000005},"min":{"1min":0.51000000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.98699999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.66500000000000004,"15min":0.64300000000000002},"min":{"1min":0.47299999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.64800000000000002}]},{"osd":3,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.65000000000000002},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.97999999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.46400000000000002}]},{"osd":4,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67600000000000005,"15min":0.67200000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.76300000000000001,"15min":0.751},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.105,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.753}]},{"osd":5,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69399999999999995,"15min":0.68200000000000005},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.93899999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.74099999999999999,"15min":0.73099999999999998},"min":{"1min":0.51600000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Mar 30 00:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81799999999999995,"15min":0.81399999999999995},"min":{"1min":0.64300000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.1200000000000001,"5min":1.127,"15min":1.127},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.77600000000000002,"15min":0.77200000000000002},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.73599999999999999}]},{"osd":8,"last update":"Sat Mar 30 00:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.84499999999999997,"15min":0.85199999999999998},"min":{"1min":0.52200000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.2789999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.876,"5min":0.89000000000000001,"15min":0.89200000000000002},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.155,"5min":1.206,"15min":1.206},"last":0.85099999999999998}]},{"osd":9,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.94299999999999995,"15min":0.95199999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.244,"5min":1.268,"15min":1.268},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.347,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.875}]},{"osd":10,"last update":"Sat Mar 30 00:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.89300000000000002}]},{"osd":11,"last update":"Sat Mar 30 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.80200000000000005}]}]},{"osd":1,"up_from":12,"seq":51539607612,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":518465,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55500000000000005,"15min":0.53500000000000003},"min":{"1min":0.42399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72099999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.58099999999999996,"15min":0.55300000000000005},"min":{"1min":0.47199999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.79100000000000004,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.68500000000000005}]},{"osd":2,"last update":"Sat Mar 30 00:56:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.52800000000000002,"15min":0.49299999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.53400000000000003,"15min":0.5},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.877}]},{"osd":3,"last update":"Sat Mar 30 00:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56200000000000006,"15min":0.52800000000000002},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.69199999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57399999999999995,"15min":0.55000000000000004},"min":{"1min":0.501,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.66900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.64500000000000002}]},{"osd":4,"last update":"Sat Mar 30 00:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.41799999999999998,"15min":0.40799999999999997},"min":{"1min":0.33500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.61899999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.48199999999999998,"15min":0.46600000000000003},"min":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.66900000000000004}]},{"osd":5,"last update":"Sat Mar 30 00:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72199999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64100000000000001,"15min":0.63600000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83899999999999997,"5min":0.877,"15min":0.877},"last":0.89800000000000002}]},{"osd":6,"last update":"Sat Mar 30 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.55700000000000005,"5min":0.435,"15min":0.435},"max":{"1min":0.81699999999999995,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.54800000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.84799999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.752}]},{"osd":7,"last update":"Sat Mar 30 00:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55400000000000005,"15min":0.55100000000000005},"min":{"1min":0.35799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79400000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.84899999999999998,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.81799999999999995}]},{"osd":8,"last update":"Sat Mar 30 00:56:28 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.79100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.624,"15min":0.625},"min":{"1min":0.41999999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81599999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.73899999999999999}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.746,"15min":0.73699999999999999},"min":{"1min":0.56899999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.91600000000000004,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72999999999999998,"15min":0.72399999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.86099999999999999}]},{"osd":10,"last update":"Sat Mar 30 00:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.94299999999999995}]},{"osd":11,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.96399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32020,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713132,"statfs":{"total":95995035648,"available":95962247168,"internally_reserved":0,"allocated":925696,"data_stored":458911,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 00:56:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.40000000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.88600000000000001,"5min":1.038,"15min":1.038},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.57099999999999995,"15min":0.56000000000000005},"min":{"1min":0.49399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.60299999999999998}]},{"osd":2,"last update":"Sat Mar 30 00:56:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.56299999999999994,"15min":0.53200000000000003},"min":{"1min":0.41899999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.495},{"interface":"front","average":{"1min":1.0940000000000001,"5min":0.68300000000000005,"15min":0.59299999999999997},"min":{"1min":0.48399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":9.0150000000000006,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":0.83799999999999997}]},{"osd":3,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.9139999999999997,"5min":7.9139999999999997,"15min":7.9139999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.96,"5min":7.96,"15min":7.96},"last":0.58099999999999996}]},{"osd":4,"last update":"Sat Mar 30 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65100000000000002,"15min":0.623},"min":{"1min":0.503,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90000000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.64100000000000001,"15min":0.60999999999999999},"min":{"1min":0.51400000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.93300000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.626}]},{"osd":5,"last update":"Sat Mar 30 00:56:35 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":0.74399999999999999,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":9.0579999999999998,"5min":9.0579999999999998,"15min":9.0579999999999998},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.871,"5min":0.68400000000000005,"15min":0.63600000000000001},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":3.8039999999999998,"5min":3.8039999999999998,"15min":3.8039999999999998},"last":0.73999999999999999}]},{"osd":6,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":0.72099999999999997,"15min":0.65900000000000003},"min":{"1min":0.42399999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":8.0050000000000008,"5min":8.0050000000000008,"15min":8.0050000000000008},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.66800000000000004,"15min":0.65800000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.41699999999999998}]},{"osd":7,"last update":"Sat Mar 30 00:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.80900000000000005}]},{"osd":8,"last update":"Sat Mar 30 00:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.55000000000000004,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0489999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.83799999999999997,"15min":0.85199999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0589999999999999,"5min":1.276,"15min":1.276},"last":0.877}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":0.88700000000000001,"15min":0.83699999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":8.0389999999999997,"5min":8.0389999999999997,"15min":8.0389999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.78400000000000003,"15min":0.76600000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":0.59399999999999997}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.56200000000000006}]},{"osd":11,"last update":"Sat Mar 30 00:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.749},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.94099999999999995}]}]},{"osd":2,"up_from":17,"seq":73014444089,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":519730,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53700000000000003,"15min":0.52800000000000002},"min":{"1min":0.36099999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.84899999999999998,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.503},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.52500000000000002},"min":{"1min":0.38300000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76200000000000001,"5min":1.097,"15min":1.097},"last":0.51300000000000001}]},{"osd":1,"last update":"Sat Mar 30 00:56:38 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.53700000000000003,"15min":0.51400000000000001},"min":{"1min":0.36899999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81000000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54300000000000004,"15min":0.52900000000000003},"min":{"1min":0.45600000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70199999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.46200000000000002}]},{"osd":3,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57899999999999996,"15min":0.55900000000000005},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.58899999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.53100000000000003}]},{"osd":4,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.443},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.47899999999999998}]},{"osd":5,"last update":"Sat Mar 30 00:56:30 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.442,"15min":0.43099999999999999},"min":{"1min":0.30399999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60199999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.46700000000000003,"15min":0.45600000000000002},"min":{"1min":0.28000000000000003,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.59899999999999998}]},{"osd":6,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65100000000000002,"15min":0.65000000000000002},"min":{"1min":0.44600000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86399999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66500000000000004,"15min":0.66100000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.73299999999999998}]},{"osd":7,"last update":"Sat Mar 30 00:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.69699999999999995}]},{"osd":8,"last update":"Sat Mar 30 00:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82799999999999996,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59699999999999998,"15min":0.59599999999999997},"min":{"1min":0.35799999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81100000000000005,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.628}]},{"osd":9,"last update":"Sat Mar 30 00:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70399999999999996,"15min":0.70099999999999996},"min":{"1min":0.52100000000000002,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.73499999999999999,"15min":0.73099999999999998},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.80200000000000005}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.753},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.68700000000000006}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.77000000000000002}]}]},{"osd":3,"up_from":24,"seq":103079215157,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":457717,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.46000000000000002,"15min":0.41399999999999998},"min":{"1min":0.32900000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46400000000000002,"15min":0.432},"min":{"1min":0.35199999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.997,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.376}]},{"osd":1,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65100000000000002,"15min":0.60299999999999998},"min":{"1min":0.45000000000000001,"5min":0.375,"15min":0.375},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.628,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.439,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.59099999999999997}]},{"osd":2,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.67200000000000004,"15min":0.629},"min":{"1min":0.51400000000000001,"5min":0.371,"15min":0.371},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64800000000000002,"15min":0.622},"min":{"1min":0.47599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.69899999999999995}]},{"osd":4,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.69499999999999995,"15min":0.66300000000000003},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0600000000000001,"5min":1.113,"15min":1.113},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67500000000000004,"15min":0.66400000000000003},"min":{"1min":0.501,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.83399999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.63100000000000001}]},{"osd":5,"last update":"Sat Mar 30 00:56:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65500000000000003,"15min":0.64600000000000002},"min":{"1min":0.46300000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93000000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.69799999999999995,"15min":0.66400000000000003},"min":{"1min":0.48799999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.117,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.67700000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.65000000000000002,"15min":0.63800000000000001},"min":{"1min":0.46800000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.44,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.81200000000000006}]},{"osd":7,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.97899999999999998,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.751},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83999999999999997,"15min":0.84099999999999997},"min":{"1min":0.66000000000000003,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.1890000000000001,"5min":1.292,"15min":1.292},"last":0.75900000000000001}]},{"osd":8,"last update":"Sat Mar 30 00:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.57399999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.92400000000000004,"15min":0.94399999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.2410000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.84499999999999997,"15min":0.85899999999999999},"min":{"1min":0.54700000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.224,"5min":1.347,"15min":1.347},"last":0.66900000000000004}]},{"osd":10,"last update":"Sat Mar 30 00:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.82599999999999996}]},{"osd":11,"last update":"Sat Mar 30 00:56:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.74299999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986226,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":909312,"data_stored":462187,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.66700000000000004,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.79300000000000004,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73099999999999998,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39700000000000002,"15min":0.39100000000000001},"min":{"1min":0.29999999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.61199999999999999,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.32000000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.46899999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.47299999999999998}]},{"osd":2,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.55400000000000005,"15min":0.54500000000000004},"min":{"1min":0.36399999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81799999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.55400000000000005,"15min":0.53900000000000003},"min":{"1min":0.441,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76000000000000001,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.71699999999999997}]},{"osd":3,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.66000000000000003,"15min":0.69799999999999995},"min":{"1min":0.42399999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.72999999999999998,"5min":3.887,"15min":3.887},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.39600000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78400000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.53300000000000003}]},{"osd":5,"last update":"Sat Mar 30 00:56:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57499999999999996,"15min":0.56399999999999995},"min":{"1min":0.39800000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.78100000000000003,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58399999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.76300000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.752}]},{"osd":6,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.623,"15min":0.625},"min":{"1min":0.42699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.64800000000000002,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82599999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58199999999999996}]},{"osd":7,"last update":"Sat Mar 30 00:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54400000000000004,"15min":0.54000000000000004},"min":{"1min":0.35199999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.79500000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.499},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52700000000000002,"15min":0.52100000000000002},"min":{"1min":0.34599999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.69499999999999995}]},{"osd":8,"last update":"Sat Mar 30 00:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78700000000000003,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.46000000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.85199999999999998,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.78900000000000003}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.69899999999999995},"min":{"1min":0.53500000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.73299999999999998,"15min":0.73699999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.98999999999999999,"5min":1.012,"15min":1.012},"last":0.80000000000000004}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.77800000000000002}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.83299999999999996}]}]},{"osd":5,"up_from":35,"seq":150323855405,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":978944,"data_stored":1053660,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51400000000000001,"15min":0.502},"min":{"1min":0.37,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78700000000000003,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.50900000000000001,"15min":0.497},"min":{"1min":0.38500000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.78800000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.44400000000000001}]},{"osd":1,"last update":"Sat Mar 30 00:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56799999999999995,"15min":0.56399999999999995},"min":{"1min":0.378,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55500000000000005,"15min":0.54100000000000004},"min":{"1min":0.45700000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72899999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.55200000000000005}]},{"osd":2,"last update":"Sat Mar 30 00:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.39700000000000002,"15min":0.38900000000000001},"min":{"1min":0.313,"5min":0.255,"15min":0.255},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.39800000000000002,"15min":0.39600000000000002},"min":{"1min":0.307,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.64400000000000002,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.41599999999999998}]},{"osd":3,"last update":"Sat Mar 30 00:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54000000000000004,"15min":0.53700000000000003},"min":{"1min":0.36799999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75800000000000001,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54200000000000004,"15min":0.53000000000000003},"min":{"1min":0.40799999999999997,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.78300000000000003,"5min":0.877,"15min":0.877},"last":0.46200000000000002}]},{"osd":4,"last update":"Sat Mar 30 00:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57299999999999995,"15min":0.56100000000000005},"min":{"1min":0.49299999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.59399999999999997,"15min":0.58199999999999996},"min":{"1min":0.47399999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76000000000000001,"5min":1.038,"15min":1.038},"last":0.50800000000000001}]},{"osd":6,"last update":"Sat Mar 30 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59899999999999998,"15min":0.58099999999999996},"min":{"1min":0.42799999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81200000000000006,"5min":0.997,"15min":0.997},"last":0.623},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60499999999999998,"15min":0.59799999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.72999999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.63400000000000001}]},{"osd":7,"last update":"Sat Mar 30 00:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.42699999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82199999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63600000000000001,"15min":0.63700000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89500000000000002,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.61299999999999999}]},{"osd":8,"last update":"Sat Mar 30 00:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.53700000000000003,"15min":0.53100000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.55300000000000005,"15min":0.54400000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85399999999999998,"5min":0.872,"15min":0.872},"last":0.66700000000000004}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.67300000000000004,"15min":0.66700000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68400000000000005,"15min":0.68100000000000005},"min":{"1min":0.58099999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.66000000000000003}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.71899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":62693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T00:57:01.010 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T00:57:01.371 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:57:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.503+0000 7f67cd800700 1 -- 172.21.15.116:0/1015283807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c8103b90 msgr2=0x7f67c8107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.503+0000 7f67cd800700 1 --2- 172.21.15.116:0/1015283807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c8103b90 0x7f67c8107be0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009990 tx=0x7f67b4009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.503+0000 7f67cd800700 1 -- 172.21.15.116:0/1015283807 shutdown_connections 2024-03-30T00:57:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.503+0000 7f67cd800700 1 --2- 172.21.15.116:0/1015283807 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67c81031e0 0x7f67c81035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.503+0000 7f67cd800700 1 --2- 172.21.15.116:0/1015283807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67c81082c0 0x7f67c810fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.503+0000 7f67cd800700 1 --2- 172.21.15.116:0/1015283807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c8103b90 0x7f67c8107be0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.503+0000 7f67cd800700 1 -- 172.21.15.116:0/1015283807 >> 172.21.15.116:0/1015283807 conn(0x7f67c80fea70 msgr2=0x7f67c8100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.503+0000 7f67cd800700 1 -- 172.21.15.116:0/1015283807 shutdown_connections 2024-03-30T00:57:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.503+0000 7f67cd800700 1 -- 172.21.15.116:0/1015283807 wait complete. 2024-03-30T00:57:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.504+0000 7f67cd800700 1 Processor -- start 2024-03-30T00:57:02.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.504+0000 7f67cd800700 1 -- start start 2024-03-30T00:57:02.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67cd800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c81031e0 0x7f67c8199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:02.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67cd800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67c8103b90 0x7f67c8199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:02.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67cd800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67c81082c0 0x7f67c819dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:02.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67cd800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c819e350 con 0x7f67c81031e0 2024-03-30T00:57:02.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67cd800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c819e4c0 con 0x7f67c8103b90 2024-03-30T00:57:02.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67cd800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c819e680 con 0x7f67c81082c0 2024-03-30T00:57:02.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67c6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c81031e0 0x7f67c8199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:02.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67c6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c81031e0 0x7f67c8199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59296/0 (socket says 172.21.15.116:59296) 2024-03-30T00:57:02.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67c6a00700 1 -- 172.21.15.116:0/3283847013 learned_addr learned my addr 172.21.15.116:0/3283847013 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:02.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67c6000700 1 --2- 172.21.15.116:0/3283847013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67c8103b90 0x7f67c8199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:02.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67c7400700 1 --2- 172.21.15.116:0/3283847013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67c81082c0 0x7f67c819dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:02.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67c6a00700 1 -- 172.21.15.116:0/3283847013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67c81082c0 msgr2=0x7f67c819dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:02.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67c6a00700 1 --2- 172.21.15.116:0/3283847013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67c81082c0 0x7f67c819dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:02.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67c6a00700 1 -- 172.21.15.116:0/3283847013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67c8103b90 msgr2=0x7f67c8199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:02.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67c6a00700 1 --2- 172.21.15.116:0/3283847013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67c8103b90 0x7f67c8199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:02.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.505+0000 7f67c6a00700 1 -- 172.21.15.116:0/3283847013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b40095f0 con 0x7f67c81031e0 2024-03-30T00:57:02.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.506+0000 7f67c6a00700 1 --2- 172.21.15.116:0/3283847013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c81031e0 0x7f67c8199450 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f67b800e9a0 tx=0x7f67b800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:02.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.506+0000 7f67bf400700 1 -- 172.21.15.116:0/3283847013 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b8009790 con 0x7f67c81031e0 2024-03-30T00:57:02.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.506+0000 7f67bf400700 1 -- 172.21.15.116:0/3283847013 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67b8010430 con 0x7f67c81031e0 2024-03-30T00:57:02.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.506+0000 7f67cd800700 1 -- 172.21.15.116:0/3283847013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c819e960 con 0x7f67c81031e0 2024-03-30T00:57:02.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.506+0000 7f67bf400700 1 -- 172.21.15.116:0/3283847013 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b8018680 con 0x7f67c81031e0 2024-03-30T00:57:02.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.506+0000 7f67cd800700 1 -- 172.21.15.116:0/3283847013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c81a61c0 con 0x7f67c81031e0 2024-03-30T00:57:02.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.507+0000 7f67cd800700 1 -- 172.21.15.116:0/3283847013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c810b5e0 con 0x7f67c81031e0 2024-03-30T00:57:02.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.509+0000 7f67bf400700 1 -- 172.21.15.116:0/3283847013 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f67b80105a0 con 0x7f67c81031e0 2024-03-30T00:57:02.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.511+0000 7f67bf400700 1 --2- 172.21.15.116:0/3283847013 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f67ac0a11b0 0x7f67ac0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:02.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.511+0000 7f67bf400700 1 -- 172.21.15.116:0/3283847013 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f67b8014070 con 0x7f67c81031e0 2024-03-30T00:57:02.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.511+0000 7f67c6000700 1 --2- 172.21.15.116:0/3283847013 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f67ac0a11b0 0x7f67ac0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:02.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.512+0000 7f67c6000700 1 --2- 172.21.15.116:0/3283847013 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f67ac0a11b0 0x7f67ac0a3670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f67c819a9b0 tx=0x7f67b4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:02.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.515+0000 7f67bf400700 1 -- 172.21.15.116:0/3283847013 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f67b80529f0 con 0x7f67c81031e0 2024-03-30T00:57:02.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:02 smithi130 ceph-mon[47626]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:02.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:02 smithi130 ceph-mon[47626]: pgmap v266: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-30T00:57:02.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:02 smithi116 ceph-mon[38086]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:02.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:02 smithi116 ceph-mon[38086]: pgmap v266: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-30T00:57:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.715+0000 7f67cd800700 1 -- 172.21.15.116:0/3283847013 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f67c819a7d0 con 0x7f67ac0a11b0 2024-03-30T00:57:02.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.727+0000 7f67bf400700 1 -- 172.21.15.116:0/3283847013 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426664 (secure 0 0 0) 0x7f67c819a7d0 con 0x7f67ac0a11b0 2024-03-30T00:57:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:02.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.734+0000 7f67cd800700 1 -- 172.21.15.116:0/3283847013 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f67ac0a11b0 msgr2=0x7f67ac0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.734+0000 7f67cd800700 1 --2- 172.21.15.116:0/3283847013 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f67ac0a11b0 0x7f67ac0a3670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f67c819a9b0 tx=0x7f67b4005fd0 comp rx=0 tx=0).stop 2024-03-30T00:57:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.734+0000 7f67cd800700 1 -- 172.21.15.116:0/3283847013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c81031e0 msgr2=0x7f67c8199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.734+0000 7f67cd800700 1 --2- 172.21.15.116:0/3283847013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c81031e0 0x7f67c8199450 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f67b800e9a0 tx=0x7f67b800ecb0 comp rx=0 tx=0).stop 2024-03-30T00:57:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.734+0000 7f67cd800700 1 -- 172.21.15.116:0/3283847013 shutdown_connections 2024-03-30T00:57:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.734+0000 7f67cd800700 1 --2- 172.21.15.116:0/3283847013 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f67ac0a11b0 0x7f67ac0a3670 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.734+0000 7f67cd800700 1 --2- 172.21.15.116:0/3283847013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67c81082c0 0x7f67c819dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.734+0000 7f67cd800700 1 --2- 172.21.15.116:0/3283847013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c81031e0 0x7f67c8199450 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.734+0000 7f67cd800700 1 --2- 172.21.15.116:0/3283847013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67c8103b90 0x7f67c8199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.734+0000 7f67cd800700 1 -- 172.21.15.116:0/3283847013 >> 172.21.15.116:0/3283847013 conn(0x7f67c80fea70 msgr2=0x7f67c80ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.734+0000 7f67cd800700 1 -- 172.21.15.116:0/3283847013 shutdown_connections 2024-03-30T00:57:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:02.735+0000 7f67cd800700 1 -- 172.21.15.116:0/3283847013 wait complete. 2024-03-30T00:57:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-30T00:57:02.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:02 smithi181 ceph-mon[47622]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:02.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:02 smithi181 ceph-mon[47622]: pgmap v266: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-30T00:57:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-03-30T00:57:01.118007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655934,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":377,"num_read_kb":302,"num_write":339,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180887,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":277,"ondisk_log_size":277,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":351736,"kb_used_data":11104,"kb_used_omap":18,"kb_used_meta":340461,"kb_avail":1124590088,"statfs":{"total":1151940427776,"available":1151580250112,"internally_reserved":0,"allocated":11370496,"data_stored":7488924,"data_compressed":320673,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19128,"internal_metadata":348632392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1652,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":42,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003486"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460643+0000","last_change":"2024-03-30T00:55:51.809659+0000","last_active":"2024-03-30T00:56:35.460643+0000","last_peered":"2024-03-30T00:56:35.460643+0000","last_clean":"2024-03-30T00:56:35.460643+0000","last_became_active":"2024-03-30T00:55:51.809337+0000","last_became_peered":"2024-03-30T00:55:51.809337+0000","last_unstale":"2024-03-30T00:56:35.460643+0000","last_undegraded":"2024-03-30T00:56:35.460643+0000","last_fullsized":"2024-03-30T00:56:35.460643+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:20:50.640033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495951+0000","last_change":"2024-03-30T00:55:51.813778+0000","last_active":"2024-03-30T00:56:35.495951+0000","last_peered":"2024-03-30T00:56:35.495951+0000","last_clean":"2024-03-30T00:56:35.495951+0000","last_became_active":"2024-03-30T00:55:51.810347+0000","last_became_peered":"2024-03-30T00:55:51.810347+0000","last_unstale":"2024-03-30T00:56:35.495951+0000","last_undegraded":"2024-03-30T00:56:35.495951+0000","last_fullsized":"2024-03-30T00:56:35.495951+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:49:07.031477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495722+0000","last_change":"2024-03-30T00:55:51.817647+0000","last_active":"2024-03-30T00:56:35.495722+0000","last_peered":"2024-03-30T00:56:35.495722+0000","last_clean":"2024-03-30T00:56:35.495722+0000","last_became_active":"2024-03-30T00:55:51.816495+0000","last_became_peered":"2024-03-30T00:55:51.816495+0000","last_unstale":"2024-03-30T00:56:35.495722+0000","last_undegraded":"2024-03-30T00:56:35.495722+0000","last_fullsized":"2024-03-30T00:56:35.495722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:59:42.965346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463453+0000","last_change":"2024-03-30T00:55:51.817581+0000","last_active":"2024-03-30T00:56:35.463453+0000","last_peered":"2024-03-30T00:56:35.463453+0000","last_clean":"2024-03-30T00:56:35.463453+0000","last_became_active":"2024-03-30T00:55:51.816223+0000","last_became_peered":"2024-03-30T00:55:51.816223+0000","last_unstale":"2024-03-30T00:56:35.463453+0000","last_undegraded":"2024-03-30T00:56:35.463453+0000","last_fullsized":"2024-03-30T00:56:35.463453+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:48:32.203975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462897+0000","last_change":"2024-03-30T00:55:51.811707+0000","last_active":"2024-03-30T00:56:35.462897+0000","last_peered":"2024-03-30T00:56:35.462897+0000","last_clean":"2024-03-30T00:56:35.462897+0000","last_became_active":"2024-03-30T00:55:51.811366+0000","last_became_peered":"2024-03-30T00:55:51.811366+0000","last_unstale":"2024-03-30T00:56:35.462897+0000","last_undegraded":"2024-03-30T00:56:35.462897+0000","last_fullsized":"2024-03-30T00:56:35.462897+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:20.438505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467445+0000","last_change":"2024-03-30T00:55:51.826849+0000","last_active":"2024-03-30T00:56:35.467445+0000","last_peered":"2024-03-30T00:56:35.467445+0000","last_clean":"2024-03-30T00:56:35.467445+0000","last_became_active":"2024-03-30T00:55:51.826520+0000","last_became_peered":"2024-03-30T00:55:51.826520+0000","last_unstale":"2024-03-30T00:56:35.467445+0000","last_undegraded":"2024-03-30T00:56:35.467445+0000","last_fullsized":"2024-03-30T00:56:35.467445+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:54:03.423725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466080+0000","last_change":"2024-03-30T00:55:51.812880+0000","last_active":"2024-03-30T00:56:35.466080+0000","last_peered":"2024-03-30T00:56:35.466080+0000","last_clean":"2024-03-30T00:56:35.466080+0000","last_became_active":"2024-03-30T00:55:51.812572+0000","last_became_peered":"2024-03-30T00:55:51.812572+0000","last_unstale":"2024-03-30T00:56:35.466080+0000","last_undegraded":"2024-03-30T00:56:35.466080+0000","last_fullsized":"2024-03-30T00:56:35.466080+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:44:49.670641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470643+0000","last_change":"2024-03-30T00:55:51.815480+0000","last_active":"2024-03-30T00:56:35.470643+0000","last_peered":"2024-03-30T00:56:35.470643+0000","last_clean":"2024-03-30T00:56:35.470643+0000","last_became_active":"2024-03-30T00:55:51.814052+0000","last_became_peered":"2024-03-30T00:55:51.814052+0000","last_unstale":"2024-03-30T00:56:35.470643+0000","last_undegraded":"2024-03-30T00:56:35.470643+0000","last_fullsized":"2024-03-30T00:56:35.470643+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:27:49.930928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468858+0000","last_change":"2024-03-30T00:55:51.814127+0000","last_active":"2024-03-30T00:56:35.468858+0000","last_peered":"2024-03-30T00:56:35.468858+0000","last_clean":"2024-03-30T00:56:35.468858+0000","last_became_active":"2024-03-30T00:55:51.813820+0000","last_became_peered":"2024-03-30T00:55:51.813820+0000","last_unstale":"2024-03-30T00:56:35.468858+0000","last_undegraded":"2024-03-30T00:56:35.468858+0000","last_fullsized":"2024-03-30T00:56:35.468858+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:21:31.605935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468423+0000","last_change":"2024-03-30T00:55:51.815198+0000","last_active":"2024-03-30T00:56:35.468423+0000","last_peered":"2024-03-30T00:56:35.468423+0000","last_clean":"2024-03-30T00:56:35.468423+0000","last_became_active":"2024-03-30T00:55:51.814901+0000","last_became_peered":"2024-03-30T00:55:51.814901+0000","last_unstale":"2024-03-30T00:56:35.468423+0000","last_undegraded":"2024-03-30T00:56:35.468423+0000","last_fullsized":"2024-03-30T00:56:35.468423+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:45:51.224346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466742+0000","last_change":"2024-03-30T00:55:51.809884+0000","last_active":"2024-03-30T00:56:35.466742+0000","last_peered":"2024-03-30T00:56:35.466742+0000","last_clean":"2024-03-30T00:56:35.466742+0000","last_became_active":"2024-03-30T00:55:51.809576+0000","last_became_peered":"2024-03-30T00:55:51.809576+0000","last_unstale":"2024-03-30T00:56:35.466742+0000","last_undegraded":"2024-03-30T00:56:35.466742+0000","last_fullsized":"2024-03-30T00:56:35.466742+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:39:51.767744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463361+0000","last_change":"2024-03-30T00:55:51.817384+0000","last_active":"2024-03-30T00:56:35.463361+0000","last_peered":"2024-03-30T00:56:35.463361+0000","last_clean":"2024-03-30T00:56:35.463361+0000","last_became_active":"2024-03-30T00:55:51.816564+0000","last_became_peered":"2024-03-30T00:55:51.816564+0000","last_unstale":"2024-03-30T00:56:35.463361+0000","last_undegraded":"2024-03-30T00:56:35.463361+0000","last_fullsized":"2024-03-30T00:56:35.463361+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:47:07.195650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.475257+0000","last_change":"2024-03-30T00:55:51.814310+0000","last_active":"2024-03-30T00:56:35.475257+0000","last_peered":"2024-03-30T00:56:35.475257+0000","last_clean":"2024-03-30T00:56:35.475257+0000","last_became_active":"2024-03-30T00:55:51.813968+0000","last_became_peered":"2024-03-30T00:55:51.813968+0000","last_unstale":"2024-03-30T00:56:35.475257+0000","last_undegraded":"2024-03-30T00:56:35.475257+0000","last_fullsized":"2024-03-30T00:56:35.475257+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:59:55.955502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495223+0000","last_change":"2024-03-30T00:55:51.815655+0000","last_active":"2024-03-30T00:56:35.495223+0000","last_peered":"2024-03-30T00:56:35.495223+0000","last_clean":"2024-03-30T00:56:35.495223+0000","last_became_active":"2024-03-30T00:55:51.814099+0000","last_became_peered":"2024-03-30T00:55:51.814099+0000","last_unstale":"2024-03-30T00:56:35.495223+0000","last_undegraded":"2024-03-30T00:56:35.495223+0000","last_fullsized":"2024-03-30T00:56:35.495223+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:42.958153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.472491+0000","last_change":"2024-03-30T00:56:36.472491+0000","last_active":"2024-03-30T00:56:36.472491+0000","last_peered":"2024-03-30T00:56:36.472491+0000","last_clean":"2024-03-30T00:56:36.472491+0000","last_became_active":"2024-03-30T00:56:36.472037+0000","last_became_peered":"2024-03-30T00:56:36.472037+0000","last_unstale":"2024-03-30T00:56:36.472491+0000","last_undegraded":"2024-03-30T00:56:36.472491+0000","last_fullsized":"2024-03-30T00:56:36.472491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:52:08.402775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467785+0000","last_change":"2024-03-30T00:55:51.822278+0000","last_active":"2024-03-30T00:56:35.467785+0000","last_peered":"2024-03-30T00:56:35.467785+0000","last_clean":"2024-03-30T00:56:35.467785+0000","last_became_active":"2024-03-30T00:55:51.821996+0000","last_became_peered":"2024-03-30T00:55:51.821996+0000","last_unstale":"2024-03-30T00:56:35.467785+0000","last_undegraded":"2024-03-30T00:56:35.467785+0000","last_fullsized":"2024-03-30T00:56:35.467785+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:37:45.841853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460610+0000","last_change":"2024-03-30T00:55:51.809547+0000","last_active":"2024-03-30T00:56:35.460610+0000","last_peered":"2024-03-30T00:56:35.460610+0000","last_clean":"2024-03-30T00:56:35.460610+0000","last_became_active":"2024-03-30T00:55:51.809314+0000","last_became_peered":"2024-03-30T00:55:51.809314+0000","last_unstale":"2024-03-30T00:56:35.460610+0000","last_undegraded":"2024-03-30T00:56:35.460610+0000","last_fullsized":"2024-03-30T00:56:35.460610+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:39:31.661362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.488652+0000","last_change":"2024-03-30T00:55:51.810857+0000","last_active":"2024-03-30T00:56:35.488652+0000","last_peered":"2024-03-30T00:56:35.488652+0000","last_clean":"2024-03-30T00:56:35.488652+0000","last_became_active":"2024-03-30T00:55:51.809420+0000","last_became_peered":"2024-03-30T00:55:51.809420+0000","last_unstale":"2024-03-30T00:56:35.488652+0000","last_undegraded":"2024-03-30T00:56:35.488652+0000","last_fullsized":"2024-03-30T00:56:35.488652+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:36:48.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.471006+0000","last_change":"2024-03-30T00:55:51.814693+0000","last_active":"2024-03-30T00:56:35.471006+0000","last_peered":"2024-03-30T00:56:35.471006+0000","last_clean":"2024-03-30T00:56:35.471006+0000","last_became_active":"2024-03-30T00:55:51.814359+0000","last_became_peered":"2024-03-30T00:55:51.814359+0000","last_unstale":"2024-03-30T00:56:35.471006+0000","last_undegraded":"2024-03-30T00:56:35.471006+0000","last_fullsized":"2024-03-30T00:56:35.471006+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:20:50.856313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470709+0000","last_change":"2024-03-30T00:55:55.839089+0000","last_active":"2024-03-30T00:56:35.470709+0000","last_peered":"2024-03-30T00:56:35.470709+0000","last_clean":"2024-03-30T00:56:35.470709+0000","last_became_active":"2024-03-30T00:55:55.838705+0000","last_became_peered":"2024-03-30T00:55:55.838705+0000","last_unstale":"2024-03-30T00:56:35.470709+0000","last_undegraded":"2024-03-30T00:56:35.470709+0000","last_fullsized":"2024-03-30T00:56:35.470709+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:33:56.010272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'9","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.487878+0000","last_change":"2024-03-30T00:55:51.298577+0000","last_active":"2024-03-30T00:56:35.487878+0000","last_peered":"2024-03-30T00:56:35.487878+0000","last_clean":"2024-03-30T00:56:35.487878+0000","last_became_active":"2024-03-30T00:55:51.298269+0000","last_became_peered":"2024-03-30T00:55:51.298269+0000","last_unstale":"2024-03-30T00:56:35.487878+0000","last_undegraded":"2024-03-30T00:56:35.487878+0000","last_fullsized":"2024-03-30T00:56:35.487878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:10:04.998621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470406+0000","last_change":"2024-03-30T00:55:51.818517+0000","last_active":"2024-03-30T00:56:35.470406+0000","last_peered":"2024-03-30T00:56:35.470406+0000","last_clean":"2024-03-30T00:56:35.470406+0000","last_became_active":"2024-03-30T00:55:51.818174+0000","last_became_peered":"2024-03-30T00:55:51.818174+0000","last_unstale":"2024-03-30T00:56:35.470406+0000","last_undegraded":"2024-03-30T00:56:35.470406+0000","last_fullsized":"2024-03-30T00:56:35.470406+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:03:01.604135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.623912+0000","last_change":"2024-03-30T00:56:36.623912+0000","last_active":"2024-03-30T00:56:36.623912+0000","last_peered":"2024-03-30T00:56:36.623912+0000","last_clean":"2024-03-30T00:56:36.623912+0000","last_became_active":"2024-03-30T00:56:36.623266+0000","last_became_peered":"2024-03-30T00:56:36.623266+0000","last_unstale":"2024-03-30T00:56:36.623912+0000","last_undegraded":"2024-03-30T00:56:36.623912+0000","last_fullsized":"2024-03-30T00:56:36.623912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:46:47.166079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.471579+0000","last_change":"2024-03-30T00:55:51.312686+0000","last_active":"2024-03-30T00:56:35.471579+0000","last_peered":"2024-03-30T00:56:35.471579+0000","last_clean":"2024-03-30T00:56:35.471579+0000","last_became_active":"2024-03-30T00:55:51.312296+0000","last_became_peered":"2024-03-30T00:55:51.312296+0000","last_unstale":"2024-03-30T00:56:35.471579+0000","last_undegraded":"2024-03-30T00:56:35.471579+0000","last_fullsized":"2024-03-30T00:56:35.471579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:28:58.398577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495565+0000","last_change":"2024-03-30T00:55:51.812197+0000","last_active":"2024-03-30T00:56:35.495565+0000","last_peered":"2024-03-30T00:56:35.495565+0000","last_clean":"2024-03-30T00:56:35.495565+0000","last_became_active":"2024-03-30T00:55:51.811892+0000","last_became_peered":"2024-03-30T00:55:51.811892+0000","last_unstale":"2024-03-30T00:56:35.495565+0000","last_undegraded":"2024-03-30T00:56:35.495565+0000","last_fullsized":"2024-03-30T00:56:35.495565+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:06:53.459760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462318+0000","last_change":"2024-03-30T00:55:55.839015+0000","last_active":"2024-03-30T00:56:35.462318+0000","last_peered":"2024-03-30T00:56:35.462318+0000","last_clean":"2024-03-30T00:56:35.462318+0000","last_became_active":"2024-03-30T00:55:55.838608+0000","last_became_peered":"2024-03-30T00:55:55.838608+0000","last_unstale":"2024-03-30T00:56:35.462318+0000","last_undegraded":"2024-03-30T00:56:35.462318+0000","last_fullsized":"2024-03-30T00:56:35.462318+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:24:23.804754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'4","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463392+0000","last_change":"2024-03-30T00:55:51.304199+0000","last_active":"2024-03-30T00:56:35.463392+0000","last_peered":"2024-03-30T00:56:35.463392+0000","last_clean":"2024-03-30T00:56:35.463392+0000","last_became_active":"2024-03-30T00:55:51.303837+0000","last_became_peered":"2024-03-30T00:55:51.303837+0000","last_unstale":"2024-03-30T00:56:35.463392+0000","last_undegraded":"2024-03-30T00:56:35.463392+0000","last_fullsized":"2024-03-30T00:56:35.463392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:19.154957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.469403+0000","last_change":"2024-03-30T00:55:51.817984+0000","last_active":"2024-03-30T00:56:35.469403+0000","last_peered":"2024-03-30T00:56:35.469403+0000","last_clean":"2024-03-30T00:56:35.469403+0000","last_became_active":"2024-03-30T00:55:51.817767+0000","last_became_peered":"2024-03-30T00:55:51.817767+0000","last_unstale":"2024-03-30T00:56:35.469403+0000","last_undegraded":"2024-03-30T00:56:35.469403+0000","last_fullsized":"2024-03-30T00:56:35.469403+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:12:48.558024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.623581+0000","last_change":"2024-03-30T00:56:36.623581+0000","last_active":"2024-03-30T00:56:36.623581+0000","last_peered":"2024-03-30T00:56:36.623581+0000","last_clean":"2024-03-30T00:56:36.623581+0000","last_became_active":"2024-03-30T00:56:36.623168+0000","last_became_peered":"2024-03-30T00:56:36.623168+0000","last_unstale":"2024-03-30T00:56:36.623581+0000","last_undegraded":"2024-03-30T00:56:36.623581+0000","last_fullsized":"2024-03-30T00:56:36.623581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:41.448935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"76'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.480921+0000","last_change":"2024-03-30T00:56:36.480478+0000","last_active":"2024-03-30T00:56:36.480921+0000","last_peered":"2024-03-30T00:56:36.480921+0000","last_clean":"2024-03-30T00:56:36.480921+0000","last_became_active":"2024-03-30T00:56:36.473297+0000","last_became_peered":"2024-03-30T00:56:36.473297+0000","last_unstale":"2024-03-30T00:56:36.480921+0000","last_undegraded":"2024-03-30T00:56:36.480921+0000","last_fullsized":"2024-03-30T00:56:36.480921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:37:08.090281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466888+0000","last_change":"2024-03-30T00:55:51.827985+0000","last_active":"2024-03-30T00:56:35.466888+0000","last_peered":"2024-03-30T00:56:35.466888+0000","last_clean":"2024-03-30T00:56:35.466888+0000","last_became_active":"2024-03-30T00:55:51.827678+0000","last_became_peered":"2024-03-30T00:55:51.827678+0000","last_unstale":"2024-03-30T00:56:35.466888+0000","last_undegraded":"2024-03-30T00:56:35.466888+0000","last_fullsized":"2024-03-30T00:56:35.466888+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:47.088009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470870+0000","last_change":"2024-03-30T00:55:55.827267+0000","last_active":"2024-03-30T00:56:35.470870+0000","last_peered":"2024-03-30T00:56:35.470870+0000","last_clean":"2024-03-30T00:56:35.470870+0000","last_became_active":"2024-03-30T00:55:55.826821+0000","last_became_peered":"2024-03-30T00:55:55.826821+0000","last_unstale":"2024-03-30T00:56:35.470870+0000","last_undegraded":"2024-03-30T00:56:35.470870+0000","last_fullsized":"2024-03-30T00:56:35.470870+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:15:49.084359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":64,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:58.207188+0000","last_change":"2024-03-30T00:55:51.305177+0000","last_active":"2024-03-30T00:56:58.207188+0000","last_peered":"2024-03-30T00:56:58.207188+0000","last_clean":"2024-03-30T00:56:58.207188+0000","last_became_active":"2024-03-30T00:55:51.304813+0000","last_became_peered":"2024-03-30T00:55:51.304813+0000","last_unstale":"2024-03-30T00:56:58.207188+0000","last_undegraded":"2024-03-30T00:56:58.207188+0000","last_fullsized":"2024-03-30T00:56:58.207188+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:05:20.578897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463357+0000","last_change":"2024-03-30T00:55:51.812714+0000","last_active":"2024-03-30T00:56:35.463357+0000","last_peered":"2024-03-30T00:56:35.463357+0000","last_clean":"2024-03-30T00:56:35.463357+0000","last_became_active":"2024-03-30T00:55:51.812406+0000","last_became_peered":"2024-03-30T00:55:51.812406+0000","last_unstale":"2024-03-30T00:56:35.463357+0000","last_undegraded":"2024-03-30T00:56:35.463357+0000","last_fullsized":"2024-03-30T00:56:35.463357+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:47:01.691412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470799+0000","last_change":"2024-03-30T00:55:55.831994+0000","last_active":"2024-03-30T00:56:35.470799+0000","last_peered":"2024-03-30T00:56:35.470799+0000","last_clean":"2024-03-30T00:56:35.470799+0000","last_became_active":"2024-03-30T00:55:55.831537+0000","last_became_peered":"2024-03-30T00:55:55.831537+0000","last_unstale":"2024-03-30T00:56:35.470799+0000","last_undegraded":"2024-03-30T00:56:35.470799+0000","last_fullsized":"2024-03-30T00:56:35.470799+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:44:47.240981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467196+0000","last_change":"2024-03-30T00:55:51.303468+0000","last_active":"2024-03-30T00:56:35.467196+0000","last_peered":"2024-03-30T00:56:35.467196+0000","last_clean":"2024-03-30T00:56:35.467196+0000","last_became_active":"2024-03-30T00:55:51.303068+0000","last_became_peered":"2024-03-30T00:55:51.303068+0000","last_unstale":"2024-03-30T00:56:35.467196+0000","last_undegraded":"2024-03-30T00:56:35.467196+0000","last_fullsized":"2024-03-30T00:56:35.467196+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:45:25.278196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466931+0000","last_change":"2024-03-30T00:55:51.825099+0000","last_active":"2024-03-30T00:56:35.466931+0000","last_peered":"2024-03-30T00:56:35.466931+0000","last_clean":"2024-03-30T00:56:35.466931+0000","last_became_active":"2024-03-30T00:55:51.824717+0000","last_became_peered":"2024-03-30T00:55:51.824717+0000","last_unstale":"2024-03-30T00:56:35.466931+0000","last_undegraded":"2024-03-30T00:56:35.466931+0000","last_fullsized":"2024-03-30T00:56:35.466931+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:49:31.542430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460499+0000","last_change":"2024-03-30T00:55:55.826106+0000","last_active":"2024-03-30T00:56:35.460499+0000","last_peered":"2024-03-30T00:56:35.460499+0000","last_clean":"2024-03-30T00:56:35.460499+0000","last_became_active":"2024-03-30T00:55:55.825597+0000","last_became_peered":"2024-03-30T00:55:55.825597+0000","last_unstale":"2024-03-30T00:56:35.460499+0000","last_undegraded":"2024-03-30T00:56:35.460499+0000","last_fullsized":"2024-03-30T00:56:35.460499+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:00:18.991732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.473981+0000","last_change":"2024-03-30T00:55:51.294090+0000","last_active":"2024-03-30T00:56:35.473981+0000","last_peered":"2024-03-30T00:56:35.473981+0000","last_clean":"2024-03-30T00:56:35.473981+0000","last_became_active":"2024-03-30T00:55:51.293668+0000","last_became_peered":"2024-03-30T00:55:51.293668+0000","last_unstale":"2024-03-30T00:56:35.473981+0000","last_undegraded":"2024-03-30T00:56:35.473981+0000","last_fullsized":"2024-03-30T00:56:35.473981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:33:36.661760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470557+0000","last_change":"2024-03-30T00:55:51.823428+0000","last_active":"2024-03-30T00:56:35.470557+0000","last_peered":"2024-03-30T00:56:35.470557+0000","last_clean":"2024-03-30T00:56:35.470557+0000","last_became_active":"2024-03-30T00:55:51.822222+0000","last_became_peered":"2024-03-30T00:55:51.822222+0000","last_unstale":"2024-03-30T00:56:35.470557+0000","last_undegraded":"2024-03-30T00:56:35.470557+0000","last_fullsized":"2024-03-30T00:56:35.470557+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:05:34.798940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495876+0000","last_change":"2024-03-30T00:55:55.828115+0000","last_active":"2024-03-30T00:56:35.495876+0000","last_peered":"2024-03-30T00:56:35.495876+0000","last_clean":"2024-03-30T00:56:35.495876+0000","last_became_active":"2024-03-30T00:55:55.827681+0000","last_became_peered":"2024-03-30T00:55:55.827681+0000","last_unstale":"2024-03-30T00:56:35.495876+0000","last_undegraded":"2024-03-30T00:56:35.495876+0000","last_fullsized":"2024-03-30T00:56:35.495876+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:18:25.751350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'1","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466298+0000","last_change":"2024-03-30T00:55:51.305163+0000","last_active":"2024-03-30T00:56:35.466298+0000","last_peered":"2024-03-30T00:56:35.466298+0000","last_clean":"2024-03-30T00:56:35.466298+0000","last_became_active":"2024-03-30T00:55:51.304860+0000","last_became_peered":"2024-03-30T00:55:51.304860+0000","last_unstale":"2024-03-30T00:56:35.466298+0000","last_undegraded":"2024-03-30T00:56:35.466298+0000","last_fullsized":"2024-03-30T00:56:35.466298+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:44:41.850998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.483962+0000","last_change":"2024-03-30T00:55:51.819401+0000","last_active":"2024-03-30T00:56:35.483962+0000","last_peered":"2024-03-30T00:56:35.483962+0000","last_clean":"2024-03-30T00:56:35.483962+0000","last_became_active":"2024-03-30T00:55:51.819021+0000","last_became_peered":"2024-03-30T00:55:51.819021+0000","last_unstale":"2024-03-30T00:56:35.483962+0000","last_undegraded":"2024-03-30T00:56:35.483962+0000","last_fullsized":"2024-03-30T00:56:35.483962+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:01:45.195719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.469445+0000","last_change":"2024-03-30T00:55:55.836806+0000","last_active":"2024-03-30T00:56:35.469445+0000","last_peered":"2024-03-30T00:56:35.469445+0000","last_clean":"2024-03-30T00:56:35.469445+0000","last_became_active":"2024-03-30T00:55:55.836403+0000","last_became_peered":"2024-03-30T00:55:55.836403+0000","last_unstale":"2024-03-30T00:56:35.469445+0000","last_undegraded":"2024-03-30T00:56:35.469445+0000","last_fullsized":"2024-03-30T00:56:35.469445+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:17:13.913675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'3","reported_seq":58,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:58.206698+0000","last_change":"2024-03-30T00:55:51.300833+0000","last_active":"2024-03-30T00:56:58.206698+0000","last_peered":"2024-03-30T00:56:58.206698+0000","last_clean":"2024-03-30T00:56:58.206698+0000","last_became_active":"2024-03-30T00:55:51.300473+0000","last_became_peered":"2024-03-30T00:55:51.300473+0000","last_unstale":"2024-03-30T00:56:58.206698+0000","last_undegraded":"2024-03-30T00:56:58.206698+0000","last_fullsized":"2024-03-30T00:56:58.206698+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:00:39.211981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468719+0000","last_change":"2024-03-30T00:55:51.810781+0000","last_active":"2024-03-30T00:56:35.468719+0000","last_peered":"2024-03-30T00:56:35.468719+0000","last_clean":"2024-03-30T00:56:35.468719+0000","last_became_active":"2024-03-30T00:55:51.810215+0000","last_became_peered":"2024-03-30T00:55:51.810215+0000","last_unstale":"2024-03-30T00:56:35.468719+0000","last_undegraded":"2024-03-30T00:56:35.468719+0000","last_fullsized":"2024-03-30T00:56:35.468719+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:53:50.283209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.484181+0000","last_change":"2024-03-30T00:55:55.834505+0000","last_active":"2024-03-30T00:56:35.484181+0000","last_peered":"2024-03-30T00:56:35.484181+0000","last_clean":"2024-03-30T00:56:35.484181+0000","last_became_active":"2024-03-30T00:55:55.829319+0000","last_became_peered":"2024-03-30T00:55:55.829319+0000","last_unstale":"2024-03-30T00:56:35.484181+0000","last_undegraded":"2024-03-30T00:56:35.484181+0000","last_fullsized":"2024-03-30T00:56:35.484181+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:51:37.595523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466067+0000","last_change":"2024-03-30T00:55:51.302070+0000","last_active":"2024-03-30T00:56:35.466067+0000","last_peered":"2024-03-30T00:56:35.466067+0000","last_clean":"2024-03-30T00:56:35.466067+0000","last_became_active":"2024-03-30T00:55:51.298940+0000","last_became_peered":"2024-03-30T00:55:51.298940+0000","last_unstale":"2024-03-30T00:56:35.466067+0000","last_undegraded":"2024-03-30T00:56:35.466067+0000","last_fullsized":"2024-03-30T00:56:35.466067+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:07:47.915506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495434+0000","last_change":"2024-03-30T00:55:51.819918+0000","last_active":"2024-03-30T00:56:35.495434+0000","last_peered":"2024-03-30T00:56:35.495434+0000","last_clean":"2024-03-30T00:56:35.495434+0000","last_became_active":"2024-03-30T00:55:51.819493+0000","last_became_peered":"2024-03-30T00:55:51.819493+0000","last_unstale":"2024-03-30T00:56:35.495434+0000","last_undegraded":"2024-03-30T00:56:35.495434+0000","last_fullsized":"2024-03-30T00:56:35.495434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:43:43.067031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462710+0000","last_change":"2024-03-30T00:55:55.827176+0000","last_active":"2024-03-30T00:56:35.462710+0000","last_peered":"2024-03-30T00:56:35.462710+0000","last_clean":"2024-03-30T00:56:35.462710+0000","last_became_active":"2024-03-30T00:55:55.826772+0000","last_became_peered":"2024-03-30T00:55:55.826772+0000","last_unstale":"2024-03-30T00:56:35.462710+0000","last_undegraded":"2024-03-30T00:56:35.462710+0000","last_fullsized":"2024-03-30T00:56:35.462710+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:02:04.512050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'4","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467237+0000","last_change":"2024-03-30T00:55:51.306688+0000","last_active":"2024-03-30T00:56:35.467237+0000","last_peered":"2024-03-30T00:56:35.467237+0000","last_clean":"2024-03-30T00:56:35.467237+0000","last_became_active":"2024-03-30T00:55:51.306333+0000","last_became_peered":"2024-03-30T00:55:51.306333+0000","last_unstale":"2024-03-30T00:56:35.467237+0000","last_undegraded":"2024-03-30T00:56:35.467237+0000","last_fullsized":"2024-03-30T00:56:35.467237+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:21:13.719947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.469497+0000","last_change":"2024-03-30T00:55:51.824879+0000","last_active":"2024-03-30T00:56:35.469497+0000","last_peered":"2024-03-30T00:56:35.469497+0000","last_clean":"2024-03-30T00:56:35.469497+0000","last_became_active":"2024-03-30T00:55:51.824566+0000","last_became_peered":"2024-03-30T00:55:51.824566+0000","last_unstale":"2024-03-30T00:56:35.469497+0000","last_undegraded":"2024-03-30T00:56:35.469497+0000","last_fullsized":"2024-03-30T00:56:35.469497+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:04:28.505693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.484132+0000","last_change":"2024-03-30T00:55:55.826387+0000","last_active":"2024-03-30T00:56:35.484132+0000","last_peered":"2024-03-30T00:56:35.484132+0000","last_clean":"2024-03-30T00:56:35.484132+0000","last_became_active":"2024-03-30T00:55:55.825941+0000","last_became_peered":"2024-03-30T00:55:55.825941+0000","last_unstale":"2024-03-30T00:56:35.484132+0000","last_undegraded":"2024-03-30T00:56:35.484132+0000","last_fullsized":"2024-03-30T00:56:35.484132+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:34:43.432712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"83'33","reported_seq":134,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:59.207817+0000","last_change":"2024-03-30T00:55:51.304592+0000","last_active":"2024-03-30T00:56:59.207817+0000","last_peered":"2024-03-30T00:56:59.207817+0000","last_clean":"2024-03-30T00:56:59.207817+0000","last_became_active":"2024-03-30T00:55:51.303931+0000","last_became_peered":"2024-03-30T00:55:51.303931+0000","last_unstale":"2024-03-30T00:56:59.207817+0000","last_undegraded":"2024-03-30T00:56:59.207817+0000","last_fullsized":"2024-03-30T00:56:59.207817+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:53:40.976289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61032,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":84,"num_read_kb":101,"num_write":34,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463233+0000","last_change":"2024-03-30T00:55:51.815196+0000","last_active":"2024-03-30T00:56:35.463233+0000","last_peered":"2024-03-30T00:56:35.463233+0000","last_clean":"2024-03-30T00:56:35.463233+0000","last_became_active":"2024-03-30T00:55:51.813376+0000","last_became_peered":"2024-03-30T00:55:51.813376+0000","last_unstale":"2024-03-30T00:56:35.463233+0000","last_undegraded":"2024-03-30T00:56:35.463233+0000","last_fullsized":"2024-03-30T00:56:35.463233+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:40:26.350248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467447+0000","last_change":"2024-03-30T00:55:55.836906+0000","last_active":"2024-03-30T00:56:35.467447+0000","last_peered":"2024-03-30T00:56:35.467447+0000","last_clean":"2024-03-30T00:56:35.467447+0000","last_became_active":"2024-03-30T00:55:55.836539+0000","last_became_peered":"2024-03-30T00:55:55.836539+0000","last_unstale":"2024-03-30T00:56:35.467447+0000","last_undegraded":"2024-03-30T00:56:35.467447+0000","last_fullsized":"2024-03-30T00:56:35.467447+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:21:39.320192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'6","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.478499+0000","last_change":"2024-03-30T00:56:36.478132+0000","last_active":"2024-03-30T00:56:36.478499+0000","last_peered":"2024-03-30T00:56:36.478499+0000","last_clean":"2024-03-30T00:56:36.478499+0000","last_became_active":"2024-03-30T00:56:36.467449+0000","last_became_peered":"2024-03-30T00:56:36.467449+0000","last_unstale":"2024-03-30T00:56:36.478499+0000","last_undegraded":"2024-03-30T00:56:36.478499+0000","last_fullsized":"2024-03-30T00:56:36.478499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:27:51.986112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.474554+0000","last_change":"2024-03-30T00:55:51.819825+0000","last_active":"2024-03-30T00:56:35.474554+0000","last_peered":"2024-03-30T00:56:35.474554+0000","last_clean":"2024-03-30T00:56:35.474554+0000","last_became_active":"2024-03-30T00:55:51.819481+0000","last_became_peered":"2024-03-30T00:55:51.819481+0000","last_unstale":"2024-03-30T00:56:35.474554+0000","last_undegraded":"2024-03-30T00:56:35.474554+0000","last_fullsized":"2024-03-30T00:56:35.474554+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:10:17.138541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460246+0000","last_change":"2024-03-30T00:55:55.832675+0000","last_active":"2024-03-30T00:56:35.460246+0000","last_peered":"2024-03-30T00:56:35.460246+0000","last_clean":"2024-03-30T00:56:35.460246+0000","last_became_active":"2024-03-30T00:55:55.832325+0000","last_became_peered":"2024-03-30T00:55:55.832325+0000","last_unstale":"2024-03-30T00:56:35.460246+0000","last_undegraded":"2024-03-30T00:56:35.460246+0000","last_fullsized":"2024-03-30T00:56:35.460246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:38:46.765473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467208+0000","last_change":"2024-03-30T00:55:51.313700+0000","last_active":"2024-03-30T00:56:35.467208+0000","last_peered":"2024-03-30T00:56:35.467208+0000","last_clean":"2024-03-30T00:56:35.467208+0000","last_became_active":"2024-03-30T00:55:51.313355+0000","last_became_peered":"2024-03-30T00:55:51.313355+0000","last_unstale":"2024-03-30T00:56:35.467208+0000","last_undegraded":"2024-03-30T00:56:35.467208+0000","last_fullsized":"2024-03-30T00:56:35.467208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:43:04.461398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467209+0000","last_change":"2024-03-30T00:55:51.819939+0000","last_active":"2024-03-30T00:56:35.467209+0000","last_peered":"2024-03-30T00:56:35.467209+0000","last_clean":"2024-03-30T00:56:35.467209+0000","last_became_active":"2024-03-30T00:55:51.819614+0000","last_became_peered":"2024-03-30T00:55:51.819614+0000","last_unstale":"2024-03-30T00:56:35.467209+0000","last_undegraded":"2024-03-30T00:56:35.467209+0000","last_fullsized":"2024-03-30T00:56:35.467209+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:20:37.362779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.469703+0000","last_change":"2024-03-30T00:56:36.469703+0000","last_active":"2024-03-30T00:56:36.469703+0000","last_peered":"2024-03-30T00:56:36.469703+0000","last_clean":"2024-03-30T00:56:36.469703+0000","last_became_active":"2024-03-30T00:56:36.469195+0000","last_became_peered":"2024-03-30T00:56:36.469195+0000","last_unstale":"2024-03-30T00:56:36.469703+0000","last_undegraded":"2024-03-30T00:56:36.469703+0000","last_fullsized":"2024-03-30T00:56:36.469703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:10:10.421447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"81'4","reported_seq":61,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:57.206035+0000","last_change":"2024-03-30T00:55:51.299246+0000","last_active":"2024-03-30T00:56:57.206035+0000","last_peered":"2024-03-30T00:56:57.206035+0000","last_clean":"2024-03-30T00:56:57.206035+0000","last_became_active":"2024-03-30T00:55:51.298924+0000","last_became_peered":"2024-03-30T00:55:51.298924+0000","last_unstale":"2024-03-30T00:56:57.206035+0000","last_undegraded":"2024-03-30T00:56:57.206035+0000","last_fullsized":"2024-03-30T00:56:57.206035+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:50:05.897612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466900+0000","last_change":"2024-03-30T00:55:51.813804+0000","last_active":"2024-03-30T00:56:35.466900+0000","last_peered":"2024-03-30T00:56:35.466900+0000","last_clean":"2024-03-30T00:56:35.466900+0000","last_became_active":"2024-03-30T00:55:51.813500+0000","last_became_peered":"2024-03-30T00:55:51.813500+0000","last_unstale":"2024-03-30T00:56:35.466900+0000","last_undegraded":"2024-03-30T00:56:35.466900+0000","last_fullsized":"2024-03-30T00:56:35.466900+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:42:21.614900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.477003+0000","last_change":"2024-03-30T00:55:55.837737+0000","last_active":"2024-03-30T00:56:35.477003+0000","last_peered":"2024-03-30T00:56:35.477003+0000","last_clean":"2024-03-30T00:56:35.477003+0000","last_became_active":"2024-03-30T00:55:55.837312+0000","last_became_peered":"2024-03-30T00:55:55.837312+0000","last_unstale":"2024-03-30T00:56:35.477003+0000","last_undegraded":"2024-03-30T00:56:35.477003+0000","last_fullsized":"2024-03-30T00:56:35.477003+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:11:46.139979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'15","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462932+0000","last_change":"2024-03-30T00:55:51.306398+0000","last_active":"2024-03-30T00:56:35.462932+0000","last_peered":"2024-03-30T00:56:35.462932+0000","last_clean":"2024-03-30T00:56:35.462932+0000","last_became_active":"2024-03-30T00:55:51.305878+0000","last_became_peered":"2024-03-30T00:55:51.305878+0000","last_unstale":"2024-03-30T00:56:35.462932+0000","last_undegraded":"2024-03-30T00:56:35.462932+0000","last_fullsized":"2024-03-30T00:56:35.462932+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:20:32.742423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466290+0000","last_change":"2024-03-30T00:55:51.824294+0000","last_active":"2024-03-30T00:56:35.466290+0000","last_peered":"2024-03-30T00:56:35.466290+0000","last_clean":"2024-03-30T00:56:35.466290+0000","last_became_active":"2024-03-30T00:55:51.823957+0000","last_became_peered":"2024-03-30T00:55:51.823957+0000","last_unstale":"2024-03-30T00:56:35.466290+0000","last_undegraded":"2024-03-30T00:56:35.466290+0000","last_fullsized":"2024-03-30T00:56:35.466290+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:23:16.873140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466253+0000","last_change":"2024-03-30T00:55:55.837023+0000","last_active":"2024-03-30T00:56:35.466253+0000","last_peered":"2024-03-30T00:56:35.466253+0000","last_clean":"2024-03-30T00:56:35.466253+0000","last_became_active":"2024-03-30T00:55:55.836541+0000","last_became_peered":"2024-03-30T00:55:55.836541+0000","last_unstale":"2024-03-30T00:56:35.466253+0000","last_undegraded":"2024-03-30T00:56:35.466253+0000","last_fullsized":"2024-03-30T00:56:35.466253+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:58:40.323953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463352+0000","last_change":"2024-03-30T00:55:51.300883+0000","last_active":"2024-03-30T00:56:35.463352+0000","last_peered":"2024-03-30T00:56:35.463352+0000","last_clean":"2024-03-30T00:56:35.463352+0000","last_became_active":"2024-03-30T00:55:51.300531+0000","last_became_peered":"2024-03-30T00:55:51.300531+0000","last_unstale":"2024-03-30T00:56:35.463352+0000","last_undegraded":"2024-03-30T00:56:35.463352+0000","last_fullsized":"2024-03-30T00:56:35.463352+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:38:38.874075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460541+0000","last_change":"2024-03-30T00:55:51.822970+0000","last_active":"2024-03-30T00:56:35.460541+0000","last_peered":"2024-03-30T00:56:35.460541+0000","last_clean":"2024-03-30T00:56:35.460541+0000","last_became_active":"2024-03-30T00:55:51.822653+0000","last_became_peered":"2024-03-30T00:55:51.822653+0000","last_unstale":"2024-03-30T00:56:35.460541+0000","last_undegraded":"2024-03-30T00:56:35.460541+0000","last_fullsized":"2024-03-30T00:56:35.460541+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:59:26.306672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463256+0000","last_change":"2024-03-30T00:55:55.856605+0000","last_active":"2024-03-30T00:56:35.463256+0000","last_peered":"2024-03-30T00:56:35.463256+0000","last_clean":"2024-03-30T00:56:35.463256+0000","last_became_active":"2024-03-30T00:55:55.856061+0000","last_became_peered":"2024-03-30T00:55:55.856061+0000","last_unstale":"2024-03-30T00:56:35.463256+0000","last_undegraded":"2024-03-30T00:56:35.463256+0000","last_fullsized":"2024-03-30T00:56:35.463256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:58:41.010206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.488831+0000","last_change":"2024-03-30T00:55:51.301961+0000","last_active":"2024-03-30T00:56:35.488831+0000","last_peered":"2024-03-30T00:56:35.488831+0000","last_clean":"2024-03-30T00:56:35.488831+0000","last_became_active":"2024-03-30T00:55:51.301618+0000","last_became_peered":"2024-03-30T00:55:51.301618+0000","last_unstale":"2024-03-30T00:56:35.488831+0000","last_undegraded":"2024-03-30T00:56:35.488831+0000","last_fullsized":"2024-03-30T00:56:35.488831+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:38:46.681490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467943+0000","last_change":"2024-03-30T00:55:51.821331+0000","last_active":"2024-03-30T00:56:35.467943+0000","last_peered":"2024-03-30T00:56:35.467943+0000","last_clean":"2024-03-30T00:56:35.467943+0000","last_became_active":"2024-03-30T00:55:51.820988+0000","last_became_peered":"2024-03-30T00:55:51.820988+0000","last_unstale":"2024-03-30T00:56:35.467943+0000","last_undegraded":"2024-03-30T00:56:35.467943+0000","last_fullsized":"2024-03-30T00:56:35.467943+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:25:20.150789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.475451+0000","last_change":"2024-03-30T00:55:55.826251+0000","last_active":"2024-03-30T00:56:35.475451+0000","last_peered":"2024-03-30T00:56:35.475451+0000","last_clean":"2024-03-30T00:56:35.475451+0000","last_became_active":"2024-03-30T00:55:55.822695+0000","last_became_peered":"2024-03-30T00:55:55.822695+0000","last_unstale":"2024-03-30T00:56:35.475451+0000","last_undegraded":"2024-03-30T00:56:35.475451+0000","last_fullsized":"2024-03-30T00:56:35.475451+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:32:19.011454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466393+0000","last_change":"2024-03-30T00:55:51.309412+0000","last_active":"2024-03-30T00:56:35.466393+0000","last_peered":"2024-03-30T00:56:35.466393+0000","last_clean":"2024-03-30T00:56:35.466393+0000","last_became_active":"2024-03-30T00:55:51.309028+0000","last_became_peered":"2024-03-30T00:55:51.309028+0000","last_unstale":"2024-03-30T00:56:35.466393+0000","last_undegraded":"2024-03-30T00:56:35.466393+0000","last_fullsized":"2024-03-30T00:56:35.466393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:00:49.022772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.467986+0000","last_change":"2024-03-30T00:56:36.467986+0000","last_active":"2024-03-30T00:56:36.467986+0000","last_peered":"2024-03-30T00:56:36.467986+0000","last_clean":"2024-03-30T00:56:36.467986+0000","last_became_active":"2024-03-30T00:56:36.467601+0000","last_became_peered":"2024-03-30T00:56:36.467601+0000","last_unstale":"2024-03-30T00:56:36.467986+0000","last_undegraded":"2024-03-30T00:56:36.467986+0000","last_fullsized":"2024-03-30T00:56:36.467986+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:37:32.979094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.479139+0000","last_change":"2024-03-30T00:55:55.856377+0000","last_active":"2024-03-30T00:56:35.479139+0000","last_peered":"2024-03-30T00:56:35.479139+0000","last_clean":"2024-03-30T00:56:35.479139+0000","last_became_active":"2024-03-30T00:55:55.855986+0000","last_became_peered":"2024-03-30T00:55:55.855986+0000","last_unstale":"2024-03-30T00:56:35.479139+0000","last_undegraded":"2024-03-30T00:56:35.479139+0000","last_fullsized":"2024-03-30T00:56:35.479139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:45:47.303446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.466031+0000","last_change":"2024-03-30T00:56:36.464863+0000","last_active":"2024-03-30T00:56:36.466031+0000","last_peered":"2024-03-30T00:56:36.466031+0000","last_clean":"2024-03-30T00:56:36.466031+0000","last_became_active":"2024-03-30T00:56:36.464323+0000","last_became_peered":"2024-03-30T00:56:36.464323+0000","last_unstale":"2024-03-30T00:56:36.466031+0000","last_undegraded":"2024-03-30T00:56:36.466031+0000","last_fullsized":"2024-03-30T00:56:36.466031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:43:10.899726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"81'4","reported_seq":56,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:54.201715+0000","last_change":"2024-03-30T00:55:51.314834+0000","last_active":"2024-03-30T00:56:54.201715+0000","last_peered":"2024-03-30T00:56:54.201715+0000","last_clean":"2024-03-30T00:56:54.201715+0000","last_became_active":"2024-03-30T00:55:51.314429+0000","last_became_peered":"2024-03-30T00:55:51.314429+0000","last_unstale":"2024-03-30T00:56:54.201715+0000","last_undegraded":"2024-03-30T00:56:54.201715+0000","last_fullsized":"2024-03-30T00:56:54.201715+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:17:13.987180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466483+0000","last_change":"2024-03-30T00:55:55.840721+0000","last_active":"2024-03-30T00:56:35.466483+0000","last_peered":"2024-03-30T00:56:35.466483+0000","last_clean":"2024-03-30T00:56:35.466483+0000","last_became_active":"2024-03-30T00:55:55.840269+0000","last_became_peered":"2024-03-30T00:55:55.840269+0000","last_unstale":"2024-03-30T00:56:35.466483+0000","last_undegraded":"2024-03-30T00:56:35.466483+0000","last_fullsized":"2024-03-30T00:56:35.466483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:28:55.366411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.473307+0000","last_change":"2024-03-30T00:55:51.812609+0000","last_active":"2024-03-30T00:56:35.473307+0000","last_peered":"2024-03-30T00:56:35.473307+0000","last_clean":"2024-03-30T00:56:35.473307+0000","last_became_active":"2024-03-30T00:55:51.812325+0000","last_became_peered":"2024-03-30T00:55:51.812325+0000","last_unstale":"2024-03-30T00:56:35.473307+0000","last_undegraded":"2024-03-30T00:56:35.473307+0000","last_fullsized":"2024-03-30T00:56:35.473307+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:22:22.074405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'3","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463363+0000","last_change":"2024-03-30T00:55:51.296173+0000","last_active":"2024-03-30T00:56:35.463363+0000","last_peered":"2024-03-30T00:56:35.463363+0000","last_clean":"2024-03-30T00:56:35.463363+0000","last_became_active":"2024-03-30T00:55:51.295331+0000","last_became_peered":"2024-03-30T00:55:51.295331+0000","last_unstale":"2024-03-30T00:56:35.463363+0000","last_undegraded":"2024-03-30T00:56:35.463363+0000","last_fullsized":"2024-03-30T00:56:35.463363+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:51:58.343765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":326,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466805+0000","last_change":"2024-03-30T00:55:02.487656+0000","last_active":"2024-03-30T00:56:35.466805+0000","last_peered":"2024-03-30T00:56:35.466805+0000","last_clean":"2024-03-30T00:56:35.466805+0000","last_became_active":"2024-03-30T00:55:02.487263+0000","last_became_peered":"2024-03-30T00:55:02.487263+0000","last_unstale":"2024-03-30T00:56:35.466805+0000","last_undegraded":"2024-03-30T00:56:35.466805+0000","last_fullsized":"2024-03-30T00:56:35.466805+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:52:29.691461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:52:29.691461+0000","last_clean_scrub_stamp":"2024-03-30T00:52:29.691461+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:19:20.825038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466944+0000","last_change":"2024-03-30T00:55:55.856487+0000","last_active":"2024-03-30T00:56:35.466944+0000","last_peered":"2024-03-30T00:56:35.466944+0000","last_clean":"2024-03-30T00:56:35.466944+0000","last_became_active":"2024-03-30T00:55:55.856106+0000","last_became_peered":"2024-03-30T00:55:55.856106+0000","last_unstale":"2024-03-30T00:56:35.466944+0000","last_undegraded":"2024-03-30T00:56:35.466944+0000","last_fullsized":"2024-03-30T00:56:35.466944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:14.504218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.473794+0000","last_change":"2024-03-30T00:55:51.821519+0000","last_active":"2024-03-30T00:56:35.473794+0000","last_peered":"2024-03-30T00:56:35.473794+0000","last_clean":"2024-03-30T00:56:35.473794+0000","last_became_active":"2024-03-30T00:55:51.821095+0000","last_became_peered":"2024-03-30T00:55:51.821095+0000","last_unstale":"2024-03-30T00:56:35.473794+0000","last_undegraded":"2024-03-30T00:56:35.473794+0000","last_fullsized":"2024-03-30T00:56:35.473794+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:50:22.297450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'3","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466994+0000","last_change":"2024-03-30T00:55:51.303047+0000","last_active":"2024-03-30T00:56:35.466994+0000","last_peered":"2024-03-30T00:56:35.466994+0000","last_clean":"2024-03-30T00:56:35.466994+0000","last_became_active":"2024-03-30T00:55:51.302712+0000","last_became_peered":"2024-03-30T00:55:51.302712+0000","last_unstale":"2024-03-30T00:56:35.466994+0000","last_undegraded":"2024-03-30T00:56:35.466994+0000","last_fullsized":"2024-03-30T00:56:35.466994+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:07:59.451674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460262+0000","last_change":"2024-03-30T00:55:55.823692+0000","last_active":"2024-03-30T00:56:35.460262+0000","last_peered":"2024-03-30T00:56:35.460262+0000","last_clean":"2024-03-30T00:56:35.460262+0000","last_became_active":"2024-03-30T00:55:55.823188+0000","last_became_peered":"2024-03-30T00:55:55.823188+0000","last_unstale":"2024-03-30T00:56:35.460262+0000","last_undegraded":"2024-03-30T00:56:35.460262+0000","last_fullsized":"2024-03-30T00:56:35.460262+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:32:32.089166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'1","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463201+0000","last_change":"2024-03-30T00:55:51.306935+0000","last_active":"2024-03-30T00:56:35.463201+0000","last_peered":"2024-03-30T00:56:35.463201+0000","last_clean":"2024-03-30T00:56:35.463201+0000","last_became_active":"2024-03-30T00:55:51.306595+0000","last_became_peered":"2024-03-30T00:55:51.306595+0000","last_unstale":"2024-03-30T00:56:35.463201+0000","last_undegraded":"2024-03-30T00:56:35.463201+0000","last_fullsized":"2024-03-30T00:56:35.463201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:50:30.396762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.475643+0000","last_change":"2024-03-30T00:55:51.807084+0000","last_active":"2024-03-30T00:56:35.475643+0000","last_peered":"2024-03-30T00:56:35.475643+0000","last_clean":"2024-03-30T00:56:35.475643+0000","last_became_active":"2024-03-30T00:55:51.806505+0000","last_became_peered":"2024-03-30T00:55:51.806505+0000","last_unstale":"2024-03-30T00:56:35.475643+0000","last_undegraded":"2024-03-30T00:56:35.475643+0000","last_fullsized":"2024-03-30T00:56:35.475643+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:38:44.759863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470455+0000","last_change":"2024-03-30T00:55:55.856457+0000","last_active":"2024-03-30T00:56:35.470455+0000","last_peered":"2024-03-30T00:56:35.470455+0000","last_clean":"2024-03-30T00:56:35.470455+0000","last_became_active":"2024-03-30T00:55:55.855982+0000","last_became_peered":"2024-03-30T00:55:55.855982+0000","last_unstale":"2024-03-30T00:56:35.470455+0000","last_undegraded":"2024-03-30T00:56:35.470455+0000","last_fullsized":"2024-03-30T00:56:35.470455+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:18:25.154634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"81'5","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.488541+0000","last_change":"2024-03-30T00:55:51.312581+0000","last_active":"2024-03-30T00:56:35.488541+0000","last_peered":"2024-03-30T00:56:35.488541+0000","last_clean":"2024-03-30T00:56:35.488541+0000","last_became_active":"2024-03-30T00:55:51.312225+0000","last_became_peered":"2024-03-30T00:55:51.312225+0000","last_unstale":"2024-03-30T00:56:35.488541+0000","last_undegraded":"2024-03-30T00:56:35.488541+0000","last_fullsized":"2024-03-30T00:56:35.488541+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:27:12.615250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468012+0000","last_change":"2024-03-30T00:55:51.821527+0000","last_active":"2024-03-30T00:56:35.468012+0000","last_peered":"2024-03-30T00:56:35.468012+0000","last_clean":"2024-03-30T00:56:35.468012+0000","last_became_active":"2024-03-30T00:55:51.820689+0000","last_became_peered":"2024-03-30T00:55:51.820689+0000","last_unstale":"2024-03-30T00:56:35.468012+0000","last_undegraded":"2024-03-30T00:56:35.468012+0000","last_fullsized":"2024-03-30T00:56:35.468012+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:14:35.735985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470468+0000","last_change":"2024-03-30T00:55:55.824307+0000","last_active":"2024-03-30T00:56:35.470468+0000","last_peered":"2024-03-30T00:56:35.470468+0000","last_clean":"2024-03-30T00:56:35.470468+0000","last_became_active":"2024-03-30T00:55:55.823900+0000","last_became_peered":"2024-03-30T00:55:55.823900+0000","last_unstale":"2024-03-30T00:56:35.470468+0000","last_undegraded":"2024-03-30T00:56:35.470468+0000","last_fullsized":"2024-03-30T00:56:35.470468+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:44:22.139120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'4","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463230+0000","last_change":"2024-03-30T00:55:51.296965+0000","last_active":"2024-03-30T00:56:35.463230+0000","last_peered":"2024-03-30T00:56:35.463230+0000","last_clean":"2024-03-30T00:56:35.463230+0000","last_became_active":"2024-03-30T00:55:51.296645+0000","last_became_peered":"2024-03-30T00:55:51.296645+0000","last_unstale":"2024-03-30T00:56:35.463230+0000","last_undegraded":"2024-03-30T00:56:35.463230+0000","last_fullsized":"2024-03-30T00:56:35.463230+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:53:46.519972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.496024+0000","last_change":"2024-03-30T00:55:51.826282+0000","last_active":"2024-03-30T00:56:35.496024+0000","last_peered":"2024-03-30T00:56:35.496024+0000","last_clean":"2024-03-30T00:56:35.496024+0000","last_became_active":"2024-03-30T00:55:51.823910+0000","last_became_peered":"2024-03-30T00:55:51.823910+0000","last_unstale":"2024-03-30T00:56:35.496024+0000","last_undegraded":"2024-03-30T00:56:35.496024+0000","last_fullsized":"2024-03-30T00:56:35.496024+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:47.454280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462851+0000","last_change":"2024-03-30T00:55:55.856750+0000","last_active":"2024-03-30T00:56:35.462851+0000","last_peered":"2024-03-30T00:56:35.462851+0000","last_clean":"2024-03-30T00:56:35.462851+0000","last_became_active":"2024-03-30T00:55:55.856288+0000","last_became_peered":"2024-03-30T00:55:55.856288+0000","last_unstale":"2024-03-30T00:56:35.462851+0000","last_undegraded":"2024-03-30T00:56:35.462851+0000","last_fullsized":"2024-03-30T00:56:35.462851+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:43:28.085910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467887+0000","last_change":"2024-03-30T00:55:51.306357+0000","last_active":"2024-03-30T00:56:35.467887+0000","last_peered":"2024-03-30T00:56:35.467887+0000","last_clean":"2024-03-30T00:56:35.467887+0000","last_became_active":"2024-03-30T00:55:51.305815+0000","last_became_peered":"2024-03-30T00:55:51.305815+0000","last_unstale":"2024-03-30T00:56:35.467887+0000","last_undegraded":"2024-03-30T00:56:35.467887+0000","last_fullsized":"2024-03-30T00:56:35.467887+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:16:21.753446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462871+0000","last_change":"2024-03-30T00:55:51.810686+0000","last_active":"2024-03-30T00:56:35.462871+0000","last_peered":"2024-03-30T00:56:35.462871+0000","last_clean":"2024-03-30T00:56:35.462871+0000","last_became_active":"2024-03-30T00:55:51.810331+0000","last_became_peered":"2024-03-30T00:55:51.810331+0000","last_unstale":"2024-03-30T00:56:35.462871+0000","last_undegraded":"2024-03-30T00:56:35.462871+0000","last_fullsized":"2024-03-30T00:56:35.462871+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:24:13.750905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.487787+0000","last_change":"2024-03-30T00:55:55.842894+0000","last_active":"2024-03-30T00:56:35.487787+0000","last_peered":"2024-03-30T00:56:35.487787+0000","last_clean":"2024-03-30T00:56:35.487787+0000","last_became_active":"2024-03-30T00:55:55.842118+0000","last_became_peered":"2024-03-30T00:55:55.842118+0000","last_unstale":"2024-03-30T00:56:35.487787+0000","last_undegraded":"2024-03-30T00:56:35.487787+0000","last_fullsized":"2024-03-30T00:56:35.487787+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:39.246228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'3","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470956+0000","last_change":"2024-03-30T00:55:51.313188+0000","last_active":"2024-03-30T00:56:35.470956+0000","last_peered":"2024-03-30T00:56:35.470956+0000","last_clean":"2024-03-30T00:56:35.470956+0000","last_became_active":"2024-03-30T00:55:51.312879+0000","last_became_peered":"2024-03-30T00:55:51.312879+0000","last_unstale":"2024-03-30T00:56:35.470956+0000","last_undegraded":"2024-03-30T00:56:35.470956+0000","last_fullsized":"2024-03-30T00:56:35.470956+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:41:23.420427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460160+0000","last_change":"2024-03-30T00:55:51.808998+0000","last_active":"2024-03-30T00:56:35.460160+0000","last_peered":"2024-03-30T00:56:35.460160+0000","last_clean":"2024-03-30T00:56:35.460160+0000","last_became_active":"2024-03-30T00:55:51.808636+0000","last_became_peered":"2024-03-30T00:55:51.808636+0000","last_unstale":"2024-03-30T00:56:35.460160+0000","last_undegraded":"2024-03-30T00:56:35.460160+0000","last_fullsized":"2024-03-30T00:56:35.460160+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:02.379294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.480854+0000","last_change":"2024-03-30T00:56:36.480854+0000","last_active":"2024-03-30T00:56:36.480854+0000","last_peered":"2024-03-30T00:56:36.480854+0000","last_clean":"2024-03-30T00:56:36.480854+0000","last_became_active":"2024-03-30T00:56:36.467064+0000","last_became_peered":"2024-03-30T00:56:36.467064+0000","last_unstale":"2024-03-30T00:56:36.480854+0000","last_undegraded":"2024-03-30T00:56:36.480854+0000","last_fullsized":"2024-03-30T00:56:36.480854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:12:55.154544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470669+0000","last_change":"2024-03-30T00:55:51.313708+0000","last_active":"2024-03-30T00:56:35.470669+0000","last_peered":"2024-03-30T00:56:35.470669+0000","last_clean":"2024-03-30T00:56:35.470669+0000","last_became_active":"2024-03-30T00:55:51.313362+0000","last_became_peered":"2024-03-30T00:55:51.313362+0000","last_unstale":"2024-03-30T00:56:35.470669+0000","last_undegraded":"2024-03-30T00:56:35.470669+0000","last_fullsized":"2024-03-30T00:56:35.470669+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:03:17.859640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.471657+0000","last_change":"2024-03-30T00:55:51.819638+0000","last_active":"2024-03-30T00:56:35.471657+0000","last_peered":"2024-03-30T00:56:35.471657+0000","last_clean":"2024-03-30T00:56:35.471657+0000","last_became_active":"2024-03-30T00:55:51.819282+0000","last_became_peered":"2024-03-30T00:55:51.819282+0000","last_unstale":"2024-03-30T00:56:35.471657+0000","last_undegraded":"2024-03-30T00:56:35.471657+0000","last_fullsized":"2024-03-30T00:56:35.471657+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:13:39.249550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462909+0000","last_change":"2024-03-30T00:55:55.856498+0000","last_active":"2024-03-30T00:56:35.462909+0000","last_peered":"2024-03-30T00:56:35.462909+0000","last_clean":"2024-03-30T00:56:35.462909+0000","last_became_active":"2024-03-30T00:55:55.856015+0000","last_became_peered":"2024-03-30T00:55:55.856015+0000","last_unstale":"2024-03-30T00:56:35.462909+0000","last_undegraded":"2024-03-30T00:56:35.462909+0000","last_fullsized":"2024-03-30T00:56:35.462909+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:14:00.248812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462909+0000","last_change":"2024-03-30T00:55:51.302995+0000","last_active":"2024-03-30T00:56:35.462909+0000","last_peered":"2024-03-30T00:56:35.462909+0000","last_clean":"2024-03-30T00:56:35.462909+0000","last_became_active":"2024-03-30T00:55:51.302650+0000","last_became_peered":"2024-03-30T00:55:51.302650+0000","last_unstale":"2024-03-30T00:56:35.462909+0000","last_undegraded":"2024-03-30T00:56:35.462909+0000","last_fullsized":"2024-03-30T00:56:35.462909+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:17:32.556550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466402+0000","last_change":"2024-03-30T00:55:51.808969+0000","last_active":"2024-03-30T00:56:35.466402+0000","last_peered":"2024-03-30T00:56:35.466402+0000","last_clean":"2024-03-30T00:56:35.466402+0000","last_became_active":"2024-03-30T00:55:51.808633+0000","last_became_peered":"2024-03-30T00:55:51.808633+0000","last_unstale":"2024-03-30T00:56:35.466402+0000","last_undegraded":"2024-03-30T00:56:35.466402+0000","last_fullsized":"2024-03-30T00:56:35.466402+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:16:46.128066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468267+0000","last_change":"2024-03-30T00:55:55.824964+0000","last_active":"2024-03-30T00:56:35.468267+0000","last_peered":"2024-03-30T00:56:35.468267+0000","last_clean":"2024-03-30T00:56:35.468267+0000","last_became_active":"2024-03-30T00:55:55.824559+0000","last_became_peered":"2024-03-30T00:55:55.824559+0000","last_unstale":"2024-03-30T00:56:35.468267+0000","last_undegraded":"2024-03-30T00:56:35.468267+0000","last_fullsized":"2024-03-30T00:56:35.468267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:19:24.746963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'3","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470912+0000","last_change":"2024-03-30T00:55:51.301602+0000","last_active":"2024-03-30T00:56:35.470912+0000","last_peered":"2024-03-30T00:56:35.470912+0000","last_clean":"2024-03-30T00:56:35.470912+0000","last_became_active":"2024-03-30T00:55:51.301221+0000","last_became_peered":"2024-03-30T00:55:51.301221+0000","last_unstale":"2024-03-30T00:56:35.470912+0000","last_undegraded":"2024-03-30T00:56:35.470912+0000","last_fullsized":"2024-03-30T00:56:35.470912+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:37:36.727603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466837+0000","last_change":"2024-03-30T00:55:51.810197+0000","last_active":"2024-03-30T00:56:35.466837+0000","last_peered":"2024-03-30T00:56:35.466837+0000","last_clean":"2024-03-30T00:56:35.466837+0000","last_became_active":"2024-03-30T00:55:51.809193+0000","last_became_peered":"2024-03-30T00:55:51.809193+0000","last_unstale":"2024-03-30T00:56:35.466837+0000","last_undegraded":"2024-03-30T00:56:35.466837+0000","last_fullsized":"2024-03-30T00:56:35.466837+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:52:41.078425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.494873+0000","last_change":"2024-03-30T00:55:55.830549+0000","last_active":"2024-03-30T00:56:35.494873+0000","last_peered":"2024-03-30T00:56:35.494873+0000","last_clean":"2024-03-30T00:56:35.494873+0000","last_became_active":"2024-03-30T00:55:55.827966+0000","last_became_peered":"2024-03-30T00:55:55.827966+0000","last_unstale":"2024-03-30T00:56:35.494873+0000","last_undegraded":"2024-03-30T00:56:35.494873+0000","last_fullsized":"2024-03-30T00:56:35.494873+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:30:36.531960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"76'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467092+0000","last_change":"2024-03-30T00:55:51.311998+0000","last_active":"2024-03-30T00:56:35.467092+0000","last_peered":"2024-03-30T00:56:35.467092+0000","last_clean":"2024-03-30T00:56:35.467092+0000","last_became_active":"2024-03-30T00:55:51.311627+0000","last_became_peered":"2024-03-30T00:55:51.311627+0000","last_unstale":"2024-03-30T00:56:35.467092+0000","last_undegraded":"2024-03-30T00:56:35.467092+0000","last_fullsized":"2024-03-30T00:56:35.467092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:23:06.939919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466545+0000","last_change":"2024-03-30T00:55:51.819457+0000","last_active":"2024-03-30T00:56:35.466545+0000","last_peered":"2024-03-30T00:56:35.466545+0000","last_clean":"2024-03-30T00:56:35.466545+0000","last_became_active":"2024-03-30T00:55:51.819099+0000","last_became_peered":"2024-03-30T00:55:51.819099+0000","last_unstale":"2024-03-30T00:56:35.466545+0000","last_undegraded":"2024-03-30T00:56:35.466545+0000","last_fullsized":"2024-03-30T00:56:35.466545+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:31:57.363328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466097+0000","last_change":"2024-03-30T00:55:55.824882+0000","last_active":"2024-03-30T00:56:35.466097+0000","last_peered":"2024-03-30T00:56:35.466097+0000","last_clean":"2024-03-30T00:56:35.466097+0000","last_became_active":"2024-03-30T00:55:55.824493+0000","last_became_peered":"2024-03-30T00:55:55.824493+0000","last_unstale":"2024-03-30T00:56:35.466097+0000","last_undegraded":"2024-03-30T00:56:35.466097+0000","last_fullsized":"2024-03-30T00:56:35.466097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:24:48.825776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"76'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.476919+0000","last_change":"2024-03-30T00:55:51.311787+0000","last_active":"2024-03-30T00:56:35.476919+0000","last_peered":"2024-03-30T00:56:35.476919+0000","last_clean":"2024-03-30T00:56:35.476919+0000","last_became_active":"2024-03-30T00:55:51.311402+0000","last_became_peered":"2024-03-30T00:55:51.311402+0000","last_unstale":"2024-03-30T00:56:35.476919+0000","last_undegraded":"2024-03-30T00:56:35.476919+0000","last_fullsized":"2024-03-30T00:56:35.476919+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:37:13.960453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467729+0000","last_change":"2024-03-30T00:55:51.809742+0000","last_active":"2024-03-30T00:56:35.467729+0000","last_peered":"2024-03-30T00:56:35.467729+0000","last_clean":"2024-03-30T00:56:35.467729+0000","last_became_active":"2024-03-30T00:55:51.809396+0000","last_became_peered":"2024-03-30T00:55:51.809396+0000","last_unstale":"2024-03-30T00:56:35.467729+0000","last_undegraded":"2024-03-30T00:56:35.467729+0000","last_fullsized":"2024-03-30T00:56:35.467729+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:22.535472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460238+0000","last_change":"2024-03-30T00:55:51.814429+0000","last_active":"2024-03-30T00:56:35.460238+0000","last_peered":"2024-03-30T00:56:35.460238+0000","last_clean":"2024-03-30T00:56:35.460238+0000","last_became_active":"2024-03-30T00:55:51.814044+0000","last_became_peered":"2024-03-30T00:55:51.814044+0000","last_unstale":"2024-03-30T00:56:35.460238+0000","last_undegraded":"2024-03-30T00:56:35.460238+0000","last_fullsized":"2024-03-30T00:56:35.460238+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:24:58.830738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470585+0000","last_change":"2024-03-30T00:55:51.828977+0000","last_active":"2024-03-30T00:56:35.470585+0000","last_peered":"2024-03-30T00:56:35.470585+0000","last_clean":"2024-03-30T00:56:35.470585+0000","last_became_active":"2024-03-30T00:55:51.828656+0000","last_became_peered":"2024-03-30T00:55:51.828656+0000","last_unstale":"2024-03-30T00:56:35.470585+0000","last_undegraded":"2024-03-30T00:56:35.470585+0000","last_fullsized":"2024-03-30T00:56:35.470585+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:10:23.463770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470569+0000","last_change":"2024-03-30T00:55:51.821511+0000","last_active":"2024-03-30T00:56:35.470569+0000","last_peered":"2024-03-30T00:56:35.470569+0000","last_clean":"2024-03-30T00:56:35.470569+0000","last_became_active":"2024-03-30T00:55:51.820876+0000","last_became_peered":"2024-03-30T00:55:51.820876+0000","last_unstale":"2024-03-30T00:56:35.470569+0000","last_undegraded":"2024-03-30T00:56:35.470569+0000","last_fullsized":"2024-03-30T00:56:35.470569+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:14:48.299376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495025+0000","last_change":"2024-03-30T00:55:51.823867+0000","last_active":"2024-03-30T00:56:35.495025+0000","last_peered":"2024-03-30T00:56:35.495025+0000","last_clean":"2024-03-30T00:56:35.495025+0000","last_became_active":"2024-03-30T00:55:51.820230+0000","last_became_peered":"2024-03-30T00:55:51.820230+0000","last_unstale":"2024-03-30T00:56:35.495025+0000","last_undegraded":"2024-03-30T00:56:35.495025+0000","last_fullsized":"2024-03-30T00:56:35.495025+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:38:51.257855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.475716+0000","last_change":"2024-03-30T00:55:51.821413+0000","last_active":"2024-03-30T00:56:35.475716+0000","last_peered":"2024-03-30T00:56:35.475716+0000","last_clean":"2024-03-30T00:56:35.475716+0000","last_became_active":"2024-03-30T00:55:51.820833+0000","last_became_peered":"2024-03-30T00:55:51.820833+0000","last_unstale":"2024-03-30T00:56:35.475716+0000","last_undegraded":"2024-03-30T00:56:35.475716+0000","last_fullsized":"2024-03-30T00:56:35.475716+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:34:11.969849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460621+0000","last_change":"2024-03-30T00:55:51.813674+0000","last_active":"2024-03-30T00:56:35.460621+0000","last_peered":"2024-03-30T00:56:35.460621+0000","last_clean":"2024-03-30T00:56:35.460621+0000","last_became_active":"2024-03-30T00:55:51.813341+0000","last_became_peered":"2024-03-30T00:55:51.813341+0000","last_unstale":"2024-03-30T00:56:35.460621+0000","last_undegraded":"2024-03-30T00:56:35.460621+0000","last_fullsized":"2024-03-30T00:56:35.460621+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:54:17.285997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.471317+0000","last_change":"2024-03-30T00:55:51.812472+0000","last_active":"2024-03-30T00:56:35.471317+0000","last_peered":"2024-03-30T00:56:35.471317+0000","last_clean":"2024-03-30T00:56:35.471317+0000","last_became_active":"2024-03-30T00:55:51.812061+0000","last_became_peered":"2024-03-30T00:55:51.812061+0000","last_unstale":"2024-03-30T00:56:35.471317+0000","last_undegraded":"2024-03-30T00:56:35.471317+0000","last_fullsized":"2024-03-30T00:56:35.471317+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:46.550201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468639+0000","last_change":"2024-03-30T00:55:51.829179+0000","last_active":"2024-03-30T00:56:35.468639+0000","last_peered":"2024-03-30T00:56:35.468639+0000","last_clean":"2024-03-30T00:56:35.468639+0000","last_became_active":"2024-03-30T00:55:51.828805+0000","last_became_peered":"2024-03-30T00:55:51.828805+0000","last_unstale":"2024-03-30T00:56:35.468639+0000","last_undegraded":"2024-03-30T00:56:35.468639+0000","last_fullsized":"2024-03-30T00:56:35.468639+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:23.576125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466369+0000","last_change":"2024-03-30T00:55:51.823170+0000","last_active":"2024-03-30T00:56:35.466369+0000","last_peered":"2024-03-30T00:56:35.466369+0000","last_clean":"2024-03-30T00:56:35.466369+0000","last_became_active":"2024-03-30T00:55:51.822785+0000","last_became_peered":"2024-03-30T00:55:51.822785+0000","last_unstale":"2024-03-30T00:56:35.466369+0000","last_undegraded":"2024-03-30T00:56:35.466369+0000","last_fullsized":"2024-03-30T00:56:35.466369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:34:43.393540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.469874+0000","last_change":"2024-03-30T00:55:51.817160+0000","last_active":"2024-03-30T00:56:35.469874+0000","last_peered":"2024-03-30T00:56:35.469874+0000","last_clean":"2024-03-30T00:56:35.469874+0000","last_became_active":"2024-03-30T00:55:51.816881+0000","last_became_peered":"2024-03-30T00:55:51.816881+0000","last_unstale":"2024-03-30T00:56:35.469874+0000","last_undegraded":"2024-03-30T00:56:35.469874+0000","last_fullsized":"2024-03-30T00:56:35.469874+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:09:05.211242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468766+0000","last_change":"2024-03-30T00:55:51.817670+0000","last_active":"2024-03-30T00:56:35.468766+0000","last_peered":"2024-03-30T00:56:35.468766+0000","last_clean":"2024-03-30T00:56:35.468766+0000","last_became_active":"2024-03-30T00:55:51.817380+0000","last_became_peered":"2024-03-30T00:55:51.817380+0000","last_unstale":"2024-03-30T00:56:35.468766+0000","last_undegraded":"2024-03-30T00:56:35.468766+0000","last_fullsized":"2024-03-30T00:56:35.468766+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:24.582557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.475181+0000","last_change":"2024-03-30T00:55:51.813071+0000","last_active":"2024-03-30T00:56:35.475181+0000","last_peered":"2024-03-30T00:56:35.475181+0000","last_clean":"2024-03-30T00:56:35.475181+0000","last_became_active":"2024-03-30T00:55:51.811681+0000","last_became_peered":"2024-03-30T00:55:51.811681+0000","last_unstale":"2024-03-30T00:56:35.475181+0000","last_undegraded":"2024-03-30T00:56:35.475181+0000","last_fullsized":"2024-03-30T00:56:35.475181+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:10:53.840055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.472592+0000","last_change":"2024-03-30T00:56:36.472592+0000","last_active":"2024-03-30T00:56:36.472592+0000","last_peered":"2024-03-30T00:56:36.472592+0000","last_clean":"2024-03-30T00:56:36.472592+0000","last_became_active":"2024-03-30T00:56:36.472173+0000","last_became_peered":"2024-03-30T00:56:36.472173+0000","last_unstale":"2024-03-30T00:56:36.472592+0000","last_undegraded":"2024-03-30T00:56:36.472592+0000","last_fullsized":"2024-03-30T00:56:36.472592+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:22:27.915984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":212960,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65332,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":251,"num_read_kb":193,"num_write":156,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":195996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138,"ondisk_log_size":138,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645336,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":913408,"data_stored":462418,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.376}]},{"osd":1,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.40000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":4,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.505},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.57699999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.62},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54800000000000004}]},{"osd":6,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.71799999999999997}]},{"osd":7,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.68999999999999995}]},{"osd":8,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.749},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.67700000000000005}]},{"osd":9,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.70099999999999996}]},{"osd":10,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.76900000000000002}]}]},{"osd":10,"up_from":65,"seq":279172874267,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1114794,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.64600000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.437}]},{"osd":2,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52900000000000003}]},{"osd":3,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.629}]},{"osd":4,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.60599999999999998}]},{"osd":5,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.61599999999999999}]},{"osd":6,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.69499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":8,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.67500000000000004}]},{"osd":9,"last update":"Sat Mar 30 00:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.76300000000000001}]},{"osd":11,"last update":"Sat Mar 30 00:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.81100000000000005}]}]},{"osd":9,"up_from":60,"seq":257698037789,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":925696,"data_stored":462620,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.628,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.44600000000000001}]},{"osd":1,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.94199999999999995,"5min":1.274,"15min":1.274},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.73399999999999999,"15min":0.74299999999999999},"min":{"1min":0.51600000000000001,"5min":0.504,"15min":0.504},"max":{"1min":0.90600000000000003,"5min":1.411,"15min":1.411},"last":0.622}]},{"osd":2,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.73399999999999999,"15min":0.74299999999999999},"min":{"1min":0.55400000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.84699999999999998,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.501,"5min":0.435,"15min":0.435},"max":{"1min":0.83999999999999997,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.68799999999999994}]},{"osd":3,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.105,"5min":1.391,"15min":1.391},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.73999999999999999,"15min":0.75900000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95899999999999996,"5min":1.302,"15min":1.302},"last":0.497}]},{"osd":4,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69199999999999995,"15min":0.69799999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.88700000000000001,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.81100000000000005,"15min":0.82299999999999995},"min":{"1min":0.56200000000000006,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98299999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.60999999999999999}]},{"osd":5,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.71899999999999997,"15min":0.72899999999999998},"min":{"1min":0.45800000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.88300000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.83799999999999997,"15min":0.85799999999999998},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0129999999999999,"5min":1.853,"15min":1.853},"last":0.71799999999999997}]},{"osd":6,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.623}]},{"osd":7,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.86399999999999999,"15min":0.88},"min":{"1min":0.54100000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.94099999999999995,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.83299999999999996,"15min":0.84399999999999997},"min":{"1min":0.58699999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94599999999999995,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.59899999999999998}]},{"osd":8,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.90100000000000002,"15min":0.92000000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.98099999999999998,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.90000000000000002,"15min":0.92200000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0720000000000001,"5min":1.968,"15min":1.968},"last":0.69599999999999995}]},{"osd":10,"last update":"Sat Mar 30 00:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.47599999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":8,"up_from":53,"seq":227633266723,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":462209,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.44},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.46500000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57099999999999995,"15min":0.56599999999999995},"min":{"1min":0.38600000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.46400000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.60699999999999998}]},{"osd":2,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.41999999999999998},"min":{"1min":0.29199999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.45100000000000001,"15min":0.44500000000000001},"min":{"1min":0.316,"5min":0.312,"15min":0.312},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.56100000000000005}]},{"osd":3,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.70899999999999996,"15min":0.73099999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74199999999999999,"5min":2.883,"15min":2.883},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58099999999999996,"15min":0.57699999999999996},"min":{"1min":0.436,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.52400000000000002}]},{"osd":4,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.60699999999999998,"15min":0.59899999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.73999999999999999}]},{"osd":5,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.496,"15min":0.48899999999999999},"min":{"1min":0.35199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.34599999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.53500000000000003}]},{"osd":6,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.53200000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.876,"5min":2.8940000000000001,"15min":2.8940000000000001},"last":0.751}]},{"osd":7,"last update":"Sat Mar 30 00:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.66000000000000003,"15min":0.65400000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.64900000000000002}]},{"osd":9,"last update":"Sat Mar 30 00:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.70999999999999996,"15min":0.69899999999999995},"min":{"1min":0.58999999999999997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93500000000000005,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.74299999999999999,"15min":0.72599999999999998},"min":{"1min":0.68600000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.81699999999999995}]},{"osd":10,"last update":"Sat Mar 30 00:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.84199999999999997}]},{"osd":11,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.88}]}]},{"osd":7,"up_from":46,"seq":197568495654,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":1058417,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60199999999999998}]},{"osd":1,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.45200000000000001,"15min":0.44800000000000001},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.60599999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.432},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.44700000000000001,"15min":0.441},"min":{"1min":0.32500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68799999999999994,"5min":0.747,"15min":0.747},"last":0.54700000000000004}]},{"osd":2,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.46700000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.57399999999999995,"15min":0.56100000000000005},"min":{"1min":0.51800000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.57299999999999995}]},{"osd":3,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.40400000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.89600000000000002,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61599999999999999,"15min":0.60999999999999999},"min":{"1min":0.53200000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89400000000000002,"5min":0.997,"15min":0.997},"last":0.63200000000000001}]},{"osd":4,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.36599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.39200000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.61599999999999999}]},{"osd":5,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.629,"15min":0.61899999999999999},"min":{"1min":0.505,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.63200000000000001,"15min":0.62},"min":{"1min":0.58099999999999996,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86199999999999999,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.52400000000000002}]},{"osd":6,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.66500000000000004,"15min":0.65500000000000003},"min":{"1min":0.58799999999999997,"5min":0.372,"15min":0.372},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.745,"15min":0.72099999999999997},"min":{"1min":0.55300000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":4.117,"5min":4.117,"15min":4.117},"last":0.65800000000000003}]},{"osd":8,"last update":"Sat Mar 30 00:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.56299999999999994,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95199999999999996,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.755,"15min":0.752},"min":{"1min":0.53400000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.76100000000000001}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.74399999999999999,"15min":0.73999999999999999},"min":{"1min":0.60299999999999998,"5min":0.495,"15min":0.495},"max":{"1min":0.94199999999999995,"5min":1.018,"15min":1.018},"last":0.747},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.91400000000000003,"15min":0.91900000000000004},"min":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":3.8839999999999999,"5min":4.0519999999999996,"15min":4.0519999999999996},"last":0.73099999999999998}]},{"osd":10,"last update":"Sat Mar 30 00:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.71399999999999997}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.78900000000000003}]}]},{"osd":6,"up_from":42,"seq":180388626474,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":905216,"data_stored":457796,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.308,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.499,"15min":0.48699999999999999},"min":{"1min":0.373,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.47399999999999998}]},{"osd":1,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.66300000000000003,"15min":0.63500000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60599999999999998,"15min":0.59599999999999997},"min":{"1min":0.39300000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.50600000000000001}]},{"osd":2,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70899999999999996,"15min":0.68300000000000005},"min":{"1min":0.51000000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.98699999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.66500000000000004,"15min":0.64300000000000002},"min":{"1min":0.47299999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.64800000000000002}]},{"osd":3,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.65000000000000002},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.97999999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.46400000000000002}]},{"osd":4,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67600000000000005,"15min":0.67200000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.76300000000000001,"15min":0.751},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.105,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.753}]},{"osd":5,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69399999999999995,"15min":0.68200000000000005},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.93899999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.74099999999999999,"15min":0.73099999999999998},"min":{"1min":0.51600000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Mar 30 00:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81799999999999995,"15min":0.81399999999999995},"min":{"1min":0.64300000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.1200000000000001,"5min":1.127,"15min":1.127},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.77600000000000002,"15min":0.77200000000000002},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.73599999999999999}]},{"osd":8,"last update":"Sat Mar 30 00:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.84499999999999997,"15min":0.85199999999999998},"min":{"1min":0.52200000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.2789999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.876,"5min":0.89000000000000001,"15min":0.89200000000000002},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.155,"5min":1.206,"15min":1.206},"last":0.85099999999999998}]},{"osd":9,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.94299999999999995,"15min":0.95199999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.244,"5min":1.268,"15min":1.268},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.347,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.875}]},{"osd":10,"last update":"Sat Mar 30 00:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.89300000000000002}]},{"osd":11,"last update":"Sat Mar 30 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.80200000000000005}]}]},{"osd":1,"up_from":12,"seq":51539607612,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":518465,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55500000000000005,"15min":0.53500000000000003},"min":{"1min":0.42399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72099999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.58099999999999996,"15min":0.55300000000000005},"min":{"1min":0.47199999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.79100000000000004,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.68500000000000005}]},{"osd":2,"last update":"Sat Mar 30 00:56:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.52800000000000002,"15min":0.49299999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.53400000000000003,"15min":0.5},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.877}]},{"osd":3,"last update":"Sat Mar 30 00:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56200000000000006,"15min":0.52800000000000002},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.69199999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57399999999999995,"15min":0.55000000000000004},"min":{"1min":0.501,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.66900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.64500000000000002}]},{"osd":4,"last update":"Sat Mar 30 00:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.41799999999999998,"15min":0.40799999999999997},"min":{"1min":0.33500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.61899999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.48199999999999998,"15min":0.46600000000000003},"min":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.66900000000000004}]},{"osd":5,"last update":"Sat Mar 30 00:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72199999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64100000000000001,"15min":0.63600000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83899999999999997,"5min":0.877,"15min":0.877},"last":0.89800000000000002}]},{"osd":6,"last update":"Sat Mar 30 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.55700000000000005,"5min":0.435,"15min":0.435},"max":{"1min":0.81699999999999995,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.54800000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.84799999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.752}]},{"osd":7,"last update":"Sat Mar 30 00:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55400000000000005,"15min":0.55100000000000005},"min":{"1min":0.35799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79400000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.84899999999999998,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.81799999999999995}]},{"osd":8,"last update":"Sat Mar 30 00:56:28 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.79100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.624,"15min":0.625},"min":{"1min":0.41999999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81599999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.73899999999999999}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.746,"15min":0.73699999999999999},"min":{"1min":0.56899999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.91600000000000004,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72999999999999998,"15min":0.72399999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.86099999999999999}]},{"osd":10,"last update":"Sat Mar 30 00:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.94299999999999995}]},{"osd":11,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.96399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738433,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32020,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713132,"statfs":{"total":95995035648,"available":95962247168,"internally_reserved":0,"allocated":925696,"data_stored":458911,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 00:56:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.40000000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.88600000000000001,"5min":1.038,"15min":1.038},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.57099999999999995,"15min":0.56000000000000005},"min":{"1min":0.49399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":1.125}]},{"osd":2,"last update":"Sat Mar 30 00:56:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.56299999999999994,"15min":0.53200000000000003},"min":{"1min":0.41899999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.0940000000000001,"5min":0.68300000000000005,"15min":0.59299999999999997},"min":{"1min":0.48399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":9.0150000000000006,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":0.90300000000000002}]},{"osd":3,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.9139999999999997,"5min":7.9139999999999997,"15min":7.9139999999999997},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.96,"5min":7.96,"15min":7.96},"last":1.044}]},{"osd":4,"last update":"Sat Mar 30 00:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65100000000000002,"15min":0.623},"min":{"1min":0.503,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90000000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.64100000000000001,"15min":0.60999999999999999},"min":{"1min":0.51400000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.93300000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":1.1879999999999999}]},{"osd":5,"last update":"Sat Mar 30 00:56:35 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":0.74399999999999999,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":9.0579999999999998,"5min":9.0579999999999998,"15min":9.0579999999999998},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.871,"5min":0.68400000000000005,"15min":0.63600000000000001},"min":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":3.8039999999999998,"5min":3.8039999999999998,"15min":3.8039999999999998},"last":1.157}]},{"osd":6,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":0.72099999999999997,"15min":0.65900000000000003},"min":{"1min":0.42399999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":8.0050000000000008,"5min":8.0050000000000008,"15min":8.0050000000000008},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.66800000000000004,"15min":0.65800000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.76900000000000002}]},{"osd":7,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.4140000000000001,"5min":2.4140000000000001,"15min":2.4140000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":9.2959999999999994,"5min":9.2959999999999994,"15min":9.2959999999999994},"last":1.3360000000000001}]},{"osd":8,"last update":"Sat Mar 30 00:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.55000000000000004,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0489999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.83799999999999997,"15min":0.85199999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0589999999999999,"5min":1.276,"15min":1.276},"last":1.3009999999999999}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":0.88700000000000001,"15min":0.83699999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":8.0389999999999997,"5min":8.0389999999999997,"15min":8.0389999999999997},"last":1.421},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.78400000000000003,"15min":0.76600000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":1.022}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":1.47},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.84499999999999997}]},{"osd":11,"last update":"Sat Mar 30 00:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":1.2290000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":1.2689999999999999}]}]},{"osd":2,"up_from":17,"seq":73014444090,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":519730,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53700000000000003,"15min":0.52800000000000002},"min":{"1min":0.36099999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.84899999999999998,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.52500000000000002},"min":{"1min":0.38300000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76200000000000001,"5min":1.097,"15min":1.097},"last":0.443}]},{"osd":1,"last update":"Sat Mar 30 00:56:38 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.53700000000000003,"15min":0.51400000000000001},"min":{"1min":0.36899999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81000000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54300000000000004,"15min":0.52900000000000003},"min":{"1min":0.45600000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70199999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.64100000000000001}]},{"osd":3,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57899999999999996,"15min":0.55900000000000005},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.58899999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.46899999999999997}]},{"osd":4,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.624},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.58699999999999997}]},{"osd":5,"last update":"Sat Mar 30 00:56:30 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.442,"15min":0.43099999999999999},"min":{"1min":0.30399999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60199999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.46700000000000003,"15min":0.45600000000000002},"min":{"1min":0.28000000000000003,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.56699999999999995}]},{"osd":6,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65100000000000002,"15min":0.65000000000000002},"min":{"1min":0.44600000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86399999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66500000000000004,"15min":0.66100000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.70399999999999996}]},{"osd":7,"last update":"Sat Mar 30 00:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.65800000000000003}]},{"osd":8,"last update":"Sat Mar 30 00:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82799999999999996,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59699999999999998,"15min":0.59599999999999997},"min":{"1min":0.35799999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81100000000000005,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.71899999999999997}]},{"osd":9,"last update":"Sat Mar 30 00:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70399999999999996,"15min":0.70099999999999996},"min":{"1min":0.52100000000000002,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.73499999999999999,"15min":0.73099999999999998},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.75}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.76000000000000001}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.82199999999999995}]}]},{"osd":3,"up_from":24,"seq":103079215157,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":457717,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.46000000000000002,"15min":0.41399999999999998},"min":{"1min":0.32900000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46400000000000002,"15min":0.432},"min":{"1min":0.35199999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.997,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.376}]},{"osd":1,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65100000000000002,"15min":0.60299999999999998},"min":{"1min":0.45000000000000001,"5min":0.375,"15min":0.375},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.628,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.439,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.59099999999999997}]},{"osd":2,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.67200000000000004,"15min":0.629},"min":{"1min":0.51400000000000001,"5min":0.371,"15min":0.371},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64800000000000002,"15min":0.622},"min":{"1min":0.47599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.69899999999999995}]},{"osd":4,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.69499999999999995,"15min":0.66300000000000003},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0600000000000001,"5min":1.113,"15min":1.113},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.67500000000000004,"15min":0.66400000000000003},"min":{"1min":0.501,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.83399999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.63100000000000001}]},{"osd":5,"last update":"Sat Mar 30 00:56:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65500000000000003,"15min":0.64600000000000002},"min":{"1min":0.46300000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93000000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.69799999999999995,"15min":0.66400000000000003},"min":{"1min":0.48799999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.117,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.67700000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.65000000000000002,"15min":0.63800000000000001},"min":{"1min":0.46800000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.44,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.81200000000000006}]},{"osd":7,"last update":"Sat Mar 30 00:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.97899999999999998,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.751},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83999999999999997,"15min":0.84099999999999997},"min":{"1min":0.66000000000000003,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.1890000000000001,"5min":1.292,"15min":1.292},"last":0.75900000000000001}]},{"osd":8,"last update":"Sat Mar 30 00:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.57399999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.92400000000000004,"15min":0.94399999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.2410000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.84499999999999997,"15min":0.85899999999999999},"min":{"1min":0.54700000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.224,"5min":1.347,"15min":1.347},"last":0.66900000000000004}]},{"osd":10,"last update":"Sat Mar 30 00:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.82599999999999996}]},{"osd":11,"last update":"Sat Mar 30 00:56:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.74299999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986226,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":909312,"data_stored":462187,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.66700000000000004,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.79300000000000004,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73099999999999998,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39700000000000002,"15min":0.39100000000000001},"min":{"1min":0.29999999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.61199999999999999,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.32000000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.46899999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.47299999999999998}]},{"osd":2,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.55400000000000005,"15min":0.54500000000000004},"min":{"1min":0.36399999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81799999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.55400000000000005,"15min":0.53900000000000003},"min":{"1min":0.441,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76000000000000001,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.71699999999999997}]},{"osd":3,"last update":"Sat Mar 30 00:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.66000000000000003,"15min":0.69799999999999995},"min":{"1min":0.42399999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.72999999999999998,"5min":3.887,"15min":3.887},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.39600000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78400000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.53300000000000003}]},{"osd":5,"last update":"Sat Mar 30 00:56:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57499999999999996,"15min":0.56399999999999995},"min":{"1min":0.39800000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.78100000000000003,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58399999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.76300000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.752}]},{"osd":6,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.623,"15min":0.625},"min":{"1min":0.42699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.64800000000000002,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82599999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58199999999999996}]},{"osd":7,"last update":"Sat Mar 30 00:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54400000000000004,"15min":0.54000000000000004},"min":{"1min":0.35199999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.79500000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.499},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52700000000000002,"15min":0.52100000000000002},"min":{"1min":0.34599999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.69499999999999995}]},{"osd":8,"last update":"Sat Mar 30 00:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78700000000000003,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.46000000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.85199999999999998,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.78900000000000003}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.69899999999999995},"min":{"1min":0.53500000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.73299999999999998,"15min":0.73699999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.98999999999999999,"5min":1.012,"15min":1.012},"last":0.80000000000000004}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.77800000000000002}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.83299999999999996}]}]},{"osd":5,"up_from":35,"seq":150323855406,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":978944,"data_stored":1053660,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51400000000000001,"15min":0.502},"min":{"1min":0.37,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78700000000000003,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.50900000000000001,"15min":0.497},"min":{"1min":0.38500000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.78800000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.69599999999999995}]},{"osd":1,"last update":"Sat Mar 30 00:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56799999999999995,"15min":0.56399999999999995},"min":{"1min":0.378,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55500000000000005,"15min":0.54100000000000004},"min":{"1min":0.45700000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72899999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.64800000000000002}]},{"osd":2,"last update":"Sat Mar 30 00:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.39700000000000002,"15min":0.38900000000000001},"min":{"1min":0.313,"5min":0.255,"15min":0.255},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.502},{"interface":"front","average":{"1min":0.44,"5min":0.39800000000000002,"15min":0.39600000000000002},"min":{"1min":0.307,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.64400000000000002,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.44900000000000001}]},{"osd":3,"last update":"Sat Mar 30 00:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54000000000000004,"15min":0.53700000000000003},"min":{"1min":0.36799999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75800000000000001,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54200000000000004,"15min":0.53000000000000003},"min":{"1min":0.40799999999999997,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.78300000000000003,"5min":0.877,"15min":0.877},"last":0.61099999999999999}]},{"osd":4,"last update":"Sat Mar 30 00:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57299999999999995,"15min":0.56100000000000005},"min":{"1min":0.49299999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.626},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.59399999999999997,"15min":0.58199999999999996},"min":{"1min":0.47399999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76000000000000001,"5min":1.038,"15min":1.038},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Mar 30 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59899999999999998,"15min":0.58099999999999996},"min":{"1min":0.42799999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81200000000000006,"5min":0.997,"15min":0.997},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60499999999999998,"15min":0.59799999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.72999999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.68600000000000005}]},{"osd":7,"last update":"Sat Mar 30 00:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.42699999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82199999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63600000000000001,"15min":0.63700000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89500000000000002,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.70999999999999996}]},{"osd":8,"last update":"Sat Mar 30 00:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.53700000000000003,"15min":0.53100000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.55300000000000005,"15min":0.54400000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85399999999999998,"5min":0.872,"15min":0.872},"last":0.48099999999999998}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.67300000000000004,"15min":0.66700000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68400000000000005,"15min":0.68100000000000005},"min":{"1min":0.58099999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.73299999999999998}]},{"osd":10,"last update":"Sat Mar 30 00:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.754}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":62693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T00:57:03.095 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-30T00:57:03.095 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-30T00:57:03.095 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-30T00:57:03.095 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph health --format=json 2024-03-30T00:57:03.464 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:57:04.330 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:04 smithi116 ceph-mon[38086]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:04.331 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:04 smithi116 ceph-mon[38086]: pgmap v267: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T00:57:04.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.577+0000 7f5a80a00700 1 -- 172.21.15.116:0/3336918707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7c100e60 msgr2=0x7f5a7c1012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:04.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.577+0000 7f5a80a00700 1 --2- 172.21.15.116:0/3336918707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7c100e60 0x7f5a7c1012e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5a680099c0 tx=0x7f5a68009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:04.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.578+0000 7f5a80a00700 1 -- 172.21.15.116:0/3336918707 shutdown_connections 2024-03-30T00:57:04.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.578+0000 7f5a80a00700 1 --2- 172.21.15.116:0/3336918707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a7c101820 0x7f5a7c10dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:04.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.578+0000 7f5a80a00700 1 --2- 172.21.15.116:0/3336918707 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a7c106ef0 0x7f5a7c1072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:04.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.578+0000 7f5a80a00700 1 --2- 172.21.15.116:0/3336918707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7c100e60 0x7f5a7c1012e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:04.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.578+0000 7f5a80a00700 1 -- 172.21.15.116:0/3336918707 >> 172.21.15.116:0/3336918707 conn(0x7f5a7c077fa0 msgr2=0x7f5a7c0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:04.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.578+0000 7f5a80a00700 1 -- 172.21.15.116:0/3336918707 shutdown_connections 2024-03-30T00:57:04.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.578+0000 7f5a80a00700 1 -- 172.21.15.116:0/3336918707 wait complete. 2024-03-30T00:57:04.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.579+0000 7f5a80a00700 1 Processor -- start 2024-03-30T00:57:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.579+0000 7f5a80a00700 1 -- start start 2024-03-30T00:57:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.579+0000 7f5a80a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a7c100e60 0x7f5a7c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.579+0000 7f5a80a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a7c101820 0x7f5a7c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.579+0000 7f5a80a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7c106ef0 0x7f5a7c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.579+0000 7f5a80a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c1a55a0 con 0x7f5a7c106ef0 2024-03-30T00:57:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.579+0000 7f5a80a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c1a5710 con 0x7f5a7c100e60 2024-03-30T00:57:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.579+0000 7f5a80a00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c1a58d0 con 0x7f5a7c101820 2024-03-30T00:57:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.580+0000 7f5a79e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a7c100e60 0x7f5a7c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.580+0000 7f5a79e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a7c100e60 0x7f5a7c0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:35672/0 (socket says 172.21.15.116:35672) 2024-03-30T00:57:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.580+0000 7f5a79e00700 1 -- 172.21.15.116:0/3867133406 learned_addr learned my addr 172.21.15.116:0/3867133406 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.580+0000 7f5a7a800700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7c106ef0 0x7f5a7c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.580+0000 7f5a79400700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a7c101820 0x7f5a7c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.580+0000 7f5a79400700 1 -- 172.21.15.116:0/3867133406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a7c100e60 msgr2=0x7f5a7c0730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.580+0000 7f5a79400700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a7c100e60 0x7f5a7c0730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.580+0000 7f5a79400700 1 -- 172.21.15.116:0/3867133406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7c106ef0 msgr2=0x7f5a7c073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.580+0000 7f5a79400700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7c106ef0 0x7f5a7c073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.580+0000 7f5a79400700 1 -- 172.21.15.116:0/3867133406 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a680095f0 con 0x7f5a7c101820 2024-03-30T00:57:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.580+0000 7f5a7a800700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7c106ef0 0x7f5a7c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.580+0000 7f5a79400700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a7c101820 0x7f5a7c0735f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5a680098e0 tx=0x7f5a6800b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.581+0000 7f5a72a00700 1 -- 172.21.15.116:0/3867133406 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6801d070 con 0x7f5a7c101820 2024-03-30T00:57:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.581+0000 7f5a80a00700 1 -- 172.21.15.116:0/3867133406 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7c1a5b50 con 0x7f5a7c101820 2024-03-30T00:57:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.581+0000 7f5a72a00700 1 -- 172.21.15.116:0/3867133406 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a6800bde0 con 0x7f5a7c101820 2024-03-30T00:57:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.581+0000 7f5a72a00700 1 -- 172.21.15.116:0/3867133406 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6800fa10 con 0x7f5a7c101820 2024-03-30T00:57:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.581+0000 7f5a80a00700 1 -- 172.21.15.116:0/3867133406 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7c1a5fc0 con 0x7f5a7c101820 2024-03-30T00:57:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.582+0000 7f5a80a00700 1 -- 172.21.15.116:0/3867133406 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7c1092f0 con 0x7f5a7c101820 2024-03-30T00:57:04.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.584+0000 7f5a72a00700 1 -- 172.21.15.116:0/3867133406 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f5a68004cd0 con 0x7f5a7c101820 2024-03-30T00:57:04.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.585+0000 7f5a72a00700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5a600a11b0 0x7f5a600a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:04.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.585+0000 7f5a72a00700 1 -- 172.21.15.116:0/3867133406 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5a680cda60 con 0x7f5a7c101820 2024-03-30T00:57:04.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.585+0000 7f5a79e00700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5a600a11b0 0x7f5a600a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:04.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.586+0000 7f5a79e00700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5a600a11b0 0x7f5a600a3670 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009ca0 tx=0x7f5a6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:04.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.589+0000 7f5a72a00700 1 -- 172.21.15.116:0/3867133406 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5a68096db0 con 0x7f5a7c101820 2024-03-30T00:57:04.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:04 smithi130 ceph-mon[47626]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:04.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:04 smithi130 ceph-mon[47626]: pgmap v267: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T00:57:04.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:04 smithi181 ceph-mon[47622]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:04.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:04 smithi181 ceph-mon[47622]: pgmap v267: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T00:57:04.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.831+0000 7f5a80a00700 1 -- 172.21.15.116:0/3867133406 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5a7c074590 con 0x7f5a7c101820 2024-03-30T00:57:04.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.832+0000 7f5a72a00700 1 -- 172.21.15.116:0/3867133406 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f5a6809a3d0 con 0x7f5a7c101820 2024-03-30T00:57:04.838 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:04.838 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-30T00:57:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.834+0000 7f5a80a00700 1 -- 172.21.15.116:0/3867133406 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5a600a11b0 msgr2=0x7f5a600a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.834+0000 7f5a80a00700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5a600a11b0 0x7f5a600a3670 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009ca0 tx=0x7f5a6c009260 comp rx=0 tx=0).stop 2024-03-30T00:57:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.834+0000 7f5a80a00700 1 -- 172.21.15.116:0/3867133406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a7c101820 msgr2=0x7f5a7c0735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.834+0000 7f5a80a00700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a7c101820 0x7f5a7c0735f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5a680098e0 tx=0x7f5a6800b9a0 comp rx=0 tx=0).stop 2024-03-30T00:57:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.834+0000 7f5a80a00700 1 -- 172.21.15.116:0/3867133406 shutdown_connections 2024-03-30T00:57:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.834+0000 7f5a80a00700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5a600a11b0 0x7f5a600a3670 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.834+0000 7f5a80a00700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7c106ef0 0x7f5a7c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.834+0000 7f5a80a00700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a7c100e60 0x7f5a7c0730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.834+0000 7f5a80a00700 1 --2- 172.21.15.116:0/3867133406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a7c101820 0x7f5a7c0735f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.834+0000 7f5a80a00700 1 -- 172.21.15.116:0/3867133406 >> 172.21.15.116:0/3867133406 conn(0x7f5a7c077fa0 msgr2=0x7f5a7c0ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.835+0000 7f5a80a00700 1 -- 172.21.15.116:0/3867133406 shutdown_connections 2024-03-30T00:57:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:04.835+0000 7f5a80a00700 1 -- 172.21.15.116:0/3867133406 wait complete. 2024-03-30T00:57:05.227 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-30T00:57:05.228 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:57:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.423+0000 7f797c800700 1 -- 172.21.15.116:0/1096781461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7974100c80 msgr2=0x7f79741010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.423+0000 7f797c800700 1 --2- 172.21.15.116:0/1096781461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7974100c80 0x7f79741010a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f79680099c0 tx=0x7f7968009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.423+0000 7f797c800700 1 -- 172.21.15.116:0/1096781461 shutdown_connections 2024-03-30T00:57:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.423+0000 7f797c800700 1 --2- 172.21.15.116:0/1096781461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7974102810 0x7f79741090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.423+0000 7f797c800700 1 --2- 172.21.15.116:0/1096781461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7974100c80 0x7f79741010a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.423+0000 7f797c800700 1 --2- 172.21.15.116:0/1096781461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7974101e70 0x7f79741022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.423+0000 7f797c800700 1 -- 172.21.15.116:0/1096781461 >> 172.21.15.116:0/1096781461 conn(0x7f79740fc1f0 msgr2=0x7f79740fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.423+0000 7f797c800700 1 -- 172.21.15.116:0/1096781461 shutdown_connections 2024-03-30T00:57:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.423+0000 7f797c800700 1 -- 172.21.15.116:0/1096781461 wait complete. 2024-03-30T00:57:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.424+0000 7f797c800700 1 Processor -- start 2024-03-30T00:57:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.424+0000 7f797c800700 1 -- start start 2024-03-30T00:57:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.424+0000 7f797c800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7974100c80 0x7f7974198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.424+0000 7f797c800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7974101e70 0x7f7974199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.424+0000 7f797c800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7974102810 0x7f79741a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.424+0000 7f797c800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79741a0930 con 0x7f7974100c80 2024-03-30T00:57:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.424+0000 7f797c800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79741a0a70 con 0x7f7974101e70 2024-03-30T00:57:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.424+0000 7f797c800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79741a0c00 con 0x7f7974102810 2024-03-30T00:57:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.425+0000 7f7979e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7974100c80 0x7f7974198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.425+0000 7f7979400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7974101e70 0x7f7974199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.425+0000 7f7979e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7974100c80 0x7f7974198e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45854/0 (socket says 172.21.15.116:45854) 2024-03-30T00:57:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.425+0000 7f7979e00700 1 -- 172.21.15.116:0/2212647451 learned_addr learned my addr 172.21.15.116:0/2212647451 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.425+0000 7f7979400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7974101e70 0x7f7974199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34418/0 (socket says 172.21.15.116:34418) 2024-03-30T00:57:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.425+0000 7f797a800700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7974102810 0x7f79741a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.425+0000 7f7979e00700 1 -- 172.21.15.116:0/2212647451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7974102810 msgr2=0x7f79741a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.425+0000 7f7979e00700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7974102810 0x7f79741a0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.425+0000 7f7979e00700 1 -- 172.21.15.116:0/2212647451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7974101e70 msgr2=0x7f7974199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.425+0000 7f7979e00700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7974101e70 0x7f7974199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.425+0000 7f7979e00700 1 -- 172.21.15.116:0/2212647451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79680095f0 con 0x7f7974100c80 2024-03-30T00:57:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.425+0000 7f7979400700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7974101e70 0x7f7974199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.425+0000 7f797a800700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7974102810 0x7f79741a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.425+0000 7f7979e00700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7974100c80 0x7f7974198e00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7968006010 tx=0x7f796800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.426+0000 7f796ea00700 1 -- 172.21.15.116:0/2212647451 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796801d070 con 0x7f7974100c80 2024-03-30T00:57:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.426+0000 7f796ea00700 1 -- 172.21.15.116:0/2212647451 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7968005020 con 0x7f7974100c80 2024-03-30T00:57:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.426+0000 7f797c800700 1 -- 172.21.15.116:0/2212647451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79741a0e80 con 0x7f7974100c80 2024-03-30T00:57:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.426+0000 7f796ea00700 1 -- 172.21.15.116:0/2212647451 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968004bb0 con 0x7f7974100c80 2024-03-30T00:57:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.426+0000 7f797c800700 1 -- 172.21.15.116:0/2212647451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79741a11f0 con 0x7f7974100c80 2024-03-30T00:57:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.427+0000 7f797c800700 1 -- 172.21.15.116:0/2212647451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7974067a80 con 0x7f7974100c80 2024-03-30T00:57:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.429+0000 7f796ea00700 1 -- 172.21.15.116:0/2212647451 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f7968005190 con 0x7f7974100c80 2024-03-30T00:57:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.430+0000 7f796ea00700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f795c0a11b0 0x7f795c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.430+0000 7f796ea00700 1 -- 172.21.15.116:0/2212647451 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f79680cd840 con 0x7f7974100c80 2024-03-30T00:57:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.430+0000 7f7979400700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f795c0a11b0 0x7f795c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.431+0000 7f7979400700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f795c0a11b0 0x7f795c0a3670 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7964011490 tx=0x7f7964009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.434+0000 7f796ea00700 1 -- 172.21.15.116:0/2212647451 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7974067a80 con 0x7f7974100c80 2024-03-30T00:57:05.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:05 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3867133406' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T00:57:05.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:05 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3867133406' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T00:57:05.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.675+0000 7f797c800700 1 -- 172.21.15.116:0/2212647451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f797410c4e0 con 0x7f7974100c80 2024-03-30T00:57:05.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.676+0000 7f796ea00700 1 -- 172.21.15.116:0/2212647451 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+24556 (secure 0 0 0) 0x7f7968096bf0 con 0x7f7974100c80 2024-03-30T00:57:05.682 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:05.682 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:56:36.456069+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:57:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.679+0000 7f797c800700 1 -- 172.21.15.116:0/2212647451 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f795c0a11b0 msgr2=0x7f795c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.679+0000 7f797c800700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f795c0a11b0 0x7f795c0a3670 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7964011490 tx=0x7f7964009260 comp rx=0 tx=0).stop 2024-03-30T00:57:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.679+0000 7f797c800700 1 -- 172.21.15.116:0/2212647451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7974100c80 msgr2=0x7f7974198e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.679+0000 7f797c800700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7974100c80 0x7f7974198e00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7968006010 tx=0x7f796800f690 comp rx=0 tx=0).stop 2024-03-30T00:57:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.679+0000 7f797c800700 1 -- 172.21.15.116:0/2212647451 shutdown_connections 2024-03-30T00:57:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.679+0000 7f797c800700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f795c0a11b0 0x7f795c0a3670 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.679+0000 7f797c800700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7974102810 0x7f79741a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.679+0000 7f797c800700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7974100c80 0x7f7974198e00 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.679+0000 7f797c800700 1 --2- 172.21.15.116:0/2212647451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7974101e70 0x7f7974199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.679+0000 7f797c800700 1 -- 172.21.15.116:0/2212647451 >> 172.21.15.116:0/2212647451 conn(0x7f79740fc1f0 msgr2=0x7f7974072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.679+0000 7f797c800700 1 -- 172.21.15.116:0/2212647451 shutdown_connections 2024-03-30T00:57:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.679+0000 7f797c800700 1 -- 172.21.15.116:0/2212647451 wait complete. 2024-03-30T00:57:05.704 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T00:52:29.602428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T00:55:50.220543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T00:55:51.274498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T00:55:54.301591+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T00:57:05.704 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T00:57:05.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:05 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3867133406' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T00:57:05.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.900+0000 7f8805200700 1 -- 172.21.15.116:0/3876599614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8800102910 msgr2=0x7f88001091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:05.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.900+0000 7f8805200700 1 --2- 172.21.15.116:0/3876599614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8800102910 0x7f88001091c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f87f800b0d0 tx=0x7f87f800b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.900+0000 7f8805200700 1 -- 172.21.15.116:0/3876599614 shutdown_connections 2024-03-30T00:57:05.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.900+0000 7f8805200700 1 --2- 172.21.15.116:0/3876599614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8800102910 0x7f88001091c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.900+0000 7f8805200700 1 --2- 172.21.15.116:0/3876599614 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8800100d80 0x7f88001011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.900+0000 7f8805200700 1 --2- 172.21.15.116:0/3876599614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8800101f70 0x7f88001023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.900+0000 7f8805200700 1 -- 172.21.15.116:0/3876599614 >> 172.21.15.116:0/3876599614 conn(0x7f88000fc2f0 msgr2=0x7f88000fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:05.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.900+0000 7f8805200700 1 -- 172.21.15.116:0/3876599614 shutdown_connections 2024-03-30T00:57:05.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.900+0000 7f8805200700 1 -- 172.21.15.116:0/3876599614 wait complete. 2024-03-30T00:57:05.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.901+0000 7f8805200700 1 Processor -- start 2024-03-30T00:57:05.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.901+0000 7f8805200700 1 -- start start 2024-03-30T00:57:05.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.901+0000 7f8805200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8800100d80 0x7f8800198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:05.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.901+0000 7f8805200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8800101f70 0x7f8800199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:05.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.901+0000 7f8805200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8800102910 0x7f88001a0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:05.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.901+0000 7f8805200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88001a0b20 con 0x7f8800101f70 2024-03-30T00:57:05.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.901+0000 7f8805200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88001a0c60 con 0x7f8800102910 2024-03-30T00:57:05.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.901+0000 7f8805200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88001a0df0 con 0x7f8800100d80 2024-03-30T00:57:05.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87fd400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8800101f70 0x7f8800199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:05.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87fd400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8800101f70 0x7f8800199530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45886/0 (socket says 172.21.15.116:45886) 2024-03-30T00:57:05.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87fe800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8800102910 0x7f88001a0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:05.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87fd400700 1 -- 172.21.15.116:0/671666026 learned_addr learned my addr 172.21.15.116:0/671666026 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:05.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87fde00700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8800100d80 0x7f8800198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:05.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87fe800700 1 -- 172.21.15.116:0/671666026 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8800100d80 msgr2=0x7f8800198ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:05.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87fe800700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8800100d80 0x7f8800198ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87fe800700 1 -- 172.21.15.116:0/671666026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8800101f70 msgr2=0x7f8800199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:05.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87fe800700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8800101f70 0x7f8800199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:05.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87fe800700 1 -- 172.21.15.116:0/671666026 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f8009d00 con 0x7f8800102910 2024-03-30T00:57:05.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87fde00700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8800100d80 0x7f8800198ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:05.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87fd400700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8800101f70 0x7f8800199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:05.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87fe800700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8800102910 0x7f88001a0550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f87f8007b60 tx=0x7f87f8008e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:05.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87f6a00700 1 -- 172.21.15.116:0/671666026 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f800e040 con 0x7f8800102910 2024-03-30T00:57:05.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f8805200700 1 -- 172.21.15.116:0/671666026 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88001a1070 con 0x7f8800102910 2024-03-30T00:57:05.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.902+0000 7f87f6a00700 1 -- 172.21.15.116:0/671666026 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87f8005070 con 0x7f8800102910 2024-03-30T00:57:05.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.903+0000 7f87f6a00700 1 -- 172.21.15.116:0/671666026 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f801d6d0 con 0x7f8800102910 2024-03-30T00:57:05.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.903+0000 7f8805200700 1 -- 172.21.15.116:0/671666026 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88001a1440 con 0x7f8800102910 2024-03-30T00:57:05.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.904+0000 7f8805200700 1 -- 172.21.15.116:0/671666026 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8800067af0 con 0x7f8800102910 2024-03-30T00:57:05.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.906+0000 7f87f6a00700 1 -- 172.21.15.116:0/671666026 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f87f80051e0 con 0x7f8800102910 2024-03-30T00:57:05.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.907+0000 7f87f6a00700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f87e40a11b0 0x7f87e40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:05.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.907+0000 7f87f6a00700 1 -- 172.21.15.116:0/671666026 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f87f80cdcf0 con 0x7f8800102910 2024-03-30T00:57:05.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.907+0000 7f87fde00700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f87e40a11b0 0x7f87e40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:05.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.908+0000 7f87fde00700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f87e40a11b0 0x7f87e40a3670 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8800101dd0 tx=0x7f87f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:05.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:05.911+0000 7f87f6a00700 1 -- 172.21.15.116:0/671666026 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f87f8097040 con 0x7f8800102910 2024-03-30T00:57:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.114+0000 7f8805200700 1 -- 172.21.15.116:0/671666026 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f880010c7d0 con 0x7f8800102910 2024-03-30T00:57:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.115+0000 7f87f6a00700 1 -- 172.21.15.116:0/671666026 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f87f809a660 con 0x7f8800102910 2024-03-30T00:57:06.121 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-30T00:57:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.117+0000 7f8805200700 1 -- 172.21.15.116:0/671666026 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f87e40a11b0 msgr2=0x7f87e40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.117+0000 7f8805200700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f87e40a11b0 0x7f87e40a3670 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8800101dd0 tx=0x7f87f0009260 comp rx=0 tx=0).stop 2024-03-30T00:57:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.117+0000 7f8805200700 1 -- 172.21.15.116:0/671666026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8800102910 msgr2=0x7f88001a0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.117+0000 7f8805200700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8800102910 0x7f88001a0550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f87f8007b60 tx=0x7f87f8008e40 comp rx=0 tx=0).stop 2024-03-30T00:57:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.117+0000 7f8805200700 1 -- 172.21.15.116:0/671666026 shutdown_connections 2024-03-30T00:57:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.117+0000 7f8805200700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f87e40a11b0 0x7f87e40a3670 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.117+0000 7f8805200700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8800100d80 0x7f8800198ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.117+0000 7f8805200700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8800102910 0x7f88001a0550 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.117+0000 7f8805200700 1 --2- 172.21.15.116:0/671666026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8800101f70 0x7f8800199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.117+0000 7f8805200700 1 -- 172.21.15.116:0/671666026 >> 172.21.15.116:0/671666026 conn(0x7f88000fc2f0 msgr2=0x7f8800072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.118+0000 7f8805200700 1 -- 172.21.15.116:0/671666026 shutdown_connections 2024-03-30T00:57:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.118+0000 7f8805200700 1 -- 172.21.15.116:0/671666026 wait complete. 2024-03-30T00:57:06.142 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T00:57:06.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.331+0000 7f3996a00700 1 -- 172.21.15.116:0/223554949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3990102910 msgr2=0x7f39901091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:06.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.331+0000 7f3996a00700 1 --2- 172.21.15.116:0/223554949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3990102910 0x7f39901091c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f398800b0d0 tx=0x7f398800b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.332+0000 7f3996a00700 1 -- 172.21.15.116:0/223554949 shutdown_connections 2024-03-30T00:57:06.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.332+0000 7f3996a00700 1 --2- 172.21.15.116:0/223554949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3990102910 0x7f39901091c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.332+0000 7f3996a00700 1 --2- 172.21.15.116:0/223554949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3990100d80 0x7f39901011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.332+0000 7f3996a00700 1 --2- 172.21.15.116:0/223554949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3990101f70 0x7f39901023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.332+0000 7f3996a00700 1 -- 172.21.15.116:0/223554949 >> 172.21.15.116:0/223554949 conn(0x7f39900fc2f0 msgr2=0x7f39900fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:06.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.332+0000 7f3996a00700 1 -- 172.21.15.116:0/223554949 shutdown_connections 2024-03-30T00:57:06.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.332+0000 7f3996a00700 1 -- 172.21.15.116:0/223554949 wait complete. 2024-03-30T00:57:06.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.333+0000 7f3996a00700 1 Processor -- start 2024-03-30T00:57:06.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.333+0000 7f3996a00700 1 -- start start 2024-03-30T00:57:06.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.333+0000 7f3996a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3990100d80 0x7f3990074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.333+0000 7f3996a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3990101f70 0x7f39900731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.333+0000 7f3996a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3990102910 0x7f39900736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.333+0000 7f3996a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3990073e80 con 0x7f3990100d80 2024-03-30T00:57:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.333+0000 7f3996a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3990073ff0 con 0x7f3990102910 2024-03-30T00:57:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.333+0000 7f3996a00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39901a0e60 con 0x7f3990101f70 2024-03-30T00:57:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.333+0000 7f398fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3990100d80 0x7f3990074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.333+0000 7f398fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3990100d80 0x7f3990074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45898/0 (socket says 172.21.15.116:45898) 2024-03-30T00:57:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.333+0000 7f398fe00700 1 -- 172.21.15.116:0/3213250418 learned_addr learned my addr 172.21.15.116:0/3213250418 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.333+0000 7f398f400700 1 --2- 172.21.15.116:0/3213250418 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3990101f70 0x7f39900731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.333+0000 7f3994a00700 1 --2- 172.21.15.116:0/3213250418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3990102910 0x7f39900736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.334+0000 7f398fe00700 1 -- 172.21.15.116:0/3213250418 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3990101f70 msgr2=0x7f39900731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:06.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.334+0000 7f398fe00700 1 --2- 172.21.15.116:0/3213250418 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3990101f70 0x7f39900731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.334+0000 7f398fe00700 1 -- 172.21.15.116:0/3213250418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3990102910 msgr2=0x7f39900736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:06.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.334+0000 7f398fe00700 1 --2- 172.21.15.116:0/3213250418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3990102910 0x7f39900736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.334+0000 7f398fe00700 1 -- 172.21.15.116:0/3213250418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3988009d00 con 0x7f3990100d80 2024-03-30T00:57:06.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.334+0000 7f398fe00700 1 --2- 172.21.15.116:0/3213250418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3990100d80 0x7f3990074b10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f398000e970 tx=0x7f398000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:06.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.334+0000 7f398cc00700 1 -- 172.21.15.116:0/3213250418 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980009790 con 0x7f3990100d80 2024-03-30T00:57:06.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.334+0000 7f398cc00700 1 -- 172.21.15.116:0/3213250418 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f398000c780 con 0x7f3990100d80 2024-03-30T00:57:06.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.334+0000 7f398cc00700 1 -- 172.21.15.116:0/3213250418 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980003be0 con 0x7f3990100d80 2024-03-30T00:57:06.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.334+0000 7f3996a00700 1 -- 172.21.15.116:0/3213250418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39901a10f0 con 0x7f3990100d80 2024-03-30T00:57:06.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.334+0000 7f3996a00700 1 -- 172.21.15.116:0/3213250418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39901a1600 con 0x7f3990100d80 2024-03-30T00:57:06.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.336+0000 7f3996a00700 1 -- 172.21.15.116:0/3213250418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3990108130 con 0x7f3990100d80 2024-03-30T00:57:06.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.337+0000 7f398cc00700 1 -- 172.21.15.116:0/3213250418 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f398000cbf0 con 0x7f3990100d80 2024-03-30T00:57:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.339+0000 7f398cc00700 1 --2- 172.21.15.116:0/3213250418 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f39740a1230 0x7f39740a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.340+0000 7f398cc00700 1 -- 172.21.15.116:0/3213250418 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3980014070 con 0x7f3990100d80 2024-03-30T00:57:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.340+0000 7f398f400700 1 --2- 172.21.15.116:0/3213250418 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f39740a1230 0x7f39740a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:06.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.340+0000 7f398f400700 1 --2- 172.21.15.116:0/3213250418 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f39740a1230 0x7f39740a36f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f397c009c60 tx=0x7f397c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:06.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.343+0000 7f398cc00700 1 -- 172.21.15.116:0/3213250418 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f39800961b0 con 0x7f3990100d80 2024-03-30T00:57:06.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:06 smithi116 ceph-mon[38086]: pgmap v268: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-30T00:57:06.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:06 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2212647451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:06.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:06 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/671666026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:06.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.547+0000 7f3996a00700 1 -- 172.21.15.116:0/3213250418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3990107b10 con 0x7f3990100d80 2024-03-30T00:57:06.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.547+0000 7f398cc00700 1 -- 172.21.15.116:0/3213250418 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f39800997d0 con 0x7f3990100d80 2024-03-30T00:57:06.553 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:57:06.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.550+0000 7f3996a00700 1 -- 172.21.15.116:0/3213250418 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f39740a1230 msgr2=0x7f39740a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:06.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.550+0000 7f3996a00700 1 --2- 172.21.15.116:0/3213250418 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f39740a1230 0x7f39740a36f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f397c009c60 tx=0x7f397c009260 comp rx=0 tx=0).stop 2024-03-30T00:57:06.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.550+0000 7f3996a00700 1 -- 172.21.15.116:0/3213250418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3990100d80 msgr2=0x7f3990074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:06.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.550+0000 7f3996a00700 1 --2- 172.21.15.116:0/3213250418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3990100d80 0x7f3990074b10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f398000e970 tx=0x7f398000ec80 comp rx=0 tx=0).stop 2024-03-30T00:57:06.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.550+0000 7f3996a00700 1 -- 172.21.15.116:0/3213250418 shutdown_connections 2024-03-30T00:57:06.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.550+0000 7f3996a00700 1 --2- 172.21.15.116:0/3213250418 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f39740a1230 0x7f39740a36f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.550+0000 7f3996a00700 1 --2- 172.21.15.116:0/3213250418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3990102910 0x7f39900736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.550+0000 7f3996a00700 1 --2- 172.21.15.116:0/3213250418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3990100d80 0x7f3990074b10 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.550+0000 7f3996a00700 1 --2- 172.21.15.116:0/3213250418 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3990101f70 0x7f39900731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.550+0000 7f3996a00700 1 -- 172.21.15.116:0/3213250418 >> 172.21.15.116:0/3213250418 conn(0x7f39900fc2f0 msgr2=0x7f3990105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:06.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.550+0000 7f3996a00700 1 -- 172.21.15.116:0/3213250418 shutdown_connections 2024-03-30T00:57:06.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.550+0000 7f3996a00700 1 -- 172.21.15.116:0/3213250418 wait complete. 2024-03-30T00:57:06.575 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T00:57:06.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:06 smithi130 ceph-mon[47626]: pgmap v268: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-30T00:57:06.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:06 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2212647451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:06.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:06 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/671666026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:06.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:06 smithi181 ceph-mon[47622]: pgmap v268: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-30T00:57:06.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:06 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2212647451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:06.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:06 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/671666026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.763+0000 7f054da00700 1 -- 172.21.15.116:0/940159912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0548102910 msgr2=0x7f05481091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.763+0000 7f054da00700 1 --2- 172.21.15.116:0/940159912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0548102910 0x7f05481091c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f054000b0d0 tx=0x7f054000b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.763+0000 7f054da00700 1 -- 172.21.15.116:0/940159912 shutdown_connections 2024-03-30T00:57:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.763+0000 7f054da00700 1 --2- 172.21.15.116:0/940159912 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0548100d80 0x7f05481011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.763+0000 7f054da00700 1 --2- 172.21.15.116:0/940159912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0548102910 0x7f05481091c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.763+0000 7f054da00700 1 --2- 172.21.15.116:0/940159912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0548101f70 0x7f05481023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.763+0000 7f054da00700 1 -- 172.21.15.116:0/940159912 >> 172.21.15.116:0/940159912 conn(0x7f05480fc2f0 msgr2=0x7f05480fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.763+0000 7f054da00700 1 -- 172.21.15.116:0/940159912 shutdown_connections 2024-03-30T00:57:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.763+0000 7f054da00700 1 -- 172.21.15.116:0/940159912 wait complete. 2024-03-30T00:57:06.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.764+0000 7f054da00700 1 Processor -- start 2024-03-30T00:57:06.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.764+0000 7f054da00700 1 -- start start 2024-03-30T00:57:06.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.764+0000 7f054da00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0548100d80 0x7f0548199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:06.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.764+0000 7f054da00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0548101f70 0x7f0548199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:06.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.764+0000 7f054da00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0548102910 0x7f05481a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.764+0000 7f054da00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05481a0b50 con 0x7f0548102910 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.764+0000 7f054da00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05481a0c90 con 0x7f0548100d80 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.764+0000 7f054da00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05481a0e20 con 0x7f0548101f70 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.765+0000 7f0547400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0548102910 0x7f05481a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.765+0000 7f0547400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0548102910 0x7f05481a0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45916/0 (socket says 172.21.15.116:45916) 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.765+0000 7f0547400700 1 -- 172.21.15.116:0/1525579784 learned_addr learned my addr 172.21.15.116:0/1525579784 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.765+0000 7f0546000700 1 --2- 172.21.15.116:0/1525579784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0548101f70 0x7f0548199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.765+0000 7f0546a00700 1 --2- 172.21.15.116:0/1525579784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0548100d80 0x7f0548199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.765+0000 7f0547400700 1 -- 172.21.15.116:0/1525579784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0548101f70 msgr2=0x7f0548199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.765+0000 7f0547400700 1 --2- 172.21.15.116:0/1525579784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0548101f70 0x7f0548199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.765+0000 7f0547400700 1 -- 172.21.15.116:0/1525579784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0548100d80 msgr2=0x7f0548199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.765+0000 7f0547400700 1 --2- 172.21.15.116:0/1525579784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0548100d80 0x7f0548199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.765+0000 7f0547400700 1 -- 172.21.15.116:0/1525579784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0540009d00 con 0x7f0548102910 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.765+0000 7f0547400700 1 --2- 172.21.15.116:0/1525579784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0548102910 0x7f05481a0580 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0540007b60 tx=0x7f0540008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.765+0000 7f053f400700 1 -- 172.21.15.116:0/1525579784 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054000e040 con 0x7f0548102910 2024-03-30T00:57:06.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.765+0000 7f053f400700 1 -- 172.21.15.116:0/1525579784 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f054001d930 con 0x7f0548102910 2024-03-30T00:57:06.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.765+0000 7f054da00700 1 -- 172.21.15.116:0/1525579784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05481a10a0 con 0x7f0548102910 2024-03-30T00:57:06.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.766+0000 7f053f400700 1 -- 172.21.15.116:0/1525579784 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054001b9f0 con 0x7f0548102910 2024-03-30T00:57:06.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.766+0000 7f054da00700 1 -- 172.21.15.116:0/1525579784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05481a1520 con 0x7f0548102910 2024-03-30T00:57:06.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.767+0000 7f054da00700 1 -- 172.21.15.116:0/1525579784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0548067af0 con 0x7f0548102910 2024-03-30T00:57:06.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.768+0000 7f053f400700 1 -- 172.21.15.116:0/1525579784 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f054001daa0 con 0x7f0548102910 2024-03-30T00:57:06.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.770+0000 7f053f400700 1 --2- 172.21.15.116:0/1525579784 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f052c0a1230 0x7f052c0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:06.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.770+0000 7f053f400700 1 -- 172.21.15.116:0/1525579784 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f05400ce8a0 con 0x7f0548102910 2024-03-30T00:57:06.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.770+0000 7f0546a00700 1 --2- 172.21.15.116:0/1525579784 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f052c0a1230 0x7f052c0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:06.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.770+0000 7f0546a00700 1 --2- 172.21.15.116:0/1525579784 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f052c0a1230 0x7f052c0a36f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0548101dd0 tx=0x7f0538009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:06.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.772+0000 7f053f400700 1 -- 172.21.15.116:0/1525579784 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0540097bf0 con 0x7f0548102910 2024-03-30T00:57:06.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.976+0000 7f054da00700 1 -- 172.21.15.116:0/1525579784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f054810c7d0 con 0x7f0548102910 2024-03-30T00:57:06.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.976+0000 7f053f400700 1 -- 172.21.15.116:0/1525579784 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f054009b210 con 0x7f0548102910 2024-03-30T00:57:06.982 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-30T00:57:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.979+0000 7f054da00700 1 -- 172.21.15.116:0/1525579784 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f052c0a1230 msgr2=0x7f052c0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.979+0000 7f054da00700 1 --2- 172.21.15.116:0/1525579784 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f052c0a1230 0x7f052c0a36f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0548101dd0 tx=0x7f0538009260 comp rx=0 tx=0).stop 2024-03-30T00:57:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.979+0000 7f054da00700 1 -- 172.21.15.116:0/1525579784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0548102910 msgr2=0x7f05481a0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.979+0000 7f054da00700 1 --2- 172.21.15.116:0/1525579784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0548102910 0x7f05481a0580 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0540007b60 tx=0x7f0540008e40 comp rx=0 tx=0).stop 2024-03-30T00:57:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.979+0000 7f054da00700 1 -- 172.21.15.116:0/1525579784 shutdown_connections 2024-03-30T00:57:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.979+0000 7f054da00700 1 --2- 172.21.15.116:0/1525579784 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f052c0a1230 0x7f052c0a36f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.979+0000 7f054da00700 1 --2- 172.21.15.116:0/1525579784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0548102910 0x7f05481a0580 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.979+0000 7f054da00700 1 --2- 172.21.15.116:0/1525579784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0548100d80 0x7f0548199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.979+0000 7f054da00700 1 --2- 172.21.15.116:0/1525579784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0548101f70 0x7f0548199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.979+0000 7f054da00700 1 -- 172.21.15.116:0/1525579784 >> 172.21.15.116:0/1525579784 conn(0x7f05480fc2f0 msgr2=0x7f0548072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.979+0000 7f054da00700 1 -- 172.21.15.116:0/1525579784 shutdown_connections 2024-03-30T00:57:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:06.979+0000 7f054da00700 1 -- 172.21.15.116:0/1525579784 wait complete. 2024-03-30T00:57:07.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T00:57:07.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.196+0000 7f040ae00700 1 -- 172.21.15.116:0/3090637169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0404102910 msgr2=0x7f04041091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:07.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.196+0000 7f040ae00700 1 --2- 172.21.15.116:0/3090637169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0404102910 0x7f04041091c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00b0d0 tx=0x7f03fc00b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.196+0000 7f040ae00700 1 -- 172.21.15.116:0/3090637169 shutdown_connections 2024-03-30T00:57:07.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.196+0000 7f040ae00700 1 --2- 172.21.15.116:0/3090637169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0404100d80 0x7f04041011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.196+0000 7f040ae00700 1 --2- 172.21.15.116:0/3090637169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0404102910 0x7f04041091c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.196+0000 7f040ae00700 1 --2- 172.21.15.116:0/3090637169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0404101f70 0x7f04041023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.196+0000 7f040ae00700 1 -- 172.21.15.116:0/3090637169 >> 172.21.15.116:0/3090637169 conn(0x7f04040fc2f0 msgr2=0x7f04040fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:07.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.196+0000 7f040ae00700 1 -- 172.21.15.116:0/3090637169 shutdown_connections 2024-03-30T00:57:07.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.196+0000 7f040ae00700 1 -- 172.21.15.116:0/3090637169 wait complete. 2024-03-30T00:57:07.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.197+0000 7f040ae00700 1 Processor -- start 2024-03-30T00:57:07.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.197+0000 7f040ae00700 1 -- start start 2024-03-30T00:57:07.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.197+0000 7f040ae00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0404100d80 0x7f0404074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:07.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.197+0000 7f040ae00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0404101f70 0x7f04040731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:07.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.197+0000 7f040ae00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0404102910 0x7f04040736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:07.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.197+0000 7f040ae00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0404073e80 con 0x7f0404100d80 2024-03-30T00:57:07.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.197+0000 7f040ae00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0404073ff0 con 0x7f0404102910 2024-03-30T00:57:07.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.197+0000 7f040ae00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04041a0e80 con 0x7f0404101f70 2024-03-30T00:57:07.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.198+0000 7f0403e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0404100d80 0x7f0404074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:07.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.198+0000 7f0403e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0404100d80 0x7f0404074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45944/0 (socket says 172.21.15.116:45944) 2024-03-30T00:57:07.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.198+0000 7f0403e00700 1 -- 172.21.15.116:0/770903080 learned_addr learned my addr 172.21.15.116:0/770903080 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:07.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.198+0000 7f0403400700 1 --2- 172.21.15.116:0/770903080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0404101f70 0x7f04040731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:07.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.198+0000 7f0408e00700 1 --2- 172.21.15.116:0/770903080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0404102910 0x7f04040736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:07.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.198+0000 7f0403e00700 1 -- 172.21.15.116:0/770903080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0404101f70 msgr2=0x7f04040731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:07.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.198+0000 7f0403e00700 1 --2- 172.21.15.116:0/770903080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0404101f70 0x7f04040731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.198+0000 7f0403e00700 1 -- 172.21.15.116:0/770903080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0404102910 msgr2=0x7f04040736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:07.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.198+0000 7f0403e00700 1 --2- 172.21.15.116:0/770903080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0404102910 0x7f04040736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.198+0000 7f0403e00700 1 -- 172.21.15.116:0/770903080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03fc009d00 con 0x7f0404100d80 2024-03-30T00:57:07.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.198+0000 7f0403e00700 1 --2- 172.21.15.116:0/770903080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0404100d80 0x7f0404074b10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f03f400e970 tx=0x7f03f400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:07.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.198+0000 7f0400c00700 1 -- 172.21.15.116:0/770903080 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f4009790 con 0x7f0404100d80 2024-03-30T00:57:07.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.199+0000 7f0400c00700 1 -- 172.21.15.116:0/770903080 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03f400c780 con 0x7f0404100d80 2024-03-30T00:57:07.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.199+0000 7f040ae00700 1 -- 172.21.15.116:0/770903080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04041a1110 con 0x7f0404100d80 2024-03-30T00:57:07.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.199+0000 7f0400c00700 1 -- 172.21.15.116:0/770903080 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f4003be0 con 0x7f0404100d80 2024-03-30T00:57:07.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.199+0000 7f040ae00700 1 -- 172.21.15.116:0/770903080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04041a1620 con 0x7f0404100d80 2024-03-30T00:57:07.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.200+0000 7f040ae00700 1 -- 172.21.15.116:0/770903080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0404108130 con 0x7f0404100d80 2024-03-30T00:57:07.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.201+0000 7f0400c00700 1 -- 172.21.15.116:0/770903080 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f03f400cbf0 con 0x7f0404100d80 2024-03-30T00:57:07.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.204+0000 7f0400c00700 1 --2- 172.21.15.116:0/770903080 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f03e80a11b0 0x7f03e80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:07.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.204+0000 7f0400c00700 1 -- 172.21.15.116:0/770903080 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f03f4014070 con 0x7f0404100d80 2024-03-30T00:57:07.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.204+0000 7f0403400700 1 --2- 172.21.15.116:0/770903080 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f03e80a11b0 0x7f03e80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:07.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.204+0000 7f0403400700 1 --2- 172.21.15.116:0/770903080 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f03e80a11b0 0x7f03e80a3670 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f03f0009c60 tx=0x7f03f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:07.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.208+0000 7f0400c00700 1 -- 172.21.15.116:0/770903080 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f03f4096260 con 0x7f0404100d80 2024-03-30T00:57:07.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.416+0000 7f040ae00700 1 -- 172.21.15.116:0/770903080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f040410c820 con 0x7f0404100d80 2024-03-30T00:57:07.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.417+0000 7f0400c00700 1 -- 172.21.15.116:0/770903080 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v83) v1 ==== 103+0+11 (secure 0 0 0) 0x7f03f4099880 con 0x7f0404100d80 2024-03-30T00:57:07.423 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:57:07.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.419+0000 7f040ae00700 1 -- 172.21.15.116:0/770903080 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f03e80a11b0 msgr2=0x7f03e80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:07.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.419+0000 7f040ae00700 1 --2- 172.21.15.116:0/770903080 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f03e80a11b0 0x7f03e80a3670 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f03f0009c60 tx=0x7f03f0009260 comp rx=0 tx=0).stop 2024-03-30T00:57:07.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.419+0000 7f040ae00700 1 -- 172.21.15.116:0/770903080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0404100d80 msgr2=0x7f0404074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:07.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.419+0000 7f040ae00700 1 --2- 172.21.15.116:0/770903080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0404100d80 0x7f0404074b10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f03f400e970 tx=0x7f03f400ec80 comp rx=0 tx=0).stop 2024-03-30T00:57:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.419+0000 7f040ae00700 1 -- 172.21.15.116:0/770903080 shutdown_connections 2024-03-30T00:57:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.419+0000 7f040ae00700 1 --2- 172.21.15.116:0/770903080 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f03e80a11b0 0x7f03e80a3670 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.419+0000 7f040ae00700 1 --2- 172.21.15.116:0/770903080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0404102910 0x7f04040736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.419+0000 7f040ae00700 1 --2- 172.21.15.116:0/770903080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0404100d80 0x7f0404074b10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.420+0000 7f040ae00700 1 --2- 172.21.15.116:0/770903080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0404101f70 0x7f04040731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.420+0000 7f040ae00700 1 -- 172.21.15.116:0/770903080 >> 172.21.15.116:0/770903080 conn(0x7f04040fc2f0 msgr2=0x7f0404105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.420+0000 7f040ae00700 1 -- 172.21.15.116:0/770903080 shutdown_connections 2024-03-30T00:57:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.420+0000 7f040ae00700 1 -- 172.21.15.116:0/770903080 wait complete. 2024-03-30T00:57:07.444 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:57:07.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:07 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3213250418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:07.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:07 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1525579784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.630+0000 7fb92ca00700 1 -- 172.21.15.116:0/4120094652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb928102810 msgr2=0x7fb9281090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.630+0000 7fb92ca00700 1 --2- 172.21.15.116:0/4120094652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb928102810 0x7fb9281090c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb92000b0d0 tx=0x7fb92000b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.630+0000 7fb92ca00700 1 -- 172.21.15.116:0/4120094652 shutdown_connections 2024-03-30T00:57:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.630+0000 7fb92ca00700 1 --2- 172.21.15.116:0/4120094652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb928102810 0x7fb9281090c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.630+0000 7fb92ca00700 1 --2- 172.21.15.116:0/4120094652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb928100c80 0x7fb9281010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.630+0000 7fb92ca00700 1 --2- 172.21.15.116:0/4120094652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb928101e70 0x7fb9281022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.630+0000 7fb92ca00700 1 -- 172.21.15.116:0/4120094652 >> 172.21.15.116:0/4120094652 conn(0x7fb9280fc210 msgr2=0x7fb9280fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.630+0000 7fb92ca00700 1 -- 172.21.15.116:0/4120094652 shutdown_connections 2024-03-30T00:57:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.630+0000 7fb92ca00700 1 -- 172.21.15.116:0/4120094652 wait complete. 2024-03-30T00:57:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.631+0000 7fb92ca00700 1 Processor -- start 2024-03-30T00:57:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.631+0000 7fb92ca00700 1 -- start start 2024-03-30T00:57:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.631+0000 7fb92ca00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb928100c80 0x7fb928198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.631+0000 7fb92ca00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb928101e70 0x7fb928199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb92ca00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb928102810 0x7fb9281a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb92ca00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9281a0a50 con 0x7fb928101e70 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb92ca00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9281a0b90 con 0x7fb928102810 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb92ca00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9281a0d20 con 0x7fb928100c80 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb925400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb928101e70 0x7fb928199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb925400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb928101e70 0x7fb928199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45964/0 (socket says 172.21.15.116:45964) 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb925400700 1 -- 172.21.15.116:0/1523115764 learned_addr learned my addr 172.21.15.116:0/1523115764 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb925e00700 1 --2- 172.21.15.116:0/1523115764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb928100c80 0x7fb928198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb926800700 1 --2- 172.21.15.116:0/1523115764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb928102810 0x7fb9281a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb925400700 1 -- 172.21.15.116:0/1523115764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb928100c80 msgr2=0x7fb928198f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb925400700 1 --2- 172.21.15.116:0/1523115764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb928100c80 0x7fb928198f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb925400700 1 -- 172.21.15.116:0/1523115764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb928102810 msgr2=0x7fb9281a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb925400700 1 --2- 172.21.15.116:0/1523115764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb928102810 0x7fb9281a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb925400700 1 -- 172.21.15.116:0/1523115764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb920009d00 con 0x7fb928101e70 2024-03-30T00:57:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.632+0000 7fb925400700 1 --2- 172.21.15.116:0/1523115764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb928101e70 0x7fb928199460 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb91400e9a0 tx=0x7fb91400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.633+0000 7fb91ea00700 1 -- 172.21.15.116:0/1523115764 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb914009790 con 0x7fb928101e70 2024-03-30T00:57:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.633+0000 7fb91ea00700 1 -- 172.21.15.116:0/1523115764 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb914010430 con 0x7fb928101e70 2024-03-30T00:57:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.633+0000 7fb91ea00700 1 -- 172.21.15.116:0/1523115764 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb914018680 con 0x7fb928101e70 2024-03-30T00:57:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.633+0000 7fb92ca00700 1 -- 172.21.15.116:0/1523115764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9281a0fa0 con 0x7fb928101e70 2024-03-30T00:57:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.633+0000 7fb92ca00700 1 -- 172.21.15.116:0/1523115764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9281a1430 con 0x7fb928101e70 2024-03-30T00:57:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.634+0000 7fb92ca00700 1 -- 172.21.15.116:0/1523115764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb928067180 con 0x7fb928101e70 2024-03-30T00:57:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.636+0000 7fb91ea00700 1 -- 172.21.15.116:0/1523115764 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fb9140105a0 con 0x7fb928101e70 2024-03-30T00:57:07.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.637+0000 7fb91ea00700 1 --2- 172.21.15.116:0/1523115764 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb90c0a11b0 0x7fb90c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:07.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.637+0000 7fb91ea00700 1 -- 172.21.15.116:0/1523115764 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb914014070 con 0x7fb928101e70 2024-03-30T00:57:07.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.637+0000 7fb925e00700 1 --2- 172.21.15.116:0/1523115764 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb90c0a11b0 0x7fb90c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:07.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.638+0000 7fb925e00700 1 --2- 172.21.15.116:0/1523115764 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb90c0a11b0 0x7fb90c0a3670 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb928101cd0 tx=0x7fb918009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:07.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.641+0000 7fb91ea00700 1 -- 172.21.15.116:0/1523115764 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb9140529f0 con 0x7fb928101e70 2024-03-30T00:57:07.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:07 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3213250418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:07.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:07 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1525579784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:07.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:07 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3213250418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:07.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:07 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1525579784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:07.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.900+0000 7fb92ca00700 1 -- 172.21.15.116:0/1523115764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9281a1620 con 0x7fb928101e70 2024-03-30T00:57:07.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.900+0000 7fb91ea00700 1 -- 172.21.15.116:0/1523115764 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+10886 (secure 0 0 0) 0x7fb914096250 con 0x7fb928101e70 2024-03-30T00:57:07.906 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:07.906 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:56:22.144079+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T00:57:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.903+0000 7fb92ca00700 1 -- 172.21.15.116:0/1523115764 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb90c0a11b0 msgr2=0x7fb90c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.903+0000 7fb92ca00700 1 --2- 172.21.15.116:0/1523115764 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb90c0a11b0 0x7fb90c0a3670 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb928101cd0 tx=0x7fb918009260 comp rx=0 tx=0).stop 2024-03-30T00:57:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.903+0000 7fb92ca00700 1 -- 172.21.15.116:0/1523115764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb928101e70 msgr2=0x7fb928199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.903+0000 7fb92ca00700 1 --2- 172.21.15.116:0/1523115764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb928101e70 0x7fb928199460 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb91400e9a0 tx=0x7fb91400ecb0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.903+0000 7fb92ca00700 1 -- 172.21.15.116:0/1523115764 shutdown_connections 2024-03-30T00:57:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.903+0000 7fb92ca00700 1 --2- 172.21.15.116:0/1523115764 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb90c0a11b0 0x7fb90c0a3670 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.903+0000 7fb92ca00700 1 --2- 172.21.15.116:0/1523115764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb928100c80 0x7fb928198f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.903+0000 7fb92ca00700 1 --2- 172.21.15.116:0/1523115764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb928102810 0x7fb9281a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.903+0000 7fb92ca00700 1 --2- 172.21.15.116:0/1523115764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb928101e70 0x7fb928199460 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.903+0000 7fb92ca00700 1 -- 172.21.15.116:0/1523115764 >> 172.21.15.116:0/1523115764 conn(0x7fb9280fc210 msgr2=0x7fb928072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.903+0000 7fb92ca00700 1 -- 172.21.15.116:0/1523115764 shutdown_connections 2024-03-30T00:57:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:07.903+0000 7fb92ca00700 1 -- 172.21.15.116:0/1523115764 wait complete. 2024-03-30T00:57:07.910 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 22 2024-03-30T00:57:07.929 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:57:08.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.115+0000 7fd28a600700 1 -- 172.21.15.116:0/3032097000 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd284106970 msgr2=0x7fd284108de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:08.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.115+0000 7fd28a600700 1 --2- 172.21.15.116:0/3032097000 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd284106970 0x7fd284108de0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009f90 tx=0x7fd27c00b330 comp rx=0 tx=0).stop 2024-03-30T00:57:08.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.115+0000 7fd28a600700 1 -- 172.21.15.116:0/3032097000 shutdown_connections 2024-03-30T00:57:08.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.115+0000 7fd28a600700 1 --2- 172.21.15.116:0/3032097000 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd284106970 0x7fd284108de0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.115+0000 7fd28a600700 1 --2- 172.21.15.116:0/3032097000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284101710 0x7fd284103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.115+0000 7fd28a600700 1 --2- 172.21.15.116:0/3032097000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd284104040 0x7fd284106430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.115+0000 7fd28a600700 1 -- 172.21.15.116:0/3032097000 >> 172.21.15.116:0/3032097000 conn(0x7fd2840fb100 msgr2=0x7fd2840fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:08.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.115+0000 7fd28a600700 1 -- 172.21.15.116:0/3032097000 shutdown_connections 2024-03-30T00:57:08.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.115+0000 7fd28a600700 1 -- 172.21.15.116:0/3032097000 wait complete. 2024-03-30T00:57:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.116+0000 7fd28a600700 1 Processor -- start 2024-03-30T00:57:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.117+0000 7fd28a600700 1 -- start start 2024-03-30T00:57:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.117+0000 7fd28a600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd284101710 0x7fd284198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.117+0000 7fd28a600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284104040 0x7fd284199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.117+0000 7fd28a600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd284106970 0x7fd2841a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.117+0000 7fd28a600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2841a0a50 con 0x7fd284104040 2024-03-30T00:57:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.117+0000 7fd28a600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2841a0b90 con 0x7fd284101710 2024-03-30T00:57:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.117+0000 7fd28a600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2841a0d20 con 0x7fd284106970 2024-03-30T00:57:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.117+0000 7fd282a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284104040 0x7fd284199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.117+0000 7fd282a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284104040 0x7fd284199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45986/0 (socket says 172.21.15.116:45986) 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.117+0000 7fd282a00700 1 -- 172.21.15.116:0/1570777475 learned_addr learned my addr 172.21.15.116:0/1570777475 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.117+0000 7fd283400700 1 --2- 172.21.15.116:0/1570777475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd284101710 0x7fd284198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.117+0000 7fd283e00700 1 --2- 172.21.15.116:0/1570777475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd284106970 0x7fd2841a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.118+0000 7fd282a00700 1 -- 172.21.15.116:0/1570777475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd284106970 msgr2=0x7fd2841a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.118+0000 7fd282a00700 1 --2- 172.21.15.116:0/1570777475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd284106970 0x7fd2841a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.118+0000 7fd282a00700 1 -- 172.21.15.116:0/1570777475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd284101710 msgr2=0x7fd284198f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.118+0000 7fd282a00700 1 --2- 172.21.15.116:0/1570777475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd284101710 0x7fd284198f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.118+0000 7fd282a00700 1 -- 172.21.15.116:0/1570777475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27c009c40 con 0x7fd284104040 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.118+0000 7fd282a00700 1 --2- 172.21.15.116:0/1570777475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284104040 0x7fd284199460 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd27000e9a0 tx=0x7fd27000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.118+0000 7fd27be00700 1 -- 172.21.15.116:0/1570777475 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270009790 con 0x7fd284104040 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.118+0000 7fd27be00700 1 -- 172.21.15.116:0/1570777475 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd27000ce80 con 0x7fd284104040 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.118+0000 7fd28a600700 1 -- 172.21.15.116:0/1570777475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2841a0fa0 con 0x7fd284104040 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.118+0000 7fd27be00700 1 -- 172.21.15.116:0/1570777475 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270018680 con 0x7fd284104040 2024-03-30T00:57:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.118+0000 7fd28a600700 1 -- 172.21.15.116:0/1570777475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2841a1480 con 0x7fd284104040 2024-03-30T00:57:08.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.119+0000 7fd28a600700 1 -- 172.21.15.116:0/1570777475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2841084f0 con 0x7fd284104040 2024-03-30T00:57:08.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.121+0000 7fd27be00700 1 -- 172.21.15.116:0/1570777475 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fd270010430 con 0x7fd284104040 2024-03-30T00:57:08.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.122+0000 7fd27be00700 1 --2- 172.21.15.116:0/1570777475 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd2680a11b0 0x7fd2680a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:08.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.123+0000 7fd27be00700 1 -- 172.21.15.116:0/1570777475 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd270014070 con 0x7fd284104040 2024-03-30T00:57:08.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.123+0000 7fd283400700 1 --2- 172.21.15.116:0/1570777475 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd2680a11b0 0x7fd2680a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:08.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.123+0000 7fd283400700 1 --2- 172.21.15.116:0/1570777475 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd2680a11b0 0x7fd2680a3670 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd274005fd0 tx=0x7fd274009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:08.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.126+0000 7fd27be00700 1 -- 172.21.15.116:0/1570777475 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd2700529f0 con 0x7fd284104040 2024-03-30T00:57:08.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.372+0000 7fd28a600700 1 -- 172.21.15.116:0/1570777475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd28410c4c0 con 0x7fd284104040 2024-03-30T00:57:08.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.374+0000 7fd27be00700 1 -- 172.21.15.116:0/1570777475 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+24556 (secure 0 0 0) 0x7fd2700d1070 con 0x7fd284104040 2024-03-30T00:57:08.380 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:08.380 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:56:36.456069+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:57:08.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.376+0000 7fd28a600700 1 -- 172.21.15.116:0/1570777475 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd2680a11b0 msgr2=0x7fd2680a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:08.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.376+0000 7fd28a600700 1 --2- 172.21.15.116:0/1570777475 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd2680a11b0 0x7fd2680a3670 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd274005fd0 tx=0x7fd274009310 comp rx=0 tx=0).stop 2024-03-30T00:57:08.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.376+0000 7fd28a600700 1 -- 172.21.15.116:0/1570777475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284104040 msgr2=0x7fd284199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:08.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.376+0000 7fd28a600700 1 --2- 172.21.15.116:0/1570777475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284104040 0x7fd284199460 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd27000e9a0 tx=0x7fd27000ecb0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.377+0000 7fd28a600700 1 -- 172.21.15.116:0/1570777475 shutdown_connections 2024-03-30T00:57:08.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.377+0000 7fd28a600700 1 --2- 172.21.15.116:0/1570777475 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd2680a11b0 0x7fd2680a3670 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.377+0000 7fd28a600700 1 --2- 172.21.15.116:0/1570777475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd284106970 0x7fd2841a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.377+0000 7fd28a600700 1 --2- 172.21.15.116:0/1570777475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd284101710 0x7fd284198f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.377+0000 7fd28a600700 1 --2- 172.21.15.116:0/1570777475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284104040 0x7fd284199460 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.377+0000 7fd28a600700 1 -- 172.21.15.116:0/1570777475 >> 172.21.15.116:0/1570777475 conn(0x7fd2840fb100 msgr2=0x7fd284100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:08.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.377+0000 7fd28a600700 1 -- 172.21.15.116:0/1570777475 shutdown_connections 2024-03-30T00:57:08.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.377+0000 7fd28a600700 1 -- 172.21.15.116:0/1570777475 wait complete. 2024-03-30T00:57:08.401 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T00:52:29.602428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T00:55:50.220543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T00:55:51.274498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T00:55:54.301591+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T00:57:08.402 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T00:57:08.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:08 smithi130 ceph-mon[47626]: pgmap v269: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:08.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:08 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/770903080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:08.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:08 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1523115764' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.591+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2646822329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb8100d80 msgr2=0x7fcfb81011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.591+0000 7fcfbdc00700 1 --2- 172.21.15.116:0/2646822329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb8100d80 0x7fcfb81011a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009920 tx=0x7fcfa8009c30 comp rx=0 tx=0).stop 2024-03-30T00:57:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.591+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2646822329 shutdown_connections 2024-03-30T00:57:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.591+0000 7fcfbdc00700 1 --2- 172.21.15.116:0/2646822329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfb8102910 0x7fcfb81091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.591+0000 7fcfbdc00700 1 --2- 172.21.15.116:0/2646822329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb8100d80 0x7fcfb81011a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.591+0000 7fcfbdc00700 1 --2- 172.21.15.116:0/2646822329 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcfb8101f70 0x7fcfb81023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.591+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2646822329 >> 172.21.15.116:0/2646822329 conn(0x7fcfb80fc2f0 msgr2=0x7fcfb80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.591+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2646822329 shutdown_connections 2024-03-30T00:57:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.591+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2646822329 wait complete. 2024-03-30T00:57:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.592+0000 7fcfbdc00700 1 Processor -- start 2024-03-30T00:57:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.592+0000 7fcfbdc00700 1 -- start start 2024-03-30T00:57:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.592+0000 7fcfbdc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb8100d80 0x7fcfb8199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.592+0000 7fcfbdc00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfb8101f70 0x7fcfb8199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.592+0000 7fcfbdc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcfb8102910 0x7fcfb81a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.592+0000 7fcfbdc00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb81a0b40 con 0x7fcfb8100d80 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.592+0000 7fcfbdc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb81a0c80 con 0x7fcfb8102910 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.592+0000 7fcfbdc00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb81a0e10 con 0x7fcfb8101f70 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.592+0000 7fcfb7400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcfb8102910 0x7fcfb81a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfb6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb8100d80 0x7fcfb8199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfb7400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcfb8102910 0x7fcfb81a0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34536/0 (socket says 172.21.15.116:34536) 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfb7400700 1 -- 172.21.15.116:0/2975112454 learned_addr learned my addr 172.21.15.116:0/2975112454 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfb6000700 1 --2- 172.21.15.116:0/2975112454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfb8101f70 0x7fcfb8199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfb6a00700 1 -- 172.21.15.116:0/2975112454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfb8101f70 msgr2=0x7fcfb8199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfb6a00700 1 --2- 172.21.15.116:0/2975112454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfb8101f70 0x7fcfb8199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfb6a00700 1 -- 172.21.15.116:0/2975112454 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcfb8102910 msgr2=0x7fcfb81a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfb6a00700 1 --2- 172.21.15.116:0/2975112454 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcfb8102910 0x7fcfb81a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfb6a00700 1 -- 172.21.15.116:0/2975112454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa40095f0 con 0x7fcfb8100d80 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfb6000700 1 --2- 172.21.15.116:0/2975112454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfb8101f70 0x7fcfb8199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfb6a00700 1 --2- 172.21.15.116:0/2975112454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb8100d80 0x7fcfb8199010 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009230 tx=0x7fcfa800bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfaf400700 1 -- 172.21.15.116:0/2975112454 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa801c070 con 0x7fcfb8100d80 2024-03-30T00:57:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2975112454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa8009520 con 0x7fcfb8100d80 2024-03-30T00:57:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfaf400700 1 -- 172.21.15.116:0/2975112454 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfa800bba0 con 0x7fcfb8100d80 2024-03-30T00:57:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfaf400700 1 -- 172.21.15.116:0/2975112454 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa80049c0 con 0x7fcfb8100d80 2024-03-30T00:57:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.593+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2975112454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb81a12d0 con 0x7fcfb8100d80 2024-03-30T00:57:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.594+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2975112454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb8067af0 con 0x7fcfb8100d80 2024-03-30T00:57:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.596+0000 7fcfaf400700 1 -- 172.21.15.116:0/2975112454 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fcfa8020070 con 0x7fcfb8100d80 2024-03-30T00:57:08.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.597+0000 7fcfaf400700 1 --2- 172.21.15.116:0/2975112454 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcf9c0a11b0 0x7fcf9c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:08.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.598+0000 7fcfaf400700 1 -- 172.21.15.116:0/2975112454 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcfa80cde70 con 0x7fcfb8100d80 2024-03-30T00:57:08.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.598+0000 7fcfb6000700 1 --2- 172.21.15.116:0/2975112454 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcf9c0a11b0 0x7fcf9c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:08.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.598+0000 7fcfb6000700 1 --2- 172.21.15.116:0/2975112454 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcf9c0a11b0 0x7fcf9c0a3670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4005fd0 tx=0x7fcfa4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:08.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.600+0000 7fcfaf400700 1 -- 172.21.15.116:0/2975112454 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcfa80971c0 con 0x7fcfb8100d80 2024-03-30T00:57:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:08 smithi116 ceph-mon[38086]: pgmap v269: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:08 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/770903080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:08.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:08 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1523115764' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:08 smithi181 ceph-mon[47622]: pgmap v269: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:08 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/770903080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:08 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1523115764' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:08.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.805+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2975112454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcfb810c770 con 0x7fcfb8100d80 2024-03-30T00:57:08.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.806+0000 7fcfaf400700 1 -- 172.21.15.116:0/2975112454 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcfa809a7e0 con 0x7fcfb8100d80 2024-03-30T00:57:08.811 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-30T00:57:08.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.808+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2975112454 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcf9c0a11b0 msgr2=0x7fcf9c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:08.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.808+0000 7fcfbdc00700 1 --2- 172.21.15.116:0/2975112454 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcf9c0a11b0 0x7fcf9c0a3670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4005fd0 tx=0x7fcfa4009310 comp rx=0 tx=0).stop 2024-03-30T00:57:08.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.808+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2975112454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb8100d80 msgr2=0x7fcfb8199010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:08.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.808+0000 7fcfbdc00700 1 --2- 172.21.15.116:0/2975112454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb8100d80 0x7fcfb8199010 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009230 tx=0x7fcfa800bef0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.808+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2975112454 shutdown_connections 2024-03-30T00:57:08.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.808+0000 7fcfbdc00700 1 --2- 172.21.15.116:0/2975112454 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcf9c0a11b0 0x7fcf9c0a3670 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.808+0000 7fcfbdc00700 1 --2- 172.21.15.116:0/2975112454 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcfb8102910 0x7fcfb81a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.808+0000 7fcfbdc00700 1 --2- 172.21.15.116:0/2975112454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb8100d80 0x7fcfb8199010 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.808+0000 7fcfbdc00700 1 --2- 172.21.15.116:0/2975112454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfb8101f70 0x7fcfb8199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:08.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.808+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2975112454 >> 172.21.15.116:0/2975112454 conn(0x7fcfb80fc2f0 msgr2=0x7fcfb81077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:08.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.808+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2975112454 shutdown_connections 2024-03-30T00:57:08.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:08.808+0000 7fcfbdc00700 1 -- 172.21.15.116:0/2975112454 wait complete. 2024-03-30T00:57:08.833 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T00:57:09.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.022+0000 7f6f55200700 1 -- 172.21.15.116:0/2802082164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f50102910 msgr2=0x7f6f501091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:09.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.022+0000 7f6f55200700 1 --2- 172.21.15.116:0/2802082164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f50102910 0x7f6f501091c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800b0d0 tx=0x7f6f4800b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.022+0000 7f6f55200700 1 -- 172.21.15.116:0/2802082164 shutdown_connections 2024-03-30T00:57:09.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.022+0000 7f6f55200700 1 --2- 172.21.15.116:0/2802082164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f50102910 0x7f6f501091c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.022+0000 7f6f55200700 1 --2- 172.21.15.116:0/2802082164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f50100d80 0x7f6f501011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.022+0000 7f6f55200700 1 --2- 172.21.15.116:0/2802082164 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f50101f70 0x7f6f501023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.022+0000 7f6f55200700 1 -- 172.21.15.116:0/2802082164 >> 172.21.15.116:0/2802082164 conn(0x7f6f500fc2f0 msgr2=0x7f6f500fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:09.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.022+0000 7f6f55200700 1 -- 172.21.15.116:0/2802082164 shutdown_connections 2024-03-30T00:57:09.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.022+0000 7f6f55200700 1 -- 172.21.15.116:0/2802082164 wait complete. 2024-03-30T00:57:09.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.024+0000 7f6f55200700 1 Processor -- start 2024-03-30T00:57:09.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.024+0000 7f6f55200700 1 -- start start 2024-03-30T00:57:09.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.024+0000 7f6f55200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f50100d80 0x7f6f50196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:09.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.024+0000 7f6f55200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f50101f70 0x7f6f50197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:09.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f55200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f50102910 0x7f6f5019e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:09.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f55200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5019e8e0 con 0x7f6f50101f70 2024-03-30T00:57:09.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f55200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5019ea50 con 0x7f6f50102910 2024-03-30T00:57:09.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f55200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5019ec10 con 0x7f6f50100d80 2024-03-30T00:57:09.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f4d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f50101f70 0x7f6f50197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:09.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f4de00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f50100d80 0x7f6f50196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:09.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f4d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f50101f70 0x7f6f50197330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46022/0 (socket says 172.21.15.116:46022) 2024-03-30T00:57:09.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f4d400700 1 -- 172.21.15.116:0/1454972696 learned_addr learned my addr 172.21.15.116:0/1454972696 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:09.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f4e800700 1 --2- 172.21.15.116:0/1454972696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f50102910 0x7f6f5019e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:09.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f4de00700 1 -- 172.21.15.116:0/1454972696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f50102910 msgr2=0x7f6f5019e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:09.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f4de00700 1 --2- 172.21.15.116:0/1454972696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f50102910 0x7f6f5019e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f4de00700 1 -- 172.21.15.116:0/1454972696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f50101f70 msgr2=0x7f6f50197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:09.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f4de00700 1 --2- 172.21.15.116:0/1454972696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f50101f70 0x7f6f50197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f4de00700 1 -- 172.21.15.116:0/1454972696 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f48009d00 con 0x7f6f50100d80 2024-03-30T00:57:09.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f4d400700 1 --2- 172.21.15.116:0/1454972696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f50101f70 0x7f6f50197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:09.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.025+0000 7f6f4de00700 1 --2- 172.21.15.116:0/1454972696 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f50100d80 0x7f6f50196df0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000e9c0 tx=0x7f6f4000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:09.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.026+0000 7f6f46a00700 1 -- 172.21.15.116:0/1454972696 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40004220 con 0x7f6f50100d80 2024-03-30T00:57:09.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.026+0000 7f6f46a00700 1 -- 172.21.15.116:0/1454972696 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f40004380 con 0x7f6f50100d80 2024-03-30T00:57:09.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.026+0000 7f6f55200700 1 -- 172.21.15.116:0/1454972696 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5019eef0 con 0x7f6f50100d80 2024-03-30T00:57:09.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.026+0000 7f6f46a00700 1 -- 172.21.15.116:0/1454972696 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40003be0 con 0x7f6f50100d80 2024-03-30T00:57:09.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.026+0000 7f6f55200700 1 -- 172.21.15.116:0/1454972696 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5019f310 con 0x7f6f50100d80 2024-03-30T00:57:09.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.027+0000 7f6f55200700 1 -- 172.21.15.116:0/1454972696 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f50108130 con 0x7f6f50100d80 2024-03-30T00:57:09.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.029+0000 7f6f46a00700 1 -- 172.21.15.116:0/1454972696 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f6f40003d40 con 0x7f6f50100d80 2024-03-30T00:57:09.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.030+0000 7f6f46a00700 1 --2- 172.21.15.116:0/1454972696 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6f340a11b0 0x7f6f340a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:09.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.030+0000 7f6f46a00700 1 -- 172.21.15.116:0/1454972696 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6f40014070 con 0x7f6f50100d80 2024-03-30T00:57:09.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.030+0000 7f6f4d400700 1 --2- 172.21.15.116:0/1454972696 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6f340a11b0 0x7f6f340a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:09.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.031+0000 7f6f4d400700 1 --2- 172.21.15.116:0/1454972696 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6f340a11b0 0x7f6f340a3670 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c011550 tx=0x7f6f3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:09.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.032+0000 7f6f46a00700 1 -- 172.21.15.116:0/1454972696 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6f40096470 con 0x7f6f50100d80 2024-03-30T00:57:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.239+0000 7f6f55200700 1 -- 172.21.15.116:0/1454972696 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6f5010c6b0 con 0x7f6f50100d80 2024-03-30T00:57:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.240+0000 7f6f46a00700 1 -- 172.21.15.116:0/1454972696 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6f40099a90 con 0x7f6f50100d80 2024-03-30T00:57:09.245 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:57:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.242+0000 7f6f55200700 1 -- 172.21.15.116:0/1454972696 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6f340a11b0 msgr2=0x7f6f340a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.242+0000 7f6f55200700 1 --2- 172.21.15.116:0/1454972696 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6f340a11b0 0x7f6f340a3670 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c011550 tx=0x7f6f3c009260 comp rx=0 tx=0).stop 2024-03-30T00:57:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.242+0000 7f6f55200700 1 -- 172.21.15.116:0/1454972696 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f50100d80 msgr2=0x7f6f50196df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.242+0000 7f6f55200700 1 --2- 172.21.15.116:0/1454972696 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f50100d80 0x7f6f50196df0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000e9c0 tx=0x7f6f4000ecd0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.242+0000 7f6f55200700 1 -- 172.21.15.116:0/1454972696 shutdown_connections 2024-03-30T00:57:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.242+0000 7f6f55200700 1 --2- 172.21.15.116:0/1454972696 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6f340a11b0 0x7f6f340a3670 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.242+0000 7f6f55200700 1 --2- 172.21.15.116:0/1454972696 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f50100d80 0x7f6f50196df0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.242+0000 7f6f55200700 1 --2- 172.21.15.116:0/1454972696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f50102910 0x7f6f5019e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.242+0000 7f6f55200700 1 --2- 172.21.15.116:0/1454972696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f50101f70 0x7f6f50197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.242+0000 7f6f55200700 1 -- 172.21.15.116:0/1454972696 >> 172.21.15.116:0/1454972696 conn(0x7f6f500fc2f0 msgr2=0x7f6f50105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.242+0000 7f6f55200700 1 -- 172.21.15.116:0/1454972696 shutdown_connections 2024-03-30T00:57:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.242+0000 7f6f55200700 1 -- 172.21.15.116:0/1454972696 wait complete. 2024-03-30T00:57:09.266 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T00:57:09.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.457+0000 7f3b5f000700 1 -- 172.21.15.116:0/4088589061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b58100d80 msgr2=0x7f3b581011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.457+0000 7f3b5f000700 1 --2- 172.21.15.116:0/4088589061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b58100d80 0x7f3b581011a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3b480099c0 tx=0x7f3b48009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.457+0000 7f3b5f000700 1 -- 172.21.15.116:0/4088589061 shutdown_connections 2024-03-30T00:57:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.457+0000 7f3b5f000700 1 --2- 172.21.15.116:0/4088589061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b58102910 0x7f3b581091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.457+0000 7f3b5f000700 1 --2- 172.21.15.116:0/4088589061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b58100d80 0x7f3b581011a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.457+0000 7f3b5f000700 1 --2- 172.21.15.116:0/4088589061 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b58101f70 0x7f3b581023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.457+0000 7f3b5f000700 1 -- 172.21.15.116:0/4088589061 >> 172.21.15.116:0/4088589061 conn(0x7f3b580fc2f0 msgr2=0x7f3b580fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.457+0000 7f3b5f000700 1 -- 172.21.15.116:0/4088589061 shutdown_connections 2024-03-30T00:57:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.457+0000 7f3b5f000700 1 -- 172.21.15.116:0/4088589061 wait complete. 2024-03-30T00:57:09.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.459+0000 7f3b5f000700 1 Processor -- start 2024-03-30T00:57:09.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.459+0000 7f3b5f000700 1 -- start start 2024-03-30T00:57:09.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.459+0000 7f3b5f000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b58100d80 0x7f3b58198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:09.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.459+0000 7f3b5f000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b58101f70 0x7f3b581994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:09.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.459+0000 7f3b5f000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b58102910 0x7f3b581a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:09.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.459+0000 7f3b5f000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b581a0a20 con 0x7f3b58101f70 2024-03-30T00:57:09.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.459+0000 7f3b5f000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b581a0b60 con 0x7f3b58100d80 2024-03-30T00:57:09.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.459+0000 7f3b5f000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b581a0cf0 con 0x7f3b58102910 2024-03-30T00:57:09.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.460+0000 7f3b57400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b58101f70 0x7f3b581994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:09.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.460+0000 7f3b57400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b58101f70 0x7f3b581994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46042/0 (socket says 172.21.15.116:46042) 2024-03-30T00:57:09.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.460+0000 7f3b57400700 1 -- 172.21.15.116:0/1010941050 learned_addr learned my addr 172.21.15.116:0/1010941050 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:09.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.460+0000 7f3b57e00700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b58100d80 0x7f3b58198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:09.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.460+0000 7f3b5d000700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b58102910 0x7f3b581a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:09.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.460+0000 7f3b57e00700 1 -- 172.21.15.116:0/1010941050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b58102910 msgr2=0x7f3b581a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:09.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.460+0000 7f3b57e00700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b58102910 0x7f3b581a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.460+0000 7f3b57e00700 1 -- 172.21.15.116:0/1010941050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b58101f70 msgr2=0x7f3b581994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:09.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.460+0000 7f3b57e00700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b58101f70 0x7f3b581994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.460+0000 7f3b57e00700 1 -- 172.21.15.116:0/1010941050 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b480095f0 con 0x7f3b58100d80 2024-03-30T00:57:09.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.460+0000 7f3b57400700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b58101f70 0x7f3b581994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:09.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.460+0000 7f3b57e00700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b58100d80 0x7f3b58198f90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3b480098e0 tx=0x7f3b4800f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:09.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.461+0000 7f3b54c00700 1 -- 172.21.15.116:0/1010941050 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4801d070 con 0x7f3b58100d80 2024-03-30T00:57:09.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.461+0000 7f3b5f000700 1 -- 172.21.15.116:0/1010941050 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b581a0f70 con 0x7f3b58100d80 2024-03-30T00:57:09.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.461+0000 7f3b54c00700 1 -- 172.21.15.116:0/1010941050 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b48005050 con 0x7f3b58100d80 2024-03-30T00:57:09.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.461+0000 7f3b54c00700 1 -- 172.21.15.116:0/1010941050 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b48004a50 con 0x7f3b58100d80 2024-03-30T00:57:09.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.461+0000 7f3b5f000700 1 -- 172.21.15.116:0/1010941050 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b581a1330 con 0x7f3b58100d80 2024-03-30T00:57:09.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.462+0000 7f3b5f000700 1 -- 172.21.15.116:0/1010941050 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b58067af0 con 0x7f3b58100d80 2024-03-30T00:57:09.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.464+0000 7f3b54c00700 1 -- 172.21.15.116:0/1010941050 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f3b480051c0 con 0x7f3b58100d80 2024-03-30T00:57:09.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.465+0000 7f3b54c00700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3b3c0a11b0 0x7f3b3c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:09.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.465+0000 7f3b54c00700 1 -- 172.21.15.116:0/1010941050 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3b480cd980 con 0x7f3b58100d80 2024-03-30T00:57:09.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.465+0000 7f3b57400700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3b3c0a11b0 0x7f3b3c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:09.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.466+0000 7f3b57400700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3b3c0a11b0 0x7f3b3c0a3670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3b44011550 tx=0x7f3b44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:09.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.467+0000 7f3b54c00700 1 -- 172.21.15.116:0/1010941050 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3b48096cd0 con 0x7f3b58100d80 2024-03-30T00:57:09.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:09 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1570777475' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:09.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:09 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2975112454' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:09.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:09 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1454972696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:09.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.673+0000 7f3b5f000700 1 -- 172.21.15.116:0/1010941050 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3b581a1620 con 0x7f3b58100d80 2024-03-30T00:57:09.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.674+0000 7f3b54c00700 1 -- 172.21.15.116:0/1010941050 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3b4809a2f0 con 0x7f3b58100d80 2024-03-30T00:57:09.679 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-30T00:57:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.676+0000 7f3b5f000700 1 -- 172.21.15.116:0/1010941050 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3b3c0a11b0 msgr2=0x7f3b3c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.676+0000 7f3b5f000700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3b3c0a11b0 0x7f3b3c0a3670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3b44011550 tx=0x7f3b44009260 comp rx=0 tx=0).stop 2024-03-30T00:57:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.676+0000 7f3b5f000700 1 -- 172.21.15.116:0/1010941050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b58100d80 msgr2=0x7f3b58198f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.676+0000 7f3b5f000700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b58100d80 0x7f3b58198f90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3b480098e0 tx=0x7f3b4800f690 comp rx=0 tx=0).stop 2024-03-30T00:57:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.676+0000 7f3b5f000700 1 -- 172.21.15.116:0/1010941050 shutdown_connections 2024-03-30T00:57:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.676+0000 7f3b5f000700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3b3c0a11b0 0x7f3b3c0a3670 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.676+0000 7f3b5f000700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b58102910 0x7f3b581a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.676+0000 7f3b5f000700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b58100d80 0x7f3b58198f90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.676+0000 7f3b5f000700 1 --2- 172.21.15.116:0/1010941050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b58101f70 0x7f3b581994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.676+0000 7f3b5f000700 1 -- 172.21.15.116:0/1010941050 >> 172.21.15.116:0/1010941050 conn(0x7f3b580fc2f0 msgr2=0x7f3b58072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.676+0000 7f3b5f000700 1 -- 172.21.15.116:0/1010941050 shutdown_connections 2024-03-30T00:57:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.676+0000 7f3b5f000700 1 -- 172.21.15.116:0/1010941050 wait complete. 2024-03-30T00:57:09.701 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T00:57:09.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:09 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1570777475' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:09.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:09 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2975112454' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:09.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:09 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1454972696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:09.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:09 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1570777475' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:09.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:09 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2975112454' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:09.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:09 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1454972696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.889+0000 7ff184c00700 1 -- 172.21.15.116:0/4086917691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff180100d80 msgr2=0x7ff1801011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.889+0000 7ff184c00700 1 --2- 172.21.15.116:0/4086917691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff180100d80 0x7ff1801011a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff170009920 tx=0x7ff170009c30 comp rx=0 tx=0).stop 2024-03-30T00:57:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.889+0000 7ff184c00700 1 -- 172.21.15.116:0/4086917691 shutdown_connections 2024-03-30T00:57:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.889+0000 7ff184c00700 1 --2- 172.21.15.116:0/4086917691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180102910 0x7ff1801091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.889+0000 7ff184c00700 1 --2- 172.21.15.116:0/4086917691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff180100d80 0x7ff1801011a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.889+0000 7ff184c00700 1 --2- 172.21.15.116:0/4086917691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff180101f70 0x7ff1801023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.889+0000 7ff184c00700 1 -- 172.21.15.116:0/4086917691 >> 172.21.15.116:0/4086917691 conn(0x7ff1800fc2f0 msgr2=0x7ff1800fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.889+0000 7ff184c00700 1 -- 172.21.15.116:0/4086917691 shutdown_connections 2024-03-30T00:57:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.889+0000 7ff184c00700 1 -- 172.21.15.116:0/4086917691 wait complete. 2024-03-30T00:57:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.890+0000 7ff184c00700 1 Processor -- start 2024-03-30T00:57:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.890+0000 7ff184c00700 1 -- start start 2024-03-30T00:57:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.890+0000 7ff184c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180100d80 0x7ff180074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.890+0000 7ff184c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff180101f70 0x7ff1800731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.890+0000 7ff184c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff180102910 0x7ff1800736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.890+0000 7ff184c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff180073e80 con 0x7ff180101f70 2024-03-30T00:57:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.890+0000 7ff184c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff180073ff0 con 0x7ff180100d80 2024-03-30T00:57:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.890+0000 7ff184c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1801a0e20 con 0x7ff180102910 2024-03-30T00:57:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.890+0000 7ff17d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff180101f70 0x7ff1800731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.890+0000 7ff17d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff180101f70 0x7ff1800731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46062/0 (socket says 172.21.15.116:46062) 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.890+0000 7ff17d400700 1 -- 172.21.15.116:0/2505447139 learned_addr learned my addr 172.21.15.116:0/2505447139 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.891+0000 7ff17e800700 1 --2- 172.21.15.116:0/2505447139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff180102910 0x7ff1800736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.891+0000 7ff17de00700 1 --2- 172.21.15.116:0/2505447139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180100d80 0x7ff180074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.891+0000 7ff17d400700 1 -- 172.21.15.116:0/2505447139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff180102910 msgr2=0x7ff1800736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.891+0000 7ff17d400700 1 --2- 172.21.15.116:0/2505447139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff180102910 0x7ff1800736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.891+0000 7ff17d400700 1 -- 172.21.15.116:0/2505447139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180100d80 msgr2=0x7ff180074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.891+0000 7ff17d400700 1 --2- 172.21.15.116:0/2505447139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180100d80 0x7ff180074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.891+0000 7ff17d400700 1 -- 172.21.15.116:0/2505447139 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff170009520 con 0x7ff180101f70 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.891+0000 7ff17d400700 1 --2- 172.21.15.116:0/2505447139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff180101f70 0x7ff1800731a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff16c00e9a0 tx=0x7ff16c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.891+0000 7ff176a00700 1 -- 172.21.15.116:0/2505447139 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c009790 con 0x7ff180101f70 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.891+0000 7ff176a00700 1 -- 172.21.15.116:0/2505447139 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff16c00ce70 con 0x7ff180101f70 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.891+0000 7ff176a00700 1 -- 172.21.15.116:0/2505447139 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c018420 con 0x7ff180101f70 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.891+0000 7ff184c00700 1 -- 172.21.15.116:0/2505447139 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1801a10b0 con 0x7ff180101f70 2024-03-30T00:57:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.892+0000 7ff184c00700 1 -- 172.21.15.116:0/2505447139 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1801a1570 con 0x7ff180101f70 2024-03-30T00:57:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.892+0000 7ff184c00700 1 -- 172.21.15.116:0/2505447139 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff180108240 con 0x7ff180101f70 2024-03-30T00:57:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.894+0000 7ff176a00700 1 -- 172.21.15.116:0/2505447139 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7ff16c0106f0 con 0x7ff180101f70 2024-03-30T00:57:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.896+0000 7ff176a00700 1 --2- 172.21.15.116:0/2505447139 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1640a1230 0x7ff1640a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.896+0000 7ff176a00700 1 -- 172.21.15.116:0/2505447139 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff16c014070 con 0x7ff180101f70 2024-03-30T00:57:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.896+0000 7ff17de00700 1 --2- 172.21.15.116:0/2505447139 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1640a1230 0x7ff1640a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.896+0000 7ff17de00700 1 --2- 172.21.15.116:0/2505447139 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1640a1230 0x7ff1640a36f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff1700098f0 tx=0x7ff170019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:09.898+0000 7ff176a00700 1 -- 172.21.15.116:0/2505447139 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff16c0965f0 con 0x7ff180101f70 2024-03-30T00:57:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.102+0000 7ff184c00700 1 -- 172.21.15.116:0/2505447139 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff18010c5e0 con 0x7ff180101f70 2024-03-30T00:57:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.102+0000 7ff176a00700 1 -- 172.21.15.116:0/2505447139 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v83) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff16c0260a0 con 0x7ff180101f70 2024-03-30T00:57:10.108 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:57:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.105+0000 7ff184c00700 1 -- 172.21.15.116:0/2505447139 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1640a1230 msgr2=0x7ff1640a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.105+0000 7ff184c00700 1 --2- 172.21.15.116:0/2505447139 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1640a1230 0x7ff1640a36f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff1700098f0 tx=0x7ff170019040 comp rx=0 tx=0).stop 2024-03-30T00:57:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.105+0000 7ff184c00700 1 -- 172.21.15.116:0/2505447139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff180101f70 msgr2=0x7ff1800731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.105+0000 7ff184c00700 1 --2- 172.21.15.116:0/2505447139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff180101f70 0x7ff1800731a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff16c00e9a0 tx=0x7ff16c00ecb0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.105+0000 7ff184c00700 1 -- 172.21.15.116:0/2505447139 shutdown_connections 2024-03-30T00:57:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.105+0000 7ff184c00700 1 --2- 172.21.15.116:0/2505447139 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1640a1230 0x7ff1640a36f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.105+0000 7ff184c00700 1 --2- 172.21.15.116:0/2505447139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff180102910 0x7ff1800736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.105+0000 7ff184c00700 1 --2- 172.21.15.116:0/2505447139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180100d80 0x7ff180074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.105+0000 7ff184c00700 1 --2- 172.21.15.116:0/2505447139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff180101f70 0x7ff1800731a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.105+0000 7ff184c00700 1 -- 172.21.15.116:0/2505447139 >> 172.21.15.116:0/2505447139 conn(0x7ff1800fc2f0 msgr2=0x7ff180105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.105+0000 7ff184c00700 1 -- 172.21.15.116:0/2505447139 shutdown_connections 2024-03-30T00:57:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.105+0000 7ff184c00700 1 -- 172.21.15.116:0/2505447139 wait complete. 2024-03-30T00:57:10.130 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:57:10.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.319+0000 7f77f6000700 1 -- 172.21.15.116:0/1184223685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77f0074d90 msgr2=0x7f77f00731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:10.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.319+0000 7f77f6000700 1 --2- 172.21.15.116:0/1184223685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77f0074d90 0x7f77f00731e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f77e0009990 tx=0x7f77e0009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.319+0000 7f77f6000700 1 -- 172.21.15.116:0/1184223685 shutdown_connections 2024-03-30T00:57:10.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.319+0000 7f77f6000700 1 --2- 172.21.15.116:0/1184223685 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77f0108b60 0x7f77f010afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.319+0000 7f77f6000700 1 --2- 172.21.15.116:0/1184223685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77f0074d90 0x7f77f00731e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.319+0000 7f77f6000700 1 --2- 172.21.15.116:0/1184223685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77f00737b0 0x7f77f0073c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.319+0000 7f77f6000700 1 -- 172.21.15.116:0/1184223685 >> 172.21.15.116:0/1184223685 conn(0x7f77f00fc200 msgr2=0x7f77f00fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:10.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.319+0000 7f77f6000700 1 -- 172.21.15.116:0/1184223685 shutdown_connections 2024-03-30T00:57:10.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.319+0000 7f77f6000700 1 -- 172.21.15.116:0/1184223685 wait complete. 2024-03-30T00:57:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.320+0000 7f77f6000700 1 Processor -- start 2024-03-30T00:57:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.320+0000 7f77f6000700 1 -- start start 2024-03-30T00:57:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.320+0000 7f77f6000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77f00737b0 0x7f77f0198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.320+0000 7f77f6000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77f0074d90 0x7f77f0199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.320+0000 7f77f6000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77f0108b60 0x7f77f01a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.320+0000 7f77f6000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f01a08f0 con 0x7f77f0108b60 2024-03-30T00:57:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.320+0000 7f77f6000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f01a0a60 con 0x7f77f00737b0 2024-03-30T00:57:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.320+0000 7f77f6000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f01a0c20 con 0x7f77f0074d90 2024-03-30T00:57:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.321+0000 7f77efe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77f0108b60 0x7f77f01a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.321+0000 7f77eea00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77f0074d90 0x7f77f0199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.321+0000 7f77ef400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77f00737b0 0x7f77f0198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.321+0000 7f77efe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77f0108b60 0x7f77f01a0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46080/0 (socket says 172.21.15.116:46080) 2024-03-30T00:57:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.321+0000 7f77eea00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77f0074d90 0x7f77f0199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:35262/0 (socket says 172.21.15.116:35262) 2024-03-30T00:57:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.321+0000 7f77ef400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77f00737b0 0x7f77f0198e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34618/0 (socket says 172.21.15.116:34618) 2024-03-30T00:57:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.321+0000 7f77efe00700 1 -- 172.21.15.116:0/1120519611 learned_addr learned my addr 172.21.15.116:0/1120519611 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.321+0000 7f77eea00700 1 -- 172.21.15.116:0/1120519611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77f00737b0 msgr2=0x7f77f0198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.321+0000 7f77eea00700 1 --2- 172.21.15.116:0/1120519611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77f00737b0 0x7f77f0198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.321+0000 7f77eea00700 1 -- 172.21.15.116:0/1120519611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77f0108b60 msgr2=0x7f77f01a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.321+0000 7f77eea00700 1 --2- 172.21.15.116:0/1120519611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77f0108b60 0x7f77f01a0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.321+0000 7f77eea00700 1 -- 172.21.15.116:0/1120519611 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77dc009520 con 0x7f77f0074d90 2024-03-30T00:57:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.322+0000 7f77efe00700 1 --2- 172.21.15.116:0/1120519611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77f0108b60 0x7f77f01a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.322+0000 7f77eea00700 1 --2- 172.21.15.116:0/1120519611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77f0074d90 0x7f77f0199340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00e9f0 tx=0x7f77dc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.322+0000 7f77e7e00700 1 -- 172.21.15.116:0/1120519611 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc00cb70 con 0x7f77f0074d90 2024-03-30T00:57:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.322+0000 7f77e7e00700 1 -- 172.21.15.116:0/1120519611 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77dc00ee90 con 0x7f77f0074d90 2024-03-30T00:57:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.322+0000 7f77f6000700 1 -- 172.21.15.116:0/1120519611 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e00095f0 con 0x7f77f0074d90 2024-03-30T00:57:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.322+0000 7f77e7e00700 1 -- 172.21.15.116:0/1120519611 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc003cf0 con 0x7f77f0074d90 2024-03-30T00:57:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.322+0000 7f77f6000700 1 -- 172.21.15.116:0/1120519611 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f01a11d0 con 0x7f77f0074d90 2024-03-30T00:57:10.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.323+0000 7f77f6000700 1 -- 172.21.15.116:0/1120519611 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f0107970 con 0x7f77f0074d90 2024-03-30T00:57:10.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.325+0000 7f77e7e00700 1 -- 172.21.15.116:0/1120519611 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f77dc00ccd0 con 0x7f77f0074d90 2024-03-30T00:57:10.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.326+0000 7f77e7e00700 1 --2- 172.21.15.116:0/1120519611 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f77d40a11b0 0x7f77d40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:10.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.327+0000 7f77e7e00700 1 -- 172.21.15.116:0/1120519611 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f77dc014070 con 0x7f77f0074d90 2024-03-30T00:57:10.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.327+0000 7f77ef400700 1 --2- 172.21.15.116:0/1120519611 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f77d40a11b0 0x7f77d40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:10.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.327+0000 7f77ef400700 1 --2- 172.21.15.116:0/1120519611 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f77d40a11b0 0x7f77d40a3670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f77e000b680 tx=0x7f77e0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:10.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.329+0000 7f77e7e00700 1 -- 172.21.15.116:0/1120519611 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f77dc096490 con 0x7f77f0074d90 2024-03-30T00:57:10.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:10 smithi116 ceph-mon[38086]: pgmap v270: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:57:10.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:10 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1010941050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:10.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:10 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2505447139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:10.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.582+0000 7f77f6000700 1 -- 172.21.15.116:0/1120519611 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77f0067a10 con 0x7f77f0074d90 2024-03-30T00:57:10.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.583+0000 7f77e7e00700 1 -- 172.21.15.116:0/1120519611 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+10886 (secure 0 0 0) 0x7f77dc00fe60 con 0x7f77f0074d90 2024-03-30T00:57:10.588 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:10.588 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:56:22.144079+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T00:57:10.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.585+0000 7f77f6000700 1 -- 172.21.15.116:0/1120519611 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f77d40a11b0 msgr2=0x7f77d40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:10.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.585+0000 7f77f6000700 1 --2- 172.21.15.116:0/1120519611 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f77d40a11b0 0x7f77d40a3670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f77e000b680 tx=0x7f77e0005fd0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.585+0000 7f77f6000700 1 -- 172.21.15.116:0/1120519611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77f0074d90 msgr2=0x7f77f0199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:10.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.585+0000 7f77f6000700 1 --2- 172.21.15.116:0/1120519611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77f0074d90 0x7f77f0199340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00e9f0 tx=0x7f77dc00ed00 comp rx=0 tx=0).stop 2024-03-30T00:57:10.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.585+0000 7f77f6000700 1 -- 172.21.15.116:0/1120519611 shutdown_connections 2024-03-30T00:57:10.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.585+0000 7f77f6000700 1 --2- 172.21.15.116:0/1120519611 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f77d40a11b0 0x7f77d40a3670 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.585+0000 7f77f6000700 1 --2- 172.21.15.116:0/1120519611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77f0108b60 0x7f77f01a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.585+0000 7f77f6000700 1 --2- 172.21.15.116:0/1120519611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77f00737b0 0x7f77f0198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.585+0000 7f77f6000700 1 --2- 172.21.15.116:0/1120519611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77f0074d90 0x7f77f0199340 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.585+0000 7f77f6000700 1 -- 172.21.15.116:0/1120519611 >> 172.21.15.116:0/1120519611 conn(0x7f77f00fc200 msgr2=0x7f77f0104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:10.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.585+0000 7f77f6000700 1 -- 172.21.15.116:0/1120519611 shutdown_connections 2024-03-30T00:57:10.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.585+0000 7f77f6000700 1 -- 172.21.15.116:0/1120519611 wait complete. 2024-03-30T00:57:10.592 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 22 2024-03-30T00:57:10.610 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:57:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:10 smithi181 ceph-mon[47622]: pgmap v270: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:57:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:10 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1010941050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:10 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2505447139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.796+0000 7f5a07400700 1 -- 172.21.15.116:0/3802293095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a00101e70 msgr2=0x7f5a001022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.796+0000 7f5a07400700 1 --2- 172.21.15.116:0/3802293095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a00101e70 0x7f5a001022d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0099c0 tx=0x7f59ec009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.796+0000 7f5a07400700 1 -- 172.21.15.116:0/3802293095 shutdown_connections 2024-03-30T00:57:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.796+0000 7f5a07400700 1 --2- 172.21.15.116:0/3802293095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a00102810 0x7f5a001090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.796+0000 7f5a07400700 1 --2- 172.21.15.116:0/3802293095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a00100c80 0x7f5a001010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.796+0000 7f5a07400700 1 --2- 172.21.15.116:0/3802293095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a00101e70 0x7f5a001022d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.796+0000 7f5a07400700 1 -- 172.21.15.116:0/3802293095 >> 172.21.15.116:0/3802293095 conn(0x7f5a000fc1f0 msgr2=0x7f5a000fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.796+0000 7f5a07400700 1 -- 172.21.15.116:0/3802293095 shutdown_connections 2024-03-30T00:57:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.796+0000 7f5a07400700 1 -- 172.21.15.116:0/3802293095 wait complete. 2024-03-30T00:57:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.798+0000 7f5a07400700 1 Processor -- start 2024-03-30T00:57:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.798+0000 7f5a07400700 1 -- start start 2024-03-30T00:57:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.798+0000 7f5a07400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a00100c80 0x7f5a00198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.798+0000 7f5a07400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a00101e70 0x7f5a00199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.798+0000 7f5a07400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a00102810 0x7f5a001a0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.798+0000 7f5a07400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a001a0950 con 0x7f5a00101e70 2024-03-30T00:57:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.798+0000 7f5a07400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a001a0a90 con 0x7f5a00100c80 2024-03-30T00:57:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.798+0000 7f5a07400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a001a0c20 con 0x7f5a00102810 2024-03-30T00:57:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.799+0000 7f59ffe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a00101e70 0x7f5a00199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.799+0000 7f5a04a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a00100c80 0x7f5a00198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.799+0000 7f59ffe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a00101e70 0x7f5a00199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46094/0 (socket says 172.21.15.116:46094) 2024-03-30T00:57:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.799+0000 7f59ffe00700 1 -- 172.21.15.116:0/177344594 learned_addr learned my addr 172.21.15.116:0/177344594 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.799+0000 7f5a05400700 1 --2- 172.21.15.116:0/177344594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a00102810 0x7f5a001a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.799+0000 7f59ffe00700 1 -- 172.21.15.116:0/177344594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a00102810 msgr2=0x7f5a001a0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.799+0000 7f59ffe00700 1 --2- 172.21.15.116:0/177344594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a00102810 0x7f5a001a0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.799+0000 7f59ffe00700 1 -- 172.21.15.116:0/177344594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a00100c80 msgr2=0x7f5a00198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.799+0000 7f59ffe00700 1 --2- 172.21.15.116:0/177344594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a00100c80 0x7f5a00198e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:10.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.799+0000 7f59ffe00700 1 -- 172.21.15.116:0/177344594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec0095f0 con 0x7f5a00101e70 2024-03-30T00:57:10.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.799+0000 7f5a04a00700 1 --2- 172.21.15.116:0/177344594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a00100c80 0x7f5a00198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:10.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.799+0000 7f59ffe00700 1 --2- 172.21.15.116:0/177344594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a00101e70 0x7f5a00199360 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f59ec006010 tx=0x7f59ec00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:10.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.799+0000 7f59fd600700 1 -- 172.21.15.116:0/177344594 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec01d070 con 0x7f5a00101e70 2024-03-30T00:57:10.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.800+0000 7f5a07400700 1 -- 172.21.15.116:0/177344594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a001a0ea0 con 0x7f5a00101e70 2024-03-30T00:57:10.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.800+0000 7f59fd600700 1 -- 172.21.15.116:0/177344594 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f59ec005030 con 0x7f5a00101e70 2024-03-30T00:57:10.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.800+0000 7f59fd600700 1 -- 172.21.15.116:0/177344594 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec004b10 con 0x7f5a00101e70 2024-03-30T00:57:10.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.800+0000 7f5a07400700 1 -- 172.21.15.116:0/177344594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a001a1270 con 0x7f5a00101e70 2024-03-30T00:57:10.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.801+0000 7f5a07400700 1 -- 172.21.15.116:0/177344594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a00067a10 con 0x7f5a00101e70 2024-03-30T00:57:10.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.803+0000 7f59fd600700 1 -- 172.21.15.116:0/177344594 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f59ec0051a0 con 0x7f5a00101e70 2024-03-30T00:57:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.804+0000 7f59fd600700 1 --2- 172.21.15.116:0/177344594 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f59e80a11b0 0x7f59e80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.804+0000 7f59fd600700 1 -- 172.21.15.116:0/177344594 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f59ec0cda40 con 0x7f5a00101e70 2024-03-30T00:57:10.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.804+0000 7f5a04a00700 1 --2- 172.21.15.116:0/177344594 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f59e80a11b0 0x7f59e80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:10.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.805+0000 7f5a04a00700 1 --2- 172.21.15.116:0/177344594 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f59e80a11b0 0x7f59e80a3670 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5a00101cd0 tx=0x7f59f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:10.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:10.808+0000 7f59fd600700 1 -- 172.21.15.116:0/177344594 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f59ec096d90 con 0x7f5a00101e70 2024-03-30T00:57:10.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:10 smithi130 ceph-mon[47626]: pgmap v270: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:57:10.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:10 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1010941050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:10.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:10 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2505447139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.042+0000 7f5a07400700 1 -- 172.21.15.116:0/177344594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a0010c730 con 0x7f5a00101e70 2024-03-30T00:57:11.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.043+0000 7f59fd600700 1 -- 172.21.15.116:0/177344594 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+24556 (secure 0 0 0) 0x7f59ec09a3b0 con 0x7f5a00101e70 2024-03-30T00:57:11.049 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:11.050 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:56:36.456069+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:57:11.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.046+0000 7f5a07400700 1 -- 172.21.15.116:0/177344594 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f59e80a11b0 msgr2=0x7f59e80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.046+0000 7f5a07400700 1 --2- 172.21.15.116:0/177344594 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f59e80a11b0 0x7f59e80a3670 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5a00101cd0 tx=0x7f59f4009260 comp rx=0 tx=0).stop 2024-03-30T00:57:11.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.046+0000 7f5a07400700 1 -- 172.21.15.116:0/177344594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a00101e70 msgr2=0x7f5a00199360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.046+0000 7f5a07400700 1 --2- 172.21.15.116:0/177344594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a00101e70 0x7f5a00199360 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f59ec006010 tx=0x7f59ec00f690 comp rx=0 tx=0).stop 2024-03-30T00:57:11.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.046+0000 7f5a07400700 1 -- 172.21.15.116:0/177344594 shutdown_connections 2024-03-30T00:57:11.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.046+0000 7f5a07400700 1 --2- 172.21.15.116:0/177344594 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f59e80a11b0 0x7f59e80a3670 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.046+0000 7f5a07400700 1 --2- 172.21.15.116:0/177344594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a00102810 0x7f5a001a0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.046+0000 7f5a07400700 1 --2- 172.21.15.116:0/177344594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a00100c80 0x7f5a00198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.046+0000 7f5a07400700 1 --2- 172.21.15.116:0/177344594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a00101e70 0x7f5a00199360 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.046+0000 7f5a07400700 1 -- 172.21.15.116:0/177344594 >> 172.21.15.116:0/177344594 conn(0x7f5a000fc1f0 msgr2=0x7f5a00072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:11.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.046+0000 7f5a07400700 1 -- 172.21.15.116:0/177344594 shutdown_connections 2024-03-30T00:57:11.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.047+0000 7f5a07400700 1 -- 172.21.15.116:0/177344594 wait complete. 2024-03-30T00:57:11.071 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:57:11.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.257+0000 7f8a41400700 1 -- 172.21.15.116:0/238488960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a3c102810 msgr2=0x7f8a3c1090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.257+0000 7f8a41400700 1 --2- 172.21.15.116:0/238488960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a3c102810 0x7f8a3c1090c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8a34009f90 tx=0x7f8a3400b330 comp rx=0 tx=0).stop 2024-03-30T00:57:11.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.257+0000 7f8a41400700 1 -- 172.21.15.116:0/238488960 shutdown_connections 2024-03-30T00:57:11.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.257+0000 7f8a41400700 1 --2- 172.21.15.116:0/238488960 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a3c102810 0x7f8a3c1090c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.257+0000 7f8a41400700 1 --2- 172.21.15.116:0/238488960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a3c100c80 0x7f8a3c1010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.257+0000 7f8a41400700 1 --2- 172.21.15.116:0/238488960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a3c101e70 0x7f8a3c1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.257+0000 7f8a41400700 1 -- 172.21.15.116:0/238488960 >> 172.21.15.116:0/238488960 conn(0x7f8a3c0fc1f0 msgr2=0x7f8a3c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:11.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.257+0000 7f8a41400700 1 -- 172.21.15.116:0/238488960 shutdown_connections 2024-03-30T00:57:11.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.257+0000 7f8a41400700 1 -- 172.21.15.116:0/238488960 wait complete. 2024-03-30T00:57:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.258+0000 7f8a41400700 1 Processor -- start 2024-03-30T00:57:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.258+0000 7f8a41400700 1 -- start start 2024-03-30T00:57:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a41400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a3c100c80 0x7f8a3c196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a41400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a3c101e70 0x7f8a3c1972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a41400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a3c102810 0x7f8a3c19e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a41400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c19e860 con 0x7f8a3c101e70 2024-03-30T00:57:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a41400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c19e9d0 con 0x7f8a3c102810 2024-03-30T00:57:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a41400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c19eb90 con 0x7f8a3c100c80 2024-03-30T00:57:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a39e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a3c101e70 0x7f8a3c1972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a39e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a3c101e70 0x7f8a3c1972b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46114/0 (socket says 172.21.15.116:46114) 2024-03-30T00:57:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a39e00700 1 -- 172.21.15.116:0/3517340285 learned_addr learned my addr 172.21.15.116:0/3517340285 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a3a800700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a3c100c80 0x7f8a3c196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a3b200700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a3c102810 0x7f8a3c19e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a39e00700 1 -- 172.21.15.116:0/3517340285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a3c100c80 msgr2=0x7f8a3c196d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a39e00700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a3c100c80 0x7f8a3c196d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a39e00700 1 -- 172.21.15.116:0/3517340285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a3c102810 msgr2=0x7f8a3c19e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a39e00700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a3c102810 0x7f8a3c19e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a39e00700 1 -- 172.21.15.116:0/3517340285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a34009c40 con 0x7f8a3c101e70 2024-03-30T00:57:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.259+0000 7f8a3b200700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a3c102810 0x7f8a3c19e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.260+0000 7f8a39e00700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a3c101e70 0x7f8a3c1972b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800e9a0 tx=0x7f8a2800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.260+0000 7f8a33400700 1 -- 172.21.15.116:0/3517340285 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a28009790 con 0x7f8a3c101e70 2024-03-30T00:57:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.260+0000 7f8a33400700 1 -- 172.21.15.116:0/3517340285 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a2800cbd0 con 0x7f8a3c101e70 2024-03-30T00:57:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.260+0000 7f8a41400700 1 -- 172.21.15.116:0/3517340285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3c19ee70 con 0x7f8a3c101e70 2024-03-30T00:57:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.260+0000 7f8a33400700 1 -- 172.21.15.116:0/3517340285 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2800eee0 con 0x7f8a3c101e70 2024-03-30T00:57:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.260+0000 7f8a41400700 1 -- 172.21.15.116:0/3517340285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3c19f290 con 0x7f8a3c101e70 2024-03-30T00:57:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.261+0000 7f8a41400700 1 -- 172.21.15.116:0/3517340285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a3c108080 con 0x7f8a3c101e70 2024-03-30T00:57:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.263+0000 7f8a33400700 1 -- 172.21.15.116:0/3517340285 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f8a2800cd40 con 0x7f8a3c101e70 2024-03-30T00:57:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.264+0000 7f8a33400700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8a200a11b0 0x7f8a200a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.264+0000 7f8a33400700 1 -- 172.21.15.116:0/3517340285 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8a28014070 con 0x7f8a3c101e70 2024-03-30T00:57:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.264+0000 7f8a3a800700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8a200a11b0 0x7f8a200a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.265+0000 7f8a3a800700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8a200a11b0 0x7f8a200a3670 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c1981f0 tx=0x7f8a2c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:11.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.268+0000 7f8a33400700 1 -- 172.21.15.116:0/3517340285 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8a280529f0 con 0x7f8a3c101e70 2024-03-30T00:57:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.523+0000 7f8a41400700 1 -- 172.21.15.116:0/3517340285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a3c067a10 con 0x7f8a3c101e70 2024-03-30T00:57:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.524+0000 7f8a33400700 1 -- 172.21.15.116:0/3517340285 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+10886 (secure 0 0 0) 0x7f8a28099d80 con 0x7f8a3c101e70 2024-03-30T00:57:11.530 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:11.530 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:56:22.144079+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T00:57:11.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.527+0000 7f8a41400700 1 -- 172.21.15.116:0/3517340285 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8a200a11b0 msgr2=0x7f8a200a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.527+0000 7f8a41400700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8a200a11b0 0x7f8a200a3670 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c1981f0 tx=0x7f8a2c009310 comp rx=0 tx=0).stop 2024-03-30T00:57:11.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.527+0000 7f8a41400700 1 -- 172.21.15.116:0/3517340285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a3c101e70 msgr2=0x7f8a3c1972b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.527+0000 7f8a41400700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a3c101e70 0x7f8a3c1972b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800e9a0 tx=0x7f8a2800ed60 comp rx=0 tx=0).stop 2024-03-30T00:57:11.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.527+0000 7f8a41400700 1 -- 172.21.15.116:0/3517340285 shutdown_connections 2024-03-30T00:57:11.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.527+0000 7f8a41400700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f8a200a11b0 0x7f8a200a3670 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.527+0000 7f8a41400700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a3c100c80 0x7f8a3c196d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.527+0000 7f8a41400700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a3c102810 0x7f8a3c19e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.527+0000 7f8a41400700 1 --2- 172.21.15.116:0/3517340285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a3c101e70 0x7f8a3c1972b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.527+0000 7f8a41400700 1 -- 172.21.15.116:0/3517340285 >> 172.21.15.116:0/3517340285 conn(0x7f8a3c0fc1f0 msgr2=0x7f8a3c1051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:11.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.527+0000 7f8a41400700 1 -- 172.21.15.116:0/3517340285 shutdown_connections 2024-03-30T00:57:11.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.527+0000 7f8a41400700 1 -- 172.21.15.116:0/3517340285 wait complete. 2024-03-30T00:57:11.533 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 22 2024-03-30T00:57:11.552 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-30T00:55:51.790326+0000', 'modified': '2024-03-30T00:56:22.144079+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24587, 'mds_1': 24548, 'mds_2': 24541}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24511': {'gid': 24511, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6835/702133015', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6834', 'nonce': 702133015}, {'type': 'v1', 'addr': '172.21.15.181:6835', 'nonce': 702133015}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24517': {'gid': 24517, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6837/941347789', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6836', 'nonce': 941347789}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 941347789}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24524': {'gid': 24524, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6835/3346365474', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6834', 'nonce': 3346365474}, {'type': 'v1', 'addr': '172.21.15.130:6835', 'nonce': 3346365474}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24541': {'gid': 24541, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.181:6839/3114781386', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6838', 'nonce': 3114781386}, {'type': 'v1', 'addr': '172.21.15.181:6839', 'nonce': 3114781386}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24548': {'gid': 24548, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.130:6837/3389690261', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6836', 'nonce': 3389690261}, {'type': 'v1', 'addr': '172.21.15.130:6837', 'nonce': 3389690261}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.130:6839/1184545085', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 1184545085}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 1184545085}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-30T00:57:11.552 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-30T00:57:11.553 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-30T00:57:11.609 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:11 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1120519611' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:11.609 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:11 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/177344594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.740+0000 7fe7fda00700 1 -- 172.21.15.116:0/845171160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f8102910 msgr2=0x7fe7f81091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.740+0000 7fe7fda00700 1 --2- 172.21.15.116:0/845171160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f8102910 0x7fe7f81091c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe7f000b0d0 tx=0x7fe7f000b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.740+0000 7fe7fda00700 1 -- 172.21.15.116:0/845171160 shutdown_connections 2024-03-30T00:57:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.740+0000 7fe7fda00700 1 --2- 172.21.15.116:0/845171160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7f8100d80 0x7fe7f81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.740+0000 7fe7fda00700 1 --2- 172.21.15.116:0/845171160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f8102910 0x7fe7f81091c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.740+0000 7fe7fda00700 1 --2- 172.21.15.116:0/845171160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f8101f70 0x7fe7f81023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.740+0000 7fe7fda00700 1 -- 172.21.15.116:0/845171160 >> 172.21.15.116:0/845171160 conn(0x7fe7f80fc2f0 msgr2=0x7fe7f80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.740+0000 7fe7fda00700 1 -- 172.21.15.116:0/845171160 shutdown_connections 2024-03-30T00:57:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.740+0000 7fe7fda00700 1 -- 172.21.15.116:0/845171160 wait complete. 2024-03-30T00:57:11.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.741+0000 7fe7fda00700 1 Processor -- start 2024-03-30T00:57:11.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.741+0000 7fe7fda00700 1 -- start start 2024-03-30T00:57:11.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.741+0000 7fe7fda00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f8100d80 0x7fe7f8199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.741+0000 7fe7fda00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7f8101f70 0x7fe7f8199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.741+0000 7fe7fda00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f8102910 0x7fe7f81a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.741+0000 7fe7fda00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f81a0b80 con 0x7fe7f8102910 2024-03-30T00:57:11.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.742+0000 7fe7fda00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f81a0cc0 con 0x7fe7f8100d80 2024-03-30T00:57:11.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.742+0000 7fe7fda00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f81a0e50 con 0x7fe7f8101f70 2024-03-30T00:57:11.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.742+0000 7fe7f7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f8102910 0x7fe7f81a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.742+0000 7fe7f7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f8102910 0x7fe7f81a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46144/0 (socket says 172.21.15.116:46144) 2024-03-30T00:57:11.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.742+0000 7fe7f7400700 1 -- 172.21.15.116:0/2995597983 learned_addr learned my addr 172.21.15.116:0/2995597983 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:11.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.742+0000 7fe7f6000700 1 --2- 172.21.15.116:0/2995597983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7f8101f70 0x7fe7f8199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.742+0000 7fe7f6a00700 1 --2- 172.21.15.116:0/2995597983 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f8100d80 0x7fe7f8199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.742+0000 7fe7f7400700 1 -- 172.21.15.116:0/2995597983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7f8101f70 msgr2=0x7fe7f8199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.742+0000 7fe7f7400700 1 --2- 172.21.15.116:0/2995597983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7f8101f70 0x7fe7f8199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.742+0000 7fe7f7400700 1 -- 172.21.15.116:0/2995597983 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f8100d80 msgr2=0x7fe7f8199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.742+0000 7fe7f7400700 1 --2- 172.21.15.116:0/2995597983 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f8100d80 0x7fe7f8199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.742+0000 7fe7f7400700 1 -- 172.21.15.116:0/2995597983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e4009520 con 0x7fe7f8102910 2024-03-30T00:57:11.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.742+0000 7fe7f7400700 1 --2- 172.21.15.116:0/2995597983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f8102910 0x7fe7f81a05b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0007b60 tx=0x7fe7f0008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:11.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.743+0000 7fe7ef400700 1 -- 172.21.15.116:0/2995597983 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f000e040 con 0x7fe7f8102910 2024-03-30T00:57:11.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.743+0000 7fe7ef400700 1 -- 172.21.15.116:0/2995597983 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7f001d930 con 0x7fe7f8102910 2024-03-30T00:57:11.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.743+0000 7fe7ef400700 1 -- 172.21.15.116:0/2995597983 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f001b9f0 con 0x7fe7f8102910 2024-03-30T00:57:11.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.743+0000 7fe7fda00700 1 -- 172.21.15.116:0/2995597983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f0009d00 con 0x7fe7f8102910 2024-03-30T00:57:11.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.743+0000 7fe7fda00700 1 -- 172.21.15.116:0/2995597983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f81a1370 con 0x7fe7f8102910 2024-03-30T00:57:11.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.746+0000 7fe7ef400700 1 -- 172.21.15.116:0/2995597983 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fe7f001daa0 con 0x7fe7f8102910 2024-03-30T00:57:11.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:11 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1120519611' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:11.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:11 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/177344594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:11.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.747+0000 7fe81143ab80 1 Processor -- start 2024-03-30T00:57:11.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.747+0000 7fe81143ab80 1 -- start start 2024-03-30T00:57:11.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.747+0000 7fe7ef400700 1 --2- 172.21.15.116:0/2995597983 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe7dc0a1230 0x7fe7dc0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.747+0000 7fe81143ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979a9780 0x5587979a9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.747+0000 7fe81143ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aa0e0 0x5587979aa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.747+0000 7fe81143ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979ab400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.747+0000 7fe81143ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558797901940 con 0x5587979aaf80 2024-03-30T00:57:11.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.747+0000 7fe81143ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587978ff300 con 0x5587979aa0e0 2024-03-30T00:57:11.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.747+0000 7fe81143ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587978fabf0 con 0x5587979a9780 2024-03-30T00:57:11.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.747+0000 7fe7ef400700 1 -- 172.21.15.116:0/2995597983 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe7f00ce290 con 0x7fe7f8102910 2024-03-30T00:57:11.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.747+0000 7fe7f7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979ab400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.748+0000 7fe7f6000700 1 --2- 172.21.15.116:0/2995597983 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe7dc0a1230 0x7fe7dc0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.748+0000 7fe7f7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979ab400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46160/0 (socket says 172.21.15.116:46160) 2024-03-30T00:57:11.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.748+0000 7fe7f7400700 1 -- 172.21.15.116:0/3897944745 learned_addr learned my addr 172.21.15.116:0/3897944745 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:11.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.748+0000 7fe7f6a00700 1 --2- 172.21.15.116:0/3897944745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979a9780 0x5587979a9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.748+0000 7fe7f6000700 1 --2- 172.21.15.116:0/3897944745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aa0e0 0x5587979aa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.748+0000 7fe7f7400700 1 -- 172.21.15.116:0/3897944745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979a9780 msgr2=0x5587979a9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:11.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.748+0000 7fe7f7400700 1 --2- 172.21.15.116:0/3897944745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979a9780 0x5587979a9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.748+0000 7fe7f7400700 1 -- 172.21.15.116:0/3897944745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aa0e0 msgr2=0x5587979aa9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:11.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.748+0000 7fe7f7400700 1 --2- 172.21.15.116:0/3897944745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aa0e0 0x5587979aa9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.748+0000 7fe7f7400700 1 -- 172.21.15.116:0/3897944745 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55879777c360 con 0x5587979aaf80 2024-03-30T00:57:11.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.748+0000 7fe7f6a00700 1 --2- 172.21.15.116:0/3897944745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979a9780 0x5587979a9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:11.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.748+0000 7fe7f7400700 1 --2- 172.21.15.116:0/3897944745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979ab400 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0097d10 tx=0x7fe7f009b0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6ece171f7e0a93 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:11.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.748+0000 7fe7f6000700 1 --2- 172.21.15.116:0/2995597983 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe7dc0a1230 0x7fe7dc0a36f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009fd0 tx=0x7fe7e4009a60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:11.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.748+0000 7fe7e3e00700 1 -- 172.21.15.116:0/3897944745 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f009b9a0 con 0x5587979aaf80 2024-03-30T00:57:11.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.749+0000 7fe7e3e00700 1 -- 172.21.15.116:0/3897944745 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7f009b2a0 con 0x5587979aaf80 2024-03-30T00:57:11.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.749+0000 7fe7e3e00700 1 -- 172.21.15.116:0/3897944745 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f00cbb20 con 0x5587979aaf80 2024-03-30T00:57:11.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.749+0000 7fe81143ab80 1 -- 172.21.15.116:0/3897944745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 msgr2=0x5587979ab400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.749+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3897944745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979ab400 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0097d10 tx=0x7fe7f009b0f0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.749+0000 7fe81143ab80 1 -- 172.21.15.116:0/3897944745 shutdown_connections 2024-03-30T00:57:11.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.749+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3897944745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979a9780 0x5587979a9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.749+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3897944745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979ab400 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.749+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3897944745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aa0e0 0x5587979aa9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.749+0000 7fe81143ab80 1 -- 172.21.15.116:0/3897944745 >> 172.21.15.116:0/3897944745 conn(0x558797908650 msgr2=0x5587979a4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.749+0000 7fe81143ab80 1 -- 172.21.15.116:0/3897944745 shutdown_connections 2024-03-30T00:57:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.749+0000 7fe81143ab80 1 -- 172.21.15.116:0/3897944745 wait complete. 2024-03-30T00:57:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.750+0000 7fe81143ab80 1 -- start start 2024-03-30T00:57:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.750+0000 7fe81143ab80 1 Processor -- start 2024-03-30T00:57:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.750+0000 7fe81143ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979a9780 0x5587979b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe81143ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aa0e0 0x5587979b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe81143ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979ba920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe81143ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558797896d30 con 0x5587979aaf80 2024-03-30T00:57:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe81143ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587978969d0 con 0x5587979aa0e0 2024-03-30T00:57:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe81143ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55879777e650 con 0x5587979a9780 2024-03-30T00:57:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe81143ab80 10 client.0 _ll_register_callbacks cb 0x5587978a0420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe7f7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979ba920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe7f7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979ba920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46172/0 (socket says 172.21.15.116:46172) 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe7f7400700 1 -- 172.21.15.116:0/3260938465 learned_addr learned my addr 172.21.15.116:0/3260938465 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe7f6a00700 1 --2- 172.21.15.116:0/3260938465 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979a9780 0x5587979b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe7f6000700 1 --2- 172.21.15.116:0/3260938465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aa0e0 0x5587979b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe7f7400700 1 -- 172.21.15.116:0/3260938465 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979a9780 msgr2=0x5587979b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe7f7400700 1 --2- 172.21.15.116:0/3260938465 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979a9780 0x5587979b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe7f7400700 1 -- 172.21.15.116:0/3260938465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aa0e0 msgr2=0x5587979b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe7f7400700 1 --2- 172.21.15.116:0/3260938465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aa0e0 0x5587979b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.751+0000 7fe7f7400700 1 -- 172.21.15.116:0/3260938465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e80098e0 con 0x5587979aaf80 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe7f7400700 1 --2- 172.21.15.116:0/3260938465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979ba920 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0009fd0 tx=0x7fe7f0007550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d52e057c2c2f5b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe7e0c00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe7e0c00700 1 -- 172.21.15.116:0/3260938465 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f009f0d0 con 0x5587979aaf80 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe7e0c00700 1 -- 172.21.15.116:0/3260938465 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe7f00a3690 con 0x5587979aaf80 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe81143ab80 1 -- 172.21.15.116:0/3260938465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5587978bb330 con 0x5587979aaf80 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe7e0c00700 1 -- 172.21.15.116:0/3260938465 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7f009b710 con 0x5587979aaf80 2024-03-30T00:57:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe81143ab80 10 client.15093 fetch_fsmap learned FSMap version 22 2024-03-30T00:57:11.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe81143ab80 1 -- 172.21.15.116:0/3260938465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7fe7f81a1370 con 0x5587979aaf80 2024-03-30T00:57:11.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe7e0c00700 1 -- 172.21.15.116:0/3260938465 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5587978bb330 con 0x5587979aaf80 2024-03-30T00:57:11.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe7e0c00700 1 -- 172.21.15.116:0/3260938465 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 22) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe7f009d070 con 0x5587979aaf80 2024-03-30T00:57:11.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe81143ab80 10 client.15093 fetch_fsmap finished waiting for FSMap version 22 2024-03-30T00:57:11.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe81143ab80 10 client.15093 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-30T00:57:11.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe81143ab80 20 client.15093 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:11.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3260938465 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5587979be1f0 0x5587979c3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe81143ab80 4 client.15093 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:57:11.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.752+0000 7fe81143ab80 1 -- 172.21.15.116:0/3260938465 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5587978f36b0 con 0x5587979be1f0 2024-03-30T00:57:11.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.753+0000 7fe7f6a00700 1 --2- 172.21.15.116:0/3260938465 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5587979be1f0 0x5587979c3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.753+0000 7fe7f6a00700 1 --2- 172.21.15.116:0/3260938465 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5587979be1f0 0x5587979c3490 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d795d592fd9d631e server_cookie=21e4c2865bab72cc in_seq=0 out_seq=0 2024-03-30T00:57:11.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.753+0000 7fe7e0c00700 10 client.15093 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T00:57:11.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.754+0000 7fe7e0c00700 1 -- 172.21.15.116:0/3260938465 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5587978f36b0 con 0x5587979be1f0 2024-03-30T00:57:11.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.754+0000 7fe7e0c00700 10 client.15093 handle_command_reply: tid=0 2024-03-30T00:57:11.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.754+0000 7fe81143ab80 1 client.15093 shutdown 2024-03-30T00:57:11.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.754+0000 7fe81143ab80 1 -- 172.21.15.116:0/3260938465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 msgr2=0x5587979ba920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.754+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3260938465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979ba920 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0009fd0 tx=0x7fe7f0007550 comp rx=0 tx=0).stop 2024-03-30T00:57:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.754+0000 7fe81143ab80 1 -- 172.21.15.116:0/3260938465 shutdown_connections 2024-03-30T00:57:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.754+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3260938465 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5587979be1f0 0x5587979c3490 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.754+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3260938465 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979a9780 0x5587979b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.754+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3260938465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979ba920 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.754+0000 7fe7e0c00700 0 client.15093 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T00:57:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.754+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3260938465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aa0e0 0x5587979b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.754+0000 7fe81143ab80 1 -- 172.21.15.116:0/3260938465 >> 172.21.15.116:0/3260938465 conn(0x558797908650 msgr2=0x5587979a8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.755+0000 7fe81143ab80 1 -- 172.21.15.116:0/3260938465 shutdown_connections 2024-03-30T00:57:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.755+0000 7fe81143ab80 1 -- 172.21.15.116:0/3260938465 wait complete. 2024-03-30T00:57:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.755+0000 7fe81143ab80 20 client.15093 trim_cache size 0 max 16384 2024-03-30T00:57:11.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.772+0000 7fe81143ab80 1 Processor -- start 2024-03-30T00:57:11.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.772+0000 7fe81143ab80 1 -- start start 2024-03-30T00:57:11.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe81143ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979a9780 0x5587979c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe81143ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979aa0e0 0x5587979ba8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe81143ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aaf80 0x5587979ab360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe81143ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587979a5610 con 0x5587979a9780 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe81143ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587979a4ed0 con 0x5587979aaf80 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe81143ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587978f36e0 con 0x5587979aa0e0 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe7f6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979a9780 0x5587979c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe7f6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979a9780 0x5587979c7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46184/0 (socket says 172.21.15.116:46184) 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe7f6a00700 1 -- 172.21.15.116:0/2609761355 learned_addr learned my addr 172.21.15.116:0/2609761355 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe7f7400700 1 --2- 172.21.15.116:0/2609761355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aaf80 0x5587979ab360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe7f6000700 1 --2- 172.21.15.116:0/2609761355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979aa0e0 0x5587979ba8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe7f6a00700 1 -- 172.21.15.116:0/2609761355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979aa0e0 msgr2=0x5587979ba8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe7f6a00700 1 --2- 172.21.15.116:0/2609761355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979aa0e0 0x5587979ba8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe7f6a00700 1 -- 172.21.15.116:0/2609761355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aaf80 msgr2=0x5587979ab360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe7f6a00700 1 --2- 172.21.15.116:0/2609761355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aaf80 0x5587979ab360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.773+0000 7fe7f6a00700 1 -- 172.21.15.116:0/2609761355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558797728b10 con 0x5587979a9780 2024-03-30T00:57:11.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.774+0000 7fe7f6a00700 1 --2- 172.21.15.116:0/2609761355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979a9780 0x5587979c7bc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8000c00 tx=0x7fe7e8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b6a0fabb2b517b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:11.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.774+0000 7fe7ecc00700 1 -- 172.21.15.116:0/2609761355 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8017070 con 0x5587979a9780 2024-03-30T00:57:11.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.774+0000 7fe7ecc00700 1 -- 172.21.15.116:0/2609761355 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7e8018040 con 0x5587979a9780 2024-03-30T00:57:11.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.774+0000 7fe7ecc00700 1 -- 172.21.15.116:0/2609761355 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e801e8b0 con 0x5587979a9780 2024-03-30T00:57:11.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.774+0000 7fe81143ab80 1 -- 172.21.15.116:0/2609761355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979a9780 msgr2=0x5587979c7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.774+0000 7fe81143ab80 1 --2- 172.21.15.116:0/2609761355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979a9780 0x5587979c7bc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8000c00 tx=0x7fe7e8009260 comp rx=0 tx=0).stop 2024-03-30T00:57:11.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.774+0000 7fe81143ab80 1 -- 172.21.15.116:0/2609761355 shutdown_connections 2024-03-30T00:57:11.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.774+0000 7fe81143ab80 1 --2- 172.21.15.116:0/2609761355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979aaf80 0x5587979ab360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.774+0000 7fe81143ab80 1 --2- 172.21.15.116:0/2609761355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979a9780 0x5587979c7bc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.774+0000 7fe81143ab80 1 --2- 172.21.15.116:0/2609761355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979aa0e0 0x5587979ba8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.774+0000 7fe81143ab80 1 -- 172.21.15.116:0/2609761355 >> 172.21.15.116:0/2609761355 conn(0x558797908650 msgr2=0x5587979c74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:11.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.775+0000 7fe81143ab80 1 -- 172.21.15.116:0/2609761355 shutdown_connections 2024-03-30T00:57:11.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.775+0000 7fe81143ab80 1 -- 172.21.15.116:0/2609761355 wait complete. 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.775+0000 7fe81143ab80 1 -- start start 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.775+0000 7fe81143ab80 1 Processor -- start 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.776+0000 7fe81143ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979a9780 0x5587979ba730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.776+0000 7fe81143ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979aa0e0 0x5587979b73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.776+0000 7fe81143ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.776+0000 7fe81143ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587979b8010 con 0x5587979aaf80 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.776+0000 7fe81143ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558797a12ab0 con 0x5587979a9780 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.776+0000 7fe81143ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558797a12c70 con 0x5587979aa0e0 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.776+0000 7fe81143ab80 10 client.0 _ll_register_callbacks cb 0x5587978a0420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.776+0000 7fe7f7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.776+0000 7fe7f7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979b78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46186/0 (socket says 172.21.15.116:46186) 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.776+0000 7fe7f7400700 1 -- 172.21.15.116:0/3070166369 learned_addr learned my addr 172.21.15.116:0/3070166369 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.776+0000 7fe7f6000700 1 --2- 172.21.15.116:0/3070166369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979aa0e0 0x5587979b73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.776+0000 7fe7f6a00700 1 --2- 172.21.15.116:0/3070166369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979a9780 0x5587979ba730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.777+0000 7fe7f7400700 1 -- 172.21.15.116:0/3070166369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979aa0e0 msgr2=0x5587979b73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.777+0000 7fe7f7400700 1 --2- 172.21.15.116:0/3070166369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979aa0e0 0x5587979b73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.777+0000 7fe7f7400700 1 -- 172.21.15.116:0/3070166369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979a9780 msgr2=0x5587979ba730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.777+0000 7fe7f7400700 1 --2- 172.21.15.116:0/3070166369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979a9780 0x5587979ba730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.777+0000 7fe7f7400700 1 -- 172.21.15.116:0/3070166369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e8010430 con 0x5587979aaf80 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.777+0000 7fe7f7400700 1 --2- 172.21.15.116:0/3070166369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979b78f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00ce4f0 tx=0x7fe7f0009c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98b0240a45969d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.777+0000 7fe7e0c00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.777+0000 7fe7e0c00700 1 -- 172.21.15.116:0/3070166369 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f00a34d0 con 0x5587979aaf80 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.777+0000 7fe7e0c00700 1 -- 172.21.15.116:0/3070166369 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe7f00a3200 con 0x5587979aaf80 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.777+0000 7fe81143ab80 1 -- 172.21.15.116:0/3070166369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe7f009b2a0 con 0x5587979aaf80 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.777+0000 7fe7e0c00700 1 -- 172.21.15.116:0/3070166369 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7f009ade0 con 0x5587979aaf80 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.777+0000 7fe81143ab80 10 client.15099 fetch_fsmap learned FSMap version 22 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.777+0000 7fe81143ab80 1 -- 172.21.15.116:0/3070166369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5587978c0e50 con 0x5587979aaf80 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.777+0000 7fe7e0c00700 1 -- 172.21.15.116:0/3070166369 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7f009b2a0 con 0x5587979aaf80 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.778+0000 7fe7e0c00700 1 -- 172.21.15.116:0/3070166369 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 22) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe7f009d070 con 0x5587979aaf80 2024-03-30T00:57:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.778+0000 7fe81143ab80 10 client.15099 fetch_fsmap finished waiting for FSMap version 22 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.778+0000 7fe81143ab80 10 client.15099 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.778+0000 7fe81143ab80 20 client.15099 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.778+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3070166369 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x558797a17480 0x558797a13f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.778+0000 7fe81143ab80 4 client.15099 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.778+0000 7fe81143ab80 1 -- 172.21.15.116:0/3070166369 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558797749ca0 con 0x558797a17480 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.778+0000 7fe7f6a00700 1 --2- 172.21.15.116:0/3070166369 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x558797a17480 0x558797a13f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.778+0000 7fe7f6a00700 1 --2- 172.21.15.116:0/3070166369 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x558797a17480 0x558797a13f20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c3939f99935b07f9 server_cookie=75b8cd2418667967 in_seq=0 out_seq=0 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.778+0000 7fe7e0c00700 10 client.15099 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe7e0c00700 1 -- 172.21.15.116:0/3070166369 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x558797749ca0 con 0x558797a17480 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe7e0c00700 10 client.15099 handle_command_reply: tid=0 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe81143ab80 1 client.15099 shutdown 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe81143ab80 1 -- 172.21.15.116:0/3070166369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 msgr2=0x5587979b78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3070166369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979b78f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00ce4f0 tx=0x7fe7f0009c10 comp rx=0 tx=0).stop 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe81143ab80 1 -- 172.21.15.116:0/3070166369 shutdown_connections 2024-03-30T00:57:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3070166369 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x558797a17480 0x558797a13f20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3070166369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587979aaf80 0x5587979b78f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3070166369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587979a9780 0x5587979ba730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe81143ab80 1 --2- 172.21.15.116:0/3070166369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5587979aa0e0 0x5587979b73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe81143ab80 1 -- 172.21.15.116:0/3070166369 >> 172.21.15.116:0/3070166369 conn(0x558797908650 msgr2=0x5587979a8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:11.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe7e0c00700 0 client.15099 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T00:57:11.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe81143ab80 1 -- 172.21.15.116:0/3070166369 shutdown_connections 2024-03-30T00:57:11.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe81143ab80 1 -- 172.21.15.116:0/3070166369 wait complete. 2024-03-30T00:57:11.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.779+0000 7fe81143ab80 20 client.15099 trim_cache size 0 max 16384 2024-03-30T00:57:11.789 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:57:11.789 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "4909f004-ee2f-11ee-b647-cb9ed24678a4", 2024-03-30T00:57:11.789 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 2, 2024-03-30T00:57:11.790 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24541, 2024-03-30T00:57:11.790 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-30T00:57:11.790 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-30T00:57:11.790 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-30T00:57:11.790 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 50.378426625000003, 2024-03-30T00:57:11.790 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 22, 2024-03-30T00:57:11.790 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 83, 2024-03-30T00:57:11.790 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 81, 2024-03-30T00:57:11.790 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 91.200453437999997 2024-03-30T00:57:11.790 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:57:11.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.780+0000 7fe7fda00700 1 -- 172.21.15.116:0/2995597983 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe7dc0a1230 msgr2=0x7fe7dc0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.780+0000 7fe7fda00700 1 --2- 172.21.15.116:0/2995597983 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe7dc0a1230 0x7fe7dc0a36f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009fd0 tx=0x7fe7e4009a60 comp rx=0 tx=0).stop 2024-03-30T00:57:11.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.780+0000 7fe7fda00700 1 -- 172.21.15.116:0/2995597983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f8102910 msgr2=0x7fe7f81a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:11.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.780+0000 7fe7fda00700 1 --2- 172.21.15.116:0/2995597983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f8102910 0x7fe7f81a05b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0007b60 tx=0x7fe7f0008e40 comp rx=0 tx=0).stop 2024-03-30T00:57:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.780+0000 7fe7fda00700 1 -- 172.21.15.116:0/2995597983 shutdown_connections 2024-03-30T00:57:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.780+0000 7fe7fda00700 1 --2- 172.21.15.116:0/2995597983 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe7dc0a1230 0x7fe7dc0a36f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.780+0000 7fe7fda00700 1 --2- 172.21.15.116:0/2995597983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f8102910 0x7fe7f81a05b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.780+0000 7fe7fda00700 1 --2- 172.21.15.116:0/2995597983 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f8100d80 0x7fe7f8199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.780+0000 7fe7fda00700 1 --2- 172.21.15.116:0/2995597983 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7f8101f70 0x7fe7f8199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.780+0000 7fe7fda00700 1 -- 172.21.15.116:0/2995597983 >> 172.21.15.116:0/2995597983 conn(0x7fe7f80fc2f0 msgr2=0x7fe7f8072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.780+0000 7fe7fda00700 1 -- 172.21.15.116:0/2995597983 shutdown_connections 2024-03-30T00:57:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.780+0000 7fe7fda00700 1 -- 172.21.15.116:0/2995597983 wait complete. 2024-03-30T00:57:11.807 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-30T00:57:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:11 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1120519611' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:11 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/177344594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.995+0000 7f02ca600700 1 -- 172.21.15.116:0/3989176036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02c4101f70 msgr2=0x7f02c41023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.995+0000 7f02ca600700 1 --2- 172.21.15.116:0/3989176036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02c4101f70 0x7f02c41023d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009920 tx=0x7f02b0009c30 comp rx=0 tx=0).stop 2024-03-30T00:57:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.996+0000 7f02ca600700 1 -- 172.21.15.116:0/3989176036 shutdown_connections 2024-03-30T00:57:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.996+0000 7f02ca600700 1 --2- 172.21.15.116:0/3989176036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02c4102910 0x7f02c41091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.996+0000 7f02ca600700 1 --2- 172.21.15.116:0/3989176036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c4100d80 0x7f02c41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.996+0000 7f02ca600700 1 --2- 172.21.15.116:0/3989176036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02c4101f70 0x7f02c41023d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.996+0000 7f02ca600700 1 -- 172.21.15.116:0/3989176036 >> 172.21.15.116:0/3989176036 conn(0x7f02c40fc2f0 msgr2=0x7f02c40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.996+0000 7f02ca600700 1 -- 172.21.15.116:0/3989176036 shutdown_connections 2024-03-30T00:57:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.996+0000 7f02ca600700 1 -- 172.21.15.116:0/3989176036 wait complete. 2024-03-30T00:57:12.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.997+0000 7f02ca600700 1 Processor -- start 2024-03-30T00:57:12.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.997+0000 7f02ca600700 1 -- start start 2024-03-30T00:57:12.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.997+0000 7f02ca600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c4100d80 0x7f02c4198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.997+0000 7f02ca600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02c4101f70 0x7f02c4199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.997+0000 7f02ca600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02c4102910 0x7f02c41a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.997+0000 7f02ca600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c41a0b00 con 0x7f02c4100d80 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.997+0000 7f02ca600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c41a0c40 con 0x7f02c4101f70 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.997+0000 7f02ca600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c41a0dd0 con 0x7f02c4102910 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.997+0000 7f02c3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c4100d80 0x7f02c4198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.997+0000 7f02c3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c4100d80 0x7f02c4198fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46212/0 (socket says 172.21.15.116:46212) 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.997+0000 7f02c3400700 1 -- 172.21.15.116:0/3667192332 learned_addr learned my addr 172.21.15.116:0/3667192332 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.998+0000 7f02c2a00700 1 --2- 172.21.15.116:0/3667192332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02c4101f70 0x7f02c4199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.998+0000 7f02c3e00700 1 --2- 172.21.15.116:0/3667192332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02c4102910 0x7f02c41a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.998+0000 7f02c3400700 1 -- 172.21.15.116:0/3667192332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02c4102910 msgr2=0x7f02c41a0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.998+0000 7f02c3400700 1 --2- 172.21.15.116:0/3667192332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02c4102910 0x7f02c41a0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.998+0000 7f02c3400700 1 -- 172.21.15.116:0/3667192332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02c4101f70 msgr2=0x7f02c4199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.998+0000 7f02c3400700 1 --2- 172.21.15.116:0/3667192332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02c4101f70 0x7f02c4199510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.998+0000 7f02c3400700 1 -- 172.21.15.116:0/3667192332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b0009520 con 0x7f02c4100d80 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.998+0000 7f02c3400700 1 --2- 172.21.15.116:0/3667192332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c4100d80 0x7f02c4198fd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f02b400e9f0 tx=0x7f02b400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.998+0000 7f02bbe00700 1 -- 172.21.15.116:0/3667192332 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b4004220 con 0x7f02c4100d80 2024-03-30T00:57:12.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.998+0000 7f02bbe00700 1 -- 172.21.15.116:0/3667192332 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02b4004380 con 0x7f02c4100d80 2024-03-30T00:57:12.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.998+0000 7f02ca600700 1 -- 172.21.15.116:0/3667192332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c41a10b0 con 0x7f02c4100d80 2024-03-30T00:57:12.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.998+0000 7f02bbe00700 1 -- 172.21.15.116:0/3667192332 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b4003be0 con 0x7f02c4100d80 2024-03-30T00:57:12.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:11.999+0000 7f02ca600700 1 -- 172.21.15.116:0/3667192332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c41a1610 con 0x7f02c4100d80 2024-03-30T00:57:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.003+0000 7f02bbe00700 1 -- 172.21.15.116:0/3667192332 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f02b4003d40 con 0x7f02c4100d80 2024-03-30T00:57:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.004+0000 7f02de09db80 1 Processor -- start 2024-03-30T00:57:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.004+0000 7f02de09db80 1 -- start start 2024-03-30T00:57:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.004+0000 7f02de09db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 0x56084012aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.004+0000 7f02de09db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 0x56084012b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.004+0000 7f02de09db80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x56084012c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.004+0000 7f02de09db80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560840082940 con 0x56084012bf80 2024-03-30T00:57:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.004+0000 7f02de09db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560840080300 con 0x56084012b0e0 2024-03-30T00:57:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.004+0000 7f02de09db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56084007bbf0 con 0x56084012a780 2024-03-30T00:57:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.004+0000 7f02c2a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x56084012c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.004+0000 7f02c3e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 0x56084012aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.005+0000 7f02c3e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 0x56084012aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:35394/0 (socket says 172.21.15.116:35394) 2024-03-30T00:57:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.005+0000 7f02c3e00700 1 -- 172.21.15.116:0/2374647946 learned_addr learned my addr 172.21.15.116:0/2374647946 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.005+0000 7f02c3400700 1 --2- 172.21.15.116:0/2374647946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 0x56084012b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.005+0000 7f02bbe00700 1 --2- 172.21.15.116:0/3667192332 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f02a80a1230 0x7f02a80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.005+0000 7f02bbe00700 1 -- 172.21.15.116:0/3667192332 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f02b4014070 con 0x7f02c4100d80 2024-03-30T00:57:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.005+0000 7f02c2a00700 1 -- 172.21.15.116:0/2374647946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 msgr2=0x56084012aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.005+0000 7f02c2a00700 1 --2- 172.21.15.116:0/2374647946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 0x56084012aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.005+0000 7f02c3e00700 1 --2- 172.21.15.116:0/3667192332 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f02a80a1230 0x7f02a80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.005+0000 7f02c2a00700 1 -- 172.21.15.116:0/2374647946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 msgr2=0x56084012b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.005+0000 7f02c2a00700 1 --2- 172.21.15.116:0/2374647946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 0x56084012b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.005+0000 7f02c2a00700 1 -- 172.21.15.116:0/2374647946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56083fefd360 con 0x56084012bf80 2024-03-30T00:57:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.005+0000 7f02c2a00700 1 --2- 172.21.15.116:0/2374647946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x56084012c400 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f02b000a010 tx=0x7f02b0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912f6d60bcb0a182 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.006+0000 7f02b8c00700 1 -- 172.21.15.116:0/2374647946 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b000f6e0 con 0x56084012bf80 2024-03-30T00:57:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.006+0000 7f02c3e00700 1 --2- 172.21.15.116:0/3667192332 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f02a80a1230 0x7f02a80a36f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009ee0 tx=0x7f02bc004b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.006+0000 7f02b8c00700 1 -- 172.21.15.116:0/2374647946 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02b001d040 con 0x56084012bf80 2024-03-30T00:57:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.006+0000 7f02b8c00700 1 -- 172.21.15.116:0/2374647946 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0005570 con 0x56084012bf80 2024-03-30T00:57:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.006+0000 7f02de09db80 1 -- 172.21.15.116:0/2374647946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 msgr2=0x56084012c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.006+0000 7f02de09db80 1 --2- 172.21.15.116:0/2374647946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x56084012c400 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f02b000a010 tx=0x7f02b0009f80 comp rx=0 tx=0).stop 2024-03-30T00:57:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.006+0000 7f02de09db80 1 -- 172.21.15.116:0/2374647946 shutdown_connections 2024-03-30T00:57:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.006+0000 7f02de09db80 1 --2- 172.21.15.116:0/2374647946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 0x56084012aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.006+0000 7f02de09db80 1 --2- 172.21.15.116:0/2374647946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x56084012c400 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.006+0000 7f02de09db80 1 --2- 172.21.15.116:0/2374647946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 0x56084012b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.006+0000 7f02de09db80 1 -- 172.21.15.116:0/2374647946 >> 172.21.15.116:0/2374647946 conn(0x560840089650 msgr2=0x560840125950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.006+0000 7f02de09db80 1 -- 172.21.15.116:0/2374647946 shutdown_connections 2024-03-30T00:57:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.007+0000 7f02de09db80 1 -- 172.21.15.116:0/2374647946 wait complete. 2024-03-30T00:57:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.007+0000 7f02de09db80 1 -- start start 2024-03-30T00:57:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.007+0000 7f02de09db80 1 Processor -- start 2024-03-30T00:57:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02de09db80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012a780 0x560840134570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02de09db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012b0e0 0x560840134ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02de09db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012bf80 0x56084013baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02de09db80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560840017d30 con 0x56084012a780 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02de09db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608400179d0 con 0x56084012bf80 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02de09db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56083feff650 con 0x56084012b0e0 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02de09db80 10 client.0 _ll_register_callbacks cb 0x560840021420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02c3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012a780 0x560840134570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02c3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012a780 0x560840134570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46232/0 (socket says 172.21.15.116:46232) 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02c3400700 1 -- 172.21.15.116:0/1761577777 learned_addr learned my addr 172.21.15.116:0/1761577777 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02c3e00700 1 --2- 172.21.15.116:0/1761577777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012bf80 0x56084013baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02c2a00700 1 --2- 172.21.15.116:0/1761577777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012b0e0 0x560840134ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02c3400700 1 -- 172.21.15.116:0/1761577777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012b0e0 msgr2=0x560840134ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02c3400700 1 --2- 172.21.15.116:0/1761577777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012b0e0 0x560840134ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02c3400700 1 -- 172.21.15.116:0/1761577777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012bf80 msgr2=0x56084013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02c3400700 1 --2- 172.21.15.116:0/1761577777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012bf80 0x56084013baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02c3400700 1 -- 172.21.15.116:0/1761577777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c4109b10 con 0x56084012a780 2024-03-30T00:57:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02c3e00700 1 --2- 172.21.15.116:0/1761577777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012bf80 0x56084013baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02c3400700 1 --2- 172.21.15.116:0/1761577777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012a780 0x560840134570 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f02b401c000 tx=0x7f02b400ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c44e5ac2d726a7c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.008+0000 7f02ad600700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:57:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.009+0000 7f02ad600700 1 -- 172.21.15.116:0/1761577777 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b409a490 con 0x56084012a780 2024-03-30T00:57:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.009+0000 7f02ad600700 1 -- 172.21.15.116:0/1761577777 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f02b40044f0 con 0x56084012a780 2024-03-30T00:57:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.009+0000 7f02de09db80 1 -- 172.21.15.116:0/1761577777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56084003c330 con 0x56084012a780 2024-03-30T00:57:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.009+0000 7f02ad600700 1 -- 172.21.15.116:0/1761577777 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02b4099d90 con 0x56084012a780 2024-03-30T00:57:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.009+0000 7f02de09db80 10 client.15111 fetch_fsmap learned FSMap version 22 2024-03-30T00:57:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.009+0000 7f02de09db80 1 -- 172.21.15.116:0/1761577777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x56083fea9b10 con 0x56084012a780 2024-03-30T00:57:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.009+0000 7f02ad600700 1 -- 172.21.15.116:0/1761577777 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56084003c330 con 0x56084012a780 2024-03-30T00:57:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.009+0000 7f02ad600700 1 -- 172.21.15.116:0/1761577777 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 22) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f02b40a1d80 con 0x56084012a780 2024-03-30T00:57:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.009+0000 7f02de09db80 10 client.15111 fetch_fsmap finished waiting for FSMap version 22 2024-03-30T00:57:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.009+0000 7f02de09db80 10 client.15111 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-30T00:57:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.009+0000 7f02de09db80 20 client.15111 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.009+0000 7f02de09db80 1 --2- 172.21.15.116:0/1761577777 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x56084013f780 0x560840144aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.009+0000 7f02de09db80 4 client.15111 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:57:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.009+0000 7f02de09db80 1 -- 172.21.15.116:0/1761577777 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5608400746b0 con 0x56084013f780 2024-03-30T00:57:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.010+0000 7f02c2a00700 1 --2- 172.21.15.116:0/1761577777 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x56084013f780 0x560840144aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.010+0000 7f02ad600700 10 client.15111 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T00:57:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.010+0000 7f02c2a00700 1 --2- 172.21.15.116:0/1761577777 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x56084013f780 0x560840144aa0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d0de7aedb93a77e3 server_cookie=4cc9fa198d5ead3c in_seq=0 out_seq=0 2024-03-30T00:57:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.011+0000 7f02ad600700 1 -- 172.21.15.116:0/1761577777 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5608400746b0 con 0x56084013f780 2024-03-30T00:57:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.011+0000 7f02ad600700 10 client.15111 handle_command_reply: tid=0 2024-03-30T00:57:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.011+0000 7f02de09db80 1 client.15111 shutdown 2024-03-30T00:57:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.011+0000 7f02de09db80 1 -- 172.21.15.116:0/1761577777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012a780 msgr2=0x560840134570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.011+0000 7f02de09db80 1 --2- 172.21.15.116:0/1761577777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012a780 0x560840134570 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f02b401c000 tx=0x7f02b400ef80 comp rx=0 tx=0).stop 2024-03-30T00:57:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.011+0000 7f02de09db80 1 -- 172.21.15.116:0/1761577777 shutdown_connections 2024-03-30T00:57:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.011+0000 7f02de09db80 1 --2- 172.21.15.116:0/1761577777 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x56084013f780 0x560840144aa0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.011+0000 7f02de09db80 1 --2- 172.21.15.116:0/1761577777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012bf80 0x56084013baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.011+0000 7f02de09db80 1 --2- 172.21.15.116:0/1761577777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012a780 0x560840134570 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.011+0000 7f02ad600700 0 client.15111 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T00:57:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.011+0000 7f02de09db80 1 --2- 172.21.15.116:0/1761577777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012b0e0 0x560840134ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.011+0000 7f02de09db80 1 -- 172.21.15.116:0/1761577777 >> 172.21.15.116:0/1761577777 conn(0x560840089650 msgr2=0x560840129120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.011+0000 7f02de09db80 1 -- 172.21.15.116:0/1761577777 shutdown_connections 2024-03-30T00:57:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.011+0000 7f02de09db80 1 -- 172.21.15.116:0/1761577777 wait complete. 2024-03-30T00:57:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.012+0000 7f02de09db80 20 client.15111 trim_cache size 0 max 16384 2024-03-30T00:57:12.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.029+0000 7f02de09db80 1 Processor -- start 2024-03-30T00:57:12.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.029+0000 7f02de09db80 1 -- start start 2024-03-30T00:57:12.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.029+0000 7f02de09db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 0x560840133a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.029+0000 7f02de09db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 0x56084013bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02de09db80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x560840132510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02de09db80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560840074990 con 0x56084012bf80 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02de09db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560840078850 con 0x56084012b0e0 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02de09db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56084001ade0 con 0x56084012a780 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02c3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x560840132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02c3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x560840132510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46246/0 (socket says 172.21.15.116:46246) 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02c3e00700 1 -- 172.21.15.116:0/966446787 learned_addr learned my addr 172.21.15.116:0/966446787 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02c2a00700 1 --2- 172.21.15.116:0/966446787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 0x56084013bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02c3400700 1 --2- 172.21.15.116:0/966446787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 0x560840133a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02c3e00700 1 -- 172.21.15.116:0/966446787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 msgr2=0x560840133a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02c3e00700 1 --2- 172.21.15.116:0/966446787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 0x560840133a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02c3e00700 1 -- 172.21.15.116:0/966446787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 msgr2=0x56084013bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02c3e00700 1 --2- 172.21.15.116:0/966446787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 0x56084013bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02c3e00700 1 -- 172.21.15.116:0/966446787 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560840041e50 con 0x56084012bf80 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02c3400700 1 --2- 172.21.15.116:0/966446787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 0x560840133a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.030+0000 7f02c3e00700 1 --2- 172.21.15.116:0/966446787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x560840132510 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f02bc004890 tx=0x7f02bc00b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7793ccffb2a204e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:12.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.031+0000 7f02b9600700 1 -- 172.21.15.116:0/966446787 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc01ab80 con 0x56084012bf80 2024-03-30T00:57:12.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.031+0000 7f02b9600700 1 -- 172.21.15.116:0/966446787 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02bc00f040 con 0x56084012bf80 2024-03-30T00:57:12.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.031+0000 7f02b9600700 1 -- 172.21.15.116:0/966446787 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc026db0 con 0x56084012bf80 2024-03-30T00:57:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.031+0000 7f02de09db80 1 -- 172.21.15.116:0/966446787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 msgr2=0x560840132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.031+0000 7f02de09db80 1 --2- 172.21.15.116:0/966446787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x560840132510 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f02bc004890 tx=0x7f02bc00b070 comp rx=0 tx=0).stop 2024-03-30T00:57:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.031+0000 7f02de09db80 1 -- 172.21.15.116:0/966446787 shutdown_connections 2024-03-30T00:57:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.031+0000 7f02de09db80 1 --2- 172.21.15.116:0/966446787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 0x560840133a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.031+0000 7f02de09db80 1 --2- 172.21.15.116:0/966446787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x560840132510 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.031+0000 7f02de09db80 1 --2- 172.21.15.116:0/966446787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 0x56084013bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.031+0000 7f02de09db80 1 -- 172.21.15.116:0/966446787 >> 172.21.15.116:0/966446787 conn(0x560840089650 msgr2=0x56084011e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.031+0000 7f02de09db80 1 -- 172.21.15.116:0/966446787 shutdown_connections 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.031+0000 7f02de09db80 1 -- 172.21.15.116:0/966446787 wait complete. 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.032+0000 7f02de09db80 1 -- start start 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.032+0000 7f02de09db80 1 Processor -- start 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.032+0000 7f02de09db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 0x560840138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.032+0000 7f02de09db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 0x560840139100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.032+0000 7f02de09db80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x560840194090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.032+0000 7f02de09db80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608401945d0 con 0x56084012bf80 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.032+0000 7f02de09db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560840194710 con 0x56084012b0e0 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.032+0000 7f02de09db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x560840194880 con 0x56084012a780 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.032+0000 7f02de09db80 10 client.0 _ll_register_callbacks cb 0x560840021420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.033+0000 7f02c3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x560840194090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.033+0000 7f02c3400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 0x560840138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.033+0000 7f02c2a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 0x560840139100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.033+0000 7f02c3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x560840194090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46262/0 (socket says 172.21.15.116:46262) 2024-03-30T00:57:12.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.033+0000 7f02c3e00700 1 -- 172.21.15.116:0/637835160 learned_addr learned my addr 172.21.15.116:0/637835160 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.033+0000 7f02c3e00700 1 -- 172.21.15.116:0/637835160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 msgr2=0x560840138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.033+0000 7f02c3e00700 1 --2- 172.21.15.116:0/637835160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 0x560840138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.033+0000 7f02c3e00700 1 -- 172.21.15.116:0/637835160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 msgr2=0x560840139100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.033+0000 7f02c3e00700 1 --2- 172.21.15.116:0/637835160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 0x560840139100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.033+0000 7f02c3e00700 1 -- 172.21.15.116:0/637835160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02bc022040 con 0x56084012bf80 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.033+0000 7f02c2a00700 1 --2- 172.21.15.116:0/637835160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 0x560840139100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.033+0000 7f02c3e00700 1 --2- 172.21.15.116:0/637835160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x560840194090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0060b0 tx=0x7f02bc027ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364a071b9309db6f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.033+0000 7f02ad600700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.033+0000 7f02ad600700 1 -- 172.21.15.116:0/637835160 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc01ae50 con 0x56084012bf80 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.034+0000 7f02ad600700 1 -- 172.21.15.116:0/637835160 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f02bc026420 con 0x56084012bf80 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.034+0000 7f02de09db80 1 -- 172.21.15.116:0/637835160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f02b001d040 con 0x56084012bf80 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.034+0000 7f02ad600700 1 -- 172.21.15.116:0/637835160 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02bc0265e0 con 0x56084012bf80 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.034+0000 7f02de09db80 10 client.15117 fetch_fsmap learned FSMap version 22 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.034+0000 7f02de09db80 1 -- 172.21.15.116:0/637835160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x56083ff01c80 con 0x56084012bf80 2024-03-30T00:57:12.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.034+0000 7f02ad600700 1 -- 172.21.15.116:0/637835160 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02b001d040 con 0x56084012bf80 2024-03-30T00:57:12.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.034+0000 7f02ad600700 1 -- 172.21.15.116:0/637835160 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 22) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f02bc01a410 con 0x56084012bf80 2024-03-30T00:57:12.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.034+0000 7f02de09db80 10 client.15117 fetch_fsmap finished waiting for FSMap version 22 2024-03-30T00:57:12.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.034+0000 7f02de09db80 10 client.15117 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-30T00:57:12.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.034+0000 7f02de09db80 20 client.15117 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:12.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.034+0000 7f02de09db80 1 --2- 172.21.15.116:0/637835160 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x5608401995d0 0x560840195f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.034+0000 7f02de09db80 4 client.15117 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T00:57:12.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.034+0000 7f02de09db80 1 -- 172.21.15.116:0/637835160 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56083fecaca0 con 0x5608401995d0 2024-03-30T00:57:12.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.035+0000 7f02c3400700 1 --2- 172.21.15.116:0/637835160 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x5608401995d0 0x560840195f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.035+0000 7f02c3400700 1 --2- 172.21.15.116:0/637835160 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x5608401995d0 0x560840195f40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9e8c9de849760ce2 server_cookie=9eeda7548d554c23 in_seq=0 out_seq=0 2024-03-30T00:57:12.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.035+0000 7f02ad600700 10 client.15117 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T00:57:12.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.035+0000 7f02ad600700 1 -- 172.21.15.116:0/637835160 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x56083fecaca0 con 0x5608401995d0 2024-03-30T00:57:12.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.035+0000 7f02ad600700 10 client.15117 handle_command_reply: tid=0 2024-03-30T00:57:12.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.035+0000 7f02de09db80 1 client.15117 shutdown 2024-03-30T00:57:12.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02de09db80 1 -- 172.21.15.116:0/637835160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 msgr2=0x560840194090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02de09db80 1 --2- 172.21.15.116:0/637835160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x560840194090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0060b0 tx=0x7f02bc027ff0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02de09db80 1 -- 172.21.15.116:0/637835160 shutdown_connections 2024-03-30T00:57:12.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02de09db80 1 --2- 172.21.15.116:0/637835160 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x5608401995d0 0x560840195f40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02de09db80 1 --2- 172.21.15.116:0/637835160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56084012a780 0x560840138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02de09db80 1 --2- 172.21.15.116:0/637835160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56084012bf80 0x560840194090 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02de09db80 1 --2- 172.21.15.116:0/637835160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084012b0e0 0x560840139100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02de09db80 1 -- 172.21.15.116:0/637835160 >> 172.21.15.116:0/637835160 conn(0x560840089650 msgr2=0x560840133a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:12.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02ad600700 0 client.15117 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T00:57:12.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02de09db80 1 -- 172.21.15.116:0/637835160 shutdown_connections 2024-03-30T00:57:12.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02de09db80 1 -- 172.21.15.116:0/637835160 wait complete. 2024-03-30T00:57:12.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02de09db80 20 client.15117 trim_cache size 0 max 16384 2024-03-30T00:57:12.047 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:57:12.047 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "4909f004-ee2f-11ee-b647-cb9ed24678a4", 2024-03-30T00:57:12.047 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 1, 2024-03-30T00:57:12.047 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24548, 2024-03-30T00:57:12.047 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-30T00:57:12.047 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-30T00:57:12.047 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-30T00:57:12.047 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 51.650721703999999, 2024-03-30T00:57:12.047 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 22, 2024-03-30T00:57:12.047 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 83, 2024-03-30T00:57:12.047 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 81, 2024-03-30T00:57:12.047 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 93.080039400000004 2024-03-30T00:57:12.047 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:57:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02ca600700 1 -- 172.21.15.116:0/3667192332 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f02a80a1230 msgr2=0x7f02a80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02ca600700 1 --2- 172.21.15.116:0/3667192332 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f02a80a1230 0x7f02a80a36f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009ee0 tx=0x7f02bc004b40 comp rx=0 tx=0).stop 2024-03-30T00:57:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02ca600700 1 -- 172.21.15.116:0/3667192332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c4100d80 msgr2=0x7f02c4198fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02ca600700 1 --2- 172.21.15.116:0/3667192332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c4100d80 0x7f02c4198fd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f02b400e9f0 tx=0x7f02b400ed00 comp rx=0 tx=0).stop 2024-03-30T00:57:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02ca600700 1 -- 172.21.15.116:0/3667192332 shutdown_connections 2024-03-30T00:57:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02ca600700 1 --2- 172.21.15.116:0/3667192332 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f02a80a1230 0x7f02a80a36f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.036+0000 7f02ca600700 1 --2- 172.21.15.116:0/3667192332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02c4102910 0x7f02c41a0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.037+0000 7f02ca600700 1 --2- 172.21.15.116:0/3667192332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c4100d80 0x7f02c4198fd0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.037+0000 7f02ca600700 1 --2- 172.21.15.116:0/3667192332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02c4101f70 0x7f02c4199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.037+0000 7f02ca600700 1 -- 172.21.15.116:0/3667192332 >> 172.21.15.116:0/3667192332 conn(0x7f02c40fc2f0 msgr2=0x7f02c4107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.037+0000 7f02ca600700 1 -- 172.21.15.116:0/3667192332 shutdown_connections 2024-03-30T00:57:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.037+0000 7f02ca600700 1 -- 172.21.15.116:0/3667192332 wait complete. 2024-03-30T00:57:12.060 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-30T00:57:12.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.248+0000 7feb48400700 1 -- 172.21.15.116:0/801373306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40101f70 msgr2=0x7feb401023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.248+0000 7feb48400700 1 --2- 172.21.15.116:0/801373306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40101f70 0x7feb401023d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0099c0 tx=0x7feb2c009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.248+0000 7feb48400700 1 -- 172.21.15.116:0/801373306 shutdown_connections 2024-03-30T00:57:12.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.248+0000 7feb48400700 1 --2- 172.21.15.116:0/801373306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb40102910 0x7feb401091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.248+0000 7feb48400700 1 --2- 172.21.15.116:0/801373306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb40100d80 0x7feb401011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.248+0000 7feb48400700 1 --2- 172.21.15.116:0/801373306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40101f70 0x7feb401023d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.248+0000 7feb48400700 1 -- 172.21.15.116:0/801373306 >> 172.21.15.116:0/801373306 conn(0x7feb400fc2f0 msgr2=0x7feb400fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:12.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.248+0000 7feb48400700 1 -- 172.21.15.116:0/801373306 shutdown_connections 2024-03-30T00:57:12.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.248+0000 7feb48400700 1 -- 172.21.15.116:0/801373306 wait complete. 2024-03-30T00:57:12.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.250+0000 7feb48400700 1 Processor -- start 2024-03-30T00:57:12.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.250+0000 7feb48400700 1 -- start start 2024-03-30T00:57:12.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.250+0000 7feb48400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb40100d80 0x7feb40196d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.250+0000 7feb48400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb40101f70 0x7feb40197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.250+0000 7feb48400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40102910 0x7feb4019e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.251+0000 7feb48400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4019e830 con 0x7feb40102910 2024-03-30T00:57:12.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.251+0000 7feb48400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4019e9a0 con 0x7feb40101f70 2024-03-30T00:57:12.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.251+0000 7feb48400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4019eb60 con 0x7feb40100d80 2024-03-30T00:57:12.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.251+0000 7feb45000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb40101f70 0x7feb40197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.251+0000 7feb45000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb40101f70 0x7feb40197250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34766/0 (socket says 172.21.15.116:34766) 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.251+0000 7feb46400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40102910 0x7feb4019e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.251+0000 7feb45000700 1 -- 172.21.15.116:0/3746390712 learned_addr learned my addr 172.21.15.116:0/3746390712 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.251+0000 7feb45a00700 1 --2- 172.21.15.116:0/3746390712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb40100d80 0x7feb40196d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.251+0000 7feb45000700 1 -- 172.21.15.116:0/3746390712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb40100d80 msgr2=0x7feb40196d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.251+0000 7feb45000700 1 --2- 172.21.15.116:0/3746390712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb40100d80 0x7feb40196d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.251+0000 7feb45000700 1 -- 172.21.15.116:0/3746390712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40102910 msgr2=0x7feb4019e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.251+0000 7feb45000700 1 --2- 172.21.15.116:0/3746390712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40102910 0x7feb4019e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.251+0000 7feb45000700 1 -- 172.21.15.116:0/3746390712 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c0095f0 con 0x7feb40101f70 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.251+0000 7feb45000700 1 --2- 172.21.15.116:0/3746390712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb40101f70 0x7feb40197250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feb2c006010 tx=0x7feb2c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.252+0000 7feb3e000700 1 -- 172.21.15.116:0/3746390712 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c01d070 con 0x7feb40101f70 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.252+0000 7feb3e000700 1 -- 172.21.15.116:0/3746390712 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb2c005030 con 0x7feb40101f70 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.252+0000 7feb3e000700 1 -- 172.21.15.116:0/3746390712 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c004b10 con 0x7feb40101f70 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.252+0000 7feb48400700 1 -- 172.21.15.116:0/3746390712 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4019ede0 con 0x7feb40101f70 2024-03-30T00:57:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.252+0000 7feb48400700 1 -- 172.21.15.116:0/3746390712 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4019f1b0 con 0x7feb40101f70 2024-03-30T00:57:12.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.255+0000 7feb3e000700 1 -- 172.21.15.116:0/3746390712 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7feb2c0051a0 con 0x7feb40101f70 2024-03-30T00:57:12.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.256+0000 7feb5be4eb80 1 Processor -- start 2024-03-30T00:57:12.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.256+0000 7feb3e000700 1 --2- 172.21.15.116:0/3746390712 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb280a1230 0x7feb280a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.256+0000 7feb5be4eb80 1 -- start start 2024-03-30T00:57:12.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.256+0000 7feb3e000700 1 -- 172.21.15.116:0/3746390712 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7feb2c0cd9f0 con 0x7feb40101f70 2024-03-30T00:57:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.256+0000 7feb46400700 1 --2- 172.21.15.116:0/3746390712 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb280a1230 0x7feb280a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.256+0000 7feb5be4eb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a5b90 0x564a108a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.256+0000 7feb5be4eb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a64f0 0x564a108a6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.256+0000 7feb5be4eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a7390 0x564a108a7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.256+0000 7feb5be4eb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a107f9940 con 0x564a108a64f0 2024-03-30T00:57:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.256+0000 7feb5be4eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a107f7300 con 0x564a108a7390 2024-03-30T00:57:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.256+0000 7feb5be4eb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a107f2bf0 con 0x564a108a5b90 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.257+0000 7feb45000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a64f0 0x564a108a6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.257+0000 7feb45a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a5b90 0x564a108a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.257+0000 7feb45a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a5b90 0x564a108a5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:35442/0 (socket says 172.21.15.116:35442) 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.257+0000 7feb45a00700 1 -- 172.21.15.116:0/2645185654 learned_addr learned my addr 172.21.15.116:0/2645185654 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.257+0000 7feb46400700 1 --2- 172.21.15.116:0/2645185654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a7390 0x564a108a7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.257+0000 7feb45a00700 1 -- 172.21.15.116:0/2645185654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a7390 msgr2=0x564a108a7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.257+0000 7feb45a00700 1 --2- 172.21.15.116:0/2645185654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a7390 0x564a108a7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.257+0000 7feb45a00700 1 -- 172.21.15.116:0/2645185654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a64f0 msgr2=0x564a108a6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.257+0000 7feb45a00700 1 --2- 172.21.15.116:0/2645185654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a64f0 0x564a108a6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.257+0000 7feb45a00700 1 -- 172.21.15.116:0/2645185654 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a10674360 con 0x564a108a5b90 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.257+0000 7feb45000700 1 --2- 172.21.15.116:0/2645185654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a64f0 0x564a108a6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.257+0000 7feb46400700 1 --2- 172.21.15.116:0/2645185654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a7390 0x564a108a7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.257+0000 7feb46400700 1 --2- 172.21.15.116:0/3746390712 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb280a1230 0x7feb280a36f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feb3800f540 tx=0x7feb38009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.257+0000 7feb45a00700 1 --2- 172.21.15.116:0/2645185654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a5b90 0x564a108a5fb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feb34009cf0 tx=0x7feb34011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39a0efb2832e5f82 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.258+0000 7feb32a00700 1 -- 172.21.15.116:0/2645185654 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34011d80 con 0x564a108a5b90 2024-03-30T00:57:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.258+0000 7feb32a00700 1 -- 172.21.15.116:0/2645185654 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb340044e0 con 0x564a108a5b90 2024-03-30T00:57:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.258+0000 7feb32a00700 1 -- 172.21.15.116:0/2645185654 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34005220 con 0x564a108a5b90 2024-03-30T00:57:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.258+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2645185654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a5b90 msgr2=0x564a108a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.258+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2645185654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a5b90 0x564a108a5fb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feb34009cf0 tx=0x7feb34011440 comp rx=0 tx=0).stop 2024-03-30T00:57:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.258+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2645185654 shutdown_connections 2024-03-30T00:57:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.258+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2645185654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a5b90 0x564a108a5fb0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.258+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2645185654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a7390 0x564a108a7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.258+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2645185654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a64f0 0x564a108a6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.258+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2645185654 >> 172.21.15.116:0/2645185654 conn(0x564a10800650 msgr2=0x564a1089c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.258+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2645185654 shutdown_connections 2024-03-30T00:57:12.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.258+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2645185654 wait complete. 2024-03-30T00:57:12.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.259+0000 7feb5be4eb80 1 -- start start 2024-03-30T00:57:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.259+0000 7feb5be4eb80 1 Processor -- start 2024-03-30T00:57:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.259+0000 7feb5be4eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb5be4eb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a64f0 0x564a108afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb5be4eb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb5be4eb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a1078ed30 con 0x564a108a64f0 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb5be4eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a1078e9d0 con 0x564a108a5b90 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb5be4eb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a10676650 con 0x564a108a7390 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb5be4eb80 10 client.0 _ll_register_callbacks cb 0x564a10798420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb45000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a64f0 0x564a108afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb45000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a64f0 0x564a108afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46306/0 (socket says 172.21.15.116:46306) 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb45000700 1 -- 172.21.15.116:0/2195307792 learned_addr learned my addr 172.21.15.116:0/2195307792 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb45a00700 1 --2- 172.21.15.116:0/2195307792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb46400700 1 --2- 172.21.15.116:0/2195307792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb45000700 1 -- 172.21.15.116:0/2195307792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 msgr2=0x564a108b6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb45000700 1 --2- 172.21.15.116:0/2195307792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb45000700 1 -- 172.21.15.116:0/2195307792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 msgr2=0x564a108af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb45000700 1 --2- 172.21.15.116:0/2195307792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108af990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb45000700 1 -- 172.21.15.116:0/2195307792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4019f1b0 con 0x564a108a64f0 2024-03-30T00:57:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb46400700 1 --2- 172.21.15.116:0/2195307792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb45000700 1 --2- 172.21.15.116:0/2195307792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a64f0 0x564a108afed0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7feb2c000c00 tx=0x7feb2c005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2874fd3d8fa5a31 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.260+0000 7feb27400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:57:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.261+0000 7feb27400700 1 -- 172.21.15.116:0/2195307792 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c09ae60 con 0x564a108a64f0 2024-03-30T00:57:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.261+0000 7feb27400700 1 -- 172.21.15.116:0/2195307792 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7feb2c0a2df0 con 0x564a108a64f0 2024-03-30T00:57:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.261+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2195307792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564a107b3330 con 0x564a108a64f0 2024-03-30T00:57:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.261+0000 7feb27400700 1 -- 172.21.15.116:0/2195307792 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb2c099ef0 con 0x564a108a64f0 2024-03-30T00:57:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.261+0000 7feb5be4eb80 10 client.15129 fetch_fsmap learned FSMap version 22 2024-03-30T00:57:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.261+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2195307792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7feb34009920 con 0x564a108a64f0 2024-03-30T00:57:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.261+0000 7feb27400700 1 -- 172.21.15.116:0/2195307792 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564a107b3330 con 0x564a108a64f0 2024-03-30T00:57:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.261+0000 7feb27400700 1 -- 172.21.15.116:0/2195307792 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 22) v1 ==== 7470+0+0 (secure 0 0 0) 0x7feb2c0ba0c0 con 0x564a108a64f0 2024-03-30T00:57:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.261+0000 7feb5be4eb80 10 client.15129 fetch_fsmap finished waiting for FSMap version 22 2024-03-30T00:57:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.261+0000 7feb5be4eb80 10 client.15129 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-30T00:57:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.261+0000 7feb5be4eb80 20 client.15129 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.261+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2195307792 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564a108ba720 0x564a108bfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.261+0000 7feb5be4eb80 4 client.15129 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.261+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2195307792 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564a107eb6b0 con 0x564a108ba720 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.262+0000 7feb45a00700 1 --2- 172.21.15.116:0/2195307792 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564a108ba720 0x564a108bfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.262+0000 7feb45a00700 1 --2- 172.21.15.116:0/2195307792 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564a108ba720 0x564a108bfa20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35cbe310e2221065 server_cookie=c69b548be52d2f08 in_seq=0 out_seq=0 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.262+0000 7feb27400700 10 client.15129 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.263+0000 7feb27400700 1 -- 172.21.15.116:0/2195307792 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564a107eb6b0 con 0x564a108ba720 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.263+0000 7feb27400700 10 client.15129 handle_command_reply: tid=0 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.263+0000 7feb5be4eb80 1 client.15129 shutdown 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.263+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2195307792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a64f0 msgr2=0x564a108afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.263+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2195307792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a64f0 0x564a108afed0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7feb2c000c00 tx=0x7feb2c005760 comp rx=0 tx=0).stop 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.263+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2195307792 shutdown_connections 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.263+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2195307792 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564a108ba720 0x564a108bfa20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.263+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2195307792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.263+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2195307792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.263+0000 7feb27400700 0 client.15129 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:12.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.263+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2195307792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a64f0 0x564a108afed0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.263+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2195307792 >> 172.21.15.116:0/2195307792 conn(0x564a10800650 msgr2=0x564a108a0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:12.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.263+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2195307792 shutdown_connections 2024-03-30T00:57:12.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.263+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2195307792 wait complete. 2024-03-30T00:57:12.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.264+0000 7feb5be4eb80 20 client.15129 trim_cache size 0 max 16384 2024-03-30T00:57:12.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.281+0000 7feb5be4eb80 1 Processor -- start 2024-03-30T00:57:12.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.281+0000 7feb5be4eb80 1 -- start start 2024-03-30T00:57:12.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb5be4eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108a7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb5be4eb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb5be4eb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a0120 0x564a108a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb5be4eb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a1078ed30 con 0x564a108a0120 2024-03-30T00:57:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb5be4eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a10896630 con 0x564a108a5b90 2024-03-30T00:57:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb5be4eb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a107b3330 con 0x564a108a7390 2024-03-30T00:57:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb46400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a0120 0x564a108a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb45000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb45000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:35464/0 (socket says 172.21.15.116:35464) 2024-03-30T00:57:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb45000700 1 -- 172.21.15.116:0/2581682793 learned_addr learned my addr 172.21.15.116:0/2581682793 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb45a00700 1 --2- 172.21.15.116:0/2581682793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108a7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb45000700 1 -- 172.21.15.116:0/2581682793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 msgr2=0x564a108a7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb45000700 1 --2- 172.21.15.116:0/2581682793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108a7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb45000700 1 -- 172.21.15.116:0/2581682793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a0120 msgr2=0x564a108a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb45000700 1 --2- 172.21.15.116:0/2581682793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a0120 0x564a108a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb45000700 1 -- 172.21.15.116:0/2581682793 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a10620b10 con 0x564a108a7390 2024-03-30T00:57:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.282+0000 7feb45a00700 1 --2- 172.21.15.116:0/2581682793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108a7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.283+0000 7feb45000700 1 --2- 172.21.15.116:0/2581682793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b6ed0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feb2c09a940 tx=0x7feb2c005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1e92f3e80980b5f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:12.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.283+0000 7feb33400700 1 -- 172.21.15.116:0/2581682793 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0a2df0 con 0x564a108a7390 2024-03-30T00:57:12.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.283+0000 7feb33400700 1 -- 172.21.15.116:0/2581682793 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb2c096940 con 0x564a108a7390 2024-03-30T00:57:12.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.283+0000 7feb33400700 1 -- 172.21.15.116:0/2581682793 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c09a130 con 0x564a108a7390 2024-03-30T00:57:12.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.283+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2581682793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 msgr2=0x564a108b6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.283+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2581682793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b6ed0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feb2c09a940 tx=0x7feb2c005ef0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.283+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2581682793 shutdown_connections 2024-03-30T00:57:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.283+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2581682793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a0120 0x564a108a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.283+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2581682793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108a7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.283+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2581682793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b6ed0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.283+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2581682793 >> 172.21.15.116:0/2581682793 conn(0x564a10800650 msgr2=0x564a10895f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.283+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2581682793 shutdown_connections 2024-03-30T00:57:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.283+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2581682793 wait complete. 2024-03-30T00:57:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.284+0000 7feb5be4eb80 1 -- start start 2024-03-30T00:57:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.284+0000 7feb5be4eb80 1 Processor -- start 2024-03-30T00:57:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.284+0000 7feb5be4eb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a0120 0x564a108b6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.284+0000 7feb5be4eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.284+0000 7feb5be4eb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.284+0000 7feb5be4eb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a1090f010 con 0x564a108a0120 2024-03-30T00:57:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.284+0000 7feb5be4eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a1090f150 con 0x564a108a5b90 2024-03-30T00:57:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.284+0000 7feb5be4eb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a1090f2e0 con 0x564a108a7390 2024-03-30T00:57:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.284+0000 7feb5be4eb80 10 client.0 _ll_register_callbacks cb 0x564a10798420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb45a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a0120 0x564a108b6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb46400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb45000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb45a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a0120 0x564a108b6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46320/0 (socket says 172.21.15.116:46320) 2024-03-30T00:57:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb46400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:35470/0 (socket says 172.21.15.116:35470) 2024-03-30T00:57:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb45a00700 1 -- 172.21.15.116:0/2884762700 learned_addr learned my addr 172.21.15.116:0/2884762700 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb45000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108b40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34802/0 (socket says 172.21.15.116:34802) 2024-03-30T00:57:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb45000700 1 -- 172.21.15.116:0/2884762700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 msgr2=0x564a108b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb45000700 1 --2- 172.21.15.116:0/2884762700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb45000700 1 -- 172.21.15.116:0/2884762700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a0120 msgr2=0x564a108b6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb45000700 1 --2- 172.21.15.116:0/2884762700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a0120 0x564a108b6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb45000700 1 -- 172.21.15.116:0/2884762700 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564a10674360 con 0x564a108a5b90 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb45000700 1 --2- 172.21.15.116:0/2884762700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108b40b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feb2c003730 tx=0x7feb2c005760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbf7a7c290731bf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb27400700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb27400700 1 -- 172.21.15.116:0/2884762700 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0a2df0 con 0x564a108a5b90 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb27400700 1 -- 172.21.15.116:0/2884762700 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7feb2c099ef0 con 0x564a108a5b90 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.285+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2884762700 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb340044e0 con 0x564a108a5b90 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.286+0000 7feb27400700 1 -- 172.21.15.116:0/2884762700 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb2c09a0b0 con 0x564a108a5b90 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.286+0000 7feb5be4eb80 10 client.25196 fetch_fsmap learned FSMap version 22 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.286+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2884762700 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7feb34009920 con 0x564a108a5b90 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.286+0000 7feb27400700 1 -- 172.21.15.116:0/2884762700 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb340044e0 con 0x564a108a5b90 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.286+0000 7feb27400700 1 -- 172.21.15.116:0/2884762700 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 22) v1 ==== 7470+0+0 (secure 0 0 0) 0x7feb2c0b3160 con 0x564a108a5b90 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.286+0000 7feb5be4eb80 10 client.25196 fetch_fsmap finished waiting for FSMap version 22 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.286+0000 7feb5be4eb80 10 client.25196 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.286+0000 7feb5be4eb80 20 client.25196 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.286+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2884762700 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564a10913ba0 0x564a10910550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.286+0000 7feb5be4eb80 4 client.25196 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.286+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2884762700 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564a10641ca0 con 0x564a10913ba0 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.286+0000 7feb45a00700 1 --2- 172.21.15.116:0/2884762700 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564a10913ba0 0x564a10910550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.287+0000 7feb45a00700 1 --2- 172.21.15.116:0/2884762700 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564a10913ba0 0x564a10910550 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdaded2ed4a86c57 server_cookie=8f21e89339d2d1da in_seq=0 out_seq=0 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.287+0000 7feb27400700 10 client.25196 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.287+0000 7feb27400700 1 -- 172.21.15.116:0/2884762700 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x564a10641ca0 con 0x564a10913ba0 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.287+0000 7feb27400700 10 client.25196 handle_command_reply: tid=0 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.287+0000 7feb5be4eb80 1 client.25196 shutdown 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2884762700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 msgr2=0x564a108b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2884762700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108b40b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feb2c003730 tx=0x7feb2c005760 comp rx=0 tx=0).stop 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2884762700 shutdown_connections 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2884762700 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564a10913ba0 0x564a10910550 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2884762700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564a108a7390 0x564a108b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2884762700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a108a0120 0x564a108b6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb27400700 0 client.25196 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:12.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb5be4eb80 1 --2- 172.21.15.116:0/2884762700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a108a5b90 0x564a108b40b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2884762700 >> 172.21.15.116:0/2884762700 conn(0x564a10800650 msgr2=0x564a108a05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:12.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2884762700 shutdown_connections 2024-03-30T00:57:12.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb5be4eb80 1 -- 172.21.15.116:0/2884762700 wait complete. 2024-03-30T00:57:12.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb5be4eb80 20 client.25196 trim_cache size 0 max 16384 2024-03-30T00:57:12.298 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:57:12.298 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "4909f004-ee2f-11ee-b647-cb9ed24678a4", 2024-03-30T00:57:12.298 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-03-30T00:57:12.298 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24587, 2024-03-30T00:57:12.298 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-30T00:57:12.298 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-30T00:57:12.299 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-30T00:57:12.299 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 80.494458093000006, 2024-03-30T00:57:12.299 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 22, 2024-03-30T00:57:12.299 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 83, 2024-03-30T00:57:12.299 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 81, 2024-03-30T00:57:12.299 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 85.256289234999997 2024-03-30T00:57:12.299 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:57:12.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb48400700 1 -- 172.21.15.116:0/3746390712 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb280a1230 msgr2=0x7feb280a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb48400700 1 --2- 172.21.15.116:0/3746390712 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb280a1230 0x7feb280a36f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feb3800f540 tx=0x7feb38009920 comp rx=0 tx=0).stop 2024-03-30T00:57:12.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb48400700 1 -- 172.21.15.116:0/3746390712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb40101f70 msgr2=0x7feb40197250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:12.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb48400700 1 --2- 172.21.15.116:0/3746390712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb40101f70 0x7feb40197250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feb2c006010 tx=0x7feb2c00f690 comp rx=0 tx=0).stop 2024-03-30T00:57:12.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb48400700 1 -- 172.21.15.116:0/3746390712 shutdown_connections 2024-03-30T00:57:12.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb48400700 1 --2- 172.21.15.116:0/3746390712 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7feb280a1230 0x7feb280a36f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb48400700 1 --2- 172.21.15.116:0/3746390712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb40100d80 0x7feb40196d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb48400700 1 --2- 172.21.15.116:0/3746390712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40102910 0x7feb4019e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb48400700 1 --2- 172.21.15.116:0/3746390712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb40101f70 0x7feb40197250 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:12.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.288+0000 7feb48400700 1 -- 172.21.15.116:0/3746390712 >> 172.21.15.116:0/3746390712 conn(0x7feb400fc2f0 msgr2=0x7feb40105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:12.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.289+0000 7feb48400700 1 -- 172.21.15.116:0/3746390712 shutdown_connections 2024-03-30T00:57:12.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:12.289+0000 7feb48400700 1 -- 172.21.15.116:0/3746390712 wait complete. 2024-03-30T00:57:12.313 INFO:tasks.cephadm:Setup complete, yielding 2024-03-30T00:57:12.313 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T00:57:12.325 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-30T00:57:12.325 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-03-30T00:57:12.383 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:12 smithi116 ceph-mon[38086]: pgmap v271: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:12.383 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:12 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3517340285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:12.681 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:57:12.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:12 smithi181 ceph-mon[47622]: pgmap v271: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:12.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:12 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3517340285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:12.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:12 smithi130 ceph-mon[47626]: pgmap v271: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:12 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3517340285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:13.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.854+0000 7f601b800700 1 -- 172.21.15.116:0/2657135703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100e00 msgr2=0x7f6014101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:13.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.854+0000 7f601b800700 1 --2- 172.21.15.116:0/2657135703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100e00 0x7f6014101220 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6008009990 tx=0x7f6008009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:13.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.854+0000 7f601b800700 1 -- 172.21.15.116:0/2657135703 shutdown_connections 2024-03-30T00:57:13.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.854+0000 7f601b800700 1 --2- 172.21.15.116:0/2657135703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6014102920 0x7f6014109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:13.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.854+0000 7f601b800700 1 --2- 172.21.15.116:0/2657135703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100e00 0x7f6014101220 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:13.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.854+0000 7f601b800700 1 --2- 172.21.15.116:0/2657135703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6014101f60 0x7f60141023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:13.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.854+0000 7f601b800700 1 -- 172.21.15.116:0/2657135703 >> 172.21.15.116:0/2657135703 conn(0x7f60140fc3c0 msgr2=0x7f60140fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:13.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.854+0000 7f601b800700 1 -- 172.21.15.116:0/2657135703 shutdown_connections 2024-03-30T00:57:13.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.854+0000 7f601b800700 1 -- 172.21.15.116:0/2657135703 wait complete. 2024-03-30T00:57:13.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.855+0000 7f601b800700 1 Processor -- start 2024-03-30T00:57:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.856+0000 7f601b800700 1 -- start start 2024-03-30T00:57:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.856+0000 7f601b800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100e00 0x7f6014198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.856+0000 7f601b800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6014101f60 0x7f6014199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.856+0000 7f601b800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6014102920 0x7f60141a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.856+0000 7f601b800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60141a0a90 con 0x7f6014100e00 2024-03-30T00:57:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.856+0000 7f601b800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60141a0bd0 con 0x7f6014101f60 2024-03-30T00:57:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.856+0000 7f601b800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60141a0d60 con 0x7f6014102920 2024-03-30T00:57:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.856+0000 7f6018e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100e00 0x7f6014198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.856+0000 7f6018e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100e00 0x7f6014198f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46334/0 (socket says 172.21.15.116:46334) 2024-03-30T00:57:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.856+0000 7f6018e00700 1 -- 172.21.15.116:0/401969222 learned_addr learned my addr 172.21.15.116:0/401969222 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.856+0000 7f6013e00700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6014101f60 0x7f6014199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.856+0000 7f6019800700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6014102920 0x7f60141a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.857+0000 7f6018e00700 1 -- 172.21.15.116:0/401969222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6014102920 msgr2=0x7f60141a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:13.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.857+0000 7f6018e00700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6014102920 0x7f60141a04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:13.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.857+0000 7f6018e00700 1 -- 172.21.15.116:0/401969222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6014101f60 msgr2=0x7f6014199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:13.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.857+0000 7f6018e00700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6014101f60 0x7f6014199490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:13.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.857+0000 7f6018e00700 1 -- 172.21.15.116:0/401969222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60080095f0 con 0x7f6014100e00 2024-03-30T00:57:13.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.857+0000 7f6019800700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6014102920 0x7f60141a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:13.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.857+0000 7f6013e00700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6014101f60 0x7f6014199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:13.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.857+0000 7f6018e00700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100e00 0x7f6014198f50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6008005910 tx=0x7f6008009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:13.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.857+0000 7f6011600700 1 -- 172.21.15.116:0/401969222 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600801d070 con 0x7f6014100e00 2024-03-30T00:57:13.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.857+0000 7f6011600700 1 -- 172.21.15.116:0/401969222 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6008022470 con 0x7f6014100e00 2024-03-30T00:57:13.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.857+0000 7f601b800700 1 -- 172.21.15.116:0/401969222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60141a0fe0 con 0x7f6014100e00 2024-03-30T00:57:13.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.857+0000 7f6011600700 1 -- 172.21.15.116:0/401969222 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008017650 con 0x7f6014100e00 2024-03-30T00:57:13.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.857+0000 7f601b800700 1 -- 172.21.15.116:0/401969222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60141a1340 con 0x7f6014100e00 2024-03-30T00:57:13.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.858+0000 7f601b800700 1 -- 172.21.15.116:0/401969222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f601404ea80 con 0x7f6014100e00 2024-03-30T00:57:13.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.860+0000 7f6011600700 1 -- 172.21.15.116:0/401969222 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f600800fad0 con 0x7f6014100e00 2024-03-30T00:57:13.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.861+0000 7f6011600700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ffc0a11b0 0x7f5ffc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:13.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.861+0000 7f6011600700 1 -- 172.21.15.116:0/401969222 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f60080ce350 con 0x7f6014100e00 2024-03-30T00:57:13.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.862+0000 7f6013e00700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ffc0a11b0 0x7f5ffc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:13.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.862+0000 7f6013e00700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ffc0a11b0 0x7f5ffc0a3670 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6000009cb0 tx=0x7f6000009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:13.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:13.866+0000 7f6011600700 1 -- 172.21.15.116:0/401969222 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6008097700 con 0x7f6014100e00 2024-03-30T00:57:14.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:14.086+0000 7f601b800700 1 -- 172.21.15.116:0/401969222 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6014074700 con 0x7f5ffc0a11b0 2024-03-30T00:57:14.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:14.087+0000 7f6011600700 1 -- 172.21.15.116:0/401969222 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f6014074700 con 0x7f5ffc0a11b0 2024-03-30T00:57:14.093 INFO:teuthology.orchestra.run.smithi116.stdout:Backend: cephadm 2024-03-30T00:57:14.093 INFO:teuthology.orchestra.run.smithi116.stdout:Available: Yes 2024-03-30T00:57:14.093 INFO:teuthology.orchestra.run.smithi116.stdout:Paused: No 2024-03-30T00:57:14.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:14.089+0000 7f601b800700 1 -- 172.21.15.116:0/401969222 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ffc0a11b0 msgr2=0x7f5ffc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:14.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:14.089+0000 7f601b800700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ffc0a11b0 0x7f5ffc0a3670 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6000009cb0 tx=0x7f6000009260 comp rx=0 tx=0).stop 2024-03-30T00:57:14.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:14.089+0000 7f601b800700 1 -- 172.21.15.116:0/401969222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100e00 msgr2=0x7f6014198f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:14.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:14.089+0000 7f601b800700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100e00 0x7f6014198f50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6008005910 tx=0x7f6008009510 comp rx=0 tx=0).stop 2024-03-30T00:57:14.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:14.089+0000 7f601b800700 1 -- 172.21.15.116:0/401969222 shutdown_connections 2024-03-30T00:57:14.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:14.089+0000 7f601b800700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ffc0a11b0 0x7f5ffc0a3670 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:14.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:14.089+0000 7f601b800700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6014102920 0x7f60141a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:14.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:14.089+0000 7f601b800700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6014100e00 0x7f6014198f50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:14.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:14.089+0000 7f601b800700 1 --2- 172.21.15.116:0/401969222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6014101f60 0x7f6014199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:14.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:14.089+0000 7f601b800700 1 -- 172.21.15.116:0/401969222 >> 172.21.15.116:0/401969222 conn(0x7f60140fc3c0 msgr2=0x7f6014072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:14.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:14.090+0000 7f601b800700 1 -- 172.21.15.116:0/401969222 shutdown_connections 2024-03-30T00:57:14.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:14.090+0000 7f601b800700 1 -- 172.21.15.116:0/401969222 wait complete. 2024-03-30T00:57:14.376 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:14 smithi116 ceph-mon[38086]: pgmap v272: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:14.407 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-03-30T00:57:14.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:14 smithi181 ceph-mon[47622]: pgmap v272: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:14.769 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:57:14.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:14 smithi130 ceph-mon[47626]: pgmap v272: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:15.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:15 smithi116 ceph-mon[38086]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:15.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:15 smithi181 ceph-mon[47622]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:15.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:15 smithi130 ceph-mon[47626]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:15.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.932+0000 7f1255e00700 1 -- 172.21.15.116:0/4274905941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1250102920 msgr2=0x7f1250109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:15.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.932+0000 7f1255e00700 1 --2- 172.21.15.116:0/4274905941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1250102920 0x7f1250109240 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f124800b0a0 tx=0x7f124800b3b0 comp rx=0 tx=0).stop 2024-03-30T00:57:15.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.932+0000 7f1255e00700 1 -- 172.21.15.116:0/4274905941 shutdown_connections 2024-03-30T00:57:15.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.932+0000 7f1255e00700 1 --2- 172.21.15.116:0/4274905941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1250100e00 0x7f1250101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:15.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.932+0000 7f1255e00700 1 --2- 172.21.15.116:0/4274905941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1250102920 0x7f1250109240 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.932+0000 7f1255e00700 1 --2- 172.21.15.116:0/4274905941 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1250101f60 0x7f12501023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.932+0000 7f1255e00700 1 -- 172.21.15.116:0/4274905941 >> 172.21.15.116:0/4274905941 conn(0x7f12500fc3c0 msgr2=0x7f12500fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.932+0000 7f1255e00700 1 -- 172.21.15.116:0/4274905941 shutdown_connections 2024-03-30T00:57:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.932+0000 7f1255e00700 1 -- 172.21.15.116:0/4274905941 wait complete. 2024-03-30T00:57:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.933+0000 7f1255e00700 1 Processor -- start 2024-03-30T00:57:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.933+0000 7f1255e00700 1 -- start start 2024-03-30T00:57:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.933+0000 7f1255e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1250100e00 0x7f1250199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f1255e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1250101f60 0x7f12501995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f1255e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1250102920 0x7f12501a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f1255e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501a0ba0 con 0x7f1250100e00 2024-03-30T00:57:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f1255e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501a0ce0 con 0x7f1250102920 2024-03-30T00:57:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f1255e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501a0e70 con 0x7f1250101f60 2024-03-30T00:57:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f124ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1250100e00 0x7f1250199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f124ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1250100e00 0x7f1250199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34482/0 (socket says 172.21.15.116:34482) 2024-03-30T00:57:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f124ea00700 1 -- 172.21.15.116:0/1890527396 learned_addr learned my addr 172.21.15.116:0/1890527396 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f124e000700 1 --2- 172.21.15.116:0/1890527396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1250101f60 0x7f12501995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f124f400700 1 --2- 172.21.15.116:0/1890527396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1250102920 0x7f12501a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f124ea00700 1 -- 172.21.15.116:0/1890527396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1250101f60 msgr2=0x7f12501995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f124ea00700 1 --2- 172.21.15.116:0/1890527396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1250101f60 0x7f12501995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f124ea00700 1 -- 172.21.15.116:0/1890527396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1250102920 msgr2=0x7f12501a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f124ea00700 1 --2- 172.21.15.116:0/1890527396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1250102920 0x7f12501a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.934+0000 7f124ea00700 1 -- 172.21.15.116:0/1890527396 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1248009d00 con 0x7f1250100e00 2024-03-30T00:57:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.935+0000 7f124ea00700 1 --2- 172.21.15.116:0/1890527396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1250100e00 0x7f1250199060 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f124000e9d0 tx=0x7f124000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:15.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.935+0000 7f1254a00700 1 -- 172.21.15.116:0/1890527396 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1240009790 con 0x7f1250100e00 2024-03-30T00:57:15.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.935+0000 7f1254a00700 1 -- 172.21.15.116:0/1890527396 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f124000ee70 con 0x7f1250100e00 2024-03-30T00:57:15.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.935+0000 7f1255e00700 1 -- 172.21.15.116:0/1890527396 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12501a1150 con 0x7f1250100e00 2024-03-30T00:57:15.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.935+0000 7f1254a00700 1 -- 172.21.15.116:0/1890527396 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1240010ea0 con 0x7f1250100e00 2024-03-30T00:57:15.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.935+0000 7f1255e00700 1 -- 172.21.15.116:0/1890527396 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12501a1650 con 0x7f1250100e00 2024-03-30T00:57:15.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.936+0000 7f1255e00700 1 -- 172.21.15.116:0/1890527396 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f125004ea80 con 0x7f1250100e00 2024-03-30T00:57:15.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.938+0000 7f1254a00700 1 -- 172.21.15.116:0/1890527396 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f124000cbd0 con 0x7f1250100e00 2024-03-30T00:57:15.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.939+0000 7f1254a00700 1 --2- 172.21.15.116:0/1890527396 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f12340a11b0 0x7f12340a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:15.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.939+0000 7f1254a00700 1 -- 172.21.15.116:0/1890527396 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1240014070 con 0x7f1250100e00 2024-03-30T00:57:15.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.939+0000 7f124e000700 1 --2- 172.21.15.116:0/1890527396 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f12340a11b0 0x7f12340a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:15.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.940+0000 7f124e000700 1 --2- 172.21.15.116:0/1890527396 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f12340a11b0 0x7f12340a3670 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f123c009c60 tx=0x7f123c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:15.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:15.943+0000 7f1254a00700 1 -- 172.21.15.116:0/1890527396 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f12400529f0 con 0x7f1250100e00 2024-03-30T00:57:16.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:16.154+0000 7f1255e00700 1 -- 172.21.15.116:0/1890527396 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1250074700 con 0x7f12340a11b0 2024-03-30T00:57:16.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:16.167+0000 7f1254a00700 1 -- 172.21.15.116:0/1890527396 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f1250074700 con 0x7f12340a11b0 2024-03-30T00:57:16.173 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T00:57:16.173 INFO:teuthology.orchestra.run.smithi116.stdout:mds.a smithi116 running (91s) 16s ago 90s 13.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 dc16e43b1112 2024-03-30T00:57:16.174 INFO:teuthology.orchestra.run.smithi116.stdout:mds.b smithi130 running (89s) 23s ago 89s 30.1M - 18.2.2-1018-g470100a4 e7c444a08cf7 a0f7f2943bde 2024-03-30T00:57:16.174 INFO:teuthology.orchestra.run.smithi116.stdout:mds.c smithi181 running (99s) 23s ago 99s 17.0M - 18.2.2-1018-g470100a4 e7c444a08cf7 2b3319046e3f 2024-03-30T00:57:16.174 INFO:teuthology.orchestra.run.smithi116.stdout:mds.d smithi116 running (104s) 16s ago 104s 13.8M - 18.2.2-1018-g470100a4 e7c444a08cf7 3b7459ec2b36 2024-03-30T00:57:16.174 INFO:teuthology.orchestra.run.smithi116.stdout:mds.e smithi130 running (102s) 23s ago 102s 18.6M - 18.2.2-1018-g470100a4 e7c444a08cf7 b900de8a8518 2024-03-30T00:57:16.174 INFO:teuthology.orchestra.run.smithi116.stdout:mds.f smithi181 running (101s) 23s ago 100s 17.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 4f2c42ace9b7 2024-03-30T00:57:16.174 INFO:teuthology.orchestra.run.smithi116.stdout:mds.g smithi116 running (93s) 16s ago 93s 13.8M - 18.2.2-1018-g470100a4 e7c444a08cf7 1aeabd56bca8 2024-03-30T00:57:16.174 INFO:teuthology.orchestra.run.smithi116.stdout:mds.h smithi130 running (97s) 23s ago 97s 18.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 b1282b01fdfa 2024-03-30T00:57:16.174 INFO:teuthology.orchestra.run.smithi116.stdout:mds.i smithi181 running (95s) 23s ago 95s 17.8M - 18.2.2-1018-g470100a4 e7c444a08cf7 7560dd3ed75f 2024-03-30T00:57:16.174 INFO:teuthology.orchestra.run.smithi116.stdout:mds.j smithi116 running (106s) 16s ago 106s 13.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 e2ee95916bb9 2024-03-30T00:57:16.174 INFO:teuthology.orchestra.run.smithi116.stdout:mds.k smithi130 running (109s) 23s ago 109s 15.1M - 18.2.2-1018-g470100a4 e7c444a08cf7 95b998cc012f 2024-03-30T00:57:16.174 INFO:teuthology.orchestra.run.smithi116.stdout:mds.l smithi181 running (111s) 23s ago 110s 15.1M - 18.2.2-1018-g470100a4 e7c444a08cf7 02f3dd0e1427 2024-03-30T00:57:16.175 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi116 *:9283,8765 running (7m) 16s ago 7m 536M - 18.2.2-1018-g470100a4 e7c444a08cf7 c520bb0ab810 2024-03-30T00:57:16.175 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi130 *:8443,8765 running (5m) 23s ago 5m 457M - 18.2.2-1018-g470100a4 e7c444a08cf7 15e936174e54 2024-03-30T00:57:16.175 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.z smithi181 *:8443,8765 running (5m) 23s ago 5m 457M - 18.2.2-1018-g470100a4 e7c444a08cf7 5de8b2a51670 2024-03-30T00:57:16.175 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (7m) 16s ago 7m 91.2M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 d8d2be658684 2024-03-30T00:57:16.175 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi130 running (6m) 23s ago 6m 63.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 d9bb73c5ec84 2024-03-30T00:57:16.175 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi181 running (6m) 23s ago 6m 63.7M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a6dbdbcf2cf9 2024-03-30T00:57:16.175 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (5m) 16s ago 5m 140M 4317M 18.2.2-1018-g470100a4 e7c444a08cf7 8f476210312d 2024-03-30T00:57:16.175 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi130 running (5m) 23s ago 5m 75.1M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 fb25f49d905d 2024-03-30T00:57:16.175 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi181 running (4m) 23s ago 4m 72.4M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 e54a61e99ed1 2024-03-30T00:57:16.175 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (4m) 16s ago 4m 64.2M 4317M 18.2.2-1018-g470100a4 e7c444a08cf7 8c113a2c59b7 2024-03-30T00:57:16.175 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi130 running (4m) 23s ago 4m 132M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 8a74cf1f72b5 2024-03-30T00:57:16.175 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi181 running (3m) 23s ago 3m 136M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 84f7cea2f830 2024-03-30T00:57:16.175 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi116 running (3m) 16s ago 3m 128M 4317M 18.2.2-1018-g470100a4 e7c444a08cf7 a8b61bef8006 2024-03-30T00:57:16.176 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi130 running (3m) 23s ago 3m 63.3M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 6e5131dc66ea 2024-03-30T00:57:16.176 INFO:teuthology.orchestra.run.smithi116.stdout:osd.8 smithi181 running (3m) 23s ago 3m 63.4M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 802c33b622a2 2024-03-30T00:57:16.176 INFO:teuthology.orchestra.run.smithi116.stdout:osd.9 smithi116 running (2m) 16s ago 2m 59.5M 4317M 18.2.2-1018-g470100a4 e7c444a08cf7 34f120e4d454 2024-03-30T00:57:16.176 INFO:teuthology.orchestra.run.smithi116.stdout:osd.10 smithi130 running (2m) 23s ago 2m 60.5M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 6f1c1365d41d 2024-03-30T00:57:16.176 INFO:teuthology.orchestra.run.smithi116.stdout:osd.11 smithi181 running (2m) 23s ago 2m 54.2M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 65d616f9ad0e 2024-03-30T00:57:16.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:16.170+0000 7f1255e00700 1 -- 172.21.15.116:0/1890527396 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f12340a11b0 msgr2=0x7f12340a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:16.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:16.170+0000 7f1255e00700 1 --2- 172.21.15.116:0/1890527396 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f12340a11b0 0x7f12340a3670 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f123c009c60 tx=0x7f123c009260 comp rx=0 tx=0).stop 2024-03-30T00:57:16.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:16.170+0000 7f1255e00700 1 -- 172.21.15.116:0/1890527396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1250100e00 msgr2=0x7f1250199060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:16.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:16.170+0000 7f1255e00700 1 --2- 172.21.15.116:0/1890527396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1250100e00 0x7f1250199060 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f124000e9d0 tx=0x7f124000ece0 comp rx=0 tx=0).stop 2024-03-30T00:57:16.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:16.170+0000 7f1255e00700 1 -- 172.21.15.116:0/1890527396 shutdown_connections 2024-03-30T00:57:16.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:16.170+0000 7f1255e00700 1 --2- 172.21.15.116:0/1890527396 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f12340a11b0 0x7f12340a3670 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:16.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:16.170+0000 7f1255e00700 1 --2- 172.21.15.116:0/1890527396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1250102920 0x7f12501a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:16.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:16.170+0000 7f1255e00700 1 --2- 172.21.15.116:0/1890527396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1250100e00 0x7f1250199060 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:16.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:16.170+0000 7f1255e00700 1 --2- 172.21.15.116:0/1890527396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1250101f60 0x7f12501995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:16.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:16.170+0000 7f1255e00700 1 -- 172.21.15.116:0/1890527396 >> 172.21.15.116:0/1890527396 conn(0x7f12500fc3c0 msgr2=0x7f1250072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:16.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:16.170+0000 7f1255e00700 1 -- 172.21.15.116:0/1890527396 shutdown_connections 2024-03-30T00:57:16.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:16.170+0000 7f1255e00700 1 -- 172.21.15.116:0/1890527396 wait complete. 2024-03-30T00:57:16.529 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:16 smithi116 ceph-mon[38086]: pgmap v273: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:16.559 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-03-30T00:57:16.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:16 smithi181 ceph-mon[47622]: pgmap v273: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:16.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:16 smithi130 ceph-mon[47626]: pgmap v273: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:16.920 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:57:17.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:17 smithi116 ceph-mon[38086]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:17.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:17 smithi181 ceph-mon[47622]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:17.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:17 smithi130 ceph-mon[47626]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:18.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.028+0000 7f1476800700 1 -- 172.21.15.116:0/1028800952 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1470102920 msgr2=0x7f1470109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:18.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.028+0000 7f1476800700 1 --2- 172.21.15.116:0/1028800952 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1470102920 0x7f1470109240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f146800b0a0 tx=0x7f146800b3b0 comp rx=0 tx=0).stop 2024-03-30T00:57:18.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.028+0000 7f1476800700 1 -- 172.21.15.116:0/1028800952 shutdown_connections 2024-03-30T00:57:18.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.028+0000 7f1476800700 1 --2- 172.21.15.116:0/1028800952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1470100e00 0x7f1470101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:18.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.028+0000 7f1476800700 1 --2- 172.21.15.116:0/1028800952 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1470102920 0x7f1470109240 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:18.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.028+0000 7f1476800700 1 --2- 172.21.15.116:0/1028800952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470101f60 0x7f14701023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:18.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.028+0000 7f1476800700 1 -- 172.21.15.116:0/1028800952 >> 172.21.15.116:0/1028800952 conn(0x7f14700fc3c0 msgr2=0x7f14700fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:18.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.028+0000 7f1476800700 1 -- 172.21.15.116:0/1028800952 shutdown_connections 2024-03-30T00:57:18.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.028+0000 7f1476800700 1 -- 172.21.15.116:0/1028800952 wait complete. 2024-03-30T00:57:18.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.029+0000 7f1476800700 1 Processor -- start 2024-03-30T00:57:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.029+0000 7f1476800700 1 -- start start 2024-03-30T00:57:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.029+0000 7f1476800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1470100e00 0x7f1470196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.029+0000 7f1476800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1470101f60 0x7f14701973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.029+0000 7f1476800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470102920 0x7f147019e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.029+0000 7f1476800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147019e980 con 0x7f1470102920 2024-03-30T00:57:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.029+0000 7f1476800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147019eac0 con 0x7f1470101f60 2024-03-30T00:57:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.029+0000 7f1476800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147019ec50 con 0x7f1470100e00 2024-03-30T00:57:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f146ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1470101f60 0x7f14701973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f146ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1470101f60 0x7f14701973d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:41642/0 (socket says 172.21.15.116:41642) 2024-03-30T00:57:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f146ea00700 1 -- 172.21.15.116:0/3218773841 learned_addr learned my addr 172.21.15.116:0/3218773841 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f146fe00700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470102920 0x7f147019e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f146f400700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1470100e00 0x7f1470196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f146ea00700 1 -- 172.21.15.116:0/3218773841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1470100e00 msgr2=0x7f1470196e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f146ea00700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1470100e00 0x7f1470196e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f146ea00700 1 -- 172.21.15.116:0/3218773841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470102920 msgr2=0x7f147019e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f146ea00700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470102920 0x7f147019e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:18.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f146ea00700 1 -- 172.21.15.116:0/3218773841 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c009520 con 0x7f1470101f60 2024-03-30T00:57:18.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f146f400700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1470100e00 0x7f1470196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:18.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f146fe00700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470102920 0x7f147019e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:18.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f146ea00700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1470101f60 0x7f14701973d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f145c00eb30 tx=0x7f145c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:18.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f1467e00700 1 -- 172.21.15.116:0/3218773841 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c009880 con 0x7f1470101f60 2024-03-30T00:57:18.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f1467e00700 1 -- 172.21.15.116:0/3218773841 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f145c0044e0 con 0x7f1470101f60 2024-03-30T00:57:18.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.030+0000 7f1467e00700 1 -- 172.21.15.116:0/3218773841 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c003e60 con 0x7f1470101f60 2024-03-30T00:57:18.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.031+0000 7f1476800700 1 -- 172.21.15.116:0/3218773841 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1468009d00 con 0x7f1470101f60 2024-03-30T00:57:18.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.031+0000 7f1476800700 1 -- 172.21.15.116:0/3218773841 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147019f1f0 con 0x7f1470101f60 2024-03-30T00:57:18.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.032+0000 7f1476800700 1 -- 172.21.15.116:0/3218773841 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14701081b0 con 0x7f1470101f60 2024-03-30T00:57:18.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.033+0000 7f1467e00700 1 -- 172.21.15.116:0/3218773841 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f145c003fc0 con 0x7f1470101f60 2024-03-30T00:57:18.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.035+0000 7f1467e00700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14540a11b0 0x7f14540a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:18.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.035+0000 7f1467e00700 1 -- 172.21.15.116:0/3218773841 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f145c014070 con 0x7f1470101f60 2024-03-30T00:57:18.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.035+0000 7f146f400700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14540a11b0 0x7f14540a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:18.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.035+0000 7f146f400700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14540a11b0 0x7f14540a3670 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1460009cb0 tx=0x7f1460009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.039+0000 7f1467e00700 1 -- 172.21.15.116:0/3218773841 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f145c0964f0 con 0x7f1470101f60 2024-03-30T00:57:18.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.252+0000 7f1476800700 1 -- 172.21.15.116:0/3218773841 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f147002cdc0 con 0x7f14540a11b0 2024-03-30T00:57:18.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.258+0000 7f1467e00700 1 -- 172.21.15.116:0/3218773841 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+985 (secure 0 0 0) 0x7f147002cdc0 con 0x7f14540a11b0 2024-03-30T00:57:18.264 INFO:teuthology.orchestra.run.smithi116.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-30T00:57:18.264 INFO:teuthology.orchestra.run.smithi116.stdout:mds.all 12/12 25s ago 115s smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2024-03-30T00:57:18.264 INFO:teuthology.orchestra.run.smithi116.stdout:mgr 3/3 25s ago 5m smithi116=x;smithi130=y;smithi181=z;count:3 2024-03-30T00:57:18.265 INFO:teuthology.orchestra.run.smithi116.stdout:mon 3/3 25s ago 6m smithi116:172.21.15.116=a;smithi130:172.21.15.130=b;smithi181:172.21.15.181=c;count:3 2024-03-30T00:57:18.265 INFO:teuthology.orchestra.run.smithi116.stdout:osd 12 25s ago - 2024-03-30T00:57:18.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.260+0000 7f1476800700 1 -- 172.21.15.116:0/3218773841 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14540a11b0 msgr2=0x7f14540a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:18.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.260+0000 7f1476800700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14540a11b0 0x7f14540a3670 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1460009cb0 tx=0x7f1460009260 comp rx=0 tx=0).stop 2024-03-30T00:57:18.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.260+0000 7f1476800700 1 -- 172.21.15.116:0/3218773841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1470101f60 msgr2=0x7f14701973d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:18.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.260+0000 7f1476800700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1470101f60 0x7f14701973d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f145c00eb30 tx=0x7f145c00ee40 comp rx=0 tx=0).stop 2024-03-30T00:57:18.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.260+0000 7f1476800700 1 -- 172.21.15.116:0/3218773841 shutdown_connections 2024-03-30T00:57:18.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.260+0000 7f1476800700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14540a11b0 0x7f14540a3670 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:18.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.260+0000 7f1476800700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1470100e00 0x7f1470196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:18.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.260+0000 7f1476800700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470102920 0x7f147019e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:18.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.260+0000 7f1476800700 1 --2- 172.21.15.116:0/3218773841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1470101f60 0x7f14701973d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:18.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.260+0000 7f1476800700 1 -- 172.21.15.116:0/3218773841 >> 172.21.15.116:0/3218773841 conn(0x7f14700fc3c0 msgr2=0x7f14701052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:18.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.260+0000 7f1476800700 1 -- 172.21.15.116:0/3218773841 shutdown_connections 2024-03-30T00:57:18.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:18.260+0000 7f1476800700 1 -- 172.21.15.116:0/3218773841 wait complete. 2024-03-30T00:57:18.586 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:18 smithi116 ceph-mon[38086]: pgmap v274: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:18.613 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-03-30T00:57:18.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:18 smithi181 ceph-mon[47622]: pgmap v274: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:18.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:18 smithi130 ceph-mon[47626]: pgmap v274: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:18.972 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:57:19.642 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:19 smithi116 ceph-mon[38086]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:19.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:19 smithi181 ceph-mon[47622]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:19.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:19 smithi130 ceph-mon[47626]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:20.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.110+0000 7f192f400700 1 -- 172.21.15.116:0/4208974729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1928102920 msgr2=0x7f1928109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:20.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.110+0000 7f192f400700 1 --2- 172.21.15.116:0/4208974729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1928102920 0x7f1928109240 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f192000b0a0 tx=0x7f192000b3b0 comp rx=0 tx=0).stop 2024-03-30T00:57:20.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.110+0000 7f192f400700 1 -- 172.21.15.116:0/4208974729 shutdown_connections 2024-03-30T00:57:20.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.110+0000 7f192f400700 1 --2- 172.21.15.116:0/4208974729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1928102920 0x7f1928109240 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:20.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.110+0000 7f192f400700 1 --2- 172.21.15.116:0/4208974729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1928100e00 0x7f1928101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:20.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.110+0000 7f192f400700 1 --2- 172.21.15.116:0/4208974729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1928101f60 0x7f19281023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:20.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.110+0000 7f192f400700 1 -- 172.21.15.116:0/4208974729 >> 172.21.15.116:0/4208974729 conn(0x7f19280fc3c0 msgr2=0x7f19280fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.110+0000 7f192f400700 1 -- 172.21.15.116:0/4208974729 shutdown_connections 2024-03-30T00:57:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.110+0000 7f192f400700 1 -- 172.21.15.116:0/4208974729 wait complete. 2024-03-30T00:57:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.111+0000 7f192f400700 1 Processor -- start 2024-03-30T00:57:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.111+0000 7f192f400700 1 -- start start 2024-03-30T00:57:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f192f400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1928100e00 0x7f1928074ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f192f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1928101f60 0x7f19280731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f192f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1928102920 0x7f19280736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f192f400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1928073dc0 con 0x7f1928101f60 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f192f400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1928073f30 con 0x7f1928102920 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f192f400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19281a0f30 con 0x7f1928100e00 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f1927e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1928101f60 0x7f19280731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f1927e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1928101f60 0x7f19280731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34518/0 (socket says 172.21.15.116:34518) 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f1927e00700 1 -- 172.21.15.116:0/4187553819 learned_addr learned my addr 172.21.15.116:0/4187553819 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f192d400700 1 --2- 172.21.15.116:0/4187553819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1928102920 0x7f19280736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f192ca00700 1 --2- 172.21.15.116:0/4187553819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1928100e00 0x7f1928074ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f1927e00700 1 -- 172.21.15.116:0/4187553819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1928100e00 msgr2=0x7f1928074ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f1927e00700 1 --2- 172.21.15.116:0/4187553819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1928100e00 0x7f1928074ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f1927e00700 1 -- 172.21.15.116:0/4187553819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1928102920 msgr2=0x7f19280736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f1927e00700 1 --2- 172.21.15.116:0/4187553819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1928102920 0x7f19280736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.112+0000 7f1927e00700 1 -- 172.21.15.116:0/4187553819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1920009d00 con 0x7f1928101f60 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.113+0000 7f1927e00700 1 --2- 172.21.15.116:0/4187553819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1928101f60 0x7f19280731a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f191400e9c0 tx=0x7f191400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.113+0000 7f1925600700 1 -- 172.21.15.116:0/4187553819 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19140097b0 con 0x7f1928101f60 2024-03-30T00:57:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.113+0000 7f1925600700 1 -- 172.21.15.116:0/4187553819 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f191400ee60 con 0x7f1928101f60 2024-03-30T00:57:20.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.113+0000 7f1925600700 1 -- 172.21.15.116:0/4187553819 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19140054f0 con 0x7f1928101f60 2024-03-30T00:57:20.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.113+0000 7f192f400700 1 -- 172.21.15.116:0/4187553819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19281a1190 con 0x7f1928101f60 2024-03-30T00:57:20.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.113+0000 7f192f400700 1 -- 172.21.15.116:0/4187553819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19281a1610 con 0x7f1928101f60 2024-03-30T00:57:20.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.114+0000 7f192f400700 1 -- 172.21.15.116:0/4187553819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19281081b0 con 0x7f1928101f60 2024-03-30T00:57:20.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.116+0000 7f1925600700 1 -- 172.21.15.116:0/4187553819 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f1914004750 con 0x7f1928101f60 2024-03-30T00:57:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.117+0000 7f1925600700 1 --2- 172.21.15.116:0/4187553819 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f19100a11b0 0x7f19100a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.118+0000 7f1925600700 1 -- 172.21.15.116:0/4187553819 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1914014070 con 0x7f1928101f60 2024-03-30T00:57:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.118+0000 7f192ca00700 1 --2- 172.21.15.116:0/4187553819 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f19100a11b0 0x7f19100a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.118+0000 7f192ca00700 1 --2- 172.21.15.116:0/4187553819 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f19100a11b0 0x7f19100a3670 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f191c009c60 tx=0x7f191c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:20.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.121+0000 7f1925600700 1 -- 172.21.15.116:0/4187553819 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f19140962a0 con 0x7f1928101f60 2024-03-30T00:57:20.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.332+0000 7f192f400700 1 -- 172.21.15.116:0/4187553819 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f19280611f0 con 0x7f19100a11b0 2024-03-30T00:57:20.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.334+0000 7f1925600700 1 -- 172.21.15.116:0/4187553819 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f19280611f0 con 0x7f19100a11b0 2024-03-30T00:57:20.340 INFO:teuthology.orchestra.run.smithi116.stdout:HOST ADDR LABELS STATUS 2024-03-30T00:57:20.340 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116 172.21.15.116 2024-03-30T00:57:20.340 INFO:teuthology.orchestra.run.smithi116.stdout:smithi130 172.21.15.130 2024-03-30T00:57:20.340 INFO:teuthology.orchestra.run.smithi116.stdout:smithi181 172.21.15.181 2024-03-30T00:57:20.340 INFO:teuthology.orchestra.run.smithi116.stdout:3 hosts in cluster 2024-03-30T00:57:20.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.336+0000 7f192f400700 1 -- 172.21.15.116:0/4187553819 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f19100a11b0 msgr2=0x7f19100a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:20.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.336+0000 7f192f400700 1 --2- 172.21.15.116:0/4187553819 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f19100a11b0 0x7f19100a3670 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f191c009c60 tx=0x7f191c009260 comp rx=0 tx=0).stop 2024-03-30T00:57:20.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.336+0000 7f192f400700 1 -- 172.21.15.116:0/4187553819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1928101f60 msgr2=0x7f19280731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:20.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.336+0000 7f192f400700 1 --2- 172.21.15.116:0/4187553819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1928101f60 0x7f19280731a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f191400e9c0 tx=0x7f191400ecd0 comp rx=0 tx=0).stop 2024-03-30T00:57:20.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.336+0000 7f192f400700 1 -- 172.21.15.116:0/4187553819 shutdown_connections 2024-03-30T00:57:20.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.336+0000 7f192f400700 1 --2- 172.21.15.116:0/4187553819 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f19100a11b0 0x7f19100a3670 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:20.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.336+0000 7f192f400700 1 --2- 172.21.15.116:0/4187553819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1928100e00 0x7f1928074ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:20.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.336+0000 7f192f400700 1 --2- 172.21.15.116:0/4187553819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1928102920 0x7f19280736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:20.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.336+0000 7f192f400700 1 --2- 172.21.15.116:0/4187553819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1928101f60 0x7f19280731a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:20.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.336+0000 7f192f400700 1 -- 172.21.15.116:0/4187553819 >> 172.21.15.116:0/4187553819 conn(0x7f19280fc3c0 msgr2=0x7f19281052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:20.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.336+0000 7f192f400700 1 -- 172.21.15.116:0/4187553819 shutdown_connections 2024-03-30T00:57:20.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:20.336+0000 7f192f400700 1 -- 172.21.15.116:0/4187553819 wait complete. 2024-03-30T00:57:20.666 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:20 smithi116 ceph-mon[38086]: pgmap v275: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:57:20.694 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-03-30T00:57:20.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:20 smithi181 ceph-mon[47622]: pgmap v275: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:57:20.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:20 smithi130 ceph-mon[47626]: pgmap v275: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:57:21.057 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:57:21.542 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:21 smithi116 ceph-mon[38086]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:21 smithi181 ceph-mon[47622]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:21.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:21 smithi130 ceph-mon[47626]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.220+0000 7fd504c00700 1 -- 172.21.15.116:0/4240294676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd500101f60 msgr2=0x7fd5001023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.220+0000 7fd504c00700 1 --2- 172.21.15.116:0/4240294676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd500101f60 0x7fd5001023e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec0098f0 tx=0x7fd4ec009c00 comp rx=0 tx=0).stop 2024-03-30T00:57:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.220+0000 7fd504c00700 1 -- 172.21.15.116:0/4240294676 shutdown_connections 2024-03-30T00:57:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.220+0000 7fd504c00700 1 --2- 172.21.15.116:0/4240294676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500102920 0x7fd500109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.220+0000 7fd504c00700 1 --2- 172.21.15.116:0/4240294676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd500100e00 0x7fd500101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.220+0000 7fd504c00700 1 --2- 172.21.15.116:0/4240294676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd500101f60 0x7fd5001023e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.220+0000 7fd504c00700 1 -- 172.21.15.116:0/4240294676 >> 172.21.15.116:0/4240294676 conn(0x7fd5000fc3c0 msgr2=0x7fd5000fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.220+0000 7fd504c00700 1 -- 172.21.15.116:0/4240294676 shutdown_connections 2024-03-30T00:57:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.220+0000 7fd504c00700 1 -- 172.21.15.116:0/4240294676 wait complete. 2024-03-30T00:57:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.221+0000 7fd504c00700 1 Processor -- start 2024-03-30T00:57:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.221+0000 7fd504c00700 1 -- start start 2024-03-30T00:57:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.222+0000 7fd504c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd500100e00 0x7fd500198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.222+0000 7fd504c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd500101f60 0x7fd5001994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.222+0000 7fd504c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500102920 0x7fd5001a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.222+0000 7fd504c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5001a0ab0 con 0x7fd500102920 2024-03-30T00:57:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.222+0000 7fd504c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5001a0bf0 con 0x7fd500101f60 2024-03-30T00:57:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.222+0000 7fd504c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5001a0d80 con 0x7fd500100e00 2024-03-30T00:57:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.222+0000 7fd4fe800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500102920 0x7fd5001a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.222+0000 7fd4fde00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd500100e00 0x7fd500198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.222+0000 7fd4fe800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500102920 0x7fd5001a04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34536/0 (socket says 172.21.15.116:34536) 2024-03-30T00:57:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.222+0000 7fd4fe800700 1 -- 172.21.15.116:0/3278744104 learned_addr learned my addr 172.21.15.116:0/3278744104 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.222+0000 7fd4fd400700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd500101f60 0x7fd5001994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.223+0000 7fd4fe800700 1 -- 172.21.15.116:0/3278744104 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd500100e00 msgr2=0x7fd500198f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.223+0000 7fd4fe800700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd500100e00 0x7fd500198f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.223+0000 7fd4fe800700 1 -- 172.21.15.116:0/3278744104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd500101f60 msgr2=0x7fd5001994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.223+0000 7fd4fe800700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd500101f60 0x7fd5001994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.223+0000 7fd4fe800700 1 -- 172.21.15.116:0/3278744104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f8009c40 con 0x7fd500102920 2024-03-30T00:57:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.223+0000 7fd4fde00700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd500100e00 0x7fd500198f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.223+0000 7fd4fd400700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd500101f60 0x7fd5001994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.223+0000 7fd4fe800700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500102920 0x7fd5001a04e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800c3a0 tx=0x7fd4f800c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.223+0000 7fd4f6a00700 1 -- 172.21.15.116:0/3278744104 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8017070 con 0x7fd500102920 2024-03-30T00:57:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.223+0000 7fd504c00700 1 -- 172.21.15.116:0/3278744104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ec009520 con 0x7fd500102920 2024-03-30T00:57:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.223+0000 7fd4f6a00700 1 -- 172.21.15.116:0/3278744104 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd4f800f040 con 0x7fd500102920 2024-03-30T00:57:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.223+0000 7fd4f6a00700 1 -- 172.21.15.116:0/3278744104 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f800e050 con 0x7fd500102920 2024-03-30T00:57:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.223+0000 7fd504c00700 1 -- 172.21.15.116:0/3278744104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5001a1370 con 0x7fd500102920 2024-03-30T00:57:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.224+0000 7fd504c00700 1 -- 172.21.15.116:0/3278744104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50004ea80 con 0x7fd500102920 2024-03-30T00:57:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.226+0000 7fd4f6a00700 1 -- 172.21.15.116:0/3278744104 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fd4f8003a00 con 0x7fd500102920 2024-03-30T00:57:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.227+0000 7fd4f6a00700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd4e40a1230 0x7fd4e40a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.228+0000 7fd4f6a00700 1 -- 172.21.15.116:0/3278744104 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd4f80cc680 con 0x7fd500102920 2024-03-30T00:57:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.228+0000 7fd4fde00700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd4e40a1230 0x7fd4e40a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.228+0000 7fd4fde00700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd4e40a1230 0x7fd4e40a36f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0005fd0 tx=0x7fd4f0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.232+0000 7fd4f6a00700 1 -- 172.21.15.116:0/3278744104 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd4f8095940 con 0x7fd500102920 2024-03-30T00:57:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.442+0000 7fd504c00700 1 -- 172.21.15.116:0/3278744104 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd500108ee0 con 0x7fd4e40a1230 2024-03-30T00:57:22.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.444+0000 7fd4f6a00700 1 -- 172.21.15.116:0/3278744104 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7fd500108ee0 con 0x7fd4e40a1230 2024-03-30T00:57:22.451 INFO:teuthology.orchestra.run.smithi116.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-30T00:57:22.451 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VJ400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-30T00:57:22.451 INFO:teuthology.orchestra.run.smithi116.stdout:smithi130 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500Q1400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-30T00:57:22.451 INFO:teuthology.orchestra.run.smithi116.stdout:smithi181 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6246000E400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-30T00:57:22.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.446+0000 7fd504c00700 1 -- 172.21.15.116:0/3278744104 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd4e40a1230 msgr2=0x7fd4e40a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:22.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.446+0000 7fd504c00700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd4e40a1230 0x7fd4e40a36f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0005fd0 tx=0x7fd4f0009310 comp rx=0 tx=0).stop 2024-03-30T00:57:22.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.446+0000 7fd504c00700 1 -- 172.21.15.116:0/3278744104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500102920 msgr2=0x7fd5001a04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:22.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.446+0000 7fd504c00700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500102920 0x7fd5001a04e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800c3a0 tx=0x7fd4f800c6b0 comp rx=0 tx=0).stop 2024-03-30T00:57:22.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.447+0000 7fd504c00700 1 -- 172.21.15.116:0/3278744104 shutdown_connections 2024-03-30T00:57:22.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.447+0000 7fd504c00700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd4e40a1230 0x7fd4e40a36f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:22.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.447+0000 7fd504c00700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd500100e00 0x7fd500198f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:22.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.447+0000 7fd504c00700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500102920 0x7fd5001a04e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:22.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.447+0000 7fd504c00700 1 --2- 172.21.15.116:0/3278744104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd500101f60 0x7fd5001994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:22.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.447+0000 7fd504c00700 1 -- 172.21.15.116:0/3278744104 >> 172.21.15.116:0/3278744104 conn(0x7fd5000fc3c0 msgr2=0x7fd5001077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:22.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.447+0000 7fd504c00700 1 -- 172.21.15.116:0/3278744104 shutdown_connections 2024-03-30T00:57:22.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:22.447+0000 7fd504c00700 1 -- 172.21.15.116:0/3278744104 wait complete. 2024-03-30T00:57:22.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:22 smithi116 ceph-mon[38086]: pgmap v276: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:22 smithi181 ceph-mon[47622]: pgmap v276: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:22.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:22 smithi130 ceph-mon[47626]: pgmap v276: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:22.850 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T00:57:22.860 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-30T00:57:22.860 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-03-30T00:57:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:57:23.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:23 smithi116 ceph-mon[38086]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:23.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:23 smithi181 ceph-mon[47622]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:23.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:23 smithi130 ceph-mon[47626]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T00:57:24.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.377+0000 7f5972000700 1 -- 172.21.15.116:0/3874134771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f596c101f60 msgr2=0x7f596c1023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:24.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.377+0000 7f5972000700 1 --2- 172.21.15.116:0/3874134771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f596c101f60 0x7f596c1023e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5958009990 tx=0x7f5958009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:24.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.377+0000 7f5972000700 1 -- 172.21.15.116:0/3874134771 shutdown_connections 2024-03-30T00:57:24.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.377+0000 7f5972000700 1 --2- 172.21.15.116:0/3874134771 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f596c102920 0x7f596c109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:24.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.377+0000 7f5972000700 1 --2- 172.21.15.116:0/3874134771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f596c100e00 0x7f596c101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:24.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.377+0000 7f5972000700 1 --2- 172.21.15.116:0/3874134771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f596c101f60 0x7f596c1023e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:24.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.377+0000 7f5972000700 1 -- 172.21.15.116:0/3874134771 >> 172.21.15.116:0/3874134771 conn(0x7f596c0fc3c0 msgr2=0x7f596c0fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:24.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.377+0000 7f5972000700 1 -- 172.21.15.116:0/3874134771 shutdown_connections 2024-03-30T00:57:24.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.377+0000 7f5972000700 1 -- 172.21.15.116:0/3874134771 wait complete. 2024-03-30T00:57:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.378+0000 7f5972000700 1 Processor -- start 2024-03-30T00:57:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.378+0000 7f5972000700 1 -- start start 2024-03-30T00:57:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f5972000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f596c100e00 0x7f596c198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f5972000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f596c101f60 0x7f596c1994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f5972000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f596c102920 0x7f596c1a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f5972000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c1a0ae0 con 0x7f596c101f60 2024-03-30T00:57:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f5972000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c1a0c20 con 0x7f596c100e00 2024-03-30T00:57:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f5972000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c1a0db0 con 0x7f596c102920 2024-03-30T00:57:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f596aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f596c101f60 0x7f596c1994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f596aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f596c101f60 0x7f596c1994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34548/0 (socket says 172.21.15.116:34548) 2024-03-30T00:57:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f596aa00700 1 -- 172.21.15.116:0/1816803342 learned_addr learned my addr 172.21.15.116:0/1816803342 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f596b400700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f596c100e00 0x7f596c198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f596be00700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f596c102920 0x7f596c1a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f596aa00700 1 -- 172.21.15.116:0/1816803342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f596c102920 msgr2=0x7f596c1a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f596aa00700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f596c102920 0x7f596c1a0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f596aa00700 1 -- 172.21.15.116:0/1816803342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f596c100e00 msgr2=0x7f596c198fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f596aa00700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f596c100e00 0x7f596c198fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f596aa00700 1 -- 172.21.15.116:0/1816803342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59580095f0 con 0x7f596c101f60 2024-03-30T00:57:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.379+0000 7f596b400700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f596c100e00 0x7f596c198fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.380+0000 7f596aa00700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f596c101f60 0x7f596c1994e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5958005910 tx=0x7f5958009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.380+0000 7f5963e00700 1 -- 172.21.15.116:0/1816803342 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595801d070 con 0x7f596c101f60 2024-03-30T00:57:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.380+0000 7f5963e00700 1 -- 172.21.15.116:0/1816803342 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5958022470 con 0x7f596c101f60 2024-03-30T00:57:24.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.380+0000 7f5963e00700 1 -- 172.21.15.116:0/1816803342 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958017650 con 0x7f596c101f60 2024-03-30T00:57:24.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.380+0000 7f5972000700 1 -- 172.21.15.116:0/1816803342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c1a1030 con 0x7f596c101f60 2024-03-30T00:57:24.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.380+0000 7f5972000700 1 -- 172.21.15.116:0/1816803342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c1a13f0 con 0x7f596c101f60 2024-03-30T00:57:24.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.381+0000 7f5972000700 1 -- 172.21.15.116:0/1816803342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c04ea80 con 0x7f596c101f60 2024-03-30T00:57:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.383+0000 7f5963e00700 1 -- 172.21.15.116:0/1816803342 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f595800fad0 con 0x7f596c101f60 2024-03-30T00:57:24.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.384+0000 7f5963e00700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f59500a11b0 0x7f59500a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:24.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.385+0000 7f5963e00700 1 -- 172.21.15.116:0/1816803342 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f59580ceb30 con 0x7f596c101f60 2024-03-30T00:57:24.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.385+0000 7f596b400700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f59500a11b0 0x7f59500a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:24.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.385+0000 7f596b400700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f59500a11b0 0x7f59500a3670 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f595c009cb0 tx=0x7f595c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:24.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.386+0000 7f5963e00700 1 -- 172.21.15.116:0/1816803342 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5958097e80 con 0x7f596c101f60 2024-03-30T00:57:24.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.639+0000 7f5972000700 1 -- 172.21.15.116:0/1816803342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f596c10c8b0 con 0x7f596c101f60 2024-03-30T00:57:24.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.640+0000 7f5963e00700 1 -- 172.21.15.116:0/1816803342 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+2911 (secure 0 0 0) 0x7f5958027b70 con 0x7f596c101f60 2024-03-30T00:57:24.647 INFO:teuthology.orchestra.run.smithi116.stdout:e22 2024-03-30T00:57:24.647 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T00:57:24.647 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T00:57:24.647 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-30T00:57:24.647 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:24.648 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-30T00:57:24.648 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-30T00:57:24.648 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 22 2024-03-30T00:57:24.648 INFO:teuthology.orchestra.run.smithi116.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T00:57:24.648 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-30T00:55:51.790326+0000 2024-03-30T00:57:24.648 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-30T00:56:22.144079+0000 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 300 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 0 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 3 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:in 0,1,2 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24587,1=24548,2=24541} 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-30T00:57:24.649 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-30T00:57:24.650 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-30T00:57:24.650 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3,4] 2024-03-30T00:57:24.650 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-30T00:57:24.650 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-30T00:57:24.650 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-30T00:57:24.650 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-30T00:57:24.650 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 0 2024-03-30T00:57:24.650 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b{0:24587} state up:active seq 3 addr [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T00:57:24.650 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.e{0:24524} state up:standby-replay seq 1 addr [v2:172.21.15.130:6834/3346365474,v1:172.21.15.130:6835/3346365474] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T00:57:24.650 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.h{1:24548} state up:active seq 12 addr [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T00:57:24.650 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c{1:24517} state up:standby-replay seq 1 addr [v2:172.21.15.181:6836/941347789,v1:172.21.15.181:6837/941347789] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T00:57:24.650 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.i{2:24541} state up:active seq 12 addr [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T00:57:24.650 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.f{2:24511} state up:standby-replay seq 1 addr [v2:172.21.15.181:6834/702133015,v1:172.21.15.181:6835/702133015] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T00:57:24.651 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:24.651 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:24.651 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-30T00:57:24.651 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:24.651 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d{-1:14457} state up:standby seq 1 addr [v2:172.21.15.116:6836/3445796607,v1:172.21.15.116:6837/3445796607] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T00:57:24.651 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.g{-1:14505} state up:standby seq 1 addr [v2:172.21.15.116:6838/1448499059,v1:172.21.15.116:6839/1448499059] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T00:57:24.651 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a{-1:14517} state up:standby seq 1 addr [v2:172.21.15.116:6840/2469944136,v1:172.21.15.116:6841/2469944136] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T00:57:24.651 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.l{-1:24460} state up:standby seq 1 addr [v2:172.21.15.181:6832/1642031781,v1:172.21.15.181:6833/1642031781] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T00:57:24.651 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.j{-1:24475} state up:standby seq 1 addr [v2:172.21.15.116:6834/459507942,v1:172.21.15.116:6835/459507942] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T00:57:24.651 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.k{-1:24488} state up:standby seq 1 addr [v2:172.21.15.130:6832/2156314548,v1:172.21.15.130:6833/2156314548] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T00:57:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.642+0000 7f5972000700 1 -- 172.21.15.116:0/1816803342 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f59500a11b0 msgr2=0x7f59500a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.642+0000 7f5972000700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f59500a11b0 0x7f59500a3670 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f595c009cb0 tx=0x7f595c009260 comp rx=0 tx=0).stop 2024-03-30T00:57:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.642+0000 7f5972000700 1 -- 172.21.15.116:0/1816803342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f596c101f60 msgr2=0x7f596c1994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.642+0000 7f5972000700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f596c101f60 0x7f596c1994e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5958005910 tx=0x7f5958009510 comp rx=0 tx=0).stop 2024-03-30T00:57:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.642+0000 7f5972000700 1 -- 172.21.15.116:0/1816803342 shutdown_connections 2024-03-30T00:57:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.642+0000 7f5972000700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f59500a11b0 0x7f59500a3670 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.642+0000 7f5972000700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f596c102920 0x7f596c1a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.642+0000 7f5972000700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f596c100e00 0x7f596c198fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.642+0000 7f5972000700 1 --2- 172.21.15.116:0/1816803342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f596c101f60 0x7f596c1994e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.642+0000 7f5972000700 1 -- 172.21.15.116:0/1816803342 >> 172.21.15.116:0/1816803342 conn(0x7f596c0fc3c0 msgr2=0x7f596c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.642+0000 7f5972000700 1 -- 172.21.15.116:0/1816803342 shutdown_connections 2024-03-30T00:57:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:24.642+0000 7f5972000700 1 -- 172.21.15.116:0/1816803342 wait complete. 2024-03-30T00:57:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 22 2024-03-30T00:57:24.672 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:24 smithi116 ceph-mon[38086]: pgmap v277: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:24 smithi181 ceph-mon[47622]: pgmap v277: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:24 smithi130 ceph-mon[47626]: pgmap v277: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:25.020 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- bash -c 'ceph osd dump' 2024-03-30T00:57:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:57:25.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:25 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1816803342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T00:57:25.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:25 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1816803342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T00:57:25.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:25 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1816803342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T00:57:26.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:26 smithi116 ceph-mon[38086]: pgmap v278: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.676+0000 7f86cfc00700 1 -- 172.21.15.116:0/502377377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c8108cd0 msgr2=0x7f86c810b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.676+0000 7f86cfc00700 1 --2- 172.21.15.116:0/502377377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c8108cd0 0x7f86c810b160 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f86c000b0a0 tx=0x7f86c000b3b0 comp rx=0 tx=0).stop 2024-03-30T00:57:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.676+0000 7f86cfc00700 1 -- 172.21.15.116:0/502377377 shutdown_connections 2024-03-30T00:57:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.676+0000 7f86cfc00700 1 --2- 172.21.15.116:0/502377377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86c8074de0 0x7f86c8073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.676+0000 7f86cfc00700 1 --2- 172.21.15.116:0/502377377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c8108cd0 0x7f86c810b160 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.676+0000 7f86cfc00700 1 --2- 172.21.15.116:0/502377377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86c8073810 0x7f86c8073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.676+0000 7f86cfc00700 1 -- 172.21.15.116:0/502377377 >> 172.21.15.116:0/502377377 conn(0x7f86c80fc3c0 msgr2=0x7f86c80fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.676+0000 7f86cfc00700 1 -- 172.21.15.116:0/502377377 shutdown_connections 2024-03-30T00:57:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.676+0000 7f86cfc00700 1 -- 172.21.15.116:0/502377377 wait complete. 2024-03-30T00:57:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.677+0000 7f86cfc00700 1 Processor -- start 2024-03-30T00:57:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.677+0000 7f86cfc00700 1 -- start start 2024-03-30T00:57:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.677+0000 7f86cfc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86c8073810 0x7f86c81990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.677+0000 7f86cfc00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86c8074de0 0x7f86c8199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.677+0000 7f86cfc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c8108cd0 0x7f86c81a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.677+0000 7f86cfc00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c81a0c00 con 0x7f86c8108cd0 2024-03-30T00:57:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.677+0000 7f86cfc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c81a0d40 con 0x7f86c8073810 2024-03-30T00:57:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.677+0000 7f86cfc00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c81a0ed0 con 0x7f86c8074de0 2024-03-30T00:57:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.678+0000 7f86c7e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86c8074de0 0x7f86c8199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.678+0000 7f86cd200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86c8073810 0x7f86c81990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.678+0000 7f86c7e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86c8074de0 0x7f86c8199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:51000/0 (socket says 172.21.15.116:51000) 2024-03-30T00:57:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.678+0000 7f86c7e00700 1 -- 172.21.15.116:0/2555797540 learned_addr learned my addr 172.21.15.116:0/2555797540 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.678+0000 7f86cdc00700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c8108cd0 0x7f86c81a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.678+0000 7f86c7e00700 1 -- 172.21.15.116:0/2555797540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86c8073810 msgr2=0x7f86c81990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.678+0000 7f86c7e00700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86c8073810 0x7f86c81990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.678+0000 7f86c7e00700 1 -- 172.21.15.116:0/2555797540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c8108cd0 msgr2=0x7f86c81a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.678+0000 7f86c7e00700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c8108cd0 0x7f86c81a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.678+0000 7f86c7e00700 1 -- 172.21.15.116:0/2555797540 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c0009d00 con 0x7f86c8074de0 2024-03-30T00:57:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.678+0000 7f86cdc00700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c8108cd0 0x7f86c81a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.678+0000 7f86cd200700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86c8073810 0x7f86c81990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.678+0000 7f86c7e00700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86c8074de0 0x7f86c8199600 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f86b400e9a0 tx=0x7f86b400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:26.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.679+0000 7f86c5600700 1 -- 172.21.15.116:0/2555797540 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4009790 con 0x7f86c8074de0 2024-03-30T00:57:26.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.679+0000 7f86c5600700 1 -- 172.21.15.116:0/2555797540 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86b400c810 con 0x7f86c8074de0 2024-03-30T00:57:26.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.679+0000 7f86cfc00700 1 -- 172.21.15.116:0/2555797540 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c81a11b0 con 0x7f86c8074de0 2024-03-30T00:57:26.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.679+0000 7f86c5600700 1 -- 172.21.15.116:0/2555797540 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4003c70 con 0x7f86c8074de0 2024-03-30T00:57:26.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.679+0000 7f86cfc00700 1 -- 172.21.15.116:0/2555797540 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c81a1700 con 0x7f86c8074de0 2024-03-30T00:57:26.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.680+0000 7f86cfc00700 1 -- 172.21.15.116:0/2555797540 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c8107a90 con 0x7f86c8074de0 2024-03-30T00:57:26.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.682+0000 7f86c5600700 1 -- 172.21.15.116:0/2555797540 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f86b400cc80 con 0x7f86c8074de0 2024-03-30T00:57:26.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.683+0000 7f86c5600700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f86b00a1230 0x7f86b00a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:26.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.683+0000 7f86c5600700 1 -- 172.21.15.116:0/2555797540 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f86b4014070 con 0x7f86c8074de0 2024-03-30T00:57:26.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.683+0000 7f86cd200700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f86b00a1230 0x7f86b00a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:26.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.684+0000 7f86cd200700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f86b00a1230 0x7f86b00a36f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f86c80751c0 tx=0x7f86bc0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:26.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.685+0000 7f86c5600700 1 -- 172.21.15.116:0/2555797540 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f86b4096260 con 0x7f86c8074de0 2024-03-30T00:57:26.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:26 smithi181 ceph-mon[47622]: pgmap v278: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:26.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:26 smithi130 ceph-mon[47626]: pgmap v278: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:26.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.886+0000 7f86cfc00700 1 -- 172.21.15.116:0/2555797540 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f86c804ea80 con 0x7f86c8074de0 2024-03-30T00:57:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.887+0000 7f86c5600700 1 -- 172.21.15.116:0/2555797540 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v83) v1 ==== 56+0+5630 (secure 0 0 0) 0x7f86b4099880 con 0x7f86c8074de0 2024-03-30T00:57:26.896 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 83 2024-03-30T00:57:26.896 INFO:teuthology.orchestra.run.smithi116.stdout:fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T00:57:26.896 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-30T00:49:38.164101+0000 2024-03-30T00:57:26.897 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-30T00:56:36.456069+0000 2024-03-30T00:57:26.897 INFO:teuthology.orchestra.run.smithi116.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-30T00:57:26.897 INFO:teuthology.orchestra.run.smithi116.stdout:crush_version 27 2024-03-30T00:57:26.898 INFO:teuthology.orchestra.run.smithi116.stdout:full_ratio 0.95 2024-03-30T00:57:26.901 INFO:teuthology.orchestra.run.smithi116.stdout:backfillfull_ratio 0.9 2024-03-30T00:57:26.901 INFO:teuthology.orchestra.run.smithi116.stdout:nearfull_ratio 0.85 2024-03-30T00:57:26.901 INFO:teuthology.orchestra.run.smithi116.stdout:require_min_compat_client luminous 2024-03-30T00:57:26.901 INFO:teuthology.orchestra.run.smithi116.stdout:min_compat_client luminous 2024-03-30T00:57:26.902 INFO:teuthology.orchestra.run.smithi116.stdout:require_osd_release reef 2024-03-30T00:57:26.902 INFO:teuthology.orchestra.run.smithi116.stdout:stretch_mode_enabled false 2024-03-30T00:57:26.902 INFO:teuthology.orchestra.run.smithi116.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 20 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-30T00:57:26.903 INFO:teuthology.orchestra.run.smithi116.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 76 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-30T00:57:26.903 INFO:teuthology.orchestra.run.smithi116.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 77 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-30T00:57:26.903 INFO:teuthology.orchestra.run.smithi116.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 81 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-30T00:57:26.903 INFO:teuthology.orchestra.run.smithi116.stdout:max_osd 12 2024-03-30T00:57:26.903 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 up in weight 1 up_from 8 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6802/3901884018,v1:172.21.15.116:6803/3901884018] [v2:172.21.15.116:6804/3901884018,v1:172.21.15.116:6805/3901884018] exists,up ec18ae5d-60d8-48ae-85ab-06e1c3294a17 2024-03-30T00:57:26.903 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 up in weight 1 up_from 12 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6800/3819551112,v1:172.21.15.130:6801/3819551112] [v2:172.21.15.130:6802/3819551112,v1:172.21.15.130:6803/3819551112] exists,up e0047501-7c23-42fe-9e48-a949d3523427 2024-03-30T00:57:26.903 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 up in weight 1 up_from 17 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6800/3331770060,v1:172.21.15.181:6801/3331770060] [v2:172.21.15.181:6802/3331770060,v1:172.21.15.181:6803/3331770060] exists,up d7679f78-2053-4395-952b-efbdabfc12fd 2024-03-30T00:57:26.903 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 up in weight 1 up_from 24 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6810/2931358072,v1:172.21.15.116:6811/2931358072] [v2:172.21.15.116:6812/2931358072,v1:172.21.15.116:6813/2931358072] exists,up ec0571fc-3fb7-436a-82e6-7544e2491baf 2024-03-30T00:57:26.903 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 up in weight 1 up_from 31 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6808/81879957,v1:172.21.15.130:6809/81879957] [v2:172.21.15.130:6810/81879957,v1:172.21.15.130:6811/81879957] exists,up 3e7952d5-9d10-42ba-bea0-93dda26eff04 2024-03-30T00:57:26.903 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 up in weight 1 up_from 35 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6808/1074863027,v1:172.21.15.181:6809/1074863027] [v2:172.21.15.181:6810/1074863027,v1:172.21.15.181:6811/1074863027] exists,up d94364f4-bdb6-4c7f-9ed0-293887f07a8b 2024-03-30T00:57:26.904 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 up in weight 1 up_from 42 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6818/1426239130,v1:172.21.15.116:6819/1426239130] [v2:172.21.15.116:6820/1426239130,v1:172.21.15.116:6821/1426239130] exists,up 9866aade-0505-4465-87bc-91fa24eba9c7 2024-03-30T00:57:26.904 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 up in weight 1 up_from 46 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6816/2137401701,v1:172.21.15.130:6817/2137401701] [v2:172.21.15.130:6818/2137401701,v1:172.21.15.130:6819/2137401701] exists,up 11c9be8d-f9e3-4e20-8557-cb9153a894ed 2024-03-30T00:57:26.904 INFO:teuthology.orchestra.run.smithi116.stdout:osd.8 up in weight 1 up_from 53 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6816/3650979389,v1:172.21.15.181:6817/3650979389] [v2:172.21.15.181:6818/3650979389,v1:172.21.15.181:6819/3650979389] exists,up 78423339-0c27-4d28-bde2-440127f4730d 2024-03-30T00:57:26.905 INFO:teuthology.orchestra.run.smithi116.stdout:osd.9 up in weight 1 up_from 60 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6826/3758826178,v1:172.21.15.116:6827/3758826178] [v2:172.21.15.116:6828/3758826178,v1:172.21.15.116:6829/3758826178] exists,up ce69b948-3a4c-4e7e-905f-109698ca78e3 2024-03-30T00:57:26.905 INFO:teuthology.orchestra.run.smithi116.stdout:osd.10 up in weight 1 up_from 65 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6824/3569352262,v1:172.21.15.130:6825/3569352262] [v2:172.21.15.130:6826/3569352262,v1:172.21.15.130:6827/3569352262] exists,up 575cf370-93a6-4985-b490-b962f2870aa1 2024-03-30T00:57:26.905 INFO:teuthology.orchestra.run.smithi116.stdout:osd.11 up in weight 1 up_from 72 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6824/2545451272,v1:172.21.15.181:6825/2545451272] [v2:172.21.15.181:6826/2545451272,v1:172.21.15.181:6827/2545451272] exists,up 09ffa044-3256-41a3-98d1-ddb3ed32ecc7 2024-03-30T00:57:26.905 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 2.13 [2,11] 2024-03-30T00:57:26.905 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 2.1c [8,11] 2024-03-30T00:57:26.905 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.0 [5,9] 2024-03-30T00:57:26.906 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.3 [4,9] 2024-03-30T00:57:26.906 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.2c [4,9] 2024-03-30T00:57:26.906 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.31 [4,8] 2024-03-30T00:57:26.906 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 4.c [9,10] 2024-03-30T00:57:26.906 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 4.17 [9,8] 2024-03-30T00:57:26.906 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 4.18 [9,8] 2024-03-30T00:57:26.906 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 4.1a [5,8] 2024-03-30T00:57:26.907 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/309656254 expires 2024-03-31T00:50:35.029431+0000 2024-03-30T00:57:26.907 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6801/131150228 expires 2024-03-31T00:50:35.029431+0000 2024-03-30T00:57:26.907 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/755518827 expires 2024-03-31T00:50:35.029431+0000 2024-03-30T00:57:26.907 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6800/3183509903 expires 2024-03-31T00:50:11.610313+0000 2024-03-30T00:57:26.908 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/4184421214 expires 2024-03-31T00:50:11.610313+0000 2024-03-30T00:57:26.909 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6801/3183509903 expires 2024-03-31T00:50:11.610313+0000 2024-03-30T00:57:26.909 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6800/131150228 expires 2024-03-31T00:50:35.029431+0000 2024-03-30T00:57:26.909 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/2567186791 expires 2024-03-31T00:50:35.029431+0000 2024-03-30T00:57:26.909 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/2997035221 expires 2024-03-31T00:50:11.610313+0000 2024-03-30T00:57:26.909 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/2367387359 expires 2024-03-31T00:50:11.610313+0000 2024-03-30T00:57:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.889+0000 7f86cfc00700 1 -- 172.21.15.116:0/2555797540 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f86b00a1230 msgr2=0x7f86b00a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.889+0000 7f86cfc00700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f86b00a1230 0x7f86b00a36f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f86c80751c0 tx=0x7f86bc0073d0 comp rx=0 tx=0).stop 2024-03-30T00:57:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.889+0000 7f86cfc00700 1 -- 172.21.15.116:0/2555797540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86c8074de0 msgr2=0x7f86c8199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.889+0000 7f86cfc00700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86c8074de0 0x7f86c8199600 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f86b400e9a0 tx=0x7f86b400ecb0 comp rx=0 tx=0).stop 2024-03-30T00:57:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.889+0000 7f86cfc00700 1 -- 172.21.15.116:0/2555797540 shutdown_connections 2024-03-30T00:57:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.889+0000 7f86cfc00700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f86b00a1230 0x7f86b00a36f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.889+0000 7f86cfc00700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c8108cd0 0x7f86c81a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.889+0000 7f86cfc00700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86c8073810 0x7f86c81990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.889+0000 7f86cfc00700 1 --2- 172.21.15.116:0/2555797540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86c8074de0 0x7f86c8199600 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.889+0000 7f86cfc00700 1 -- 172.21.15.116:0/2555797540 >> 172.21.15.116:0/2555797540 conn(0x7f86c80fc3c0 msgr2=0x7f86c8104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.890+0000 7f86cfc00700 1 -- 172.21.15.116:0/2555797540 shutdown_connections 2024-03-30T00:57:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:26.890+0000 7f86cfc00700 1 -- 172.21.15.116:0/2555797540 wait complete. 2024-03-30T00:57:27.286 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-30T00:57:27.301 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:57:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.498+0000 7f2453200700 1 -- 172.21.15.116:0/3122242022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244c102810 msgr2=0x7f244c1090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.498+0000 7f2453200700 1 --2- 172.21.15.116:0/3122242022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244c102810 0x7f244c1090c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f244400b0d0 tx=0x7f244400b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.498+0000 7f2453200700 1 -- 172.21.15.116:0/3122242022 shutdown_connections 2024-03-30T00:57:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.498+0000 7f2453200700 1 --2- 172.21.15.116:0/3122242022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244c102810 0x7f244c1090c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.498+0000 7f2453200700 1 --2- 172.21.15.116:0/3122242022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f244c100c80 0x7f244c1010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.498+0000 7f2453200700 1 --2- 172.21.15.116:0/3122242022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f244c101e70 0x7f244c1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.498+0000 7f2453200700 1 -- 172.21.15.116:0/3122242022 >> 172.21.15.116:0/3122242022 conn(0x7f244c0fc1f0 msgr2=0x7f244c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.498+0000 7f2453200700 1 -- 172.21.15.116:0/3122242022 shutdown_connections 2024-03-30T00:57:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.498+0000 7f2453200700 1 -- 172.21.15.116:0/3122242022 wait complete. 2024-03-30T00:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.499+0000 7f2453200700 1 Processor -- start 2024-03-30T00:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.499+0000 7f2453200700 1 -- start start 2024-03-30T00:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.499+0000 7f2453200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f244c100c80 0x7f244c19d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.499+0000 7f2453200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244c101e70 0x7f244c19d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f2453200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f244c102810 0x7f244c1a4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f2453200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c1a4e40 con 0x7f244c101e70 2024-03-30T00:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f2453200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c1a4f80 con 0x7f244c100c80 2024-03-30T00:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f2453200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c1a5110 con 0x7f244c102810 2024-03-30T00:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f244be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f244c100c80 0x7f244c19d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f244b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244c101e70 0x7f244c19d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f244be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f244c100c80 0x7f244c19d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34316/0 (socket says 172.21.15.116:34316) 2024-03-30T00:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f244be00700 1 -- 172.21.15.116:0/3586517153 learned_addr learned my addr 172.21.15.116:0/3586517153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f2451200700 1 --2- 172.21.15.116:0/3586517153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f244c102810 0x7f244c1a4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f244be00700 1 -- 172.21.15.116:0/3586517153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f244c102810 msgr2=0x7f244c1a4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f244be00700 1 --2- 172.21.15.116:0/3586517153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f244c102810 0x7f244c1a4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f244be00700 1 -- 172.21.15.116:0/3586517153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244c101e70 msgr2=0x7f244c19d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f244be00700 1 --2- 172.21.15.116:0/3586517153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244c101e70 0x7f244c19d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f244be00700 1 -- 172.21.15.116:0/3586517153 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2444009d00 con 0x7f244c100c80 2024-03-30T00:57:27.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f244b400700 1 --2- 172.21.15.116:0/3586517153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244c101e70 0x7f244c19d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:27.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.500+0000 7f244be00700 1 --2- 172.21.15.116:0/3586517153 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f244c100c80 0x7f244c19d310 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f243c00e970 tx=0x7f243c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:27.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.501+0000 7f2448c00700 1 -- 172.21.15.116:0/3586517153 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243c009790 con 0x7f244c100c80 2024-03-30T00:57:27.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.501+0000 7f2453200700 1 -- 172.21.15.116:0/3586517153 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f244c1a53f0 con 0x7f244c100c80 2024-03-30T00:57:27.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.501+0000 7f2448c00700 1 -- 172.21.15.116:0/3586517153 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f243c00c7e0 con 0x7f244c100c80 2024-03-30T00:57:27.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.501+0000 7f2448c00700 1 -- 172.21.15.116:0/3586517153 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243c003c70 con 0x7f244c100c80 2024-03-30T00:57:27.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.501+0000 7f2453200700 1 -- 172.21.15.116:0/3586517153 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f244c1a5900 con 0x7f244c100c80 2024-03-30T00:57:27.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.502+0000 7f2453200700 1 -- 172.21.15.116:0/3586517153 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f244c1060a0 con 0x7f244c100c80 2024-03-30T00:57:27.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.504+0000 7f2448c00700 1 -- 172.21.15.116:0/3586517153 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f243c00cc50 con 0x7f244c100c80 2024-03-30T00:57:27.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.505+0000 7f2448c00700 1 --2- 172.21.15.116:0/3586517153 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f24300a11b0 0x7f24300a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:27.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.505+0000 7f2448c00700 1 -- 172.21.15.116:0/3586517153 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f243c014070 con 0x7f244c100c80 2024-03-30T00:57:27.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.505+0000 7f244b400700 1 --2- 172.21.15.116:0/3586517153 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f24300a11b0 0x7f24300a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:27.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.506+0000 7f244b400700 1 --2- 172.21.15.116:0/3586517153 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f24300a11b0 0x7f24300a3670 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2438011490 tx=0x7f2438009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:27.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.507+0000 7f2448c00700 1 -- 172.21.15.116:0/3586517153 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f243c0969e0 con 0x7f244c100c80 2024-03-30T00:57:27.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:27 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2555797540' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T00:57:27.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:27 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2555797540' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T00:57:27.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.752+0000 7f2453200700 1 -- 172.21.15.116:0/3586517153 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f244c067a10 con 0x7f244c100c80 2024-03-30T00:57:27.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.754+0000 7f2448c00700 1 -- 172.21.15.116:0/3586517153 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+24556 (secure 0 0 0) 0x7f243c003dd0 con 0x7f244c100c80 2024-03-30T00:57:27.761 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:27.761 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:56:36.456069+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:57:27.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.756+0000 7f2453200700 1 -- 172.21.15.116:0/3586517153 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f24300a11b0 msgr2=0x7f24300a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:27.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.756+0000 7f2453200700 1 --2- 172.21.15.116:0/3586517153 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f24300a11b0 0x7f24300a3670 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2438011490 tx=0x7f2438009260 comp rx=0 tx=0).stop 2024-03-30T00:57:27.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.756+0000 7f2453200700 1 -- 172.21.15.116:0/3586517153 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f244c100c80 msgr2=0x7f244c19d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:27.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.756+0000 7f2453200700 1 --2- 172.21.15.116:0/3586517153 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f244c100c80 0x7f244c19d310 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f243c00e970 tx=0x7f243c00ec80 comp rx=0 tx=0).stop 2024-03-30T00:57:27.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.756+0000 7f2453200700 1 -- 172.21.15.116:0/3586517153 shutdown_connections 2024-03-30T00:57:27.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.756+0000 7f2453200700 1 --2- 172.21.15.116:0/3586517153 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f24300a11b0 0x7f24300a3670 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.756+0000 7f2453200700 1 --2- 172.21.15.116:0/3586517153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f244c102810 0x7f244c1a4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.756+0000 7f2453200700 1 --2- 172.21.15.116:0/3586517153 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f244c100c80 0x7f244c19d310 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.756+0000 7f2453200700 1 --2- 172.21.15.116:0/3586517153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244c101e70 0x7f244c19d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.756+0000 7f2453200700 1 -- 172.21.15.116:0/3586517153 >> 172.21.15.116:0/3586517153 conn(0x7f244c0fc1f0 msgr2=0x7f244c1031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.756+0000 7f2453200700 1 -- 172.21.15.116:0/3586517153 shutdown_connections 2024-03-30T00:57:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.757+0000 7f2453200700 1 -- 172.21.15.116:0/3586517153 wait complete. 2024-03-30T00:57:27.783 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T00:52:29.602428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T00:55:50.220543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T00:55:51.274498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T00:55:54.301591+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T00:57:27.784 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T00:57:27.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:27 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2555797540' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T00:57:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.974+0000 7fa2f5c00700 1 -- 172.21.15.116:0/3089427734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101f70 msgr2=0x7fa2f01023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.974+0000 7fa2f5c00700 1 --2- 172.21.15.116:0/3089427734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101f70 0x7fa2f01023d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc0099c0 tx=0x7fa2dc009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.974+0000 7fa2f5c00700 1 -- 172.21.15.116:0/3089427734 shutdown_connections 2024-03-30T00:57:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.974+0000 7fa2f5c00700 1 --2- 172.21.15.116:0/3089427734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f0100d80 0x7fa2f01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.974+0000 7fa2f5c00700 1 --2- 172.21.15.116:0/3089427734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2f0102910 0x7fa2f01091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.974+0000 7fa2f5c00700 1 --2- 172.21.15.116:0/3089427734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101f70 0x7fa2f01023d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.974+0000 7fa2f5c00700 1 -- 172.21.15.116:0/3089427734 >> 172.21.15.116:0/3089427734 conn(0x7fa2f00fc2f0 msgr2=0x7fa2f00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.975+0000 7fa2f5c00700 1 -- 172.21.15.116:0/3089427734 shutdown_connections 2024-03-30T00:57:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.975+0000 7fa2f5c00700 1 -- 172.21.15.116:0/3089427734 wait complete. 2024-03-30T00:57:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.976+0000 7fa2f5c00700 1 Processor -- start 2024-03-30T00:57:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.976+0000 7fa2f5c00700 1 -- start start 2024-03-30T00:57:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.976+0000 7fa2f5c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2f0100d80 0x7fa2f0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.976+0000 7fa2f5c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101f70 0x7fa2f00731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.976+0000 7fa2f5c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f0102910 0x7fa2f00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.976+0000 7fa2f5c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f0073e80 con 0x7fa2f0101f70 2024-03-30T00:57:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.976+0000 7fa2f5c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f0073ff0 con 0x7fa2f0100d80 2024-03-30T00:57:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.976+0000 7fa2f5c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f01a0d90 con 0x7fa2f0102910 2024-03-30T00:57:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.976+0000 7fa2ee000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101f70 0x7fa2f00731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.976+0000 7fa2eea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2f0100d80 0x7fa2f0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.976+0000 7fa2ee000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101f70 0x7fa2f00731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42582/0 (socket says 172.21.15.116:42582) 2024-03-30T00:57:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.976+0000 7fa2ee000700 1 -- 172.21.15.116:0/1699278449 learned_addr learned my addr 172.21.15.116:0/1699278449 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.976+0000 7fa2ef400700 1 --2- 172.21.15.116:0/1699278449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f0102910 0x7fa2f00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.977+0000 7fa2ee000700 1 -- 172.21.15.116:0/1699278449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f0102910 msgr2=0x7fa2f00736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.977+0000 7fa2ee000700 1 --2- 172.21.15.116:0/1699278449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f0102910 0x7fa2f00736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.977+0000 7fa2ee000700 1 -- 172.21.15.116:0/1699278449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2f0100d80 msgr2=0x7fa2f0074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.977+0000 7fa2ee000700 1 --2- 172.21.15.116:0/1699278449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2f0100d80 0x7fa2f0074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.977+0000 7fa2ee000700 1 -- 172.21.15.116:0/1699278449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2dc0095f0 con 0x7fa2f0101f70 2024-03-30T00:57:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.977+0000 7fa2eea00700 1 --2- 172.21.15.116:0/1699278449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2f0100d80 0x7fa2f0074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.977+0000 7fa2ee000700 1 --2- 172.21.15.116:0/1699278449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101f70 0x7fa2f00731a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc003730 tx=0x7fa2dc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.977+0000 7fa2e7400700 1 -- 172.21.15.116:0/1699278449 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc01d070 con 0x7fa2f0101f70 2024-03-30T00:57:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.977+0000 7fa2e7400700 1 -- 172.21.15.116:0/1699278449 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa2dc005050 con 0x7fa2f0101f70 2024-03-30T00:57:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.977+0000 7fa2f5c00700 1 -- 172.21.15.116:0/1699278449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f01a0fc0 con 0x7fa2f0101f70 2024-03-30T00:57:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.977+0000 7fa2e7400700 1 -- 172.21.15.116:0/1699278449 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc004b10 con 0x7fa2f0101f70 2024-03-30T00:57:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.978+0000 7fa2f5c00700 1 -- 172.21.15.116:0/1699278449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f01a1330 con 0x7fa2f0101f70 2024-03-30T00:57:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.979+0000 7fa2f5c00700 1 -- 172.21.15.116:0/1699278449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f0108220 con 0x7fa2f0101f70 2024-03-30T00:57:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.980+0000 7fa2e7400700 1 -- 172.21.15.116:0/1699278449 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fa2dc0051c0 con 0x7fa2f0101f70 2024-03-30T00:57:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.982+0000 7fa2e7400700 1 --2- 172.21.15.116:0/1699278449 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa2d40a11b0 0x7fa2d40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.982+0000 7fa2e7400700 1 -- 172.21.15.116:0/1699278449 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa2dc0cdb40 con 0x7fa2f0101f70 2024-03-30T00:57:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.982+0000 7fa2eea00700 1 --2- 172.21.15.116:0/1699278449 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa2d40a11b0 0x7fa2d40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.982+0000 7fa2eea00700 1 --2- 172.21.15.116:0/1699278449 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa2d40a11b0 0x7fa2d40a3670 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0101dd0 tx=0x7fa2e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:27.986+0000 7fa2e7400700 1 -- 172.21.15.116:0/1699278449 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa2dc096e90 con 0x7fa2f0101f70 2024-03-30T00:57:28.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.191+0000 7fa2f5c00700 1 -- 172.21.15.116:0/1699278449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa2f010c6f0 con 0x7fa2f0101f70 2024-03-30T00:57:28.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.191+0000 7fa2e7400700 1 -- 172.21.15.116:0/1699278449 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa2dc09a4b0 con 0x7fa2f0101f70 2024-03-30T00:57:28.198 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-30T00:57:28.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.193+0000 7fa2f5c00700 1 -- 172.21.15.116:0/1699278449 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa2d40a11b0 msgr2=0x7fa2d40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:28.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.193+0000 7fa2f5c00700 1 --2- 172.21.15.116:0/1699278449 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa2d40a11b0 0x7fa2d40a3670 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0101dd0 tx=0x7fa2e0009260 comp rx=0 tx=0).stop 2024-03-30T00:57:28.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.193+0000 7fa2f5c00700 1 -- 172.21.15.116:0/1699278449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101f70 msgr2=0x7fa2f00731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:28.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.193+0000 7fa2f5c00700 1 --2- 172.21.15.116:0/1699278449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101f70 0x7fa2f00731a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc003730 tx=0x7fa2dc00f690 comp rx=0 tx=0).stop 2024-03-30T00:57:28.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.193+0000 7fa2f5c00700 1 -- 172.21.15.116:0/1699278449 shutdown_connections 2024-03-30T00:57:28.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.193+0000 7fa2f5c00700 1 --2- 172.21.15.116:0/1699278449 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa2d40a11b0 0x7fa2d40a3670 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.193+0000 7fa2f5c00700 1 --2- 172.21.15.116:0/1699278449 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa2f0102910 0x7fa2f00736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.193+0000 7fa2f5c00700 1 --2- 172.21.15.116:0/1699278449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2f0100d80 0x7fa2f0074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.193+0000 7fa2f5c00700 1 --2- 172.21.15.116:0/1699278449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101f70 0x7fa2f00731a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.193+0000 7fa2f5c00700 1 -- 172.21.15.116:0/1699278449 >> 172.21.15.116:0/1699278449 conn(0x7fa2f00fc2f0 msgr2=0x7fa2f0105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:28.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.194+0000 7fa2f5c00700 1 -- 172.21.15.116:0/1699278449 shutdown_connections 2024-03-30T00:57:28.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.194+0000 7fa2f5c00700 1 -- 172.21.15.116:0/1699278449 wait complete. 2024-03-30T00:57:28.219 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T00:57:28.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.411+0000 7f4ac2800700 1 -- 172.21.15.116:0/2887225788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc102910 msgr2=0x7f4abc1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.411+0000 7f4ac2800700 1 --2- 172.21.15.116:0/2887225788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc102910 0x7f4abc1091c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400b0d0 tx=0x7f4ab400b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.412+0000 7f4ac2800700 1 -- 172.21.15.116:0/2887225788 shutdown_connections 2024-03-30T00:57:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.412+0000 7f4ac2800700 1 --2- 172.21.15.116:0/2887225788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc102910 0x7f4abc1091c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.412+0000 7f4ac2800700 1 --2- 172.21.15.116:0/2887225788 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4abc100d80 0x7f4abc1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.412+0000 7f4ac2800700 1 --2- 172.21.15.116:0/2887225788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4abc101f70 0x7f4abc1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.412+0000 7f4ac2800700 1 -- 172.21.15.116:0/2887225788 >> 172.21.15.116:0/2887225788 conn(0x7f4abc0fc2f0 msgr2=0x7f4abc0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.412+0000 7f4ac2800700 1 -- 172.21.15.116:0/2887225788 shutdown_connections 2024-03-30T00:57:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.412+0000 7f4ac2800700 1 -- 172.21.15.116:0/2887225788 wait complete. 2024-03-30T00:57:28.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.413+0000 7f4ac2800700 1 Processor -- start 2024-03-30T00:57:28.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.413+0000 7f4ac2800700 1 -- start start 2024-03-30T00:57:28.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.413+0000 7f4ac2800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4abc100d80 0x7f4abc074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:28.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.413+0000 7f4ac2800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc101f70 0x7f4abc0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:28.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.413+0000 7f4ac2800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4abc102910 0x7f4abc0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:28.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.413+0000 7f4ac2800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc073e80 con 0x7f4abc101f70 2024-03-30T00:57:28.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.413+0000 7f4ac2800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc073ff0 con 0x7f4abc102910 2024-03-30T00:57:28.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.413+0000 7f4ac2800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc1a0e80 con 0x7f4abc100d80 2024-03-30T00:57:28.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.413+0000 7f4abaa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc101f70 0x7f4abc0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:28.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.413+0000 7f4abb400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4abc100d80 0x7f4abc074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:28.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.413+0000 7f4abaa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc101f70 0x7f4abc0731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42600/0 (socket says 172.21.15.116:42600) 2024-03-30T00:57:28.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.413+0000 7f4abaa00700 1 -- 172.21.15.116:0/3274338620 learned_addr learned my addr 172.21.15.116:0/3274338620 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:28.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.413+0000 7f4abbe00700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4abc102910 0x7f4abc0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:28.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.414+0000 7f4abb400700 1 -- 172.21.15.116:0/3274338620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4abc102910 msgr2=0x7f4abc0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:28.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.414+0000 7f4abb400700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4abc102910 0x7f4abc0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.414+0000 7f4abb400700 1 -- 172.21.15.116:0/3274338620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc101f70 msgr2=0x7f4abc0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:28.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.414+0000 7f4abb400700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc101f70 0x7f4abc0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.414+0000 7f4abb400700 1 -- 172.21.15.116:0/3274338620 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa8009520 con 0x7f4abc100d80 2024-03-30T00:57:28.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.414+0000 7f4abaa00700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc101f70 0x7f4abc0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:28.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.414+0000 7f4abbe00700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4abc102910 0x7f4abc0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:28.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.414+0000 7f4abb400700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4abc100d80 0x7f4abc074b10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4aac00e9c0 tx=0x7f4aac00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:28.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.414+0000 7f4ab3e00700 1 -- 172.21.15.116:0/3274338620 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac004220 con 0x7f4abc100d80 2024-03-30T00:57:28.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.414+0000 7f4ab3e00700 1 -- 172.21.15.116:0/3274338620 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4aac004380 con 0x7f4abc100d80 2024-03-30T00:57:28.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.414+0000 7f4ac2800700 1 -- 172.21.15.116:0/3274338620 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab4009d00 con 0x7f4abc100d80 2024-03-30T00:57:28.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.414+0000 7f4ab3e00700 1 -- 172.21.15.116:0/3274338620 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac003be0 con 0x7f4abc100d80 2024-03-30T00:57:28.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.415+0000 7f4ac2800700 1 -- 172.21.15.116:0/3274338620 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4abc1a1430 con 0x7f4abc100d80 2024-03-30T00:57:28.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.415+0000 7f4ac2800700 1 -- 172.21.15.116:0/3274338620 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4abc108130 con 0x7f4abc100d80 2024-03-30T00:57:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.417+0000 7f4ab3e00700 1 -- 172.21.15.116:0/3274338620 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f4aac00c7c0 con 0x7f4abc100d80 2024-03-30T00:57:28.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.418+0000 7f4ab3e00700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4aa00a11b0 0x7f4aa00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:28.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.419+0000 7f4ab3e00700 1 -- 172.21.15.116:0/3274338620 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4aac014070 con 0x7f4abc100d80 2024-03-30T00:57:28.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.419+0000 7f4abaa00700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4aa00a11b0 0x7f4aa00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:28.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.419+0000 7f4abaa00700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4aa00a11b0 0x7f4aa00a3670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009fd0 tx=0x7f4aa8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.423+0000 7f4ab3e00700 1 -- 172.21.15.116:0/3274338620 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4aac0963d0 con 0x7f4abc100d80 2024-03-30T00:57:28.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.627+0000 7f4ac2800700 1 -- 172.21.15.116:0/3274338620 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4abc10c5f0 con 0x7f4abc100d80 2024-03-30T00:57:28.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.628+0000 7f4ab3e00700 1 -- 172.21.15.116:0/3274338620 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4aac0999f0 con 0x7f4abc100d80 2024-03-30T00:57:28.634 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:57:28.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.630+0000 7f4ac2800700 1 -- 172.21.15.116:0/3274338620 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4aa00a11b0 msgr2=0x7f4aa00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:28.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.630+0000 7f4ac2800700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4aa00a11b0 0x7f4aa00a3670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009fd0 tx=0x7f4aa8009260 comp rx=0 tx=0).stop 2024-03-30T00:57:28.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.630+0000 7f4ac2800700 1 -- 172.21.15.116:0/3274338620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4abc100d80 msgr2=0x7f4abc074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:28.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.630+0000 7f4ac2800700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4abc100d80 0x7f4abc074b10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4aac00e9c0 tx=0x7f4aac00ecd0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.630+0000 7f4ac2800700 1 -- 172.21.15.116:0/3274338620 shutdown_connections 2024-03-30T00:57:28.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.630+0000 7f4ac2800700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4aa00a11b0 0x7f4aa00a3670 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.630+0000 7f4ac2800700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4abc100d80 0x7f4abc074b10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.630+0000 7f4ac2800700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4abc102910 0x7f4abc0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.630+0000 7f4ac2800700 1 --2- 172.21.15.116:0/3274338620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc101f70 0x7f4abc0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.630+0000 7f4ac2800700 1 -- 172.21.15.116:0/3274338620 >> 172.21.15.116:0/3274338620 conn(0x7f4abc0fc2f0 msgr2=0x7f4abc105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:28.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.630+0000 7f4ac2800700 1 -- 172.21.15.116:0/3274338620 shutdown_connections 2024-03-30T00:57:28.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.630+0000 7f4ac2800700 1 -- 172.21.15.116:0/3274338620 wait complete. 2024-03-30T00:57:28.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:28 smithi116 ceph-mon[38086]: pgmap v279: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:28.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:28 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3586517153' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:28.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:28 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1699278449' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:28.656 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T00:57:28.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:28 smithi181 ceph-mon[47622]: pgmap v279: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:28.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:28 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3586517153' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:28.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:28 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1699278449' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:28.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:28 smithi130 ceph-mon[47626]: pgmap v279: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:28.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:28 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3586517153' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:28.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:28 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1699278449' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.843+0000 7fe17c600700 1 -- 172.21.15.116:0/3674896899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe1740737b0 msgr2=0x7fe174073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.843+0000 7fe17c600700 1 --2- 172.21.15.116:0/3674896899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe1740737b0 0x7fe174073c30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe160009990 tx=0x7fe160009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.844+0000 7fe17c600700 1 -- 172.21.15.116:0/3674896899 shutdown_connections 2024-03-30T00:57:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.844+0000 7fe17c600700 1 --2- 172.21.15.116:0/3674896899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe174108c50 0x7fe17410b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.844+0000 7fe17c600700 1 --2- 172.21.15.116:0/3674896899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe174074d90 0x7fe1740731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.844+0000 7fe17c600700 1 --2- 172.21.15.116:0/3674896899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe1740737b0 0x7fe174073c30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.844+0000 7fe17c600700 1 -- 172.21.15.116:0/3674896899 >> 172.21.15.116:0/3674896899 conn(0x7fe1740fc2f0 msgr2=0x7fe1740fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.844+0000 7fe17c600700 1 -- 172.21.15.116:0/3674896899 shutdown_connections 2024-03-30T00:57:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.844+0000 7fe17c600700 1 -- 172.21.15.116:0/3674896899 wait complete. 2024-03-30T00:57:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.845+0000 7fe17c600700 1 Processor -- start 2024-03-30T00:57:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.845+0000 7fe17c600700 1 -- start start 2024-03-30T00:57:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.845+0000 7fe17c600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1740737b0 0x7fe174198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.845+0000 7fe17c600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe174074d90 0x7fe174199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.845+0000 7fe17c600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe174108c50 0x7fe1741a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.845+0000 7fe17c600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1741a0a60 con 0x7fe1740737b0 2024-03-30T00:57:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.845+0000 7fe17c600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1741a0ba0 con 0x7fe174074d90 2024-03-30T00:57:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.845+0000 7fe17c600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1741a0d30 con 0x7fe174108c50 2024-03-30T00:57:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.845+0000 7fe179c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1740737b0 0x7fe174198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.845+0000 7fe179c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1740737b0 0x7fe174198f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42612/0 (socket says 172.21.15.116:42612) 2024-03-30T00:57:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.845+0000 7fe179c00700 1 -- 172.21.15.116:0/194916224 learned_addr learned my addr 172.21.15.116:0/194916224 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.845+0000 7fe179200700 1 --2- 172.21.15.116:0/194916224 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe174074d90 0x7fe174199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.845+0000 7fe17a600700 1 --2- 172.21.15.116:0/194916224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe174108c50 0x7fe1741a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.846+0000 7fe179c00700 1 -- 172.21.15.116:0/194916224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe174108c50 msgr2=0x7fe1741a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.846+0000 7fe179c00700 1 --2- 172.21.15.116:0/194916224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe174108c50 0x7fe1741a0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.846+0000 7fe179c00700 1 -- 172.21.15.116:0/194916224 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe174074d90 msgr2=0x7fe174199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.846+0000 7fe179c00700 1 --2- 172.21.15.116:0/194916224 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe174074d90 0x7fe174199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.846+0000 7fe179c00700 1 -- 172.21.15.116:0/194916224 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1600095f0 con 0x7fe1740737b0 2024-03-30T00:57:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.846+0000 7fe179c00700 1 --2- 172.21.15.116:0/194916224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1740737b0 0x7fe174198f30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe16800e9a0 tx=0x7fe16800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.846+0000 7fe172000700 1 -- 172.21.15.116:0/194916224 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe168009790 con 0x7fe1740737b0 2024-03-30T00:57:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.846+0000 7fe172000700 1 -- 172.21.15.116:0/194916224 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe16800cbd0 con 0x7fe1740737b0 2024-03-30T00:57:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.846+0000 7fe17c600700 1 -- 172.21.15.116:0/194916224 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1741a0fb0 con 0x7fe1740737b0 2024-03-30T00:57:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.846+0000 7fe172000700 1 -- 172.21.15.116:0/194916224 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe168018680 con 0x7fe1740737b0 2024-03-30T00:57:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.846+0000 7fe17c600700 1 -- 172.21.15.116:0/194916224 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1741a1430 con 0x7fe1740737b0 2024-03-30T00:57:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.847+0000 7fe17c600700 1 -- 172.21.15.116:0/194916224 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe174107a60 con 0x7fe1740737b0 2024-03-30T00:57:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.849+0000 7fe172000700 1 -- 172.21.15.116:0/194916224 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fe16800cd40 con 0x7fe1740737b0 2024-03-30T00:57:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.850+0000 7fe172000700 1 --2- 172.21.15.116:0/194916224 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe15c0a11b0 0x7fe15c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.851+0000 7fe172000700 1 -- 172.21.15.116:0/194916224 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe168014070 con 0x7fe1740737b0 2024-03-30T00:57:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.851+0000 7fe179200700 1 --2- 172.21.15.116:0/194916224 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe15c0a11b0 0x7fe15c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.851+0000 7fe179200700 1 --2- 172.21.15.116:0/194916224 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe15c0a11b0 0x7fe15c0a3670 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe16000b2e0 tx=0x7fe160005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:28.854+0000 7fe172000700 1 -- 172.21.15.116:0/194916224 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe1680529f0 con 0x7fe1740737b0 2024-03-30T00:57:29.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.057+0000 7fe17c600700 1 -- 172.21.15.116:0/194916224 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe17410e7c0 con 0x7fe1740737b0 2024-03-30T00:57:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.058+0000 7fe172000700 1 -- 172.21.15.116:0/194916224 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe1680962f0 con 0x7fe1740737b0 2024-03-30T00:57:29.077 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-30T00:57:29.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.060+0000 7fe17c600700 1 -- 172.21.15.116:0/194916224 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe15c0a11b0 msgr2=0x7fe15c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:29.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.060+0000 7fe17c600700 1 --2- 172.21.15.116:0/194916224 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe15c0a11b0 0x7fe15c0a3670 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe16000b2e0 tx=0x7fe160005fb0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.060+0000 7fe17c600700 1 -- 172.21.15.116:0/194916224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1740737b0 msgr2=0x7fe174198f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:29.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.060+0000 7fe17c600700 1 --2- 172.21.15.116:0/194916224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1740737b0 0x7fe174198f30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe16800e9a0 tx=0x7fe16800ecb0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.060+0000 7fe17c600700 1 -- 172.21.15.116:0/194916224 shutdown_connections 2024-03-30T00:57:29.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.060+0000 7fe17c600700 1 --2- 172.21.15.116:0/194916224 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe15c0a11b0 0x7fe15c0a3670 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.060+0000 7fe17c600700 1 --2- 172.21.15.116:0/194916224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe174108c50 0x7fe1741a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.060+0000 7fe17c600700 1 --2- 172.21.15.116:0/194916224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1740737b0 0x7fe174198f30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.060+0000 7fe17c600700 1 --2- 172.21.15.116:0/194916224 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe174074d90 0x7fe174199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.060+0000 7fe17c600700 1 -- 172.21.15.116:0/194916224 >> 172.21.15.116:0/194916224 conn(0x7fe1740fc2f0 msgr2=0x7fe174104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:29.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.060+0000 7fe17c600700 1 -- 172.21.15.116:0/194916224 shutdown_connections 2024-03-30T00:57:29.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.060+0000 7fe17c600700 1 -- 172.21.15.116:0/194916224 wait complete. 2024-03-30T00:57:29.087 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T00:57:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.277+0000 7fdc97600700 1 -- 172.21.15.116:0/643056986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc90102910 msgr2=0x7fdc901091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.277+0000 7fdc97600700 1 --2- 172.21.15.116:0/643056986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc90102910 0x7fdc901091c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800b0d0 tx=0x7fdc8800b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.277+0000 7fdc97600700 1 -- 172.21.15.116:0/643056986 shutdown_connections 2024-03-30T00:57:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.277+0000 7fdc97600700 1 --2- 172.21.15.116:0/643056986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc90100d80 0x7fdc901011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.277+0000 7fdc97600700 1 --2- 172.21.15.116:0/643056986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc90102910 0x7fdc901091c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.277+0000 7fdc97600700 1 --2- 172.21.15.116:0/643056986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc90101f70 0x7fdc901023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.277+0000 7fdc97600700 1 -- 172.21.15.116:0/643056986 >> 172.21.15.116:0/643056986 conn(0x7fdc900fc2f0 msgr2=0x7fdc900fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.277+0000 7fdc97600700 1 -- 172.21.15.116:0/643056986 shutdown_connections 2024-03-30T00:57:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.277+0000 7fdc97600700 1 -- 172.21.15.116:0/643056986 wait complete. 2024-03-30T00:57:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.278+0000 7fdc97600700 1 Processor -- start 2024-03-30T00:57:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.278+0000 7fdc97600700 1 -- start start 2024-03-30T00:57:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc97600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc90100d80 0x7fdc901990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc97600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc90101f70 0x7fdc901995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc97600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc90102910 0x7fdc901a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc97600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc901a0b80 con 0x7fdc90102910 2024-03-30T00:57:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc97600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc901a0cc0 con 0x7fdc90100d80 2024-03-30T00:57:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc97600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc901a0e50 con 0x7fdc90101f70 2024-03-30T00:57:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc95600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc90102910 0x7fdc901a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc95600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc90102910 0x7fdc901a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42634/0 (socket says 172.21.15.116:42634) 2024-03-30T00:57:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc95600700 1 -- 172.21.15.116:0/587181703 learned_addr learned my addr 172.21.15.116:0/587181703 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc8fe00700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc90101f70 0x7fdc901995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:29.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc94c00700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc90100d80 0x7fdc901990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:29.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc95600700 1 -- 172.21.15.116:0/587181703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc90101f70 msgr2=0x7fdc901995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:29.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc95600700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc90101f70 0x7fdc901995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc95600700 1 -- 172.21.15.116:0/587181703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc90100d80 msgr2=0x7fdc901990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:29.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.279+0000 7fdc95600700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc90100d80 0x7fdc901990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.280+0000 7fdc95600700 1 -- 172.21.15.116:0/587181703 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c009520 con 0x7fdc90102910 2024-03-30T00:57:29.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.280+0000 7fdc8fe00700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc90101f70 0x7fdc901995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.280+0000 7fdc95600700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc90102910 0x7fdc901a05b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdc88007b60 tx=0x7fdc88008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.280+0000 7fdc8d600700 1 -- 172.21.15.116:0/587181703 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8800e040 con 0x7fdc90102910 2024-03-30T00:57:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.280+0000 7fdc8d600700 1 -- 172.21.15.116:0/587181703 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc8801d930 con 0x7fdc90102910 2024-03-30T00:57:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.280+0000 7fdc8d600700 1 -- 172.21.15.116:0/587181703 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8801b9f0 con 0x7fdc90102910 2024-03-30T00:57:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.280+0000 7fdc97600700 1 -- 172.21.15.116:0/587181703 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc88009d00 con 0x7fdc90102910 2024-03-30T00:57:29.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.280+0000 7fdc97600700 1 -- 172.21.15.116:0/587181703 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc901a1370 con 0x7fdc90102910 2024-03-30T00:57:29.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.281+0000 7fdc97600700 1 -- 172.21.15.116:0/587181703 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc90067af0 con 0x7fdc90102910 2024-03-30T00:57:29.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.283+0000 7fdc8d600700 1 -- 172.21.15.116:0/587181703 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fdc8801daa0 con 0x7fdc90102910 2024-03-30T00:57:29.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.284+0000 7fdc8d600700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdc780a1230 0x7fdc780a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:29.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.285+0000 7fdc8d600700 1 -- 172.21.15.116:0/587181703 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdc880ce290 con 0x7fdc90102910 2024-03-30T00:57:29.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.285+0000 7fdc94c00700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdc780a1230 0x7fdc780a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:29.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.285+0000 7fdc94c00700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdc780a1230 0x7fdc780a36f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdc90101dd0 tx=0x7fdc84009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:29.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.288+0000 7fdc8d600700 1 -- 172.21.15.116:0/587181703 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdc880975e0 con 0x7fdc90102910 2024-03-30T00:57:29.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.496+0000 7fdc97600700 1 -- 172.21.15.116:0/587181703 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fdc901a1630 con 0x7fdc90102910 2024-03-30T00:57:29.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.496+0000 7fdc8d600700 1 -- 172.21.15.116:0/587181703 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v83) v1 ==== 103+0+11 (secure 0 0 0) 0x7fdc8809ac00 con 0x7fdc90102910 2024-03-30T00:57:29.503 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:57:29.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.499+0000 7fdc97600700 1 -- 172.21.15.116:0/587181703 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdc780a1230 msgr2=0x7fdc780a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:29.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.499+0000 7fdc97600700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdc780a1230 0x7fdc780a36f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdc90101dd0 tx=0x7fdc84009260 comp rx=0 tx=0).stop 2024-03-30T00:57:29.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.499+0000 7fdc97600700 1 -- 172.21.15.116:0/587181703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc90102910 msgr2=0x7fdc901a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:29.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.499+0000 7fdc97600700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc90102910 0x7fdc901a05b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdc88007b60 tx=0x7fdc88008e40 comp rx=0 tx=0).stop 2024-03-30T00:57:29.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.499+0000 7fdc97600700 1 -- 172.21.15.116:0/587181703 shutdown_connections 2024-03-30T00:57:29.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.499+0000 7fdc97600700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdc780a1230 0x7fdc780a36f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.499+0000 7fdc97600700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc90102910 0x7fdc901a05b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.499+0000 7fdc97600700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc90100d80 0x7fdc901990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.499+0000 7fdc97600700 1 --2- 172.21.15.116:0/587181703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc90101f70 0x7fdc901995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.499+0000 7fdc97600700 1 -- 172.21.15.116:0/587181703 >> 172.21.15.116:0/587181703 conn(0x7fdc900fc2f0 msgr2=0x7fdc90072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:29.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.499+0000 7fdc97600700 1 -- 172.21.15.116:0/587181703 shutdown_connections 2024-03-30T00:57:29.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.499+0000 7fdc97600700 1 -- 172.21.15.116:0/587181703 wait complete. 2024-03-30T00:57:29.525 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:57:29.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:29 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3274338620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:29.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:29 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/194916224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.711+0000 7f06b5c00700 1 -- 172.21.15.116:0/1600739854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b01017b0 msgr2=0x7f06b0103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:29.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.711+0000 7f06b5c00700 1 --2- 172.21.15.116:0/1600739854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b01017b0 0x7f06b0103ba0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009990 tx=0x7f06a0009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.711+0000 7f06b5c00700 1 -- 172.21.15.116:0/1600739854 shutdown_connections 2024-03-30T00:57:29.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.711+0000 7f06b5c00700 1 --2- 172.21.15.116:0/1600739854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b0106a10 0x7f06b0108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.711+0000 7f06b5c00700 1 --2- 172.21.15.116:0/1600739854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b01017b0 0x7f06b0103ba0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.711+0000 7f06b5c00700 1 --2- 172.21.15.116:0/1600739854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06b01040e0 0x7f06b01064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.711+0000 7f06b5c00700 1 -- 172.21.15.116:0/1600739854 >> 172.21.15.116:0/1600739854 conn(0x7f06b00fb100 msgr2=0x7f06b00fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:29.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.711+0000 7f06b5c00700 1 -- 172.21.15.116:0/1600739854 shutdown_connections 2024-03-30T00:57:29.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.711+0000 7f06b5c00700 1 -- 172.21.15.116:0/1600739854 wait complete. 2024-03-30T00:57:29.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.712+0000 7f06b5c00700 1 Processor -- start 2024-03-30T00:57:29.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.712+0000 7f06b5c00700 1 -- start start 2024-03-30T00:57:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.712+0000 7f06b5c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06b01017b0 0x7f06b01010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.712+0000 7f06b5c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b01040e0 0x7f06b00ff760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.712+0000 7f06b5c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b0106a10 0x7f06b00ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.712+0000 7f06b5c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b019aad0 con 0x7f06b0106a10 2024-03-30T00:57:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.712+0000 7f06b5c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b01001e0 con 0x7f06b01040e0 2024-03-30T00:57:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.712+0000 7f06b5c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b01003a0 con 0x7f06b01017b0 2024-03-30T00:57:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.713+0000 7f06af400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b0106a10 0x7f06b00ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.713+0000 7f06ae000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b01040e0 0x7f06b00ff760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.713+0000 7f06af400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b0106a10 0x7f06b00ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42662/0 (socket says 172.21.15.116:42662) 2024-03-30T00:57:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.713+0000 7f06af400700 1 -- 172.21.15.116:0/500014484 learned_addr learned my addr 172.21.15.116:0/500014484 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.713+0000 7f06aea00700 1 --2- 172.21.15.116:0/500014484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06b01017b0 0x7f06b01010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.713+0000 7f06ae000700 1 -- 172.21.15.116:0/500014484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06b01017b0 msgr2=0x7f06b01010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.713+0000 7f06ae000700 1 --2- 172.21.15.116:0/500014484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06b01017b0 0x7f06b01010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.713+0000 7f06ae000700 1 -- 172.21.15.116:0/500014484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b0106a10 msgr2=0x7f06b00ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.713+0000 7f06ae000700 1 --2- 172.21.15.116:0/500014484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b0106a10 0x7f06b00ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.713+0000 7f06ae000700 1 -- 172.21.15.116:0/500014484 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a00095f0 con 0x7f06b01040e0 2024-03-30T00:57:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.713+0000 7f06af400700 1 --2- 172.21.15.116:0/500014484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b0106a10 0x7f06b00ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.713+0000 7f06ae000700 1 --2- 172.21.15.116:0/500014484 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b01040e0 0x7f06b00ff760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f069c00e9f0 tx=0x7f069c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.714+0000 7f06a7400700 1 -- 172.21.15.116:0/500014484 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069c00c9c0 con 0x7f06b01040e0 2024-03-30T00:57:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.714+0000 7f06b5c00700 1 -- 172.21.15.116:0/500014484 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b01a5480 con 0x7f06b01040e0 2024-03-30T00:57:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.714+0000 7f06a7400700 1 -- 172.21.15.116:0/500014484 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f069c010430 con 0x7f06b01040e0 2024-03-30T00:57:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.714+0000 7f06a7400700 1 -- 172.21.15.116:0/500014484 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069c018680 con 0x7f06b01040e0 2024-03-30T00:57:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.714+0000 7f06b5c00700 1 -- 172.21.15.116:0/500014484 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b01a58a0 con 0x7f06b01040e0 2024-03-30T00:57:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.715+0000 7f06b5c00700 1 -- 172.21.15.116:0/500014484 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b00fcc80 con 0x7f06b01040e0 2024-03-30T00:57:29.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.717+0000 7f06a7400700 1 -- 172.21.15.116:0/500014484 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f069c010860 con 0x7f06b01040e0 2024-03-30T00:57:29.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.718+0000 7f06a7400700 1 --2- 172.21.15.116:0/500014484 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f06940a11b0 0x7f06940a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:29.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.718+0000 7f06a7400700 1 -- 172.21.15.116:0/500014484 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f069c014070 con 0x7f06b01040e0 2024-03-30T00:57:29.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.718+0000 7f06aea00700 1 --2- 172.21.15.116:0/500014484 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f06940a11b0 0x7f06940a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:29.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.718+0000 7f06aea00700 1 --2- 172.21.15.116:0/500014484 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f06940a11b0 0x7f06940a3670 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f06a000b2e0 tx=0x7f06a0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:29.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.720+0000 7f06a7400700 1 -- 172.21.15.116:0/500014484 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f069c096160 con 0x7f06b01040e0 2024-03-30T00:57:29.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:29 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3274338620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:29.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:29 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/194916224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:29.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:29 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3274338620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:29.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:29 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/194916224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:29.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.967+0000 7f06b5c00700 1 -- 172.21.15.116:0/500014484 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06b0067180 con 0x7f06b01040e0 2024-03-30T00:57:29.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.968+0000 7f06a7400700 1 -- 172.21.15.116:0/500014484 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+10886 (secure 0 0 0) 0x7f069c099780 con 0x7f06b01040e0 2024-03-30T00:57:29.975 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:29.975 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:56:22.144079+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T00:57:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.970+0000 7f06b5c00700 1 -- 172.21.15.116:0/500014484 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f06940a11b0 msgr2=0x7f06940a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.970+0000 7f06b5c00700 1 --2- 172.21.15.116:0/500014484 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f06940a11b0 0x7f06940a3670 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f06a000b2e0 tx=0x7f06a0005fb0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.970+0000 7f06b5c00700 1 -- 172.21.15.116:0/500014484 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b01040e0 msgr2=0x7f06b00ff760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.970+0000 7f06b5c00700 1 --2- 172.21.15.116:0/500014484 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b01040e0 0x7f06b00ff760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f069c00e9f0 tx=0x7f069c00ed00 comp rx=0 tx=0).stop 2024-03-30T00:57:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.970+0000 7f06b5c00700 1 -- 172.21.15.116:0/500014484 shutdown_connections 2024-03-30T00:57:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.970+0000 7f06b5c00700 1 --2- 172.21.15.116:0/500014484 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f06940a11b0 0x7f06940a3670 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.970+0000 7f06b5c00700 1 --2- 172.21.15.116:0/500014484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06b01017b0 0x7f06b01010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.970+0000 7f06b5c00700 1 --2- 172.21.15.116:0/500014484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06b0106a10 0x7f06b00ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.970+0000 7f06b5c00700 1 --2- 172.21.15.116:0/500014484 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b01040e0 0x7f06b00ff760 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.970+0000 7f06b5c00700 1 -- 172.21.15.116:0/500014484 >> 172.21.15.116:0/500014484 conn(0x7f06b00fb100 msgr2=0x7f06b0102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.970+0000 7f06b5c00700 1 -- 172.21.15.116:0/500014484 shutdown_connections 2024-03-30T00:57:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:29.971+0000 7f06b5c00700 1 -- 172.21.15.116:0/500014484 wait complete. 2024-03-30T00:57:29.980 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 22 2024-03-30T00:57:29.997 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:57:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.183+0000 7efcf0200700 1 -- 172.21.15.116:0/1550806793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efce8101e70 msgr2=0x7efce81022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.183+0000 7efcf0200700 1 --2- 172.21.15.116:0/1550806793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efce8101e70 0x7efce81022d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7efcd40099c0 tx=0x7efcd4009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.184+0000 7efcf0200700 1 -- 172.21.15.116:0/1550806793 shutdown_connections 2024-03-30T00:57:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.184+0000 7efcf0200700 1 --2- 172.21.15.116:0/1550806793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efce8102810 0x7efce81090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.184+0000 7efcf0200700 1 --2- 172.21.15.116:0/1550806793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce8100c80 0x7efce81010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.184+0000 7efcf0200700 1 --2- 172.21.15.116:0/1550806793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efce8101e70 0x7efce81022d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.184+0000 7efcf0200700 1 -- 172.21.15.116:0/1550806793 >> 172.21.15.116:0/1550806793 conn(0x7efce80fc1f0 msgr2=0x7efce80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.184+0000 7efcf0200700 1 -- 172.21.15.116:0/1550806793 shutdown_connections 2024-03-30T00:57:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.184+0000 7efcf0200700 1 -- 172.21.15.116:0/1550806793 wait complete. 2024-03-30T00:57:30.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.185+0000 7efcf0200700 1 Processor -- start 2024-03-30T00:57:30.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.185+0000 7efcf0200700 1 -- start start 2024-03-30T00:57:30.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.185+0000 7efcf0200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efce8100c80 0x7efce8196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:30.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.185+0000 7efcf0200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce8101e70 0x7efce8197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:30.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.185+0000 7efcf0200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efce8102810 0x7efce819e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:30.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.185+0000 7efcf0200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce819e710 con 0x7efce8100c80 2024-03-30T00:57:30.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.185+0000 7efcf0200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce819e850 con 0x7efce8101e70 2024-03-30T00:57:30.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.185+0000 7efcf0200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce819e9e0 con 0x7efce8102810 2024-03-30T00:57:30.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.185+0000 7efced800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efce8100c80 0x7efce8196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:30.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.185+0000 7efced800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efce8100c80 0x7efce8196be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42676/0 (socket says 172.21.15.116:42676) 2024-03-30T00:57:30.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.185+0000 7efced800700 1 -- 172.21.15.116:0/2047366387 learned_addr learned my addr 172.21.15.116:0/2047366387 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:30.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.185+0000 7efcece00700 1 --2- 172.21.15.116:0/2047366387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce8101e70 0x7efce8197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:30.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.186+0000 7efcee200700 1 --2- 172.21.15.116:0/2047366387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efce8102810 0x7efce819e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:30.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.186+0000 7efced800700 1 -- 172.21.15.116:0/2047366387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efce8102810 msgr2=0x7efce819e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:30.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.186+0000 7efced800700 1 --2- 172.21.15.116:0/2047366387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efce8102810 0x7efce819e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:30.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.186+0000 7efced800700 1 -- 172.21.15.116:0/2047366387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce8101e70 msgr2=0x7efce8197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:30.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.186+0000 7efced800700 1 --2- 172.21.15.116:0/2047366387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce8101e70 0x7efce8197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:30.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.186+0000 7efced800700 1 -- 172.21.15.116:0/2047366387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd40095f0 con 0x7efce8100c80 2024-03-30T00:57:30.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.186+0000 7efced800700 1 --2- 172.21.15.116:0/2047366387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efce8100c80 0x7efce8196be0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00e970 tx=0x7efcdc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:30.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.186+0000 7efce6000700 1 -- 172.21.15.116:0/2047366387 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc009790 con 0x7efce8100c80 2024-03-30T00:57:30.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.186+0000 7efce6000700 1 -- 172.21.15.116:0/2047366387 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcdc00c7e0 con 0x7efce8100c80 2024-03-30T00:57:30.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.186+0000 7efce6000700 1 -- 172.21.15.116:0/2047366387 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc003c70 con 0x7efce8100c80 2024-03-30T00:57:30.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.186+0000 7efcf0200700 1 -- 172.21.15.116:0/2047366387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce819ecc0 con 0x7efce8100c80 2024-03-30T00:57:30.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.186+0000 7efcf0200700 1 -- 172.21.15.116:0/2047366387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce819f180 con 0x7efce8100c80 2024-03-30T00:57:30.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.187+0000 7efcf0200700 1 -- 172.21.15.116:0/2047366387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce81080b0 con 0x7efce8100c80 2024-03-30T00:57:30.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.189+0000 7efce6000700 1 -- 172.21.15.116:0/2047366387 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7efcdc00cc50 con 0x7efce8100c80 2024-03-30T00:57:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.191+0000 7efce6000700 1 --2- 172.21.15.116:0/2047366387 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efcd00a11b0 0x7efcd00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.191+0000 7efce6000700 1 -- 172.21.15.116:0/2047366387 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efcdc014070 con 0x7efce8100c80 2024-03-30T00:57:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.191+0000 7efcece00700 1 --2- 172.21.15.116:0/2047366387 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efcd00a11b0 0x7efcd00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.191+0000 7efcece00700 1 --2- 172.21.15.116:0/2047366387 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efcd00a11b0 0x7efcd00a3670 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efcd4003730 tx=0x7efcd400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.193+0000 7efce6000700 1 -- 172.21.15.116:0/2047366387 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efcdc096380 con 0x7efce8100c80 2024-03-30T00:57:30.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.431+0000 7efcf0200700 1 -- 172.21.15.116:0/2047366387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efce8067a10 con 0x7efce8100c80 2024-03-30T00:57:30.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.432+0000 7efce6000700 1 -- 172.21.15.116:0/2047366387 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+24556 (secure 0 0 0) 0x7efcdc080000 con 0x7efce8100c80 2024-03-30T00:57:30.439 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:30.440 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:56:36.456069+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:57:30.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.435+0000 7efcf0200700 1 -- 172.21.15.116:0/2047366387 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efcd00a11b0 msgr2=0x7efcd00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:30.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.435+0000 7efcf0200700 1 --2- 172.21.15.116:0/2047366387 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efcd00a11b0 0x7efcd00a3670 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efcd4003730 tx=0x7efcd400b540 comp rx=0 tx=0).stop 2024-03-30T00:57:30.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.435+0000 7efcf0200700 1 -- 172.21.15.116:0/2047366387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efce8100c80 msgr2=0x7efce8196be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:30.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.435+0000 7efcf0200700 1 --2- 172.21.15.116:0/2047366387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efce8100c80 0x7efce8196be0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00e970 tx=0x7efcdc00ec80 comp rx=0 tx=0).stop 2024-03-30T00:57:30.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.435+0000 7efcf0200700 1 -- 172.21.15.116:0/2047366387 shutdown_connections 2024-03-30T00:57:30.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.435+0000 7efcf0200700 1 --2- 172.21.15.116:0/2047366387 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efcd00a11b0 0x7efcd00a3670 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:30.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.435+0000 7efcf0200700 1 --2- 172.21.15.116:0/2047366387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efce8102810 0x7efce819e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:30.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.435+0000 7efcf0200700 1 --2- 172.21.15.116:0/2047366387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efce8100c80 0x7efce8196be0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:30.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.435+0000 7efcf0200700 1 --2- 172.21.15.116:0/2047366387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce8101e70 0x7efce8197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:30.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.435+0000 7efcf0200700 1 -- 172.21.15.116:0/2047366387 >> 172.21.15.116:0/2047366387 conn(0x7efce80fc1f0 msgr2=0x7efce81051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:30.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.435+0000 7efcf0200700 1 -- 172.21.15.116:0/2047366387 shutdown_connections 2024-03-30T00:57:30.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.435+0000 7efcf0200700 1 -- 172.21.15.116:0/2047366387 wait complete. 2024-03-30T00:57:30.461 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T00:52:29.602428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T00:55:50.220543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T00:55:51.274498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T00:55:54.301591+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T00:57:30.462 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T00:57:30.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.650+0000 7f112b200700 1 -- 172.21.15.116:0/3881237242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1124102750 msgr2=0x7f1124109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:31.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.650+0000 7f112b200700 1 --2- 172.21.15.116:0/3881237242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1124102750 0x7f1124109000 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f112000b0d0 tx=0x7f112000b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.650+0000 7f112b200700 1 -- 172.21.15.116:0/3881237242 shutdown_connections 2024-03-30T00:57:31.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.650+0000 7f112b200700 1 --2- 172.21.15.116:0/3881237242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124100bc0 0x7f1124100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.650+0000 7f112b200700 1 --2- 172.21.15.116:0/3881237242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1124102750 0x7f1124109000 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.650+0000 7f112b200700 1 --2- 172.21.15.116:0/3881237242 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1124101db0 0x7f1124102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.650+0000 7f112b200700 1 -- 172.21.15.116:0/3881237242 >> 172.21.15.116:0/3881237242 conn(0x7f11240fc110 msgr2=0x7f11240fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:31.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.650+0000 7f112b200700 1 -- 172.21.15.116:0/3881237242 shutdown_connections 2024-03-30T00:57:31.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.650+0000 7f112b200700 1 -- 172.21.15.116:0/3881237242 wait complete. 2024-03-30T00:57:31.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.651+0000 7f112b200700 1 Processor -- start 2024-03-30T00:57:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f112b200700 1 -- start start 2024-03-30T00:57:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f112b200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1124100bc0 0x7f1124198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f112b200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124101db0 0x7f11241993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f112b200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1124102750 0x7f11241a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f112b200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11241a0990 con 0x7f1124100bc0 2024-03-30T00:57:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f112b200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11241a0ad0 con 0x7f1124102750 2024-03-30T00:57:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f112b200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11241a0c60 con 0x7f1124101db0 2024-03-30T00:57:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f1129e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1124100bc0 0x7f1124198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f1129e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1124100bc0 0x7f1124198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42686/0 (socket says 172.21.15.116:42686) 2024-03-30T00:57:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f1129e00700 1 -- 172.21.15.116:0/1432558412 learned_addr learned my addr 172.21.15.116:0/1432558412 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f1129400700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124101db0 0x7f11241993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f112a800700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1124102750 0x7f11241a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f1129400700 1 -- 172.21.15.116:0/1432558412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1124102750 msgr2=0x7f11241a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f1129400700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1124102750 0x7f11241a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f1129400700 1 -- 172.21.15.116:0/1432558412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1124100bc0 msgr2=0x7f1124198e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.652+0000 7f1129400700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1124100bc0 0x7f1124198e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.653+0000 7f1129400700 1 -- 172.21.15.116:0/1432558412 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1120009d00 con 0x7f1124101db0 2024-03-30T00:57:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.653+0000 7f1129e00700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1124100bc0 0x7f1124198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.653+0000 7f1129400700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124101db0 0x7f11241993a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f111400e9f0 tx=0x7f111400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.653+0000 7f111ea00700 1 -- 172.21.15.116:0/1432558412 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111400c9c0 con 0x7f1124101db0 2024-03-30T00:57:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.653+0000 7f112b200700 1 -- 172.21.15.116:0/1432558412 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11241a0f40 con 0x7f1124101db0 2024-03-30T00:57:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.653+0000 7f111ea00700 1 -- 172.21.15.116:0/1432558412 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1114010430 con 0x7f1124101db0 2024-03-30T00:57:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.653+0000 7f111ea00700 1 -- 172.21.15.116:0/1432558412 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1114018680 con 0x7f1124101db0 2024-03-30T00:57:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.653+0000 7f112b200700 1 -- 172.21.15.116:0/1432558412 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11241a1410 con 0x7f1124101db0 2024-03-30T00:57:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.654+0000 7f112b200700 1 -- 172.21.15.116:0/1432558412 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11240679a0 con 0x7f1124101db0 2024-03-30T00:57:31.158 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:30 smithi130 ceph-mon[47626]: pgmap v280: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:57:31.158 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:30 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/587181703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:31.158 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:30 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/500014484' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:31.158 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:30 smithi181 ceph-mon[47622]: pgmap v280: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:57:31.158 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:30 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/587181703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:31.158 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:30 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/500014484' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.657+0000 7f111ea00700 1 -- 172.21.15.116:0/1432558412 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f1114010860 con 0x7f1124101db0 2024-03-30T00:57:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.658+0000 7f111ea00700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f110c0a11b0 0x7f110c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.658+0000 7f111ea00700 1 -- 172.21.15.116:0/1432558412 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1114014070 con 0x7f1124101db0 2024-03-30T00:57:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.658+0000 7f1129e00700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f110c0a11b0 0x7f110c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.659+0000 7f1129e00700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f110c0a11b0 0x7f110c0a3670 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1118005950 tx=0x7f1118009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:31.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.660+0000 7f111ea00700 1 -- 172.21.15.116:0/1432558412 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1114096300 con 0x7f1124101db0 2024-03-30T00:57:31.160 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:30 smithi116 ceph-mon[38086]: pgmap v280: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:57:31.160 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:30 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/587181703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:31.160 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:30 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/500014484' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.863+0000 7f112b200700 1 -- 172.21.15.116:0/1432558412 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f112410c430 con 0x7f1124101db0 2024-03-30T00:57:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.864+0000 7f111ea00700 1 -- 172.21.15.116:0/1432558412 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1114099920 con 0x7f1124101db0 2024-03-30T00:57:31.161 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-30T00:57:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.866+0000 7f112b200700 1 -- 172.21.15.116:0/1432558412 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f110c0a11b0 msgr2=0x7f110c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.866+0000 7f112b200700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f110c0a11b0 0x7f110c0a3670 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1118005950 tx=0x7f1118009260 comp rx=0 tx=0).stop 2024-03-30T00:57:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.866+0000 7f112b200700 1 -- 172.21.15.116:0/1432558412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124101db0 msgr2=0x7f11241993a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.866+0000 7f112b200700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124101db0 0x7f11241993a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f111400e9f0 tx=0x7f111400ed00 comp rx=0 tx=0).stop 2024-03-30T00:57:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.867+0000 7f112b200700 1 -- 172.21.15.116:0/1432558412 shutdown_connections 2024-03-30T00:57:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.867+0000 7f112b200700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f110c0a11b0 0x7f110c0a3670 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.867+0000 7f112b200700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1124102750 0x7f11241a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.867+0000 7f112b200700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1124100bc0 0x7f1124198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.867+0000 7f112b200700 1 --2- 172.21.15.116:0/1432558412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124101db0 0x7f11241993a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.867+0000 7f112b200700 1 -- 172.21.15.116:0/1432558412 >> 172.21.15.116:0/1432558412 conn(0x7f11240fc110 msgr2=0x7f1124072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.867+0000 7f112b200700 1 -- 172.21.15.116:0/1432558412 shutdown_connections 2024-03-30T00:57:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:30.867+0000 7f112b200700 1 -- 172.21.15.116:0/1432558412 wait complete. 2024-03-30T00:57:31.165 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T00:57:31.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.400+0000 7f8782200700 1 -- 172.21.15.116:0/3706301003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f877c102910 msgr2=0x7f877c1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:31.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.400+0000 7f8782200700 1 --2- 172.21.15.116:0/3706301003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f877c102910 0x7f877c1091c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f877400b0d0 tx=0x7f877400b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.400+0000 7f8782200700 1 -- 172.21.15.116:0/3706301003 shutdown_connections 2024-03-30T00:57:31.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.400+0000 7f8782200700 1 --2- 172.21.15.116:0/3706301003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f877c102910 0x7f877c1091c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.400+0000 7f8782200700 1 --2- 172.21.15.116:0/3706301003 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f877c100d80 0x7f877c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.400+0000 7f8782200700 1 --2- 172.21.15.116:0/3706301003 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f877c101f70 0x7f877c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.400+0000 7f8782200700 1 -- 172.21.15.116:0/3706301003 >> 172.21.15.116:0/3706301003 conn(0x7f877c0fc2f0 msgr2=0x7f877c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:31.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.400+0000 7f8782200700 1 -- 172.21.15.116:0/3706301003 shutdown_connections 2024-03-30T00:57:31.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.400+0000 7f8782200700 1 -- 172.21.15.116:0/3706301003 wait complete. 2024-03-30T00:57:31.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.401+0000 7f8782200700 1 Processor -- start 2024-03-30T00:57:31.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.401+0000 7f8782200700 1 -- start start 2024-03-30T00:57:31.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.402+0000 7f8782200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f877c100d80 0x7f877c196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:31.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.402+0000 7f8782200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f877c101f70 0x7f877c197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:31.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.402+0000 7f8782200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f877c102910 0x7f877c19e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:31.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.402+0000 7f8782200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c19e950 con 0x7f877c102910 2024-03-30T00:57:31.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.402+0000 7f8782200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c19ea90 con 0x7f877c100d80 2024-03-30T00:57:31.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.402+0000 7f8782200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c19ec20 con 0x7f877c101f70 2024-03-30T00:57:31.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.402+0000 7f877be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f877c102910 0x7f877c19e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:31.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.402+0000 7f877be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f877c102910 0x7f877c19e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42704/0 (socket says 172.21.15.116:42704) 2024-03-30T00:57:31.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.402+0000 7f877be00700 1 -- 172.21.15.116:0/1028101858 learned_addr learned my addr 172.21.15.116:0/1028101858 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:31.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.402+0000 7f877b400700 1 --2- 172.21.15.116:0/1028101858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f877c100d80 0x7f877c196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:31.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.402+0000 7f877aa00700 1 --2- 172.21.15.116:0/1028101858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f877c101f70 0x7f877c197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:31.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.403+0000 7f877be00700 1 -- 172.21.15.116:0/1028101858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f877c101f70 msgr2=0x7f877c197360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:31.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.403+0000 7f877be00700 1 --2- 172.21.15.116:0/1028101858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f877c101f70 0x7f877c197360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.403+0000 7f877be00700 1 -- 172.21.15.116:0/1028101858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f877c100d80 msgr2=0x7f877c196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:31.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.403+0000 7f877be00700 1 --2- 172.21.15.116:0/1028101858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f877c100d80 0x7f877c196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.403+0000 7f877be00700 1 -- 172.21.15.116:0/1028101858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8774009d00 con 0x7f877c102910 2024-03-30T00:57:31.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.403+0000 7f877be00700 1 --2- 172.21.15.116:0/1028101858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f877c102910 0x7f877c19e380 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8774007b60 tx=0x7f8774008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:31.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.403+0000 7f8773e00700 1 -- 172.21.15.116:0/1028101858 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877400e040 con 0x7f877c102910 2024-03-30T00:57:31.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.403+0000 7f8773e00700 1 -- 172.21.15.116:0/1028101858 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f877401d930 con 0x7f877c102910 2024-03-30T00:57:31.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.403+0000 7f8773e00700 1 -- 172.21.15.116:0/1028101858 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877401b9f0 con 0x7f877c102910 2024-03-30T00:57:31.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.403+0000 7f8782200700 1 -- 172.21.15.116:0/1028101858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877c19eea0 con 0x7f877c102910 2024-03-30T00:57:31.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.403+0000 7f8782200700 1 -- 172.21.15.116:0/1028101858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877c19f270 con 0x7f877c102910 2024-03-30T00:57:31.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.405+0000 7f8782200700 1 -- 172.21.15.116:0/1028101858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877c108130 con 0x7f877c102910 2024-03-30T00:57:31.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.406+0000 7f8773e00700 1 -- 172.21.15.116:0/1028101858 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f877401daa0 con 0x7f877c102910 2024-03-30T00:57:31.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.407+0000 7f8773e00700 1 --2- 172.21.15.116:0/1028101858 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f87600a1230 0x7f87600a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:31.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.407+0000 7f8773e00700 1 -- 172.21.15.116:0/1028101858 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f87740ce280 con 0x7f877c102910 2024-03-30T00:57:31.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.408+0000 7f877b400700 1 --2- 172.21.15.116:0/1028101858 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f87600a1230 0x7f87600a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:31.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.408+0000 7f877b400700 1 --2- 172.21.15.116:0/1028101858 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f87600a1230 0x7f87600a36f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f877c101dd0 tx=0x7f876c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:31.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.411+0000 7f8773e00700 1 -- 172.21.15.116:0/1028101858 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f87740975d0 con 0x7f877c102910 2024-03-30T00:57:31.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.621+0000 7f8782200700 1 -- 172.21.15.116:0/1028101858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f877c10c5f0 con 0x7f877c102910 2024-03-30T00:57:31.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.622+0000 7f8773e00700 1 -- 172.21.15.116:0/1028101858 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f877409abf0 con 0x7f877c102910 2024-03-30T00:57:31.645 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:57:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.624+0000 7f8782200700 1 -- 172.21.15.116:0/1028101858 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f87600a1230 msgr2=0x7f87600a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.624+0000 7f8782200700 1 --2- 172.21.15.116:0/1028101858 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f87600a1230 0x7f87600a36f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f877c101dd0 tx=0x7f876c009260 comp rx=0 tx=0).stop 2024-03-30T00:57:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.624+0000 7f8782200700 1 -- 172.21.15.116:0/1028101858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f877c102910 msgr2=0x7f877c19e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.624+0000 7f8782200700 1 --2- 172.21.15.116:0/1028101858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f877c102910 0x7f877c19e380 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8774007b60 tx=0x7f8774008e40 comp rx=0 tx=0).stop 2024-03-30T00:57:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.624+0000 7f8782200700 1 -- 172.21.15.116:0/1028101858 shutdown_connections 2024-03-30T00:57:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.624+0000 7f8782200700 1 --2- 172.21.15.116:0/1028101858 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f87600a1230 0x7f87600a36f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.624+0000 7f8782200700 1 --2- 172.21.15.116:0/1028101858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f877c102910 0x7f877c19e380 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.624+0000 7f8782200700 1 --2- 172.21.15.116:0/1028101858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f877c100d80 0x7f877c196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.624+0000 7f8782200700 1 --2- 172.21.15.116:0/1028101858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f877c101f70 0x7f877c197360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.624+0000 7f8782200700 1 -- 172.21.15.116:0/1028101858 >> 172.21.15.116:0/1028101858 conn(0x7f877c0fc2f0 msgr2=0x7f877c105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.625+0000 7f8782200700 1 -- 172.21.15.116:0/1028101858 shutdown_connections 2024-03-30T00:57:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.625+0000 7f8782200700 1 -- 172.21.15.116:0/1028101858 wait complete. 2024-03-30T00:57:31.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:31 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2047366387' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:31.648 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:31 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1432558412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:31.650 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T00:57:31.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:31 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2047366387' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:31.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:31 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1432558412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:31.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:31 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2047366387' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:31.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:31 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1432558412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:31.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.841+0000 7f09cc200700 1 -- 172.21.15.116:0/1944709457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c4102910 msgr2=0x7f09c41091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:31.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.841+0000 7f09cc200700 1 --2- 172.21.15.116:0/1944709457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c4102910 0x7f09c41091c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00b0d0 tx=0x7f09bc00b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.842+0000 7f09cc200700 1 -- 172.21.15.116:0/1944709457 shutdown_connections 2024-03-30T00:57:31.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.842+0000 7f09cc200700 1 --2- 172.21.15.116:0/1944709457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c4100d80 0x7f09c41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.842+0000 7f09cc200700 1 --2- 172.21.15.116:0/1944709457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c4102910 0x7f09c41091c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.842+0000 7f09cc200700 1 --2- 172.21.15.116:0/1944709457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09c4101f70 0x7f09c41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.842+0000 7f09cc200700 1 -- 172.21.15.116:0/1944709457 >> 172.21.15.116:0/1944709457 conn(0x7f09c40fc2f0 msgr2=0x7f09c40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:31.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.842+0000 7f09cc200700 1 -- 172.21.15.116:0/1944709457 shutdown_connections 2024-03-30T00:57:31.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.842+0000 7f09cc200700 1 -- 172.21.15.116:0/1944709457 wait complete. 2024-03-30T00:57:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.843+0000 7f09cc200700 1 Processor -- start 2024-03-30T00:57:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.843+0000 7f09cc200700 1 -- start start 2024-03-30T00:57:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.843+0000 7f09cc200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c4100d80 0x7f09c4196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.843+0000 7f09cc200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09c4101f70 0x7f09c4197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.843+0000 7f09cc200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c4102910 0x7f09c419e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.843+0000 7f09cc200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c419e900 con 0x7f09c4102910 2024-03-30T00:57:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.843+0000 7f09cc200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c419ea70 con 0x7f09c4101f70 2024-03-30T00:57:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.843+0000 7f09cc200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c419ec30 con 0x7f09c4100d80 2024-03-30T00:57:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.844+0000 7f09c8e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09c4101f70 0x7f09c4197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.844+0000 7f09c8e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09c4101f70 0x7f09c4197350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34482/0 (socket says 172.21.15.116:34482) 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.844+0000 7f09c8e00700 1 -- 172.21.15.116:0/3313515433 learned_addr learned my addr 172.21.15.116:0/3313515433 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.844+0000 7f09ca200700 1 --2- 172.21.15.116:0/3313515433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c4102910 0x7f09c419e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.844+0000 7f09c9800700 1 --2- 172.21.15.116:0/3313515433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c4100d80 0x7f09c4196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.844+0000 7f09ca200700 1 -- 172.21.15.116:0/3313515433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c4100d80 msgr2=0x7f09c4196e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.844+0000 7f09ca200700 1 --2- 172.21.15.116:0/3313515433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c4100d80 0x7f09c4196e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.844+0000 7f09ca200700 1 -- 172.21.15.116:0/3313515433 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09c4101f70 msgr2=0x7f09c4197350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.844+0000 7f09ca200700 1 --2- 172.21.15.116:0/3313515433 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09c4101f70 0x7f09c4197350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.844+0000 7f09ca200700 1 -- 172.21.15.116:0/3313515433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc009d00 con 0x7f09c4102910 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.844+0000 7f09ca200700 1 --2- 172.21.15.116:0/3313515433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c4102910 0x7f09c419e330 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f09bc007b60 tx=0x7f09bc008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.844+0000 7f09c2000700 1 -- 172.21.15.116:0/3313515433 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc00e040 con 0x7f09c4102910 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.844+0000 7f09c2000700 1 -- 172.21.15.116:0/3313515433 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09bc005070 con 0x7f09c4102910 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.844+0000 7f09cc200700 1 -- 172.21.15.116:0/3313515433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c419eeb0 con 0x7f09c4102910 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.845+0000 7f09c2000700 1 -- 172.21.15.116:0/3313515433 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc01d6d0 con 0x7f09c4102910 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.845+0000 7f09cc200700 1 -- 172.21.15.116:0/3313515433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c419f220 con 0x7f09c4102910 2024-03-30T00:57:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.846+0000 7f09cc200700 1 -- 172.21.15.116:0/3313515433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c4108130 con 0x7f09c4102910 2024-03-30T00:57:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.848+0000 7f09c2000700 1 -- 172.21.15.116:0/3313515433 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f09bc0051e0 con 0x7f09c4102910 2024-03-30T00:57:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.849+0000 7f09c2000700 1 --2- 172.21.15.116:0/3313515433 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f09ac0a11b0 0x7f09ac0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.849+0000 7f09c2000700 1 -- 172.21.15.116:0/3313515433 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f09bc0ce290 con 0x7f09c4102910 2024-03-30T00:57:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.849+0000 7f09c9800700 1 --2- 172.21.15.116:0/3313515433 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f09ac0a11b0 0x7f09ac0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.850+0000 7f09c9800700 1 --2- 172.21.15.116:0/3313515433 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f09ac0a11b0 0x7f09ac0a3670 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f09c4198290 tx=0x7f09b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:31.851+0000 7f09c2000700 1 -- 172.21.15.116:0/3313515433 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f09bc0975e0 con 0x7f09c4102910 2024-03-30T00:57:32.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.057+0000 7f09cc200700 1 -- 172.21.15.116:0/3313515433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f09c410c700 con 0x7f09c4102910 2024-03-30T00:57:32.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.057+0000 7f09c2000700 1 -- 172.21.15.116:0/3313515433 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f09bc09ac00 con 0x7f09c4102910 2024-03-30T00:57:32.065 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-30T00:57:32.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.060+0000 7f09cc200700 1 -- 172.21.15.116:0/3313515433 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f09ac0a11b0 msgr2=0x7f09ac0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.060+0000 7f09cc200700 1 --2- 172.21.15.116:0/3313515433 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f09ac0a11b0 0x7f09ac0a3670 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f09c4198290 tx=0x7f09b8009260 comp rx=0 tx=0).stop 2024-03-30T00:57:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.060+0000 7f09cc200700 1 -- 172.21.15.116:0/3313515433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c4102910 msgr2=0x7f09c419e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.060+0000 7f09cc200700 1 --2- 172.21.15.116:0/3313515433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c4102910 0x7f09c419e330 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f09bc007b60 tx=0x7f09bc008e40 comp rx=0 tx=0).stop 2024-03-30T00:57:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.060+0000 7f09cc200700 1 -- 172.21.15.116:0/3313515433 shutdown_connections 2024-03-30T00:57:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.060+0000 7f09cc200700 1 --2- 172.21.15.116:0/3313515433 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f09ac0a11b0 0x7f09ac0a3670 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.060+0000 7f09cc200700 1 --2- 172.21.15.116:0/3313515433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09c4100d80 0x7f09c4196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.060+0000 7f09cc200700 1 --2- 172.21.15.116:0/3313515433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c4102910 0x7f09c419e330 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.060+0000 7f09cc200700 1 --2- 172.21.15.116:0/3313515433 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09c4101f70 0x7f09c4197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.060+0000 7f09cc200700 1 -- 172.21.15.116:0/3313515433 >> 172.21.15.116:0/3313515433 conn(0x7f09c40fc2f0 msgr2=0x7f09c4105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.060+0000 7f09cc200700 1 -- 172.21.15.116:0/3313515433 shutdown_connections 2024-03-30T00:57:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.060+0000 7f09cc200700 1 -- 172.21.15.116:0/3313515433 wait complete. 2024-03-30T00:57:32.090 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T00:57:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.282+0000 7f0467000700 1 -- 172.21.15.116:0/1022012559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0460102910 msgr2=0x7f04601091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.282+0000 7f0467000700 1 --2- 172.21.15.116:0/1022012559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0460102910 0x7f04601091c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f045800b0d0 tx=0x7f045800b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.282+0000 7f0467000700 1 -- 172.21.15.116:0/1022012559 shutdown_connections 2024-03-30T00:57:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.282+0000 7f0467000700 1 --2- 172.21.15.116:0/1022012559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0460100d80 0x7f04601011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.282+0000 7f0467000700 1 --2- 172.21.15.116:0/1022012559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0460102910 0x7f04601091c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.282+0000 7f0467000700 1 --2- 172.21.15.116:0/1022012559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0460101f70 0x7f04601023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.282+0000 7f0467000700 1 -- 172.21.15.116:0/1022012559 >> 172.21.15.116:0/1022012559 conn(0x7f04600fc2f0 msgr2=0x7f04600fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.283+0000 7f0467000700 1 -- 172.21.15.116:0/1022012559 shutdown_connections 2024-03-30T00:57:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.283+0000 7f0467000700 1 -- 172.21.15.116:0/1022012559 wait complete. 2024-03-30T00:57:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.284+0000 7f0467000700 1 Processor -- start 2024-03-30T00:57:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.284+0000 7f0467000700 1 -- start start 2024-03-30T00:57:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.284+0000 7f0467000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0460100d80 0x7f0460198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.284+0000 7f0467000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0460101f70 0x7f0460199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.284+0000 7f0467000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0460102910 0x7f04601a0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.284+0000 7f0467000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04601a0b20 con 0x7f0460101f70 2024-03-30T00:57:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.284+0000 7f0467000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04601a0c60 con 0x7f0460102910 2024-03-30T00:57:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.284+0000 7f0467000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04601a0df0 con 0x7f0460100d80 2024-03-30T00:57:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.284+0000 7f045fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0460100d80 0x7f0460198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.284+0000 7f0457400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0460101f70 0x7f0460199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.284+0000 7f045fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0460100d80 0x7f0460198ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:51188/0 (socket says 172.21.15.116:51188) 2024-03-30T00:57:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.284+0000 7f045fe00700 1 -- 172.21.15.116:0/327574568 learned_addr learned my addr 172.21.15.116:0/327574568 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.284+0000 7f0465000700 1 --2- 172.21.15.116:0/327574568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0460102910 0x7f04601a0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.285+0000 7f045fe00700 1 -- 172.21.15.116:0/327574568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0460102910 msgr2=0x7f04601a0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.285+0000 7f045fe00700 1 --2- 172.21.15.116:0/327574568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0460102910 0x7f04601a0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.285+0000 7f045fe00700 1 -- 172.21.15.116:0/327574568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0460101f70 msgr2=0x7f0460199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.285+0000 7f045fe00700 1 --2- 172.21.15.116:0/327574568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0460101f70 0x7f0460199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.285+0000 7f045fe00700 1 -- 172.21.15.116:0/327574568 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0458009d00 con 0x7f0460100d80 2024-03-30T00:57:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.285+0000 7f045fe00700 1 --2- 172.21.15.116:0/327574568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0460100d80 0x7f0460198ff0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f045000e970 tx=0x7f045000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.285+0000 7f045d600700 1 -- 172.21.15.116:0/327574568 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450009790 con 0x7f0460100d80 2024-03-30T00:57:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.285+0000 7f0467000700 1 -- 172.21.15.116:0/327574568 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04601a10d0 con 0x7f0460100d80 2024-03-30T00:57:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.285+0000 7f045d600700 1 -- 172.21.15.116:0/327574568 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f045000eec0 con 0x7f0460100d80 2024-03-30T00:57:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.285+0000 7f045d600700 1 -- 172.21.15.116:0/327574568 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450003cf0 con 0x7f0460100d80 2024-03-30T00:57:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.285+0000 7f0467000700 1 -- 172.21.15.116:0/327574568 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04601a1630 con 0x7f0460100d80 2024-03-30T00:57:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.286+0000 7f0467000700 1 -- 172.21.15.116:0/327574568 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0460067af0 con 0x7f0460100d80 2024-03-30T00:57:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.288+0000 7f045d600700 1 -- 172.21.15.116:0/327574568 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f045000ccd0 con 0x7f0460100d80 2024-03-30T00:57:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.289+0000 7f045d600700 1 --2- 172.21.15.116:0/327574568 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f04440a11b0 0x7f04440a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.290+0000 7f045d600700 1 -- 172.21.15.116:0/327574568 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0450014070 con 0x7f0460100d80 2024-03-30T00:57:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.290+0000 7f0457400700 1 --2- 172.21.15.116:0/327574568 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f04440a11b0 0x7f04440a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.290+0000 7f0457400700 1 --2- 172.21.15.116:0/327574568 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f04440a11b0 0x7f04440a3670 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0448009c60 tx=0x7f0448009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.292+0000 7f045d600700 1 -- 172.21.15.116:0/327574568 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0450096450 con 0x7f0460100d80 2024-03-30T00:57:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.500+0000 7f0467000700 1 -- 172.21.15.116:0/327574568 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f04601a19b0 con 0x7f0460100d80 2024-03-30T00:57:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.501+0000 7f045d600700 1 -- 172.21.15.116:0/327574568 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v83) v1 ==== 103+0+11 (secure 0 0 0) 0x7f045009ee60 con 0x7f0460100d80 2024-03-30T00:57:32.514 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:57:32.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.504+0000 7f0467000700 1 -- 172.21.15.116:0/327574568 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f04440a11b0 msgr2=0x7f04440a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:32.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.504+0000 7f0467000700 1 --2- 172.21.15.116:0/327574568 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f04440a11b0 0x7f04440a3670 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0448009c60 tx=0x7f0448009260 comp rx=0 tx=0).stop 2024-03-30T00:57:32.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.504+0000 7f0467000700 1 -- 172.21.15.116:0/327574568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0460100d80 msgr2=0x7f0460198ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:32.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.504+0000 7f0467000700 1 --2- 172.21.15.116:0/327574568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0460100d80 0x7f0460198ff0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f045000e970 tx=0x7f045000ec80 comp rx=0 tx=0).stop 2024-03-30T00:57:32.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.504+0000 7f0467000700 1 -- 172.21.15.116:0/327574568 shutdown_connections 2024-03-30T00:57:32.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.504+0000 7f0467000700 1 --2- 172.21.15.116:0/327574568 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f04440a11b0 0x7f04440a3670 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.504+0000 7f0467000700 1 --2- 172.21.15.116:0/327574568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0460100d80 0x7f0460198ff0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.504+0000 7f0467000700 1 --2- 172.21.15.116:0/327574568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0460102910 0x7f04601a0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.504+0000 7f0467000700 1 --2- 172.21.15.116:0/327574568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0460101f70 0x7f0460199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.504+0000 7f0467000700 1 -- 172.21.15.116:0/327574568 >> 172.21.15.116:0/327574568 conn(0x7f04600fc2f0 msgr2=0x7f0460072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:32.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.504+0000 7f0467000700 1 -- 172.21.15.116:0/327574568 shutdown_connections 2024-03-30T00:57:32.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.504+0000 7f0467000700 1 -- 172.21.15.116:0/327574568 wait complete. 2024-03-30T00:57:32.530 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:57:32.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.716+0000 7f916c600700 1 -- 172.21.15.116:0/4151228365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9164101e70 msgr2=0x7f91641022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:32.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.716+0000 7f916c600700 1 --2- 172.21.15.116:0/4151228365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9164101e70 0x7f91641022d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f91500099c0 tx=0x7f9150009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.716+0000 7f916c600700 1 -- 172.21.15.116:0/4151228365 shutdown_connections 2024-03-30T00:57:32.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.716+0000 7f916c600700 1 --2- 172.21.15.116:0/4151228365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9164102810 0x7f91641090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.716+0000 7f916c600700 1 --2- 172.21.15.116:0/4151228365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9164100c80 0x7f91641010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.716+0000 7f916c600700 1 --2- 172.21.15.116:0/4151228365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9164101e70 0x7f91641022d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.716+0000 7f916c600700 1 -- 172.21.15.116:0/4151228365 >> 172.21.15.116:0/4151228365 conn(0x7f91640fc1f0 msgr2=0x7f91640fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:32.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.716+0000 7f916c600700 1 -- 172.21.15.116:0/4151228365 shutdown_connections 2024-03-30T00:57:32.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.716+0000 7f916c600700 1 -- 172.21.15.116:0/4151228365 wait complete. 2024-03-30T00:57:32.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.717+0000 7f916c600700 1 Processor -- start 2024-03-30T00:57:32.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.717+0000 7f916c600700 1 -- start start 2024-03-30T00:57:32.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.717+0000 7f916c600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9164100c80 0x7f9164196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:32.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.717+0000 7f916c600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9164101e70 0x7f9164197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:32.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.717+0000 7f916c600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9164102810 0x7f916419e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:32.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.717+0000 7f916c600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916419e720 con 0x7f9164101e70 2024-03-30T00:57:32.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.717+0000 7f916c600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916419e860 con 0x7f9164100c80 2024-03-30T00:57:32.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.717+0000 7f916c600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916419e9f0 con 0x7f9164102810 2024-03-30T00:57:32.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.717+0000 7f9169200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9164101e70 0x7f9164197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.717+0000 7f9169200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9164101e70 0x7f9164197130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42748/0 (socket says 172.21.15.116:42748) 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.718+0000 7f9169200700 1 -- 172.21.15.116:0/2472399738 learned_addr learned my addr 172.21.15.116:0/2472399738 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.718+0000 7f9169c00700 1 --2- 172.21.15.116:0/2472399738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9164100c80 0x7f9164196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.718+0000 7f916a600700 1 --2- 172.21.15.116:0/2472399738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9164102810 0x7f916419e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.718+0000 7f9169200700 1 -- 172.21.15.116:0/2472399738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9164102810 msgr2=0x7f916419e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.718+0000 7f9169200700 1 --2- 172.21.15.116:0/2472399738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9164102810 0x7f916419e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.718+0000 7f9169200700 1 -- 172.21.15.116:0/2472399738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9164100c80 msgr2=0x7f9164196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.718+0000 7f9169200700 1 --2- 172.21.15.116:0/2472399738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9164100c80 0x7f9164196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.718+0000 7f9169200700 1 -- 172.21.15.116:0/2472399738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91500095f0 con 0x7f9164101e70 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.718+0000 7f9169200700 1 --2- 172.21.15.116:0/2472399738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9164101e70 0x7f9164197130 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9150006010 tx=0x7f915000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.718+0000 7f9162000700 1 -- 172.21.15.116:0/2472399738 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915001d070 con 0x7f9164101e70 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.718+0000 7f9162000700 1 -- 172.21.15.116:0/2472399738 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9150005030 con 0x7f9164101e70 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.718+0000 7f9162000700 1 -- 172.21.15.116:0/2472399738 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9150004b10 con 0x7f9164101e70 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.718+0000 7f916c600700 1 -- 172.21.15.116:0/2472399738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f916419ec70 con 0x7f9164101e70 2024-03-30T00:57:32.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.719+0000 7f916c600700 1 -- 172.21.15.116:0/2472399738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916419f040 con 0x7f9164101e70 2024-03-30T00:57:32.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.719+0000 7f916c600700 1 -- 172.21.15.116:0/2472399738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9164108100 con 0x7f9164101e70 2024-03-30T00:57:32.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.721+0000 7f9162000700 1 -- 172.21.15.116:0/2472399738 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f91500051a0 con 0x7f9164101e70 2024-03-30T00:57:32.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.723+0000 7f9162000700 1 --2- 172.21.15.116:0/2472399738 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f914c0a1230 0x7f914c0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:32.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.724+0000 7f9162000700 1 -- 172.21.15.116:0/2472399738 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f91500cde10 con 0x7f9164101e70 2024-03-30T00:57:32.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.724+0000 7f9169c00700 1 --2- 172.21.15.116:0/2472399738 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f914c0a1230 0x7f914c0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:32.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.724+0000 7f9169c00700 1 --2- 172.21.15.116:0/2472399738 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f914c0a1230 0x7f914c0a36f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9164101cd0 tx=0x7f9158009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:32.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.726+0000 7f9162000700 1 -- 172.21.15.116:0/2472399738 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9150097280 con 0x7f9164101e70 2024-03-30T00:57:32.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:32 smithi181 ceph-mon[47622]: pgmap v281: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:32.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:32 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1028101858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:32.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:32 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3313515433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:32.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:32 smithi130 ceph-mon[47626]: pgmap v281: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:32.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:32 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1028101858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:32.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:32 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3313515433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:32 smithi116 ceph-mon[38086]: pgmap v281: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:32 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1028101858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:32 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3313515433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.975+0000 7f916c600700 1 -- 172.21.15.116:0/2472399738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9164067a10 con 0x7f9164101e70 2024-03-30T00:57:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.976+0000 7f9162000700 1 -- 172.21.15.116:0/2472399738 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+10886 (secure 0 0 0) 0x7f9150026090 con 0x7f9164101e70 2024-03-30T00:57:32.984 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:32.984 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:56:22.144079+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T00:57:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.979+0000 7f916c600700 1 -- 172.21.15.116:0/2472399738 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f914c0a1230 msgr2=0x7f914c0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.979+0000 7f916c600700 1 --2- 172.21.15.116:0/2472399738 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f914c0a1230 0x7f914c0a36f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9164101cd0 tx=0x7f9158009260 comp rx=0 tx=0).stop 2024-03-30T00:57:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.979+0000 7f916c600700 1 -- 172.21.15.116:0/2472399738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9164101e70 msgr2=0x7f9164197130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.979+0000 7f916c600700 1 --2- 172.21.15.116:0/2472399738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9164101e70 0x7f9164197130 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9150006010 tx=0x7f915000f690 comp rx=0 tx=0).stop 2024-03-30T00:57:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.979+0000 7f916c600700 1 -- 172.21.15.116:0/2472399738 shutdown_connections 2024-03-30T00:57:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.979+0000 7f916c600700 1 --2- 172.21.15.116:0/2472399738 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f914c0a1230 0x7f914c0a36f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.979+0000 7f916c600700 1 --2- 172.21.15.116:0/2472399738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9164102810 0x7f916419e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.979+0000 7f916c600700 1 --2- 172.21.15.116:0/2472399738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9164100c80 0x7f9164196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.979+0000 7f916c600700 1 --2- 172.21.15.116:0/2472399738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9164101e70 0x7f9164197130 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.979+0000 7f916c600700 1 -- 172.21.15.116:0/2472399738 >> 172.21.15.116:0/2472399738 conn(0x7f91640fc1f0 msgr2=0x7f9164105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.979+0000 7f916c600700 1 -- 172.21.15.116:0/2472399738 shutdown_connections 2024-03-30T00:57:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:32.979+0000 7f916c600700 1 -- 172.21.15.116:0/2472399738 wait complete. 2024-03-30T00:57:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 22 2024-03-30T00:57:33.005 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:57:33.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.196+0000 7ff53d400700 1 -- 172.21.15.116:0/4070158948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff538102810 msgr2=0x7ff5381090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.196+0000 7ff53d400700 1 --2- 172.21.15.116:0/4070158948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff538102810 0x7ff5381090c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff53000b0d0 tx=0x7ff53000b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.196+0000 7ff53d400700 1 -- 172.21.15.116:0/4070158948 shutdown_connections 2024-03-30T00:57:33.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.196+0000 7ff53d400700 1 --2- 172.21.15.116:0/4070158948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff538102810 0x7ff5381090c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.196+0000 7ff53d400700 1 --2- 172.21.15.116:0/4070158948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff538100c80 0x7ff5381010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.196+0000 7ff53d400700 1 --2- 172.21.15.116:0/4070158948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff538101e70 0x7ff5381022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.196+0000 7ff53d400700 1 -- 172.21.15.116:0/4070158948 >> 172.21.15.116:0/4070158948 conn(0x7ff5380fc1f0 msgr2=0x7ff5380fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:33.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.196+0000 7ff53d400700 1 -- 172.21.15.116:0/4070158948 shutdown_connections 2024-03-30T00:57:33.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.196+0000 7ff53d400700 1 -- 172.21.15.116:0/4070158948 wait complete. 2024-03-30T00:57:33.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.197+0000 7ff53d400700 1 Processor -- start 2024-03-30T00:57:33.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.197+0000 7ff53d400700 1 -- start start 2024-03-30T00:57:33.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.197+0000 7ff53d400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff538100c80 0x7ff538074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.197+0000 7ff53d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff538101e70 0x7ff5380731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.197+0000 7ff53d400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff538102810 0x7ff5380736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.197+0000 7ff53d400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff538073e80 con 0x7ff538101e70 2024-03-30T00:57:33.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.197+0000 7ff53d400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff538073ff0 con 0x7ff538100c80 2024-03-30T00:57:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.197+0000 7ff53d400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5381a0d70 con 0x7ff538102810 2024-03-30T00:57:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.198+0000 7ff535e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff538101e70 0x7ff5380731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.198+0000 7ff536800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff538100c80 0x7ff538074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.198+0000 7ff536800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff538100c80 0x7ff538074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34528/0 (socket says 172.21.15.116:34528) 2024-03-30T00:57:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.198+0000 7ff536800700 1 -- 172.21.15.116:0/523330850 learned_addr learned my addr 172.21.15.116:0/523330850 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.198+0000 7ff537200700 1 --2- 172.21.15.116:0/523330850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff538102810 0x7ff5380736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.198+0000 7ff535e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff538101e70 0x7ff5380731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42762/0 (socket says 172.21.15.116:42762) 2024-03-30T00:57:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.198+0000 7ff536800700 1 -- 172.21.15.116:0/523330850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff538102810 msgr2=0x7ff5380736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.198+0000 7ff536800700 1 --2- 172.21.15.116:0/523330850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff538102810 0x7ff5380736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.198+0000 7ff536800700 1 -- 172.21.15.116:0/523330850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff538101e70 msgr2=0x7ff5380731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.198+0000 7ff536800700 1 --2- 172.21.15.116:0/523330850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff538101e70 0x7ff5380731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.198+0000 7ff536800700 1 -- 172.21.15.116:0/523330850 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff530009d00 con 0x7ff538100c80 2024-03-30T00:57:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.198+0000 7ff537200700 1 --2- 172.21.15.116:0/523330850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff538102810 0x7ff5380736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.198+0000 7ff536800700 1 --2- 172.21.15.116:0/523330850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff538100c80 0x7ff538074b10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff52800e970 tx=0x7ff52800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.199+0000 7ff52f400700 1 -- 172.21.15.116:0/523330850 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528009790 con 0x7ff538100c80 2024-03-30T00:57:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.199+0000 7ff53d400700 1 -- 172.21.15.116:0/523330850 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5381a0fa0 con 0x7ff538100c80 2024-03-30T00:57:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.199+0000 7ff52f400700 1 -- 172.21.15.116:0/523330850 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff52800c780 con 0x7ff538100c80 2024-03-30T00:57:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.199+0000 7ff52f400700 1 -- 172.21.15.116:0/523330850 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528003be0 con 0x7ff538100c80 2024-03-30T00:57:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.199+0000 7ff53d400700 1 -- 172.21.15.116:0/523330850 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5381a1470 con 0x7ff538100c80 2024-03-30T00:57:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.200+0000 7ff53d400700 1 -- 172.21.15.116:0/523330850 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5381080d0 con 0x7ff538100c80 2024-03-30T00:57:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.202+0000 7ff52f400700 1 -- 172.21.15.116:0/523330850 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7ff528003d60 con 0x7ff538100c80 2024-03-30T00:57:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.203+0000 7ff52f400700 1 --2- 172.21.15.116:0/523330850 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff51c0a11b0 0x7ff51c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.203+0000 7ff52f400700 1 -- 172.21.15.116:0/523330850 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff528014070 con 0x7ff538100c80 2024-03-30T00:57:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.203+0000 7ff535e00700 1 --2- 172.21.15.116:0/523330850 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff51c0a11b0 0x7ff51c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.204+0000 7ff535e00700 1 --2- 172.21.15.116:0/523330850 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff51c0a11b0 0x7ff51c0a3670 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff524009c60 tx=0x7ff524009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.205+0000 7ff52f400700 1 -- 172.21.15.116:0/523330850 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff528096380 con 0x7ff538100c80 2024-03-30T00:57:33.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.441+0000 7ff53d400700 1 -- 172.21.15.116:0/523330850 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff53810c4f0 con 0x7ff538100c80 2024-03-30T00:57:33.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.443+0000 7ff52f400700 1 -- 172.21.15.116:0/523330850 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+24556 (secure 0 0 0) 0x7ff528080000 con 0x7ff538100c80 2024-03-30T00:57:33.450 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:33.450 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:56:36.456069+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:57:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.446+0000 7ff53d400700 1 -- 172.21.15.116:0/523330850 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff51c0a11b0 msgr2=0x7ff51c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.446+0000 7ff53d400700 1 --2- 172.21.15.116:0/523330850 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff51c0a11b0 0x7ff51c0a3670 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff524009c60 tx=0x7ff524009260 comp rx=0 tx=0).stop 2024-03-30T00:57:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.446+0000 7ff53d400700 1 -- 172.21.15.116:0/523330850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff538100c80 msgr2=0x7ff538074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.446+0000 7ff53d400700 1 --2- 172.21.15.116:0/523330850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff538100c80 0x7ff538074b10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff52800e970 tx=0x7ff52800ec80 comp rx=0 tx=0).stop 2024-03-30T00:57:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.446+0000 7ff53d400700 1 -- 172.21.15.116:0/523330850 shutdown_connections 2024-03-30T00:57:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.446+0000 7ff53d400700 1 --2- 172.21.15.116:0/523330850 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff51c0a11b0 0x7ff51c0a3670 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.446+0000 7ff53d400700 1 --2- 172.21.15.116:0/523330850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff538102810 0x7ff5380736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.446+0000 7ff53d400700 1 --2- 172.21.15.116:0/523330850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff538100c80 0x7ff538074b10 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.446+0000 7ff53d400700 1 --2- 172.21.15.116:0/523330850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff538101e70 0x7ff5380731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.446+0000 7ff53d400700 1 -- 172.21.15.116:0/523330850 >> 172.21.15.116:0/523330850 conn(0x7ff5380fc1f0 msgr2=0x7ff538105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.446+0000 7ff53d400700 1 -- 172.21.15.116:0/523330850 shutdown_connections 2024-03-30T00:57:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.446+0000 7ff53d400700 1 -- 172.21.15.116:0/523330850 wait complete. 2024-03-30T00:57:33.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-30T00:55:51.790326+0000', 'modified': '2024-03-30T00:56:22.144079+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24587, 'mds_1': 24548, 'mds_2': 24541}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24511': {'gid': 24511, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6835/702133015', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6834', 'nonce': 702133015}, {'type': 'v1', 'addr': '172.21.15.181:6835', 'nonce': 702133015}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24517': {'gid': 24517, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6837/941347789', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6836', 'nonce': 941347789}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 941347789}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24524': {'gid': 24524, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6835/3346365474', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6834', 'nonce': 3346365474}, {'type': 'v1', 'addr': '172.21.15.130:6835', 'nonce': 3346365474}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24541': {'gid': 24541, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.181:6839/3114781386', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6838', 'nonce': 3114781386}, {'type': 'v1', 'addr': '172.21.15.181:6839', 'nonce': 3114781386}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24548': {'gid': 24548, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.130:6837/3389690261', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6836', 'nonce': 3389690261}, {'type': 'v1', 'addr': '172.21.15.130:6837', 'nonce': 3389690261}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.130:6839/1184545085', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 1184545085}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 1184545085}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-30T00:57:33.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-30T00:57:33.472 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-30T00:57:33.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.661+0000 7fdb61e00700 1 -- 172.21.15.116:0/1593940145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb5c100d80 msgr2=0x7fdb5c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.661+0000 7fdb61e00700 1 --2- 172.21.15.116:0/1593940145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb5c100d80 0x7fdb5c1011a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c0099c0 tx=0x7fdb4c009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.661+0000 7fdb61e00700 1 -- 172.21.15.116:0/1593940145 shutdown_connections 2024-03-30T00:57:33.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.661+0000 7fdb61e00700 1 --2- 172.21.15.116:0/1593940145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb5c102910 0x7fdb5c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.661+0000 7fdb61e00700 1 --2- 172.21.15.116:0/1593940145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb5c100d80 0x7fdb5c1011a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.661+0000 7fdb61e00700 1 --2- 172.21.15.116:0/1593940145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb5c101f70 0x7fdb5c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.661+0000 7fdb61e00700 1 -- 172.21.15.116:0/1593940145 >> 172.21.15.116:0/1593940145 conn(0x7fdb5c0fc2f0 msgr2=0x7fdb5c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:33.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.661+0000 7fdb61e00700 1 -- 172.21.15.116:0/1593940145 shutdown_connections 2024-03-30T00:57:33.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.661+0000 7fdb61e00700 1 -- 172.21.15.116:0/1593940145 wait complete. 2024-03-30T00:57:33.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.662+0000 7fdb61e00700 1 Processor -- start 2024-03-30T00:57:33.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.662+0000 7fdb61e00700 1 -- start start 2024-03-30T00:57:33.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.662+0000 7fdb61e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb5c100d80 0x7fdb5c196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.662+0000 7fdb61e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb5c101f70 0x7fdb5c1972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.662+0000 7fdb61e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb5c102910 0x7fdb5c19e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.662+0000 7fdb61e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c19e860 con 0x7fdb5c102910 2024-03-30T00:57:33.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.662+0000 7fdb61e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c19e9d0 con 0x7fdb5c101f70 2024-03-30T00:57:33.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.662+0000 7fdb61e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c19eb90 con 0x7fdb5c100d80 2024-03-30T00:57:33.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.663+0000 7fdb5b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb5c102910 0x7fdb5c19e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.663+0000 7fdb5b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb5c102910 0x7fdb5c19e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42782/0 (socket says 172.21.15.116:42782) 2024-03-30T00:57:33.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.663+0000 7fdb5b400700 1 -- 172.21.15.116:0/2767442953 learned_addr learned my addr 172.21.15.116:0/2767442953 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:33.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.663+0000 7fdb5aa00700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb5c100d80 0x7fdb5c196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.663+0000 7fdb5a000700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb5c101f70 0x7fdb5c1972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.663+0000 7fdb5b400700 1 -- 172.21.15.116:0/2767442953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb5c100d80 msgr2=0x7fdb5c196d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.663+0000 7fdb5b400700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb5c100d80 0x7fdb5c196d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.663+0000 7fdb5b400700 1 -- 172.21.15.116:0/2767442953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb5c101f70 msgr2=0x7fdb5c1972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.663+0000 7fdb5b400700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb5c101f70 0x7fdb5c1972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.663+0000 7fdb5b400700 1 -- 172.21.15.116:0/2767442953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c0095f0 con 0x7fdb5c102910 2024-03-30T00:57:33.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.663+0000 7fdb5a000700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb5c101f70 0x7fdb5c1972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:33.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.663+0000 7fdb5b400700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb5c102910 0x7fdb5c19e290 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400ef30 tx=0x7fdb5400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:33.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.664+0000 7fdb60a00700 1 -- 172.21.15.116:0/2767442953 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb54018070 con 0x7fdb5c102910 2024-03-30T00:57:33.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.664+0000 7fdb60a00700 1 -- 172.21.15.116:0/2767442953 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb5400f040 con 0x7fdb5c102910 2024-03-30T00:57:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.664+0000 7fdb61e00700 1 -- 172.21.15.116:0/2767442953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c19ee10 con 0x7fdb5c102910 2024-03-30T00:57:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.664+0000 7fdb60a00700 1 -- 172.21.15.116:0/2767442953 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5400cdb0 con 0x7fdb5c102910 2024-03-30T00:57:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.664+0000 7fdb61e00700 1 -- 172.21.15.116:0/2767442953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c19f230 con 0x7fdb5c102910 2024-03-30T00:57:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.667+0000 7fdb60a00700 1 -- 172.21.15.116:0/2767442953 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7fdb54003a00 con 0x7fdb5c102910 2024-03-30T00:57:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.668+0000 7fdb60a00700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdb400a1230 0x7fdb400a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.668+0000 7fdb75897b80 1 Processor -- start 2024-03-30T00:57:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.668+0000 7fdb75897b80 1 -- start start 2024-03-30T00:57:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.668+0000 7fdb60a00700 1 -- 172.21.15.116:0/2767442953 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdb540cc990 con 0x7fdb5c102910 2024-03-30T00:57:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.668+0000 7fdb5aa00700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdb400a1230 0x7fdb400a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb75897b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c0b90 0x5574887c0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb75897b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c14f0 0x5574887c1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb75897b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c2390 0x5574887c2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb75897b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557488714940 con 0x5574887c0b90 2024-03-30T00:57:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb75897b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557488712300 con 0x5574887c14f0 2024-03-30T00:57:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb75897b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55748870dbf0 con 0x5574887c2390 2024-03-30T00:57:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb5aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c0b90 0x5574887c0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb5a000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c14f0 0x5574887c1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb5aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c0b90 0x5574887c0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42796/0 (socket says 172.21.15.116:42796) 2024-03-30T00:57:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb5aa00700 1 -- 172.21.15.116:0/2448221185 learned_addr learned my addr 172.21.15.116:0/2448221185 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb5b400700 1 --2- 172.21.15.116:0/2448221185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c2390 0x5574887c2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb5aa00700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdb400a1230 0x7fdb400a36f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c003730 tx=0x7fdb4c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb5a000700 1 -- 172.21.15.116:0/2448221185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c2390 msgr2=0x5574887c2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb5a000700 1 --2- 172.21.15.116:0/2448221185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c2390 0x5574887c2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb5a000700 1 -- 172.21.15.116:0/2448221185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c0b90 msgr2=0x5574887c0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb5a000700 1 --2- 172.21.15.116:0/2448221185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c0b90 0x5574887c0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb5a000700 1 -- 172.21.15.116:0/2448221185 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55748858f360 con 0x5574887c14f0 2024-03-30T00:57:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.669+0000 7fdb5aa00700 1 --2- 172.21.15.116:0/2448221185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c0b90 0x5574887c0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.670+0000 7fdb5a000700 1 --2- 172.21.15.116:0/2448221185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c14f0 0x5574887c1db0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009cc0 tx=0x7fdb480113e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bb80e48f2843ca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.670+0000 7fdb47e00700 1 -- 172.21.15.116:0/2448221185 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb48011c30 con 0x5574887c14f0 2024-03-30T00:57:33.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.670+0000 7fdb47e00700 1 -- 172.21.15.116:0/2448221185 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb48011d90 con 0x5574887c14f0 2024-03-30T00:57:33.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.670+0000 7fdb47e00700 1 -- 172.21.15.116:0/2448221185 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4800f7b0 con 0x5574887c14f0 2024-03-30T00:57:33.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.671+0000 7fdb75897b80 1 -- 172.21.15.116:0/2448221185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c14f0 msgr2=0x5574887c1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.671+0000 7fdb75897b80 1 --2- 172.21.15.116:0/2448221185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c14f0 0x5574887c1db0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009cc0 tx=0x7fdb480113e0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.671+0000 7fdb75897b80 1 -- 172.21.15.116:0/2448221185 shutdown_connections 2024-03-30T00:57:33.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.671+0000 7fdb75897b80 1 --2- 172.21.15.116:0/2448221185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c2390 0x5574887c2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.671+0000 7fdb75897b80 1 --2- 172.21.15.116:0/2448221185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c0b90 0x5574887c0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.671+0000 7fdb75897b80 1 --2- 172.21.15.116:0/2448221185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c14f0 0x5574887c1db0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.671+0000 7fdb75897b80 1 -- 172.21.15.116:0/2448221185 >> 172.21.15.116:0/2448221185 conn(0x55748871b650 msgr2=0x5574887b7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:33.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.671+0000 7fdb75897b80 1 -- 172.21.15.116:0/2448221185 shutdown_connections 2024-03-30T00:57:33.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.671+0000 7fdb75897b80 1 -- 172.21.15.116:0/2448221185 wait complete. 2024-03-30T00:57:33.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.671+0000 7fdb75897b80 1 -- start start 2024-03-30T00:57:33.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.671+0000 7fdb75897b80 1 Processor -- start 2024-03-30T00:57:33.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.672+0000 7fdb75897b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c0b90 0x5574887caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.672+0000 7fdb75897b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c14f0 0x5574887cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.672+0000 7fdb75897b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c2390 0x5574887d1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.672+0000 7fdb75897b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574886a9d30 con 0x5574887c14f0 2024-03-30T00:57:33.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.672+0000 7fdb75897b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574886a99d0 con 0x5574887c0b90 2024-03-30T00:57:33.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.672+0000 7fdb75897b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557488591650 con 0x5574887c2390 2024-03-30T00:57:33.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.672+0000 7fdb75897b80 10 client.0 _ll_register_callbacks cb 0x5574886b3420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:33.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.672+0000 7fdb5a000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c14f0 0x5574887cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.672+0000 7fdb5aa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c0b90 0x5574887caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.672+0000 7fdb5a000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c14f0 0x5574887cafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42806/0 (socket says 172.21.15.116:42806) 2024-03-30T00:57:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.672+0000 7fdb5aa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c0b90 0x5574887caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34576/0 (socket says 172.21.15.116:34576) 2024-03-30T00:57:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.672+0000 7fdb5a000700 1 -- 172.21.15.116:0/410794536 learned_addr learned my addr 172.21.15.116:0/410794536 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.672+0000 7fdb5b400700 1 --2- 172.21.15.116:0/410794536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c2390 0x5574887d1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.673+0000 7fdb5a000700 1 -- 172.21.15.116:0/410794536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c2390 msgr2=0x5574887d1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.673+0000 7fdb5a000700 1 --2- 172.21.15.116:0/410794536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c2390 0x5574887d1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.673+0000 7fdb5a000700 1 -- 172.21.15.116:0/410794536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c0b90 msgr2=0x5574887caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.673+0000 7fdb5a000700 1 --2- 172.21.15.116:0/410794536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c0b90 0x5574887caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.673+0000 7fdb5a000700 1 -- 172.21.15.116:0/410794536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4c020040 con 0x5574887c14f0 2024-03-30T00:57:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.673+0000 7fdb5aa00700 1 --2- 172.21.15.116:0/410794536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c0b90 0x5574887caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.673+0000 7fdb5a000700 1 --2- 172.21.15.116:0/410794536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c14f0 0x5574887cafc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdb48014fd0 tx=0x7fdb48005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec5a39c87de1762 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.673+0000 7fdb44c00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:57:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.673+0000 7fdb44c00700 1 -- 172.21.15.116:0/410794536 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb48004950 con 0x5574887c14f0 2024-03-30T00:57:33.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.673+0000 7fdb44c00700 1 -- 172.21.15.116:0/410794536 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdb48005670 con 0x5574887c14f0 2024-03-30T00:57:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.673+0000 7fdb75897b80 1 -- 172.21.15.116:0/410794536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5574886ce330 con 0x5574887c14f0 2024-03-30T00:57:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.673+0000 7fdb44c00700 1 -- 172.21.15.116:0/410794536 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb48004ab0 con 0x5574887c14f0 2024-03-30T00:57:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.674+0000 7fdb75897b80 10 client.15267 fetch_fsmap learned FSMap version 22 2024-03-30T00:57:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.674+0000 7fdb75897b80 1 -- 172.21.15.116:0/410794536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7fdb480098f0 con 0x5574887c14f0 2024-03-30T00:57:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.674+0000 7fdb44c00700 1 -- 172.21.15.116:0/410794536 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5574886ce330 con 0x5574887c14f0 2024-03-30T00:57:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.674+0000 7fdb44c00700 1 -- 172.21.15.116:0/410794536 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 22) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdb48012820 con 0x5574887c14f0 2024-03-30T00:57:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.674+0000 7fdb75897b80 10 client.15267 fetch_fsmap finished waiting for FSMap version 22 2024-03-30T00:57:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.674+0000 7fdb75897b80 10 client.15267 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-30T00:57:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.674+0000 7fdb75897b80 20 client.15267 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.674+0000 7fdb75897b80 1 --2- 172.21.15.116:0/410794536 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5574887d5a60 0x5574887dad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.674+0000 7fdb75897b80 4 client.15267 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:57:33.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.674+0000 7fdb75897b80 1 -- 172.21.15.116:0/410794536 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5574887066b0 con 0x5574887d5a60 2024-03-30T00:57:33.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.675+0000 7fdb5aa00700 1 --2- 172.21.15.116:0/410794536 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5574887d5a60 0x5574887dad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.675+0000 7fdb44c00700 10 client.15267 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T00:57:33.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.675+0000 7fdb5aa00700 1 --2- 172.21.15.116:0/410794536 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5574887d5a60 0x5574887dad60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4a1ec4b52b8ecdf2 server_cookie=e2f4caa65e7940eb in_seq=0 out_seq=0 2024-03-30T00:57:33.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb44c00700 1 -- 172.21.15.116:0/410794536 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5574887066b0 con 0x5574887d5a60 2024-03-30T00:57:33.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb44c00700 10 client.15267 handle_command_reply: tid=0 2024-03-30T00:57:33.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb75897b80 1 client.15267 shutdown 2024-03-30T00:57:33.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb75897b80 1 -- 172.21.15.116:0/410794536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c14f0 msgr2=0x5574887cafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb75897b80 1 --2- 172.21.15.116:0/410794536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c14f0 0x5574887cafc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdb48014fd0 tx=0x7fdb48005270 comp rx=0 tx=0).stop 2024-03-30T00:57:33.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb75897b80 1 -- 172.21.15.116:0/410794536 shutdown_connections 2024-03-30T00:57:33.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb75897b80 1 --2- 172.21.15.116:0/410794536 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5574887d5a60 0x5574887dad60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb75897b80 1 --2- 172.21.15.116:0/410794536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c2390 0x5574887d1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb75897b80 1 --2- 172.21.15.116:0/410794536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c0b90 0x5574887caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb75897b80 1 --2- 172.21.15.116:0/410794536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c14f0 0x5574887cafc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb75897b80 1 -- 172.21.15.116:0/410794536 >> 172.21.15.116:0/410794536 conn(0x55748871b650 msgr2=0x5574887bb120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:33.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb44c00700 0 client.15267 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T00:57:33.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb75897b80 1 -- 172.21.15.116:0/410794536 shutdown_connections 2024-03-30T00:57:33.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb75897b80 1 -- 172.21.15.116:0/410794536 wait complete. 2024-03-30T00:57:33.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.676+0000 7fdb75897b80 20 client.15267 trim_cache size 0 max 16384 2024-03-30T00:57:33.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.694+0000 7fdb75897b80 1 Processor -- start 2024-03-30T00:57:33.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.694+0000 7fdb75897b80 1 -- start start 2024-03-30T00:57:33.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.694+0000 7fdb75897b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c0b90 0x5574887c0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.694+0000 7fdb75897b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c14f0 0x5574887d2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.694+0000 7fdb75897b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c2390 0x5574887c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.694+0000 7fdb75897b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574887069c0 con 0x5574887c2390 2024-03-30T00:57:33.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.694+0000 7fdb75897b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55748870a880 con 0x5574887c14f0 2024-03-30T00:57:33.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.694+0000 7fdb75897b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574887b8460 con 0x5574887c0b90 2024-03-30T00:57:33.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.695+0000 7fdb5b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c2390 0x5574887c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.695+0000 7fdb5b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c2390 0x5574887c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42822/0 (socket says 172.21.15.116:42822) 2024-03-30T00:57:33.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.695+0000 7fdb5aa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c0b90 0x5574887c0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.695+0000 7fdb5b400700 1 -- 172.21.15.116:0/470360195 learned_addr learned my addr 172.21.15.116:0/470360195 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:33.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.695+0000 7fdb5a000700 1 --2- 172.21.15.116:0/470360195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c14f0 0x5574887d2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.695+0000 7fdb5aa00700 1 -- 172.21.15.116:0/470360195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c14f0 msgr2=0x5574887d2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.695+0000 7fdb5aa00700 1 --2- 172.21.15.116:0/470360195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c14f0 0x5574887d2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.695+0000 7fdb5aa00700 1 -- 172.21.15.116:0/470360195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c2390 msgr2=0x5574887c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.695+0000 7fdb5aa00700 1 --2- 172.21.15.116:0/470360195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c2390 0x5574887c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.695+0000 7fdb5aa00700 1 -- 172.21.15.116:0/470360195 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55748853bb10 con 0x5574887c0b90 2024-03-30T00:57:33.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.695+0000 7fdb5b400700 1 --2- 172.21.15.116:0/470360195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c2390 0x5574887c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:33.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.695+0000 7fdb5aa00700 1 --2- 172.21.15.116:0/470360195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c0b90 0x5574887c0f70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c026000 tx=0x7fdb4c027000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c123820cffd6a93 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:33.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.696+0000 7fdb51200700 1 -- 172.21.15.116:0/470360195 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c037070 con 0x5574887c0b90 2024-03-30T00:57:33.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.696+0000 7fdb51200700 1 -- 172.21.15.116:0/470360195 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb4c01e050 con 0x5574887c0b90 2024-03-30T00:57:33.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.696+0000 7fdb51200700 1 -- 172.21.15.116:0/470360195 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c00f6c0 con 0x5574887c0b90 2024-03-30T00:57:33.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.696+0000 7fdb75897b80 1 -- 172.21.15.116:0/470360195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c0b90 msgr2=0x5574887c0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.696+0000 7fdb75897b80 1 --2- 172.21.15.116:0/470360195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c0b90 0x5574887c0f70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c026000 tx=0x7fdb4c027000 comp rx=0 tx=0).stop 2024-03-30T00:57:33.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.696+0000 7fdb75897b80 1 -- 172.21.15.116:0/470360195 shutdown_connections 2024-03-30T00:57:33.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.696+0000 7fdb75897b80 1 --2- 172.21.15.116:0/470360195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c0b90 0x5574887c0f70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.696+0000 7fdb75897b80 1 --2- 172.21.15.116:0/470360195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c2390 0x5574887c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.696+0000 7fdb75897b80 1 --2- 172.21.15.116:0/470360195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c14f0 0x5574887d2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.696+0000 7fdb75897b80 1 -- 172.21.15.116:0/470360195 >> 172.21.15.116:0/470360195 conn(0x55748871b650 msgr2=0x5574887c8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:33.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.696+0000 7fdb75897b80 1 -- 172.21.15.116:0/470360195 shutdown_connections 2024-03-30T00:57:33.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.696+0000 7fdb75897b80 1 -- 172.21.15.116:0/470360195 wait complete. 2024-03-30T00:57:33.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb75897b80 1 -- start start 2024-03-30T00:57:33.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb75897b80 1 Processor -- start 2024-03-30T00:57:33.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb75897b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c0b90 0x5574887ce120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb75897b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c14f0 0x5574887ce660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb75897b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c2390 0x5574887d1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb75897b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574887d1a80 con 0x5574887c2390 2024-03-30T00:57:33.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb75897b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574887d1bf0 con 0x5574887c0b90 2024-03-30T00:57:33.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb75897b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574887d1db0 con 0x5574887c14f0 2024-03-30T00:57:33.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb75897b80 10 client.0 _ll_register_callbacks cb 0x5574886b3420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:33.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb5a000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c14f0 0x5574887ce660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb5aa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c0b90 0x5574887ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb5b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c2390 0x5574887d1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb5a000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c14f0 0x5574887ce660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:51276/0 (socket says 172.21.15.116:51276) 2024-03-30T00:57:33.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb5aa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c0b90 0x5574887ce120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34600/0 (socket says 172.21.15.116:34600) 2024-03-30T00:57:33.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb5b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c2390 0x5574887d1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42838/0 (socket says 172.21.15.116:42838) 2024-03-30T00:57:33.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.697+0000 7fdb5a000700 1 -- 172.21.15.116:0/2179616504 learned_addr learned my addr 172.21.15.116:0/2179616504 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:33.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.698+0000 7fdb5a000700 1 -- 172.21.15.116:0/2179616504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c0b90 msgr2=0x5574887ce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.698+0000 7fdb5a000700 1 --2- 172.21.15.116:0/2179616504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c0b90 0x5574887ce120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.698+0000 7fdb5a000700 1 -- 172.21.15.116:0/2179616504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c2390 msgr2=0x5574887d1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.698+0000 7fdb5a000700 1 --2- 172.21.15.116:0/2179616504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c2390 0x5574887d1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.698+0000 7fdb5a000700 1 -- 172.21.15.116:0/2179616504 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c19f230 con 0x5574887c14f0 2024-03-30T00:57:33.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.698+0000 7fdb5b400700 1 --2- 172.21.15.116:0/2179616504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c2390 0x5574887d1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:33.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.698+0000 7fdb5a000700 1 --2- 172.21.15.116:0/2179616504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c14f0 0x5574887ce660 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdb48012be0 tx=0x7fdb48003b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dc545f70dd89c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:33.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.698+0000 7fdb44c00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T00:57:33.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.698+0000 7fdb44c00700 1 -- 172.21.15.116:0/2179616504 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4801a5f0 con 0x5574887c14f0 2024-03-30T00:57:33.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.698+0000 7fdb75897b80 1 -- 172.21.15.116:0/2179616504 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb48011d90 con 0x5574887c14f0 2024-03-30T00:57:33.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.698+0000 7fdb44c00700 1 -- 172.21.15.116:0/2179616504 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdb48009260 con 0x5574887c14f0 2024-03-30T00:57:33.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.699+0000 7fdb44c00700 1 -- 172.21.15.116:0/2179616504 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb48005940 con 0x5574887c14f0 2024-03-30T00:57:33.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.699+0000 7fdb75897b80 10 client.25321 fetch_fsmap learned FSMap version 22 2024-03-30T00:57:33.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.699+0000 7fdb75897b80 1 -- 172.21.15.116:0/2179616504 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7fdb4c02c040 con 0x5574887c14f0 2024-03-30T00:57:33.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.699+0000 7fdb44c00700 1 -- 172.21.15.116:0/2179616504 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb48011d90 con 0x5574887c14f0 2024-03-30T00:57:33.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.699+0000 7fdb44c00700 1 -- 172.21.15.116:0/2179616504 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 22) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdb4801a750 con 0x5574887c14f0 2024-03-30T00:57:33.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.699+0000 7fdb75897b80 10 client.25321 fetch_fsmap finished waiting for FSMap version 22 2024-03-30T00:57:33.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.699+0000 7fdb75897b80 10 client.25321 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-30T00:57:33.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.699+0000 7fdb75897b80 20 client.25321 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:33.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.699+0000 7fdb75897b80 1 --2- 172.21.15.116:0/2179616504 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55748882e500 0x55748882ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.699+0000 7fdb75897b80 4 client.25321 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T00:57:33.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.699+0000 7fdb75897b80 1 -- 172.21.15.116:0/2179616504 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55748855cca0 con 0x55748882e500 2024-03-30T00:57:33.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.699+0000 7fdb5aa00700 1 --2- 172.21.15.116:0/2179616504 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55748882e500 0x55748882ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.700+0000 7fdb5aa00700 1 --2- 172.21.15.116:0/2179616504 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55748882e500 0x55748882ae70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6f7602ec9b9d9870 server_cookie=b5c0c9cf31024c1b in_seq=0 out_seq=0 2024-03-30T00:57:33.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.700+0000 7fdb44c00700 10 client.25321 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T00:57:33.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.700+0000 7fdb44c00700 1 -- 172.21.15.116:0/2179616504 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55748855cca0 con 0x55748882e500 2024-03-30T00:57:33.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.700+0000 7fdb44c00700 10 client.25321 handle_command_reply: tid=0 2024-03-30T00:57:33.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.700+0000 7fdb75897b80 1 client.25321 shutdown 2024-03-30T00:57:33.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.700+0000 7fdb75897b80 1 -- 172.21.15.116:0/2179616504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c14f0 msgr2=0x5574887ce660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.700+0000 7fdb75897b80 1 --2- 172.21.15.116:0/2179616504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c14f0 0x5574887ce660 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdb48012be0 tx=0x7fdb48003b20 comp rx=0 tx=0).stop 2024-03-30T00:57:33.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.700+0000 7fdb75897b80 1 -- 172.21.15.116:0/2179616504 shutdown_connections 2024-03-30T00:57:33.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.700+0000 7fdb75897b80 1 --2- 172.21.15.116:0/2179616504 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55748882e500 0x55748882ae70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.700+0000 7fdb75897b80 1 --2- 172.21.15.116:0/2179616504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574887c2390 0x5574887d1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.700+0000 7fdb75897b80 1 --2- 172.21.15.116:0/2179616504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574887c0b90 0x5574887ce120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.700+0000 7fdb44c00700 0 client.25321 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T00:57:33.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.700+0000 7fdb75897b80 1 --2- 172.21.15.116:0/2179616504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574887c14f0 0x5574887ce660 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.700+0000 7fdb75897b80 1 -- 172.21.15.116:0/2179616504 >> 172.21.15.116:0/2179616504 conn(0x55748871b650 msgr2=0x5574887c8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:33.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.701+0000 7fdb75897b80 1 -- 172.21.15.116:0/2179616504 shutdown_connections 2024-03-30T00:57:33.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.701+0000 7fdb75897b80 1 -- 172.21.15.116:0/2179616504 wait complete. 2024-03-30T00:57:33.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.701+0000 7fdb75897b80 20 client.25321 trim_cache size 0 max 16384 2024-03-30T00:57:33.717 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:57:33.717 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "4909f004-ee2f-11ee-b647-cb9ed24678a4", 2024-03-30T00:57:33.717 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 2, 2024-03-30T00:57:33.717 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24541, 2024-03-30T00:57:33.718 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-30T00:57:33.718 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-30T00:57:33.718 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-30T00:57:33.718 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 72.300948118999997, 2024-03-30T00:57:33.718 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 22, 2024-03-30T00:57:33.718 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 83, 2024-03-30T00:57:33.718 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 81, 2024-03-30T00:57:33.718 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 113.122974821 2024-03-30T00:57:33.718 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:57:33.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.701+0000 7fdb61e00700 1 -- 172.21.15.116:0/2767442953 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdb400a1230 msgr2=0x7fdb400a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.701+0000 7fdb61e00700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdb400a1230 0x7fdb400a36f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c003730 tx=0x7fdb4c00b540 comp rx=0 tx=0).stop 2024-03-30T00:57:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.701+0000 7fdb61e00700 1 -- 172.21.15.116:0/2767442953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb5c102910 msgr2=0x7fdb5c19e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.701+0000 7fdb61e00700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb5c102910 0x7fdb5c19e290 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400ef30 tx=0x7fdb5400c3a0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.701+0000 7fdb61e00700 1 -- 172.21.15.116:0/2767442953 shutdown_connections 2024-03-30T00:57:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.701+0000 7fdb61e00700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fdb400a1230 0x7fdb400a36f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.701+0000 7fdb61e00700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdb5c100d80 0x7fdb5c196d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.701+0000 7fdb61e00700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb5c102910 0x7fdb5c19e290 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.701+0000 7fdb61e00700 1 --2- 172.21.15.116:0/2767442953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb5c101f70 0x7fdb5c1972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.701+0000 7fdb61e00700 1 -- 172.21.15.116:0/2767442953 >> 172.21.15.116:0/2767442953 conn(0x7fdb5c0fc2f0 msgr2=0x7fdb5c105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:33.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.701+0000 7fdb61e00700 1 -- 172.21.15.116:0/2767442953 shutdown_connections 2024-03-30T00:57:33.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.702+0000 7fdb61e00700 1 -- 172.21.15.116:0/2767442953 wait complete. 2024-03-30T00:57:33.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:33 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/327574568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:33.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:33 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2472399738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:33.729 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-30T00:57:33.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:33 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/327574568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:33.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:33 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2472399738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:33 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/327574568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:33 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2472399738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:33.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.917+0000 7f4434600700 1 -- 172.21.15.116:0/3399223344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442c0737d0 msgr2=0x7f442c073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.917+0000 7f4434600700 1 --2- 172.21.15.116:0/3399223344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442c0737d0 0x7f442c073c30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4418009990 tx=0x7f4418009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.917+0000 7f4434600700 1 -- 172.21.15.116:0/3399223344 shutdown_connections 2024-03-30T00:57:33.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.917+0000 7f4434600700 1 --2- 172.21.15.116:0/3399223344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f442c074db0 0x7f442c073200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.917+0000 7f4434600700 1 --2- 172.21.15.116:0/3399223344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f442c108c30 0x7f442c10b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.917+0000 7f4434600700 1 --2- 172.21.15.116:0/3399223344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442c0737d0 0x7f442c073c30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.917+0000 7f4434600700 1 -- 172.21.15.116:0/3399223344 >> 172.21.15.116:0/3399223344 conn(0x7f442c0fc2f0 msgr2=0x7f442c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:33.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.917+0000 7f4434600700 1 -- 172.21.15.116:0/3399223344 shutdown_connections 2024-03-30T00:57:33.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.917+0000 7f4434600700 1 -- 172.21.15.116:0/3399223344 wait complete. 2024-03-30T00:57:33.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.918+0000 7f4434600700 1 Processor -- start 2024-03-30T00:57:33.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.919+0000 7f4434600700 1 -- start start 2024-03-30T00:57:33.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.919+0000 7f4434600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442c0737d0 0x7f442c198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.919+0000 7f4434600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f442c074db0 0x7f442c199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.919+0000 7f4434600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f442c108c30 0x7f442c1a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.919+0000 7f4434600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c1a09f0 con 0x7f442c0737d0 2024-03-30T00:57:33.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.919+0000 7f4434600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c1a0b30 con 0x7f442c074db0 2024-03-30T00:57:33.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.919+0000 7f4434600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c1a0cc0 con 0x7f442c108c30 2024-03-30T00:57:33.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.919+0000 7f4431c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442c0737d0 0x7f442c198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.919+0000 7f4431c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442c0737d0 0x7f442c198ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42858/0 (socket says 172.21.15.116:42858) 2024-03-30T00:57:33.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.919+0000 7f4431c00700 1 -- 172.21.15.116:0/1550308045 learned_addr learned my addr 172.21.15.116:0/1550308045 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:33.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.919+0000 7f4431200700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f442c074db0 0x7f442c199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.919+0000 7f4432600700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f442c108c30 0x7f442c1a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.920+0000 7f4431c00700 1 -- 172.21.15.116:0/1550308045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f442c108c30 msgr2=0x7f442c1a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.920+0000 7f4431c00700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f442c108c30 0x7f442c1a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.920+0000 7f4431c00700 1 -- 172.21.15.116:0/1550308045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f442c074db0 msgr2=0x7f442c199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.920+0000 7f4431c00700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f442c074db0 0x7f442c199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.920+0000 7f4431c00700 1 -- 172.21.15.116:0/1550308045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44180095f0 con 0x7f442c0737d0 2024-03-30T00:57:33.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.920+0000 7f4431200700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f442c074db0 0x7f442c199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:33.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.920+0000 7f4431c00700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442c0737d0 0x7f442c198ec0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f442000e9a0 tx=0x7f442000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:33.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.920+0000 7f442a000700 1 -- 172.21.15.116:0/1550308045 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420009790 con 0x7f442c0737d0 2024-03-30T00:57:33.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.920+0000 7f442a000700 1 -- 172.21.15.116:0/1550308045 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4420010430 con 0x7f442c0737d0 2024-03-30T00:57:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.920+0000 7f4434600700 1 -- 172.21.15.116:0/1550308045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442c1a0fa0 con 0x7f442c0737d0 2024-03-30T00:57:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.920+0000 7f442a000700 1 -- 172.21.15.116:0/1550308045 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420018680 con 0x7f442c0737d0 2024-03-30T00:57:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.920+0000 7f4434600700 1 -- 172.21.15.116:0/1550308045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442c1a13c0 con 0x7f442c0737d0 2024-03-30T00:57:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.924+0000 7f442a000700 1 -- 172.21.15.116:0/1550308045 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f44200105a0 con 0x7f442c0737d0 2024-03-30T00:57:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.925+0000 7f4448031b80 1 Processor -- start 2024-03-30T00:57:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4448031b80 1 -- start start 2024-03-30T00:57:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f442a000700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f44140a11b0 0x7f44140a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4448031b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31616ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f442a000700 1 -- 172.21.15.116:0/1550308045 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4420014070 con 0x7f442c0737d0 2024-03-30T00:57:33.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4448031b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f316179a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4448031b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31617f80 0x563f31618400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4448031b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f3156e940 con 0x563f31617f80 2024-03-30T00:57:33.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4448031b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f3156c300 con 0x563f31616780 2024-03-30T00:57:33.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4448031b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f31567bf0 con 0x563f316170e0 2024-03-30T00:57:33.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4432600700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f44140a11b0 0x7f44140a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4432600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31617f80 0x563f31618400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4431c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31616ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4431200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f316179a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4432600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31617f80 0x563f31618400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42860/0 (socket says 172.21.15.116:42860) 2024-03-30T00:57:33.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4431c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31616ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34624/0 (socket says 172.21.15.116:34624) 2024-03-30T00:57:33.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4431200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f316179a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:51306/0 (socket says 172.21.15.116:51306) 2024-03-30T00:57:33.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.926+0000 7f4432600700 1 -- 172.21.15.116:0/1723413057 learned_addr learned my addr 172.21.15.116:0/1723413057 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:33.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.927+0000 7f4431c00700 1 -- 172.21.15.116:0/1723413057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 msgr2=0x563f316179a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.927+0000 7f4431c00700 1 --2- 172.21.15.116:0/1723413057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f316179a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.927+0000 7f4431c00700 1 -- 172.21.15.116:0/1723413057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31617f80 msgr2=0x563f31618400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.927+0000 7f4431c00700 1 --2- 172.21.15.116:0/1723413057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31617f80 0x563f31618400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.927+0000 7f4431c00700 1 -- 172.21.15.116:0/1723413057 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f313e9360 con 0x563f31616780 2024-03-30T00:57:33.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.927+0000 7f4431200700 1 --2- 172.21.15.116:0/1723413057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f316179a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:33.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.927+0000 7f4432600700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f44140a11b0 0x7f44140a3670 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4424009550 tx=0x7f4424009380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:33.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.927+0000 7f4431c00700 1 --2- 172.21.15.116:0/1723413057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31616ba0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f44200ca5d0 tx=0x7f4420099b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50eecd7c182f3375 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:33.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.927+0000 7f441ea00700 1 -- 172.21.15.116:0/1723413057 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420099f10 con 0x563f31616780 2024-03-30T00:57:33.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.927+0000 7f441ea00700 1 -- 172.21.15.116:0/1723413057 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f442009dd40 con 0x563f31616780 2024-03-30T00:57:33.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.927+0000 7f441ea00700 1 -- 172.21.15.116:0/1723413057 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44200cae20 con 0x563f31616780 2024-03-30T00:57:33.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.928+0000 7f4448031b80 1 -- 172.21.15.116:0/1723413057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 msgr2=0x563f31616ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.928+0000 7f4448031b80 1 --2- 172.21.15.116:0/1723413057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31616ba0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f44200ca5d0 tx=0x7f4420099b60 comp rx=0 tx=0).stop 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.928+0000 7f4448031b80 1 -- 172.21.15.116:0/1723413057 shutdown_connections 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.928+0000 7f4448031b80 1 --2- 172.21.15.116:0/1723413057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31617f80 0x563f31618400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.928+0000 7f4448031b80 1 --2- 172.21.15.116:0/1723413057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31616ba0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.928+0000 7f4448031b80 1 --2- 172.21.15.116:0/1723413057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f316179a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.928+0000 7f4448031b80 1 -- 172.21.15.116:0/1723413057 >> 172.21.15.116:0/1723413057 conn(0x563f31575650 msgr2=0x563f31611950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.928+0000 7f4448031b80 1 -- 172.21.15.116:0/1723413057 shutdown_connections 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.928+0000 7f4448031b80 1 -- 172.21.15.116:0/1723413057 wait complete. 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.929+0000 7f4448031b80 1 -- start start 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.929+0000 7f4448031b80 1 Processor -- start 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.929+0000 7f4448031b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31620630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.929+0000 7f4448031b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f31620b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.929+0000 7f4448031b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31617f80 0x563f31627bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.929+0000 7f4448031b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f31503d30 con 0x563f31617f80 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.929+0000 7f4448031b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f315039d0 con 0x563f31616780 2024-03-30T00:57:33.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.929+0000 7f4448031b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f313eb650 con 0x563f316170e0 2024-03-30T00:57:33.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.929+0000 7f4448031b80 10 client.0 _ll_register_callbacks cb 0x563f3150d420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:33.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.929+0000 7f4431c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31620630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.929+0000 7f4432600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31617f80 0x563f31627bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.929+0000 7f4431200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f31620b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.929+0000 7f4431c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31620630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34632/0 (socket says 172.21.15.116:34632) 2024-03-30T00:57:33.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.929+0000 7f4431c00700 1 -- 172.21.15.116:0/1880191370 learned_addr learned my addr 172.21.15.116:0/1880191370 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:33.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.930+0000 7f4431c00700 1 -- 172.21.15.116:0/1880191370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 msgr2=0x563f31620b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.930+0000 7f4431c00700 1 --2- 172.21.15.116:0/1880191370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f31620b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.930+0000 7f4431c00700 1 -- 172.21.15.116:0/1880191370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31617f80 msgr2=0x563f31627bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.930+0000 7f4431c00700 1 --2- 172.21.15.116:0/1880191370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31617f80 0x563f31627bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.930+0000 7f4431c00700 1 -- 172.21.15.116:0/1880191370 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442c10bce0 con 0x563f31616780 2024-03-30T00:57:33.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.930+0000 7f4431200700 1 --2- 172.21.15.116:0/1880191370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f31620b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:33.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.930+0000 7f4431c00700 1 --2- 172.21.15.116:0/1880191370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31620630 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f442000e970 tx=0x7f44200caa50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95677e9ca5b71388 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:33.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.930+0000 7f4413400700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T00:57:33.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.930+0000 7f4413400700 1 -- 172.21.15.116:0/1880191370 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44200a2390 con 0x563f31616780 2024-03-30T00:57:33.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.930+0000 7f4413400700 1 -- 172.21.15.116:0/1880191370 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f44200b82c0 con 0x563f31616780 2024-03-30T00:57:33.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.930+0000 7f4448031b80 1 -- 172.21.15.116:0/1880191370 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f31528330 con 0x563f31616780 2024-03-30T00:57:33.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.930+0000 7f4413400700 1 -- 172.21.15.116:0/1880191370 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4420099f10 con 0x563f31616780 2024-03-30T00:57:33.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.930+0000 7f4448031b80 10 client.25346 fetch_fsmap learned FSMap version 22 2024-03-30T00:57:33.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.930+0000 7f4448031b80 1 -- 172.21.15.116:0/1880191370 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7f442c1a13c0 con 0x563f31616780 2024-03-30T00:57:33.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.931+0000 7f4413400700 1 -- 172.21.15.116:0/1880191370 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563f31528330 con 0x563f31616780 2024-03-30T00:57:33.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.931+0000 7f4413400700 1 -- 172.21.15.116:0/1880191370 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 22) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4420098e70 con 0x563f31616780 2024-03-30T00:57:33.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.931+0000 7f4448031b80 10 client.25346 fetch_fsmap finished waiting for FSMap version 22 2024-03-30T00:57:33.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.931+0000 7f4448031b80 10 client.25346 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-30T00:57:33.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.931+0000 7f4448031b80 20 client.25346 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.931+0000 7f4448031b80 1 --2- 172.21.15.116:0/1880191370 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x563f3162b450 0x563f316306f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.931+0000 7f4448031b80 4 client.25346 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:57:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.931+0000 7f4448031b80 1 -- 172.21.15.116:0/1880191370 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f315606b0 con 0x563f3162b450 2024-03-30T00:57:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.931+0000 7f4431200700 1 --2- 172.21.15.116:0/1880191370 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x563f3162b450 0x563f316306f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.932+0000 7f4431200700 1 --2- 172.21.15.116:0/1880191370 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x563f3162b450 0x563f316306f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5ec33ec74d91a3f8 server_cookie=b5ebba2636684a6d in_seq=0 out_seq=0 2024-03-30T00:57:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.932+0000 7f4413400700 10 client.25346 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T00:57:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.932+0000 7f4413400700 1 -- 172.21.15.116:0/1880191370 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563f315606b0 con 0x563f3162b450 2024-03-30T00:57:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.932+0000 7f4413400700 10 client.25346 handle_command_reply: tid=0 2024-03-30T00:57:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.933+0000 7f4448031b80 1 client.25346 shutdown 2024-03-30T00:57:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.933+0000 7f4448031b80 1 -- 172.21.15.116:0/1880191370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 msgr2=0x563f31620630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.933+0000 7f4448031b80 1 --2- 172.21.15.116:0/1880191370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31620630 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f442000e970 tx=0x7f44200caa50 comp rx=0 tx=0).stop 2024-03-30T00:57:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.933+0000 7f4448031b80 1 -- 172.21.15.116:0/1880191370 shutdown_connections 2024-03-30T00:57:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.933+0000 7f4448031b80 1 --2- 172.21.15.116:0/1880191370 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x563f3162b450 0x563f316306f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.933+0000 7f4448031b80 1 --2- 172.21.15.116:0/1880191370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31617f80 0x563f31627bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.933+0000 7f4448031b80 1 --2- 172.21.15.116:0/1880191370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31620630 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.933+0000 7f4448031b80 1 --2- 172.21.15.116:0/1880191370 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f31620b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.933+0000 7f4413400700 0 client.25346 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T00:57:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.933+0000 7f4448031b80 1 -- 172.21.15.116:0/1880191370 >> 172.21.15.116:0/1880191370 conn(0x563f31575650 msgr2=0x563f31615120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.933+0000 7f4448031b80 1 -- 172.21.15.116:0/1880191370 shutdown_connections 2024-03-30T00:57:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.933+0000 7f4448031b80 1 -- 172.21.15.116:0/1880191370 wait complete. 2024-03-30T00:57:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.933+0000 7f4448031b80 20 client.25346 trim_cache size 0 max 16384 2024-03-30T00:57:33.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.951+0000 7f4448031b80 1 Processor -- start 2024-03-30T00:57:33.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.951+0000 7f4448031b80 1 -- start start 2024-03-30T00:57:33.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.951+0000 7f4448031b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31616b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.951+0000 7f4448031b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f316170e0 0x563f31627b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.951+0000 7f4448031b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f31617f80 0x563f31618360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.951+0000 7f4448031b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f3161f670 con 0x563f316170e0 2024-03-30T00:57:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.951+0000 7f4448031b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f316174c0 con 0x563f31616780 2024-03-30T00:57:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.951+0000 7f4448031b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f315606b0 con 0x563f31617f80 2024-03-30T00:57:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.952+0000 7f4431200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f316170e0 0x563f31627b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.952+0000 7f4431200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f316170e0 0x563f31627b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42870/0 (socket says 172.21.15.116:42870) 2024-03-30T00:57:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.952+0000 7f4431200700 1 -- 172.21.15.116:0/2971127375 learned_addr learned my addr 172.21.15.116:0/2971127375 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.952+0000 7f4431c00700 1 --2- 172.21.15.116:0/2971127375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31616b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.952+0000 7f4432600700 1 --2- 172.21.15.116:0/2971127375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f31617f80 0x563f31618360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.952+0000 7f4431200700 1 -- 172.21.15.116:0/2971127375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f31617f80 msgr2=0x563f31618360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.952+0000 7f4431200700 1 --2- 172.21.15.116:0/2971127375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f31617f80 0x563f31618360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.952+0000 7f4431200700 1 -- 172.21.15.116:0/2971127375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 msgr2=0x563f31616b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.952+0000 7f4431200700 1 --2- 172.21.15.116:0/2971127375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31616b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.952+0000 7f4431200700 1 -- 172.21.15.116:0/2971127375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f31395b10 con 0x563f316170e0 2024-03-30T00:57:33.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.952+0000 7f4431200700 1 --2- 172.21.15.116:0/2971127375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f316170e0 0x563f31627b70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f441800c010 tx=0x7f4418005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3796d40e405519c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:33.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.952+0000 7f441f400700 1 -- 172.21.15.116:0/2971127375 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441802b070 con 0x563f316170e0 2024-03-30T00:57:33.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.952+0000 7f441f400700 1 -- 172.21.15.116:0/2971127375 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f441801e040 con 0x563f316170e0 2024-03-30T00:57:33.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.953+0000 7f441f400700 1 -- 172.21.15.116:0/2971127375 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441801d040 con 0x563f316170e0 2024-03-30T00:57:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.953+0000 7f4448031b80 1 -- 172.21.15.116:0/2971127375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f316170e0 msgr2=0x563f31627b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.953+0000 7f4448031b80 1 --2- 172.21.15.116:0/2971127375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f316170e0 0x563f31627b70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f441800c010 tx=0x7f4418005fd0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.953+0000 7f4448031b80 1 -- 172.21.15.116:0/2971127375 shutdown_connections 2024-03-30T00:57:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.953+0000 7f4448031b80 1 --2- 172.21.15.116:0/2971127375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f31617f80 0x563f31618360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.953+0000 7f4448031b80 1 --2- 172.21.15.116:0/2971127375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31616780 0x563f31616b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.953+0000 7f4448031b80 1 --2- 172.21.15.116:0/2971127375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f316170e0 0x563f31627b70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.953+0000 7f4448031b80 1 -- 172.21.15.116:0/2971127375 >> 172.21.15.116:0/2971127375 conn(0x563f31575650 msgr2=0x563f3160a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.953+0000 7f4448031b80 1 -- 172.21.15.116:0/2971127375 shutdown_connections 2024-03-30T00:57:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.953+0000 7f4448031b80 1 -- 172.21.15.116:0/2971127375 wait complete. 2024-03-30T00:57:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.954+0000 7f4448031b80 1 -- start start 2024-03-30T00:57:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.954+0000 7f4448031b80 1 Processor -- start 2024-03-30T00:57:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.954+0000 7f4448031b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31616780 0x563f31627c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.954+0000 7f4448031b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f31624740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.954+0000 7f4448031b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31617f80 0x563f31624d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.954+0000 7f4448031b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f3167fce0 con 0x563f31616780 2024-03-30T00:57:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.954+0000 7f4448031b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f3167fe20 con 0x563f31617f80 2024-03-30T00:57:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.954+0000 7f4448031b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f3167ffe0 con 0x563f316170e0 2024-03-30T00:57:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.954+0000 7f4448031b80 10 client.0 _ll_register_callbacks cb 0x563f3150d420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.954+0000 7f4431c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31616780 0x563f31627c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.954+0000 7f4431c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31616780 0x563f31627c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42882/0 (socket says 172.21.15.116:42882) 2024-03-30T00:57:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.954+0000 7f4431c00700 1 -- 172.21.15.116:0/434956372 learned_addr learned my addr 172.21.15.116:0/434956372 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4431200700 1 --2- 172.21.15.116:0/434956372 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f31624740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4432600700 1 --2- 172.21.15.116:0/434956372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31617f80 0x563f31624d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4431c00700 1 -- 172.21.15.116:0/434956372 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 msgr2=0x563f31624740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4431c00700 1 --2- 172.21.15.116:0/434956372 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f31624740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4431c00700 1 -- 172.21.15.116:0/434956372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31617f80 msgr2=0x563f31624d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:33.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4431c00700 1 --2- 172.21.15.116:0/434956372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31617f80 0x563f31624d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4431c00700 1 -- 172.21.15.116:0/434956372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4418005280 con 0x563f31616780 2024-03-30T00:57:33.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4431c00700 1 --2- 172.21.15.116:0/434956372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31616780 0x563f31627c50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4420000c00 tx=0x7f4420063f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b2e69769b6e492 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:33.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4413400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:57:33.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4413400700 1 -- 172.21.15.116:0/434956372 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44200b82c0 con 0x563f31616780 2024-03-30T00:57:33.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4413400700 1 -- 172.21.15.116:0/434956372 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f44200cadf0 con 0x563f31616780 2024-03-30T00:57:33.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4448031b80 1 -- 172.21.15.116:0/434956372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f442009dd40 con 0x563f31616780 2024-03-30T00:57:33.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4413400700 1 -- 172.21.15.116:0/434956372 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44200a2390 con 0x563f31616780 2024-03-30T00:57:33.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4448031b80 10 client.15291 fetch_fsmap learned FSMap version 22 2024-03-30T00:57:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.955+0000 7f4448031b80 1 -- 172.21.15.116:0/434956372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x563f3152de50 con 0x563f31616780 2024-03-30T00:57:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.956+0000 7f4413400700 1 -- 172.21.15.116:0/434956372 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f442009dd40 con 0x563f31616780 2024-03-30T00:57:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.956+0000 7f4413400700 1 -- 172.21.15.116:0/434956372 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 22) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4420094b20 con 0x563f31616780 2024-03-30T00:57:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.956+0000 7f4448031b80 10 client.15291 fetch_fsmap finished waiting for FSMap version 22 2024-03-30T00:57:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.956+0000 7f4448031b80 10 client.15291 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-30T00:57:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.956+0000 7f4448031b80 20 client.15291 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.956+0000 7f4448031b80 1 --2- 172.21.15.116:0/434956372 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x563f31684950 0x563f31681290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.956+0000 7f4448031b80 4 client.15291 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T00:57:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.956+0000 7f4448031b80 1 -- 172.21.15.116:0/434956372 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563f313b6ca0 con 0x563f31684950 2024-03-30T00:57:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.956+0000 7f4431200700 1 --2- 172.21.15.116:0/434956372 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x563f31684950 0x563f31681290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4413400700 10 client.15291 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T00:57:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4431200700 1 --2- 172.21.15.116:0/434956372 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x563f31684950 0x563f31681290 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=955c7fdb022c6b0e server_cookie=23bcb453e4d38297 in_seq=0 out_seq=0 2024-03-30T00:57:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4413400700 1 -- 172.21.15.116:0/434956372 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x563f313b6ca0 con 0x563f31684950 2024-03-30T00:57:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4413400700 10 client.15291 handle_command_reply: tid=0 2024-03-30T00:57:33.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4448031b80 1 client.15291 shutdown 2024-03-30T00:57:33.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4448031b80 1 -- 172.21.15.116:0/434956372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31616780 msgr2=0x563f31627c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4448031b80 1 --2- 172.21.15.116:0/434956372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31616780 0x563f31627c50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4420000c00 tx=0x7f4420063f40 comp rx=0 tx=0).stop 2024-03-30T00:57:33.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4448031b80 1 -- 172.21.15.116:0/434956372 shutdown_connections 2024-03-30T00:57:33.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4448031b80 1 --2- 172.21.15.116:0/434956372 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x563f31684950 0x563f31681290 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4448031b80 1 --2- 172.21.15.116:0/434956372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f31617f80 0x563f31624d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4448031b80 1 --2- 172.21.15.116:0/434956372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f31616780 0x563f31627c50 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4413400700 0 client.15291 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T00:57:33.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4448031b80 1 --2- 172.21.15.116:0/434956372 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563f316170e0 0x563f31624740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4448031b80 1 -- 172.21.15.116:0/434956372 >> 172.21.15.116:0/434956372 conn(0x563f31575650 msgr2=0x563f3161faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:33.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4448031b80 1 -- 172.21.15.116:0/434956372 shutdown_connections 2024-03-30T00:57:33.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.957+0000 7f4448031b80 1 -- 172.21.15.116:0/434956372 wait complete. 2024-03-30T00:57:33.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.958+0000 7f4448031b80 20 client.15291 trim_cache size 0 max 16384 2024-03-30T00:57:33.974 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:57:33.974 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "4909f004-ee2f-11ee-b647-cb9ed24678a4", 2024-03-30T00:57:33.974 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 1, 2024-03-30T00:57:33.975 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24548, 2024-03-30T00:57:33.975 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-30T00:57:33.975 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-30T00:57:33.975 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-30T00:57:33.975 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 73.57345076, 2024-03-30T00:57:33.975 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 22, 2024-03-30T00:57:33.975 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 83, 2024-03-30T00:57:33.975 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 81, 2024-03-30T00:57:33.975 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 115.00276861499999 2024-03-30T00:57:33.975 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:57:33.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.958+0000 7f4434600700 1 -- 172.21.15.116:0/1550308045 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f44140a11b0 msgr2=0x7f44140a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.958+0000 7f4434600700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f44140a11b0 0x7f44140a3670 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4424009550 tx=0x7f4424009380 comp rx=0 tx=0).stop 2024-03-30T00:57:33.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.958+0000 7f4434600700 1 -- 172.21.15.116:0/1550308045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442c0737d0 msgr2=0x7f442c198ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:33.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.958+0000 7f4434600700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442c0737d0 0x7f442c198ec0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f442000e9a0 tx=0x7f442000ecb0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.958+0000 7f4434600700 1 -- 172.21.15.116:0/1550308045 shutdown_connections 2024-03-30T00:57:33.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.958+0000 7f4434600700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f44140a11b0 0x7f44140a3670 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.958+0000 7f4434600700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f442c108c30 0x7f442c1a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.958+0000 7f4434600700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442c0737d0 0x7f442c198ec0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.958+0000 7f4434600700 1 --2- 172.21.15.116:0/1550308045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f442c074db0 0x7f442c199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:33.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.958+0000 7f4434600700 1 -- 172.21.15.116:0/1550308045 >> 172.21.15.116:0/1550308045 conn(0x7f442c0fc2f0 msgr2=0x7f442c104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:33.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.958+0000 7f4434600700 1 -- 172.21.15.116:0/1550308045 shutdown_connections 2024-03-30T00:57:33.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:33.958+0000 7f4434600700 1 -- 172.21.15.116:0/1550308045 wait complete. 2024-03-30T00:57:33.983 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-30T00:57:34.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.169+0000 7f48ea000700 1 -- 172.21.15.116:0/1476222331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e40ff5e0 msgr2=0x7f48e40ffa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:34.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.169+0000 7f48ea000700 1 --2- 172.21.15.116:0/1476222331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e40ff5e0 0x7f48e40ffa00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009990 tx=0x7f48d4009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.169+0000 7f48ea000700 1 -- 172.21.15.116:0/1476222331 shutdown_connections 2024-03-30T00:57:34.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.169+0000 7f48ea000700 1 --2- 172.21.15.116:0/1476222331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e4100900 0x7f48e4100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.169+0000 7f48ea000700 1 --2- 172.21.15.116:0/1476222331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e40ff5e0 0x7f48e40ffa00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.169+0000 7f48ea000700 1 --2- 172.21.15.116:0/1476222331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48e40fff40 0x7f48e41003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.169+0000 7f48ea000700 1 -- 172.21.15.116:0/1476222331 >> 172.21.15.116:0/1476222331 conn(0x7f48e40fb200 msgr2=0x7f48e40fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:34.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.169+0000 7f48ea000700 1 -- 172.21.15.116:0/1476222331 shutdown_connections 2024-03-30T00:57:34.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.169+0000 7f48ea000700 1 -- 172.21.15.116:0/1476222331 wait complete. 2024-03-30T00:57:34.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.170+0000 7f48ea000700 1 Processor -- start 2024-03-30T00:57:34.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.170+0000 7f48ea000700 1 -- start start 2024-03-30T00:57:34.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.170+0000 7f48ea000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48e40ff5e0 0x7f48e4198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.170+0000 7f48ea000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e40fff40 0x7f48e4199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.170+0000 7f48ea000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e4100900 0x7f48e41a04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.170+0000 7f48ea000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e41a0a80 con 0x7f48e4100900 2024-03-30T00:57:34.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.170+0000 7f48ea000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e41a0bc0 con 0x7f48e40ff5e0 2024-03-30T00:57:34.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.170+0000 7f48ea000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e41a0d50 con 0x7f48e40fff40 2024-03-30T00:57:34.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.171+0000 7f48e3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e4100900 0x7f48e41a04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.171+0000 7f48e3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e4100900 0x7f48e41a04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42898/0 (socket says 172.21.15.116:42898) 2024-03-30T00:57:34.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.171+0000 7f48e3e00700 1 -- 172.21.15.116:0/1101929812 learned_addr learned my addr 172.21.15.116:0/1101929812 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:34.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.171+0000 7f48e3400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48e40ff5e0 0x7f48e4198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.171+0000 7f48e2a00700 1 --2- 172.21.15.116:0/1101929812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e40fff40 0x7f48e4199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.171+0000 7f48e3e00700 1 -- 172.21.15.116:0/1101929812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e40fff40 msgr2=0x7f48e4199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:34.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.171+0000 7f48e3e00700 1 --2- 172.21.15.116:0/1101929812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e40fff40 0x7f48e4199490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.171+0000 7f48e3e00700 1 -- 172.21.15.116:0/1101929812 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48e40ff5e0 msgr2=0x7f48e4198f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:34.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.171+0000 7f48e3e00700 1 --2- 172.21.15.116:0/1101929812 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48e40ff5e0 0x7f48e4198f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.171+0000 7f48e3e00700 1 -- 172.21.15.116:0/1101929812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d40095f0 con 0x7f48e4100900 2024-03-30T00:57:34.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.171+0000 7f48e3e00700 1 --2- 172.21.15.116:0/1101929812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e4100900 0x7f48e41a04b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f48dc00ef30 tx=0x7f48dc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:34.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.171+0000 7f48dbe00700 1 -- 172.21.15.116:0/1101929812 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48dc018070 con 0x7f48e4100900 2024-03-30T00:57:34.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.171+0000 7f48dbe00700 1 -- 172.21.15.116:0/1101929812 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48dc00f040 con 0x7f48e4100900 2024-03-30T00:57:34.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.171+0000 7f48ea000700 1 -- 172.21.15.116:0/1101929812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e41a0fd0 con 0x7f48e4100900 2024-03-30T00:57:34.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.172+0000 7f48dbe00700 1 -- 172.21.15.116:0/1101929812 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48dc00cd00 con 0x7f48e4100900 2024-03-30T00:57:34.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.172+0000 7f48ea000700 1 -- 172.21.15.116:0/1101929812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e41a1400 con 0x7f48e4100900 2024-03-30T00:57:34.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.174+0000 7f48dbe00700 1 -- 172.21.15.116:0/1101929812 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 135917+0+0 (secure 0 0 0) 0x7f48dc003a00 con 0x7f48e4100900 2024-03-30T00:57:34.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.176+0000 7f48fdb5ab80 1 Processor -- start 2024-03-30T00:57:34.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48fdb5ab80 1 -- start start 2024-03-30T00:57:34.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48dbe00700 1 --2- 172.21.15.116:0/1101929812 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f48c80a11b0 0x7f48c80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48dbe00700 1 -- 172.21.15.116:0/1101929812 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f48dc0cc910 con 0x7f48e4100900 2024-03-30T00:57:34.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48fdb5ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ed780 0x55655e7edba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48e2a00700 1 --2- 172.21.15.116:0/1101929812 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f48c80a11b0 0x7f48c80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48fdb5ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7ee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48fdb5ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7eef80 0x55655e7ef400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48fdb5ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655e745940 con 0x55655e7eef80 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48fdb5ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655e743300 con 0x55655e7ee0e0 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48fdb5ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655e73ebf0 con 0x55655e7ed780 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48e3400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ed780 0x55655e7edba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48e3400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ed780 0x55655e7edba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:51354/0 (socket says 172.21.15.116:51354) 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48e3400700 1 -- 172.21.15.116:0/2881409886 learned_addr learned my addr 172.21.15.116:0/2881409886 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48e2a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7ee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.177+0000 7f48e3e00700 1 --2- 172.21.15.116:0/2881409886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7eef80 0x55655e7ef400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.178+0000 7f48e3400700 1 -- 172.21.15.116:0/2881409886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 msgr2=0x55655e7ee9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.178+0000 7f48e3400700 1 --2- 172.21.15.116:0/2881409886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7ee9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.178+0000 7f48e3400700 1 -- 172.21.15.116:0/2881409886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7eef80 msgr2=0x55655e7ef400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.178+0000 7f48e3400700 1 --2- 172.21.15.116:0/2881409886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7eef80 0x55655e7ef400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.178+0000 7f48e3400700 1 -- 172.21.15.116:0/2881409886 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55655e5c0360 con 0x55655e7ed780 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.178+0000 7f48e2a00700 1 --2- 172.21.15.116:0/1101929812 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f48c80a11b0 0x7f48c80a3670 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f48d0013570 tx=0x7f48d0009f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:34.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.178+0000 7f48e3400700 1 --2- 172.21.15.116:0/2881409886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ed780 0x55655e7edba0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009990 tx=0x7f48d4005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e222f9e34424d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:34.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.178+0000 7f48cfe00700 1 -- 172.21.15.116:0/2881409886 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d401d070 con 0x55655e7ed780 2024-03-30T00:57:34.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.178+0000 7f48cfe00700 1 -- 172.21.15.116:0/2881409886 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48d401e040 con 0x55655e7ed780 2024-03-30T00:57:34.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.178+0000 7f48cfe00700 1 -- 172.21.15.116:0/2881409886 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4004880 con 0x55655e7ed780 2024-03-30T00:57:34.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.179+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/2881409886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ed780 msgr2=0x55655e7edba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:34.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.179+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/2881409886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ed780 0x55655e7edba0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009990 tx=0x7f48d4005e80 comp rx=0 tx=0).stop 2024-03-30T00:57:34.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.179+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/2881409886 shutdown_connections 2024-03-30T00:57:34.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.179+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/2881409886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ed780 0x55655e7edba0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.179+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/2881409886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7eef80 0x55655e7ef400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.179+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/2881409886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7ee9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.179+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/2881409886 >> 172.21.15.116:0/2881409886 conn(0x55655e74c650 msgr2=0x55655e7e8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:34.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.179+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/2881409886 shutdown_connections 2024-03-30T00:57:34.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.179+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/2881409886 wait complete. 2024-03-30T00:57:34.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.180+0000 7f48fdb5ab80 1 -- start start 2024-03-30T00:57:34.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.180+0000 7f48fdb5ab80 1 Processor -- start 2024-03-30T00:57:34.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.180+0000 7f48fdb5ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7ed780 0x55655e7f7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.180+0000 7f48fdb5ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7f7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.180+0000 7f48fdb5ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7eef80 0x55655e7feaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.180+0000 7f48fdb5ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655e6dad30 con 0x55655e7ed780 2024-03-30T00:57:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.180+0000 7f48fdb5ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655e6da9d0 con 0x55655e7ee0e0 2024-03-30T00:57:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.180+0000 7f48fdb5ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655e5c2650 con 0x55655e7eef80 2024-03-30T00:57:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.180+0000 7f48fdb5ab80 10 client.0 _ll_register_callbacks cb 0x55655e6e4420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.180+0000 7f48e3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7ed780 0x55655e7f7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.180+0000 7f48e2a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7f7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.180+0000 7f48e3e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7eef80 0x55655e7feaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.180+0000 7f48e2a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7f7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34690/0 (socket says 172.21.15.116:34690) 2024-03-30T00:57:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.180+0000 7f48e2a00700 1 -- 172.21.15.116:0/3668996121 learned_addr learned my addr 172.21.15.116:0/3668996121 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.181+0000 7f48e2a00700 1 -- 172.21.15.116:0/3668996121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7eef80 msgr2=0x55655e7feaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.181+0000 7f48e2a00700 1 --2- 172.21.15.116:0/3668996121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7eef80 0x55655e7feaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.181+0000 7f48e2a00700 1 -- 172.21.15.116:0/3668996121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7ed780 msgr2=0x55655e7f7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.181+0000 7f48e2a00700 1 --2- 172.21.15.116:0/3668996121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7ed780 0x55655e7f7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.181+0000 7f48e2a00700 1 -- 172.21.15.116:0/3668996121 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e410bd70 con 0x55655e7ee0e0 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.181+0000 7f48e2a00700 1 --2- 172.21.15.116:0/3668996121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7f7a60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f48d0012250 tx=0x7f48d00047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86231c65bdb07af9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.181+0000 7f48ccc00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.181+0000 7f48ccc00700 1 -- 172.21.15.116:0/3668996121 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d0010630 con 0x55655e7ee0e0 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.181+0000 7f48ccc00700 1 -- 172.21.15.116:0/3668996121 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f48d0012280 con 0x55655e7ee0e0 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.181+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/3668996121 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55655e6ff330 con 0x55655e7ee0e0 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.182+0000 7f48ccc00700 1 -- 172.21.15.116:0/3668996121 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48d0010790 con 0x55655e7ee0e0 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.182+0000 7f48fdb5ab80 10 client.25364 fetch_fsmap learned FSMap version 22 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.182+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/3668996121 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55655e56cb10 con 0x55655e7ee0e0 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.182+0000 7f48ccc00700 1 -- 172.21.15.116:0/3668996121 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55655e6ff330 con 0x55655e7ee0e0 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.182+0000 7f48ccc00700 1 -- 172.21.15.116:0/3668996121 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 22) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f48d0020b10 con 0x55655e7ee0e0 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.182+0000 7f48fdb5ab80 10 client.25364 fetch_fsmap finished waiting for FSMap version 22 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.182+0000 7f48fdb5ab80 10 client.25364 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.182+0000 7f48fdb5ab80 20 client.25364 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.182+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/3668996121 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55655e801c40 0x55655e807ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.182+0000 7f48fdb5ab80 4 client.25364 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:57:34.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.182+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/3668996121 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55655e7376b0 con 0x55655e801c40 2024-03-30T00:57:34.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.183+0000 7f48e3400700 1 --2- 172.21.15.116:0/3668996121 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55655e801c40 0x55655e807ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.183+0000 7f48e3400700 1 --2- 172.21.15.116:0/3668996121 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55655e801c40 0x55655e807ad0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cdbe23abc55b070 server_cookie=7bbf97c3500cb031 in_seq=0 out_seq=0 2024-03-30T00:57:34.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.183+0000 7f48ccc00700 10 client.25364 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:34.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.184+0000 7f48ccc00700 1 -- 172.21.15.116:0/3668996121 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55655e7376b0 con 0x55655e801c40 2024-03-30T00:57:34.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.184+0000 7f48ccc00700 10 client.25364 handle_command_reply: tid=0 2024-03-30T00:57:34.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.184+0000 7f48fdb5ab80 1 client.25364 shutdown 2024-03-30T00:57:34.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.184+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/3668996121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 msgr2=0x55655e7f7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:34.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.184+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/3668996121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7f7a60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f48d0012250 tx=0x7f48d00047c0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.184+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/3668996121 shutdown_connections 2024-03-30T00:57:34.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.184+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/3668996121 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55655e801c40 0x55655e807ad0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.184+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/3668996121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7eef80 0x55655e7feaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.184+0000 7f48ccc00700 0 client.25364 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:34.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.184+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/3668996121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7ed780 0x55655e7f7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.184+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/3668996121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7f7a60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.184+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/3668996121 >> 172.21.15.116:0/3668996121 conn(0x55655e74c650 msgr2=0x55655e7ec120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:34.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.184+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/3668996121 shutdown_connections 2024-03-30T00:57:34.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.184+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/3668996121 wait complete. 2024-03-30T00:57:34.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.185+0000 7f48fdb5ab80 20 client.25364 trim_cache size 0 max 16384 2024-03-30T00:57:34.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.203+0000 7f48fdb5ab80 1 Processor -- start 2024-03-30T00:57:34.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.203+0000 7f48fdb5ab80 1 -- start start 2024-03-30T00:57:34.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.203+0000 7f48fdb5ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7ed780 0x55655e80c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.203+0000 7f48fdb5ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ee0e0 0x55655e7edb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.203+0000 7f48fdb5ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7eef80 0x55655e7ff150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.203+0000 7f48fdb5ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655e800830 con 0x55655e7ed780 2024-03-30T00:57:34.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.203+0000 7f48fdb5ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655e7376b0 con 0x55655e7eef80 2024-03-30T00:57:34.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.203+0000 7f48fdb5ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655e737990 con 0x55655e7ee0e0 2024-03-30T00:57:34.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.203+0000 7f48e3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7ed780 0x55655e80c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.203+0000 7f48e3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7ed780 0x55655e80c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42922/0 (socket says 172.21.15.116:42922) 2024-03-30T00:57:34.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.203+0000 7f48e3400700 1 -- 172.21.15.116:0/109193018 learned_addr learned my addr 172.21.15.116:0/109193018 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.203+0000 7f48e2a00700 1 --2- 172.21.15.116:0/109193018 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ee0e0 0x55655e7edb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.203+0000 7f48e3e00700 1 --2- 172.21.15.116:0/109193018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7eef80 0x55655e7ff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.204+0000 7f48e3400700 1 -- 172.21.15.116:0/109193018 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ee0e0 msgr2=0x55655e7edb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.204+0000 7f48e3400700 1 --2- 172.21.15.116:0/109193018 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ee0e0 0x55655e7edb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.204+0000 7f48e3400700 1 -- 172.21.15.116:0/109193018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7eef80 msgr2=0x55655e7ff150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.204+0000 7f48e3400700 1 --2- 172.21.15.116:0/109193018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7eef80 0x55655e7ff150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.204+0000 7f48e3400700 1 -- 172.21.15.116:0/109193018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55655e704e50 con 0x55655e7ed780 2024-03-30T00:57:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.204+0000 7f48e3400700 1 --2- 172.21.15.116:0/109193018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7ed780 0x55655e80c200 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f48d4005850 tx=0x7f48d4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30cef6ba044a08c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.204+0000 7f48d9200700 1 -- 172.21.15.116:0/109193018 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d401d040 con 0x55655e7ed780 2024-03-30T00:57:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.204+0000 7f48d9200700 1 -- 172.21.15.116:0/109193018 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48d400beb0 con 0x55655e7ed780 2024-03-30T00:57:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.204+0000 7f48d9200700 1 -- 172.21.15.116:0/109193018 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4005510 con 0x55655e7ed780 2024-03-30T00:57:34.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.205+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/109193018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7ed780 msgr2=0x55655e80c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:34.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.205+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/109193018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7ed780 0x55655e80c200 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f48d4005850 tx=0x7f48d4005fd0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.205+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/109193018 shutdown_connections 2024-03-30T00:57:34.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.205+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/109193018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7eef80 0x55655e7ff150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.205+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/109193018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7ed780 0x55655e80c200 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.205+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/109193018 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ee0e0 0x55655e7edb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.205+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/109193018 >> 172.21.15.116:0/109193018 conn(0x55655e74c650 msgr2=0x55655e80baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:34.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.205+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/109193018 shutdown_connections 2024-03-30T00:57:34.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.205+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/109193018 wait complete. 2024-03-30T00:57:34.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.205+0000 7f48fdb5ab80 1 -- start start 2024-03-30T00:57:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.205+0000 7f48fdb5ab80 1 Processor -- start 2024-03-30T00:57:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48fdb5ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ed780 0x55655e7faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48fdb5ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7fb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48fdb5ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7eef80 0x55655e7fe230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48fdb5ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655e7fe770 con 0x55655e7eef80 2024-03-30T00:57:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48fdb5ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655e7fe8e0 con 0x55655e7ee0e0 2024-03-30T00:57:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48fdb5ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655e7feaa0 con 0x55655e7ed780 2024-03-30T00:57:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48fdb5ab80 10 client.0 _ll_register_callbacks cb 0x55655e6e4420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48e3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7eef80 0x55655e7fe230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48e2a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7fb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48e3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7eef80 0x55655e7fe230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42930/0 (socket says 172.21.15.116:42930) 2024-03-30T00:57:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48e2a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7fb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34706/0 (socket says 172.21.15.116:34706) 2024-03-30T00:57:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48e3e00700 1 -- 172.21.15.116:0/1700233124 learned_addr learned my addr 172.21.15.116:0/1700233124 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48e3400700 1 --2- 172.21.15.116:0/1700233124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ed780 0x55655e7faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48e3e00700 1 -- 172.21.15.116:0/1700233124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ed780 msgr2=0x55655e7faef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48e3e00700 1 --2- 172.21.15.116:0/1700233124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ed780 0x55655e7faef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48e3e00700 1 -- 172.21.15.116:0/1700233124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 msgr2=0x55655e7fb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.206+0000 7f48e3e00700 1 --2- 172.21.15.116:0/1700233124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7fb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.207+0000 7f48e3e00700 1 -- 172.21.15.116:0/1700233124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55655e5c0360 con 0x55655e7eef80 2024-03-30T00:57:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.207+0000 7f48e3400700 1 --2- 172.21.15.116:0/1700233124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ed780 0x55655e7faef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.207+0000 7f48e2a00700 1 --2- 172.21.15.116:0/1700233124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7fb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.207+0000 7f48e3e00700 1 --2- 172.21.15.116:0/1700233124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7eef80 0x55655e7fe230 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f48dc095e20 tx=0x7f48dc004c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f983c6da8ef3add server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.207+0000 7f48ccc00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:57:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.207+0000 7f48ccc00700 1 -- 172.21.15.116:0/1700233124 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48dc00ce60 con 0x55655e7eef80 2024-03-30T00:57:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.207+0000 7f48ccc00700 1 -- 172.21.15.116:0/1700233124 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f48dc099920 con 0x55655e7eef80 2024-03-30T00:57:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.207+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/1700233124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48d401e040 con 0x55655e7eef80 2024-03-30T00:57:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.207+0000 7f48ccc00700 1 -- 172.21.15.116:0/1700233124 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48dc0b2240 con 0x55655e7eef80 2024-03-30T00:57:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.207+0000 7f48fdb5ab80 10 client.15309 fetch_fsmap learned FSMap version 22 2024-03-30T00:57:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.207+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/1700233124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55655e5c4c80 con 0x55655e7eef80 2024-03-30T00:57:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.207+0000 7f48ccc00700 1 -- 172.21.15.116:0/1700233124 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48d401e040 con 0x55655e7eef80 2024-03-30T00:57:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.207+0000 7f48ccc00700 1 -- 172.21.15.116:0/1700233124 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 22) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f48dc0b90b0 con 0x55655e7eef80 2024-03-30T00:57:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.208+0000 7f48fdb5ab80 10 client.15309 fetch_fsmap finished waiting for FSMap version 22 2024-03-30T00:57:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.208+0000 7f48fdb5ab80 10 client.15309 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-30T00:57:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.208+0000 7f48fdb5ab80 20 client.15309 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.208+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/1700233124 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55655e85b9f0 0x55655e858320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.208+0000 7f48fdb5ab80 4 client.15309 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T00:57:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.208+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/1700233124 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55655e58dca0 con 0x55655e85b9f0 2024-03-30T00:57:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.208+0000 7f48e3400700 1 --2- 172.21.15.116:0/1700233124 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55655e85b9f0 0x55655e858320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.208+0000 7f48ccc00700 10 client.15309 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.208+0000 7f48e3400700 1 --2- 172.21.15.116:0/1700233124 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55655e85b9f0 0x55655e858320 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ed9fe98384d6bf2 server_cookie=f39d75e7fedd9916 in_seq=0 out_seq=0 2024-03-30T00:57:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48ccc00700 1 -- 172.21.15.116:0/1700233124 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55655e58dca0 con 0x55655e85b9f0 2024-03-30T00:57:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48ccc00700 10 client.15309 handle_command_reply: tid=0 2024-03-30T00:57:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48fdb5ab80 1 client.15309 shutdown 2024-03-30T00:57:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/1700233124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7eef80 msgr2=0x55655e7fe230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/1700233124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7eef80 0x55655e7fe230 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f48dc095e20 tx=0x7f48dc004c10 comp rx=0 tx=0).stop 2024-03-30T00:57:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/1700233124 shutdown_connections 2024-03-30T00:57:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/1700233124 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55655e85b9f0 0x55655e858320 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/1700233124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55655e7ed780 0x55655e7faef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/1700233124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55655e7eef80 0x55655e7fe230 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48fdb5ab80 1 --2- 172.21.15.116:0/1700233124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55655e7ee0e0 0x55655e7fb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/1700233124 >> 172.21.15.116:0/1700233124 conn(0x55655e74c650 msgr2=0x55655e7f6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:34.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48ccc00700 0 client.15309 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:34.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/1700233124 shutdown_connections 2024-03-30T00:57:34.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48fdb5ab80 1 -- 172.21.15.116:0/1700233124 wait complete. 2024-03-30T00:57:34.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.209+0000 7f48fdb5ab80 20 client.15309 trim_cache size 0 max 16384 2024-03-30T00:57:34.222 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:57:34.222 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "4909f004-ee2f-11ee-b647-cb9ed24678a4", 2024-03-30T00:57:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-03-30T00:57:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24587, 2024-03-30T00:57:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-30T00:57:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-30T00:57:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-30T00:57:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 102.416757175, 2024-03-30T00:57:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 22, 2024-03-30T00:57:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 83, 2024-03-30T00:57:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 81, 2024-03-30T00:57:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 107.178571049 2024-03-30T00:57:34.223 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:57:34.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.210+0000 7f48ea000700 1 -- 172.21.15.116:0/1101929812 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f48c80a11b0 msgr2=0x7f48c80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:34.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.210+0000 7f48ea000700 1 --2- 172.21.15.116:0/1101929812 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f48c80a11b0 0x7f48c80a3670 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f48d0013570 tx=0x7f48d0009f90 comp rx=0 tx=0).stop 2024-03-30T00:57:34.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.210+0000 7f48ea000700 1 -- 172.21.15.116:0/1101929812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e4100900 msgr2=0x7f48e41a04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:34.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.210+0000 7f48ea000700 1 --2- 172.21.15.116:0/1101929812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e4100900 0x7f48e41a04b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f48dc00ef30 tx=0x7f48dc00c3a0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.210+0000 7f48ea000700 1 -- 172.21.15.116:0/1101929812 shutdown_connections 2024-03-30T00:57:34.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.210+0000 7f48ea000700 1 --2- 172.21.15.116:0/1101929812 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f48c80a11b0 0x7f48c80a3670 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.210+0000 7f48ea000700 1 --2- 172.21.15.116:0/1101929812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e4100900 0x7f48e41a04b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.210+0000 7f48ea000700 1 --2- 172.21.15.116:0/1101929812 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48e40ff5e0 0x7f48e4198f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.210+0000 7f48ea000700 1 --2- 172.21.15.116:0/1101929812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e40fff40 0x7f48e4199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:34.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.210+0000 7f48ea000700 1 -- 172.21.15.116:0/1101929812 >> 172.21.15.116:0/1101929812 conn(0x7f48e40fb200 msgr2=0x7f48e40fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:34.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.210+0000 7f48ea000700 1 -- 172.21.15.116:0/1101929812 shutdown_connections 2024-03-30T00:57:34.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:34.210+0000 7f48ea000700 1 -- 172.21.15.116:0/1101929812 wait complete. 2024-03-30T00:57:34.235 INFO:teuthology.run_tasks:Running task kclient... 2024-03-30T00:57:34.252 INFO:tasks.kclient:Mounting kernel clients... 2024-03-30T00:57:34.252 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-30T00:57:34.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:34.253 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-30T00:57:34.296 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-30T00:57:34.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:34.297 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-30T00:57:34.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:34.404 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-03-30T00:57:34.502 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:34 smithi116 ceph-mon[38086]: pgmap v282: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:34.502 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:34 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/523330850' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:34.511 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-30T00:57:34.511 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-30T00:57:34.512 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-30T00:57:34.512 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph fs ls 2024-03-30T00:57:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:34 smithi181 ceph-mon[47622]: pgmap v282: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:34 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/523330850' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:34.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:34 smithi130 ceph-mon[47626]: pgmap v282: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:57:34.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:34 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/523330850' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T00:57:36.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.088+0000 7f4fb1400700 1 -- 172.21.15.116:0/1345003767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fac111b30 msgr2=0x7f4fac113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:36.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.088+0000 7f4fb1400700 1 --2- 172.21.15.116:0/1345003767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fac111b30 0x7f4fac113f60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400b0a0 tx=0x7f4fa400b3b0 comp rx=0 tx=0).stop 2024-03-30T00:57:36.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.088+0000 7f4fb1400700 1 -- 172.21.15.116:0/1345003767 shutdown_connections 2024-03-30T00:57:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.088+0000 7f4fb1400700 1 --2- 172.21.15.116:0/1345003767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fac074d30 0x7f4fac072fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.088+0000 7f4fb1400700 1 --2- 172.21.15.116:0/1345003767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fac111b30 0x7f4fac113f60 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.088+0000 7f4fb1400700 1 --2- 172.21.15.116:0/1345003767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fac073520 0x7f4fac0739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.088+0000 7f4fb1400700 1 -- 172.21.15.116:0/1345003767 >> 172.21.15.116:0/1345003767 conn(0x7f4fac077fa0 msgr2=0x7f4fac0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.088+0000 7f4fb1400700 1 -- 172.21.15.116:0/1345003767 shutdown_connections 2024-03-30T00:57:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.089+0000 7f4fb1400700 1 -- 172.21.15.116:0/1345003767 wait complete. 2024-03-30T00:57:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.089+0000 7f4fb1400700 1 Processor -- start 2024-03-30T00:57:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.090+0000 7f4fb1400700 1 -- start start 2024-03-30T00:57:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.090+0000 7f4fb1400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fac073520 0x7f4fac1a3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.090+0000 7f4fb1400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fac074d30 0x7f4fac1a3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.090+0000 7f4fb1400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fac111b30 0x7f4fac19d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.090+0000 7f4fb1400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac19ddd0 con 0x7f4fac074d30 2024-03-30T00:57:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.090+0000 7f4fb1400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac19df40 con 0x7f4fac111b30 2024-03-30T00:57:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.090+0000 7f4fb1400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac19e100 con 0x7f4fac073520 2024-03-30T00:57:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.090+0000 7f4fa3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fac074d30 0x7f4fac1a3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.090+0000 7f4fa3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fac074d30 0x7f4fac1a3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53092/0 (socket says 172.21.15.116:53092) 2024-03-30T00:57:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.090+0000 7f4fa3e00700 1 -- 172.21.15.116:0/2590633561 learned_addr learned my addr 172.21.15.116:0/2590633561 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.090+0000 7f4faa800700 1 --2- 172.21.15.116:0/2590633561 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fac073520 0x7f4fac1a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:36.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.090+0000 7f4fab200700 1 --2- 172.21.15.116:0/2590633561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fac111b30 0x7f4fac19d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:36.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.091+0000 7f4fa3e00700 1 -- 172.21.15.116:0/2590633561 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fac073520 msgr2=0x7f4fac1a3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:36.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.091+0000 7f4fa3e00700 1 --2- 172.21.15.116:0/2590633561 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fac073520 0x7f4fac1a3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:36.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.091+0000 7f4fa3e00700 1 -- 172.21.15.116:0/2590633561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fac111b30 msgr2=0x7f4fac19d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:36.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.091+0000 7f4fa3e00700 1 --2- 172.21.15.116:0/2590633561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fac111b30 0x7f4fac19d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:36.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.091+0000 7f4fa3e00700 1 -- 172.21.15.116:0/2590633561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa4009d00 con 0x7f4fac074d30 2024-03-30T00:57:36.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.091+0000 7f4fa3e00700 1 --2- 172.21.15.116:0/2590633561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fac074d30 0x7f4fac1a3b70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4f9400d950 tx=0x7f4f9400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:36.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.091+0000 7f4fa3400700 1 -- 172.21.15.116:0/2590633561 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9400f890 con 0x7f4fac074d30 2024-03-30T00:57:36.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.091+0000 7f4fa3400700 1 -- 172.21.15.116:0/2590633561 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f94010460 con 0x7f4fac074d30 2024-03-30T00:57:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.091+0000 7f4fb1400700 1 -- 172.21.15.116:0/2590633561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fac19e3e0 con 0x7f4fac074d30 2024-03-30T00:57:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.091+0000 7f4fa3400700 1 -- 172.21.15.116:0/2590633561 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9400e610 con 0x7f4fac074d30 2024-03-30T00:57:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.091+0000 7f4fb1400700 1 -- 172.21.15.116:0/2590633561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fac1aa620 con 0x7f4fac074d30 2024-03-30T00:57:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.093+0000 7f4fb1400700 1 -- 172.21.15.116:0/2590633561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fac074670 con 0x7f4fac074d30 2024-03-30T00:57:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.094+0000 7f4fa3400700 1 -- 172.21.15.116:0/2590633561 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f4f940108d0 con 0x7f4fac074d30 2024-03-30T00:57:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.095+0000 7f4fa3400700 1 --2- 172.21.15.116:0/2590633561 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4f900a1120 0x7f4f900a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.096+0000 7f4fa3400700 1 -- 172.21.15.116:0/2590633561 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4f940cca00 con 0x7f4fac074d30 2024-03-30T00:57:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.096+0000 7f4faa800700 1 --2- 172.21.15.116:0/2590633561 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4f900a1120 0x7f4f900a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.096+0000 7f4faa800700 1 --2- 172.21.15.116:0/2590633561 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4f900a1120 0x7f4f900a35e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c00ac50 tx=0x7f4f9c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.100+0000 7f4fa3400700 1 -- 172.21.15.116:0/2590633561 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4f94095e20 con 0x7f4fac074d30 2024-03-30T00:57:36.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.345+0000 7f4fb1400700 1 -- 172.21.15.116:0/2590633561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4fac19ec60 con 0x7f4fac074d30 2024-03-30T00:57:36.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.346+0000 7f4fa3400700 1 -- 172.21.15.116:0/2590633561 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) v1 ==== 53+0+88 (secure 0 0 0) 0x7f4f94016070 con 0x7f4fac074d30 2024-03-30T00:57:36.353 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-30T00:57:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.348+0000 7f4fb1400700 1 -- 172.21.15.116:0/2590633561 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4f900a1120 msgr2=0x7f4f900a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.348+0000 7f4fb1400700 1 --2- 172.21.15.116:0/2590633561 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4f900a1120 0x7f4f900a35e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c00ac50 tx=0x7f4f9c00a380 comp rx=0 tx=0).stop 2024-03-30T00:57:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.348+0000 7f4fb1400700 1 -- 172.21.15.116:0/2590633561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fac074d30 msgr2=0x7f4fac1a3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.348+0000 7f4fb1400700 1 --2- 172.21.15.116:0/2590633561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fac074d30 0x7f4fac1a3b70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4f9400d950 tx=0x7f4f9400dc60 comp rx=0 tx=0).stop 2024-03-30T00:57:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.348+0000 7f4fb1400700 1 -- 172.21.15.116:0/2590633561 shutdown_connections 2024-03-30T00:57:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.348+0000 7f4fb1400700 1 --2- 172.21.15.116:0/2590633561 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4f900a1120 0x7f4f900a35e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.348+0000 7f4fb1400700 1 --2- 172.21.15.116:0/2590633561 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fac073520 0x7f4fac1a3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.348+0000 7f4fb1400700 1 --2- 172.21.15.116:0/2590633561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fac111b30 0x7f4fac19d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.348+0000 7f4fb1400700 1 --2- 172.21.15.116:0/2590633561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fac074d30 0x7f4fac1a3b70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.348+0000 7f4fb1400700 1 -- 172.21.15.116:0/2590633561 >> 172.21.15.116:0/2590633561 conn(0x7f4fac077fa0 msgr2=0x7f4fac1106c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.348+0000 7f4fb1400700 1 -- 172.21.15.116:0/2590633561 shutdown_connections 2024-03-30T00:57:36.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:36.348+0000 7f4fb1400700 1 -- 172.21.15.116:0/2590633561 wait complete. 2024-03-30T00:57:36.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:36 smithi116 ceph-mon[38086]: pgmap v283: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:36.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:36 smithi116 ceph-mon[38086]: mgrmap e18: x(active, since 7m), standbys: z, y 2024-03-30T00:57:36.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:36 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2590633561' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T00:57:36.683 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-30T00:57:36.683 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-30T00:57:36.684 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2024-03-30T00:57:36.684 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-30T00:57:36.684 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-30T00:57:36.684 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-30T00:57:36.684 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-30T00:57:36.684 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-30T00:57:36.684 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-30T00:57:36.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:36.684 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2024-03-30T00:57:36.728 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-30T00:57:36.728 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-30T00:57:36.728 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-30T00:57:36.729 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 41799sec preferred_lft 41799sec 2024-03-30T00:57:36.730 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-03-30T00:57:36.730 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-30T00:57:36.730 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-30T00:57:36.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:36.730 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-30T00:57:36.730 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-30T00:57:36.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add name ceph-brx type bridge 2024-03-30T00:57:36.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr flush dev ceph-brx 2024-03-30T00:57:36.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set ceph-brx up 2024-03-30T00:57:36.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-30T00:57:36.731 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-30T00:57:36.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:36 smithi181 ceph-mon[47622]: pgmap v283: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:36.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:36 smithi181 ceph-mon[47622]: mgrmap e18: x(active, since 7m), standbys: z, y 2024-03-30T00:57:36.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:36 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2590633561' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T00:57:36.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:36 smithi130 ceph-mon[47626]: pgmap v283: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:57:36.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:36 smithi130 ceph-mon[47626]: mgrmap e18: x(active, since 7m), standbys: z, y 2024-03-30T00:57:36.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:36 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2590633561' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T00:57:36.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:36.967 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-30T00:57:37.033 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-03-30T00:57:37.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:37.036 DEBUG:teuthology.orchestra.run.smithi116:> route 2024-03-30T00:57:37.082 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2024-03-30T00:57:37.083 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-30T00:57:37.083 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-30T00:57:37.083 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-30T00:57:37.083 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-30T00:57:37.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:37.084 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-30T00:57:37.084 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-30T00:57:37.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-30T00:57:37.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-30T00:57:37.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-30T00:57:37.084 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-30T00:57:37.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:37.289 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-30T00:57:37.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:37.332 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2024-03-30T00:57:37.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:37.415 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-30T00:57:37.415 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-30T00:57:37.415 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-30T00:57:37.415 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-30T00:57:37.415 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-30T00:57:37.591 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-30T00:57:37.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:37.591 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-30T00:57:37.591 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-30T00:57:37.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-30T00:57:37.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-30T00:57:37.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-30T00:57:37.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-30T00:57:37.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-30T00:57:37.591 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-30T00:57:37.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:37.825 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-30T00:57:37.825 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-30T00:57:37.825 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2024-03-30T00:57:37.825 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.0 master ceph-brx 2024-03-30T00:57:37.825 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-30T00:57:37.991 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-30T00:57:37.992 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-30T00:57:38.033 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-30T00:57:38.033 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-30T00:57:38.033 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-30T00:57:38.114 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-30T00:57:38.114 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-30T00:57:38.114 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64 2024-03-30T00:57:38.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:38.115 DEBUG:teuthology.orchestra.run.smithi116:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-30T00:57:38.253 INFO:teuthology.orchestra.run.smithi116.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-30T00:57:38.253 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-03-30T00:57:38.533 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:38 smithi116 ceph-mon[38086]: pgmap v284: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T00:57:38.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:38 smithi181 ceph-mon[47622]: pgmap v284: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T00:57:38.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:38 smithi130 ceph-mon[47626]: pgmap v284: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T00:57:38.866 INFO:teuthology.orchestra.run.smithi116.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-30T00:57:38.866 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-03-30T00:57:38.866 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: resolved to: "172.21.15.116:3300,172.21.15.130:3300,172.21.15.181:3300" 2024-03-30T00:57:38.866 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: trying mount with new device syntax: 0@4909f004-ee2f-11ee-b647-cb9ed24678a4.cephfs=/volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64 2024-03-30T00:57:38.866 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.116:3300/172.21.15.130:3300/172.21.15.181:3300" will pass to kernel 2024-03-30T00:57:38.872 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-30T00:57:38.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:38.872 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-30T00:57:38.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:38.940 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-30T00:57:39.022 INFO:teuthology.orchestra.run.smithi116.stdout:client15318 2024-03-30T00:57:39.023 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:57:39.023 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-30T00:57:39.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:39.065 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-30T00:57:39.190 INFO:teuthology.orchestra.run.smithi116.stdout:client15318 2024-03-30T00:57:39.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:39.190 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-30T00:57:39.273 INFO:teuthology.orchestra.run.smithi116.stdout:client15318 2024-03-30T00:57:39.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T00:57:39.274 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4.client15318/status') 2024-03-30T00:57:39.376 INFO:teuthology.orchestra.run.smithi116.stdout:instance: client.15318 (3)192.168.0.1:0/878501191 2024-03-30T00:57:39.377 INFO:teuthology.orchestra.run.smithi116.stdout:blocklisted: false 2024-03-30T00:57:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:0+1 records in 2024-03-30T00:57:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:0+1 records out 2024-03-30T00:57:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:69 bytes copied, 5.2738e-05 s, 1.3 MB/s 2024-03-30T00:57:39.381 INFO:teuthology.run_tasks:Running task exec... 2024-03-30T00:57:39.420 INFO:teuthology.task.exec:Executing custom commands... 2024-03-30T00:57:39.421 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-30T00:57:39.421 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-30T00:57:39.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.603+0000 7f8463400700 1 -- 172.21.15.116:0/1149817845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f845c100d20 msgr2=0x7f845c101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:39.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.603+0000 7f8463400700 1 --2- 172.21.15.116:0/1149817845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f845c100d20 0x7f845c101140 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f84500099c0 tx=0x7f8450009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:39.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.603+0000 7f8463400700 1 -- 172.21.15.116:0/1149817845 shutdown_connections 2024-03-30T00:57:39.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.603+0000 7f8463400700 1 --2- 172.21.15.116:0/1149817845 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f845c1028b0 0x7f845c109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:39.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.603+0000 7f8463400700 1 --2- 172.21.15.116:0/1149817845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f845c100d20 0x7f845c101140 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:39.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.603+0000 7f8463400700 1 --2- 172.21.15.116:0/1149817845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f845c101f10 0x7f845c102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:39.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.603+0000 7f8463400700 1 -- 172.21.15.116:0/1149817845 >> 172.21.15.116:0/1149817845 conn(0x7f845c0fc2b0 msgr2=0x7f845c0fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:39.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.603+0000 7f8463400700 1 -- 172.21.15.116:0/1149817845 shutdown_connections 2024-03-30T00:57:39.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.603+0000 7f8463400700 1 -- 172.21.15.116:0/1149817845 wait complete. 2024-03-30T00:57:39.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.604+0000 7f8463400700 1 Processor -- start 2024-03-30T00:57:39.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.604+0000 7f8463400700 1 -- start start 2024-03-30T00:57:39.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.605+0000 7f8463400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f845c100d20 0x7f845c198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:39.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.605+0000 7f8463400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f845c101f10 0x7f845c199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:39.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.605+0000 7f8463400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f845c1028b0 0x7f845c1a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:39.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.605+0000 7f8463400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c1a0a00 con 0x7f845c1028b0 2024-03-30T00:57:39.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.605+0000 7f8463400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c1a0b40 con 0x7f845c101f10 2024-03-30T00:57:39.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.605+0000 7f8463400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c1a0cd0 con 0x7f845c100d20 2024-03-30T00:57:39.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.605+0000 7f8461400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f845c1028b0 0x7f845c1a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:39.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.605+0000 7f8460a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f845c100d20 0x7f845c198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:39.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.605+0000 7f8461400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f845c1028b0 0x7f845c1a0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53110/0 (socket says 172.21.15.116:53110) 2024-03-30T00:57:39.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.605+0000 7f8461400700 1 -- 172.21.15.116:0/937110070 learned_addr learned my addr 172.21.15.116:0/937110070 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:39.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.605+0000 7f845be00700 1 --2- 172.21.15.116:0/937110070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f845c101f10 0x7f845c199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:39.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.606+0000 7f8461400700 1 -- 172.21.15.116:0/937110070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f845c100d20 msgr2=0x7f845c198ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:39.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.606+0000 7f8461400700 1 --2- 172.21.15.116:0/937110070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f845c100d20 0x7f845c198ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:39.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.606+0000 7f8461400700 1 -- 172.21.15.116:0/937110070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f845c101f10 msgr2=0x7f845c199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:39.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.606+0000 7f8461400700 1 --2- 172.21.15.116:0/937110070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f845c101f10 0x7f845c199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:39.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.606+0000 7f8461400700 1 -- 172.21.15.116:0/937110070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84500095f0 con 0x7f845c1028b0 2024-03-30T00:57:39.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.606+0000 7f8460a00700 1 --2- 172.21.15.116:0/937110070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f845c100d20 0x7f845c198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:39.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.606+0000 7f8461400700 1 --2- 172.21.15.116:0/937110070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f845c1028b0 0x7f845c1a0430 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8454007ee0 tx=0x7f845400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:39.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.606+0000 7f8459600700 1 -- 172.21.15.116:0/937110070 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8454017070 con 0x7f845c1028b0 2024-03-30T00:57:39.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.606+0000 7f8459600700 1 -- 172.21.15.116:0/937110070 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f845400f040 con 0x7f845c1028b0 2024-03-30T00:57:39.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.606+0000 7f8463400700 1 -- 172.21.15.116:0/937110070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845c1a0fb0 con 0x7f845c1028b0 2024-03-30T00:57:39.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.606+0000 7f8459600700 1 -- 172.21.15.116:0/937110070 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8454005580 con 0x7f845c1028b0 2024-03-30T00:57:39.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.606+0000 7f8463400700 1 -- 172.21.15.116:0/937110070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845c1a1430 con 0x7f845c1028b0 2024-03-30T00:57:39.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.607+0000 7f8463400700 1 -- 172.21.15.116:0/937110070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845c067c30 con 0x7f845c1028b0 2024-03-30T00:57:39.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.609+0000 7f8459600700 1 -- 172.21.15.116:0/937110070 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f845400db20 con 0x7f845c1028b0 2024-03-30T00:57:39.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.610+0000 7f8459600700 1 --2- 172.21.15.116:0/937110070 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f84440a1120 0x7f84440a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:39.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.610+0000 7f8459600700 1 -- 172.21.15.116:0/937110070 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f84540cc360 con 0x7f845c1028b0 2024-03-30T00:57:39.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.611+0000 7f8460a00700 1 --2- 172.21.15.116:0/937110070 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f84440a1120 0x7f84440a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:39.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.611+0000 7f8460a00700 1 --2- 172.21.15.116:0/937110070 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f84440a1120 0x7f84440a35e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8450006010 tx=0x7f845000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:39.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.615+0000 7f8459600700 1 -- 172.21.15.116:0/937110070 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8454095780 con 0x7f845c1028b0 2024-03-30T00:57:39.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:39.875+0000 7f8463400700 1 -- 172.21.15.116:0/937110070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) v1 -- 0x7f845c10c580 con 0x7f845c1028b0 2024-03-30T00:57:40.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.445+0000 7f8459600700 1 -- 172.21.15.116:0/937110070 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v23) v1 ==== 117+0+0 (secure 0 0 0) 0x7f8454098da0 con 0x7f845c1028b0 2024-03-30T00:57:40.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.447+0000 7f8463400700 1 -- 172.21.15.116:0/937110070 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f84440a1120 msgr2=0x7f84440a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:40.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.447+0000 7f8463400700 1 --2- 172.21.15.116:0/937110070 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f84440a1120 0x7f84440a35e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8450006010 tx=0x7f845000b540 comp rx=0 tx=0).stop 2024-03-30T00:57:40.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.447+0000 7f8463400700 1 -- 172.21.15.116:0/937110070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f845c1028b0 msgr2=0x7f845c1a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:40.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.447+0000 7f8463400700 1 --2- 172.21.15.116:0/937110070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f845c1028b0 0x7f845c1a0430 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8454007ee0 tx=0x7f845400d3b0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.448+0000 7f8463400700 1 -- 172.21.15.116:0/937110070 shutdown_connections 2024-03-30T00:57:40.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.448+0000 7f8463400700 1 --2- 172.21.15.116:0/937110070 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f84440a1120 0x7f84440a35e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.448+0000 7f8463400700 1 --2- 172.21.15.116:0/937110070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f845c100d20 0x7f845c198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.448+0000 7f8463400700 1 --2- 172.21.15.116:0/937110070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f845c1028b0 0x7f845c1a0430 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.448+0000 7f8463400700 1 --2- 172.21.15.116:0/937110070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f845c101f10 0x7f845c199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.448+0000 7f8463400700 1 -- 172.21.15.116:0/937110070 >> 172.21.15.116:0/937110070 conn(0x7f845c0fc2b0 msgr2=0x7f845c1076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:40.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.448+0000 7f8463400700 1 -- 172.21.15.116:0/937110070 shutdown_connections 2024-03-30T00:57:40.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.448+0000 7f8463400700 1 -- 172.21.15.116:0/937110070 wait complete. 2024-03-30T00:57:40.474 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-30T00:57:40.521 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-30T00:57:40.538 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:57:40.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.725+0000 7fb509200700 1 -- 172.21.15.116:0/3425464375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5041040e0 msgr2=0x7fb5041064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.725+0000 7fb509200700 1 --2- 172.21.15.116:0/3425464375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5041040e0 0x7fb5041064d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009990 tx=0x7fb4f0009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.725+0000 7fb509200700 1 -- 172.21.15.116:0/3425464375 shutdown_connections 2024-03-30T00:57:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.725+0000 7fb509200700 1 --2- 172.21.15.116:0/3425464375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb504106a10 0x7fb504108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.725+0000 7fb509200700 1 --2- 172.21.15.116:0/3425464375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5041017b0 0x7fb504103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.725+0000 7fb509200700 1 --2- 172.21.15.116:0/3425464375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5041040e0 0x7fb5041064d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.725+0000 7fb509200700 1 -- 172.21.15.116:0/3425464375 >> 172.21.15.116:0/3425464375 conn(0x7fb5040fb100 msgr2=0x7fb5040fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.725+0000 7fb509200700 1 -- 172.21.15.116:0/3425464375 shutdown_connections 2024-03-30T00:57:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.725+0000 7fb509200700 1 -- 172.21.15.116:0/3425464375 wait complete. 2024-03-30T00:57:40.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.726+0000 7fb509200700 1 Processor -- start 2024-03-30T00:57:40.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.726+0000 7fb509200700 1 -- start start 2024-03-30T00:57:40.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.726+0000 7fb509200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5041017b0 0x7fb504198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:40.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.726+0000 7fb509200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5041040e0 0x7fb504199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:40.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.726+0000 7fb509200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb504106a10 0x7fb5041a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.726+0000 7fb509200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5041a08f0 con 0x7fb5041040e0 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.726+0000 7fb509200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5041a0a30 con 0x7fb5041017b0 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.726+0000 7fb509200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5041a0bc0 con 0x7fb504106a10 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.726+0000 7fb501400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5041040e0 0x7fb504199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.727+0000 7fb502800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb504106a10 0x7fb5041a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.727+0000 7fb502800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb504106a10 0x7fb5041a0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:43196/0 (socket says 172.21.15.116:43196) 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.727+0000 7fb502800700 1 -- 172.21.15.116:0/3066229946 learned_addr learned my addr 172.21.15.116:0/3066229946 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.727+0000 7fb501e00700 1 --2- 172.21.15.116:0/3066229946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5041017b0 0x7fb504198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.727+0000 7fb502800700 1 -- 172.21.15.116:0/3066229946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5041017b0 msgr2=0x7fb504198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.727+0000 7fb502800700 1 --2- 172.21.15.116:0/3066229946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5041017b0 0x7fb504198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.727+0000 7fb502800700 1 -- 172.21.15.116:0/3066229946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5041040e0 msgr2=0x7fb504199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.727+0000 7fb502800700 1 --2- 172.21.15.116:0/3066229946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5041040e0 0x7fb504199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.727+0000 7fb502800700 1 -- 172.21.15.116:0/3066229946 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f00095f0 con 0x7fb504106a10 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.727+0000 7fb502800700 1 --2- 172.21.15.116:0/3066229946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb504106a10 0x7fb5041a0320 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc00ef30 tx=0x7fb4fc00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.727+0000 7fb4faa00700 1 -- 172.21.15.116:0/3066229946 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc018070 con 0x7fb504106a10 2024-03-30T00:57:40.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.727+0000 7fb4faa00700 1 -- 172.21.15.116:0/3066229946 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb4fc00f040 con 0x7fb504106a10 2024-03-30T00:57:40.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.727+0000 7fb509200700 1 -- 172.21.15.116:0/3066229946 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5041a0e40 con 0x7fb504106a10 2024-03-30T00:57:40.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.728+0000 7fb4faa00700 1 -- 172.21.15.116:0/3066229946 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc00cd00 con 0x7fb504106a10 2024-03-30T00:57:40.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.728+0000 7fb509200700 1 -- 172.21.15.116:0/3066229946 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5041a1310 con 0x7fb504106a10 2024-03-30T00:57:40.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.729+0000 7fb509200700 1 -- 172.21.15.116:0/3066229946 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb504108590 con 0x7fb504106a10 2024-03-30T00:57:40.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.730+0000 7fb4faa00700 1 -- 172.21.15.116:0/3066229946 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fb4fc003a00 con 0x7fb504106a10 2024-03-30T00:57:40.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.732+0000 7fb4faa00700 1 --2- 172.21.15.116:0/3066229946 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4e80a11a0 0x7fb4e80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:40.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.732+0000 7fb4faa00700 1 -- 172.21.15.116:0/3066229946 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb4fc0cc730 con 0x7fb504106a10 2024-03-30T00:57:40.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.732+0000 7fb501e00700 1 --2- 172.21.15.116:0/3066229946 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4e80a11a0 0x7fb4e80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:40.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.732+0000 7fb501e00700 1 --2- 172.21.15.116:0/3066229946 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4e80a11a0 0x7fb4e80a3660 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4007c60 tx=0x7fb4f40073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.734+0000 7fb4faa00700 1 -- 172.21.15.116:0/3066229946 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb4fc095c40 con 0x7fb504106a10 2024-03-30T00:57:40.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:40 smithi181 ceph-mon[47622]: pgmap v285: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-30T00:57:40.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:40 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/937110070' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-30T00:57:40.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:40 smithi130 ceph-mon[47626]: pgmap v285: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-30T00:57:40.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:40 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/937110070' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-30T00:57:40.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:40 smithi116 ceph-mon[38086]: pgmap v285: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-30T00:57:40.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:40 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/937110070' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-30T00:57:40.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.960+0000 7fb509200700 1 -- 172.21.15.116:0/3066229946 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb504067180 con 0x7fb504106a10 2024-03-30T00:57:40.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.962+0000 7fb4faa00700 1 -- 172.21.15.116:0/3066229946 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+24556 (secure 0 0 0) 0x7fb4fc04e000 con 0x7fb504106a10 2024-03-30T00:57:40.969 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:40.970 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:56:36.456069+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:57:40.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.965+0000 7fb509200700 1 -- 172.21.15.116:0/3066229946 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4e80a11a0 msgr2=0x7fb4e80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:40.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.965+0000 7fb509200700 1 --2- 172.21.15.116:0/3066229946 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4e80a11a0 0x7fb4e80a3660 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4007c60 tx=0x7fb4f40073d0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.965+0000 7fb509200700 1 -- 172.21.15.116:0/3066229946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb504106a10 msgr2=0x7fb5041a0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:40.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.965+0000 7fb509200700 1 --2- 172.21.15.116:0/3066229946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb504106a10 0x7fb5041a0320 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc00ef30 tx=0x7fb4fc00c3a0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.965+0000 7fb509200700 1 -- 172.21.15.116:0/3066229946 shutdown_connections 2024-03-30T00:57:40.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.965+0000 7fb509200700 1 --2- 172.21.15.116:0/3066229946 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4e80a11a0 0x7fb4e80a3660 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.965+0000 7fb509200700 1 --2- 172.21.15.116:0/3066229946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb504106a10 0x7fb5041a0320 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.965+0000 7fb509200700 1 --2- 172.21.15.116:0/3066229946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5041017b0 0x7fb504198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.965+0000 7fb509200700 1 --2- 172.21.15.116:0/3066229946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5041040e0 0x7fb504199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:40.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.965+0000 7fb509200700 1 -- 172.21.15.116:0/3066229946 >> 172.21.15.116:0/3066229946 conn(0x7fb5040fb100 msgr2=0x7fb504100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:40.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.965+0000 7fb509200700 1 -- 172.21.15.116:0/3066229946 shutdown_connections 2024-03-30T00:57:40.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:40.965+0000 7fb509200700 1 -- 172.21.15.116:0/3066229946 wait complete. 2024-03-30T00:57:40.990 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T00:52:29.602428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T00:55:50.220543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T00:55:51.274498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T00:55:54.301591+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T00:57:40.991 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T00:57:41.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.182+0000 7f31bf600700 1 -- 172.21.15.116:0/1705143513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31b8100d80 msgr2=0x7f31b81011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:41.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.182+0000 7f31bf600700 1 --2- 172.21.15.116:0/1705143513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31b8100d80 0x7f31b81011a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0099c0 tx=0x7f31ac009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.182+0000 7f31bf600700 1 -- 172.21.15.116:0/1705143513 shutdown_connections 2024-03-30T00:57:41.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.182+0000 7f31bf600700 1 --2- 172.21.15.116:0/1705143513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31b8102910 0x7f31b81091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.182+0000 7f31bf600700 1 --2- 172.21.15.116:0/1705143513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31b8100d80 0x7f31b81011a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.182+0000 7f31bf600700 1 --2- 172.21.15.116:0/1705143513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31b8101f70 0x7f31b81023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.182+0000 7f31bf600700 1 -- 172.21.15.116:0/1705143513 >> 172.21.15.116:0/1705143513 conn(0x7f31b80fc2f0 msgr2=0x7f31b80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:41.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.182+0000 7f31bf600700 1 -- 172.21.15.116:0/1705143513 shutdown_connections 2024-03-30T00:57:41.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.182+0000 7f31bf600700 1 -- 172.21.15.116:0/1705143513 wait complete. 2024-03-30T00:57:41.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.183+0000 7f31bf600700 1 Processor -- start 2024-03-30T00:57:41.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.183+0000 7f31bf600700 1 -- start start 2024-03-30T00:57:41.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.183+0000 7f31bf600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31b8100d80 0x7f31b8196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:41.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.183+0000 7f31bf600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31b8101f70 0x7f31b81972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:41.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.183+0000 7f31bf600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31b8102910 0x7f31b819e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:41.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.183+0000 7f31bf600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b819e870 con 0x7f31b8101f70 2024-03-30T00:57:41.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.183+0000 7f31bf600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b819e9e0 con 0x7f31b8102910 2024-03-30T00:57:41.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.183+0000 7f31bf600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b819eba0 con 0x7f31b8100d80 2024-03-30T00:57:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.184+0000 7f31b7e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31b8101f70 0x7f31b81972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.184+0000 7f31b7e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31b8101f70 0x7f31b81972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53136/0 (socket says 172.21.15.116:53136) 2024-03-30T00:57:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.184+0000 7f31b7e00700 1 -- 172.21.15.116:0/563327619 learned_addr learned my addr 172.21.15.116:0/563327619 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.184+0000 7f31bcc00700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31b8100d80 0x7f31b8196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.184+0000 7f31bd600700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31b8102910 0x7f31b819e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.184+0000 7f31b7e00700 1 -- 172.21.15.116:0/563327619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31b8100d80 msgr2=0x7f31b8196d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.184+0000 7f31b7e00700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31b8100d80 0x7f31b8196d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.184+0000 7f31b7e00700 1 -- 172.21.15.116:0/563327619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31b8102910 msgr2=0x7f31b819e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.184+0000 7f31b7e00700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31b8102910 0x7f31b819e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.184+0000 7f31b7e00700 1 -- 172.21.15.116:0/563327619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b0009c40 con 0x7f31b8101f70 2024-03-30T00:57:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.184+0000 7f31bd600700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31b8102910 0x7f31b819e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.184+0000 7f31b7e00700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31b8101f70 0x7f31b81972c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f31a400d950 tx=0x7f31a400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.184+0000 7f31b5600700 1 -- 172.21.15.116:0/563327619 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a4009940 con 0x7f31b8101f70 2024-03-30T00:57:41.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.185+0000 7f31bf600700 1 -- 172.21.15.116:0/563327619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ac0095f0 con 0x7f31b8101f70 2024-03-30T00:57:41.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.185+0000 7f31b5600700 1 -- 172.21.15.116:0/563327619 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31a4010460 con 0x7f31b8101f70 2024-03-30T00:57:41.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.185+0000 7f31b5600700 1 -- 172.21.15.116:0/563327619 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a400f620 con 0x7f31b8101f70 2024-03-30T00:57:41.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.185+0000 7f31bf600700 1 -- 172.21.15.116:0/563327619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b819f1d0 con 0x7f31b8101f70 2024-03-30T00:57:41.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.186+0000 7f31bf600700 1 -- 172.21.15.116:0/563327619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b8108250 con 0x7f31b8101f70 2024-03-30T00:57:41.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.187+0000 7f31b5600700 1 -- 172.21.15.116:0/563327619 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f31a40108d0 con 0x7f31b8101f70 2024-03-30T00:57:41.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.189+0000 7f31b5600700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f31a00a1120 0x7f31a00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:41.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.189+0000 7f31b5600700 1 -- 172.21.15.116:0/563327619 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f31a40cca80 con 0x7f31b8101f70 2024-03-30T00:57:41.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.189+0000 7f31bcc00700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f31a00a1120 0x7f31a00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:41.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.189+0000 7f31bcc00700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f31a00a1120 0x7f31a00a35e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f31ac003730 tx=0x7f31ac000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:41.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.192+0000 7f31b5600700 1 -- 172.21.15.116:0/563327619 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f31a4095f00 con 0x7f31b8101f70 2024-03-30T00:57:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.400+0000 7f31bf600700 1 -- 172.21.15.116:0/563327619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31b8107c30 con 0x7f31b8101f70 2024-03-30T00:57:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.400+0000 7f31b5600700 1 -- 172.21.15.116:0/563327619 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31a40160a0 con 0x7f31b8101f70 2024-03-30T00:57:41.407 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-30T00:57:41.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.402+0000 7f31bf600700 1 -- 172.21.15.116:0/563327619 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f31a00a1120 msgr2=0x7f31a00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:41.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.402+0000 7f31bf600700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f31a00a1120 0x7f31a00a35e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f31ac003730 tx=0x7f31ac000bc0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.402+0000 7f31bf600700 1 -- 172.21.15.116:0/563327619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31b8101f70 msgr2=0x7f31b81972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:41.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.402+0000 7f31bf600700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31b8101f70 0x7f31b81972c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f31a400d950 tx=0x7f31a400dc60 comp rx=0 tx=0).stop 2024-03-30T00:57:41.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.403+0000 7f31bf600700 1 -- 172.21.15.116:0/563327619 shutdown_connections 2024-03-30T00:57:41.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.403+0000 7f31bf600700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f31a00a1120 0x7f31a00a35e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.403+0000 7f31bf600700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31b8100d80 0x7f31b8196d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.403+0000 7f31bf600700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31b8102910 0x7f31b819e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.403+0000 7f31bf600700 1 --2- 172.21.15.116:0/563327619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31b8101f70 0x7f31b81972c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.403+0000 7f31bf600700 1 -- 172.21.15.116:0/563327619 >> 172.21.15.116:0/563327619 conn(0x7f31b80fc2f0 msgr2=0x7f31b8105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:41.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.403+0000 7f31bf600700 1 -- 172.21.15.116:0/563327619 shutdown_connections 2024-03-30T00:57:41.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.403+0000 7f31bf600700 1 -- 172.21.15.116:0/563327619 wait complete. 2024-03-30T00:57:41.431 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T00:57:41.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.621+0000 7f2ef4800700 1 -- 172.21.15.116:0/3407794856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eec101f70 msgr2=0x7f2eec1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:41.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.621+0000 7f2ef4800700 1 --2- 172.21.15.116:0/3407794856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eec101f70 0x7f2eec1023d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0099c0 tx=0x7f2edc009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.621+0000 7f2ef4800700 1 -- 172.21.15.116:0/3407794856 shutdown_connections 2024-03-30T00:57:41.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.621+0000 7f2ef4800700 1 --2- 172.21.15.116:0/3407794856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eec102910 0x7f2eec1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.621+0000 7f2ef4800700 1 --2- 172.21.15.116:0/3407794856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2eec100d80 0x7f2eec1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.621+0000 7f2ef4800700 1 --2- 172.21.15.116:0/3407794856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eec101f70 0x7f2eec1023d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.621+0000 7f2ef4800700 1 -- 172.21.15.116:0/3407794856 >> 172.21.15.116:0/3407794856 conn(0x7f2eec0fc2f0 msgr2=0x7f2eec0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:41.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.621+0000 7f2ef4800700 1 -- 172.21.15.116:0/3407794856 shutdown_connections 2024-03-30T00:57:41.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.621+0000 7f2ef4800700 1 -- 172.21.15.116:0/3407794856 wait complete. 2024-03-30T00:57:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.622+0000 7f2ef4800700 1 Processor -- start 2024-03-30T00:57:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.622+0000 7f2ef4800700 1 -- start start 2024-03-30T00:57:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.622+0000 7f2ef4800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2eec100d80 0x7f2eec198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.622+0000 7f2ef4800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eec101f70 0x7f2eec1994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.622+0000 7f2ef4800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eec102910 0x7f2eec1a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef4800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec1a0ab0 con 0x7f2eec101f70 2024-03-30T00:57:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef4800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec1a0bf0 con 0x7f2eec100d80 2024-03-30T00:57:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef4800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec1a0d80 con 0x7f2eec102910 2024-03-30T00:57:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef1400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eec101f70 0x7f2eec1994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef1400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eec101f70 0x7f2eec1994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53156/0 (socket says 172.21.15.116:53156) 2024-03-30T00:57:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef1400700 1 -- 172.21.15.116:0/2308959374 learned_addr learned my addr 172.21.15.116:0/2308959374 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef1e00700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2eec100d80 0x7f2eec198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:41.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef2800700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eec102910 0x7f2eec1a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef1400700 1 -- 172.21.15.116:0/2308959374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eec102910 msgr2=0x7f2eec1a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef1400700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eec102910 0x7f2eec1a04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef1400700 1 -- 172.21.15.116:0/2308959374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2eec100d80 msgr2=0x7f2eec198f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef1400700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2eec100d80 0x7f2eec198f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef1400700 1 -- 172.21.15.116:0/2308959374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc0095f0 con 0x7f2eec101f70 2024-03-30T00:57:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef2800700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eec102910 0x7f2eec1a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.623+0000 7f2ef1400700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eec101f70 0x7f2eec1994c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2edc003730 tx=0x7f2edc00fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.624+0000 7f2ee6a00700 1 -- 172.21.15.116:0/2308959374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc01d070 con 0x7f2eec101f70 2024-03-30T00:57:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.624+0000 7f2ee6a00700 1 -- 172.21.15.116:0/2308959374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2edc00fe70 con 0x7f2eec101f70 2024-03-30T00:57:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.624+0000 7f2ee6a00700 1 -- 172.21.15.116:0/2308959374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc004bd0 con 0x7f2eec101f70 2024-03-30T00:57:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.624+0000 7f2ef4800700 1 -- 172.21.15.116:0/2308959374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eec1a1000 con 0x7f2eec101f70 2024-03-30T00:57:41.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.624+0000 7f2ef4800700 1 -- 172.21.15.116:0/2308959374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eec1a1420 con 0x7f2eec101f70 2024-03-30T00:57:41.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.625+0000 7f2ef4800700 1 -- 172.21.15.116:0/2308959374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eec067af0 con 0x7f2eec101f70 2024-03-30T00:57:41.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.627+0000 7f2ee6a00700 1 -- 172.21.15.116:0/2308959374 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f2edc004d30 con 0x7f2eec101f70 2024-03-30T00:57:41.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.628+0000 7f2ee6a00700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2ed40a11a0 0x7f2ed40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:41.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.628+0000 7f2ee6a00700 1 -- 172.21.15.116:0/2308959374 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2edc0cdb40 con 0x7f2eec101f70 2024-03-30T00:57:41.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.628+0000 7f2ef1e00700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2ed40a11a0 0x7f2ed40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:41.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.629+0000 7f2ef1e00700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2ed40a11a0 0x7f2ed40a3660 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2eec101dd0 tx=0x7f2ee0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:41.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.631+0000 7f2ee6a00700 1 -- 172.21.15.116:0/2308959374 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2edc096ff0 con 0x7f2eec101f70 2024-03-30T00:57:41.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:41 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/937110070' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-30T00:57:41.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:41 smithi181 ceph-mon[47622]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T00:57:41.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:41 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3066229946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:41.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:41 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/563327619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:41.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.837+0000 7f2ef4800700 1 -- 172.21.15.116:0/2308959374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2eec1a1700 con 0x7f2eec101f70 2024-03-30T00:57:41.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.837+0000 7f2ee6a00700 1 -- 172.21.15.116:0/2308959374 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2edc09a610 con 0x7f2eec101f70 2024-03-30T00:57:41.844 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:57:41.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:41 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/937110070' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-30T00:57:41.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:41 smithi130 ceph-mon[47626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T00:57:41.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:41 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3066229946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:41.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:41 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/563327619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:41.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.839+0000 7f2ef4800700 1 -- 172.21.15.116:0/2308959374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2ed40a11a0 msgr2=0x7f2ed40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:41.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.839+0000 7f2ef4800700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2ed40a11a0 0x7f2ed40a3660 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2eec101dd0 tx=0x7f2ee0009260 comp rx=0 tx=0).stop 2024-03-30T00:57:41.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.840+0000 7f2ef4800700 1 -- 172.21.15.116:0/2308959374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eec101f70 msgr2=0x7f2eec1994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:41.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.840+0000 7f2ef4800700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eec101f70 0x7f2eec1994c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2edc003730 tx=0x7f2edc00fa00 comp rx=0 tx=0).stop 2024-03-30T00:57:41.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.840+0000 7f2ef4800700 1 -- 172.21.15.116:0/2308959374 shutdown_connections 2024-03-30T00:57:41.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.840+0000 7f2ef4800700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2ed40a11a0 0x7f2ed40a3660 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.840+0000 7f2ef4800700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eec102910 0x7f2eec1a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.840+0000 7f2ef4800700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2eec100d80 0x7f2eec198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.840+0000 7f2ef4800700 1 --2- 172.21.15.116:0/2308959374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eec101f70 0x7f2eec1994c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.840+0000 7f2ef4800700 1 -- 172.21.15.116:0/2308959374 >> 172.21.15.116:0/2308959374 conn(0x7f2eec0fc2f0 msgr2=0x7f2eec072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.840+0000 7f2ef4800700 1 -- 172.21.15.116:0/2308959374 shutdown_connections 2024-03-30T00:57:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:41.840+0000 7f2ef4800700 1 -- 172.21.15.116:0/2308959374 wait complete. 2024-03-30T00:57:41.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:41 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/937110070' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-30T00:57:41.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:41 smithi116 ceph-mon[38086]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T00:57:41.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:41 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3066229946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:41.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:41 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/563327619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:41.867 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T00:57:42.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.056+0000 7f7d7e000700 1 -- 172.21.15.116:0/3878314797 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d78074df0 msgr2=0x7f7d78073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.056+0000 7f7d7e000700 1 --2- 172.21.15.116:0/3878314797 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d78074df0 0x7f7d78073240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7d68009990 tx=0x7f7d68009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.056+0000 7f7d7e000700 1 -- 172.21.15.116:0/3878314797 shutdown_connections 2024-03-30T00:57:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.056+0000 7f7d7e000700 1 --2- 172.21.15.116:0/3878314797 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d78074df0 0x7f7d78073240 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.056+0000 7f7d7e000700 1 --2- 172.21.15.116:0/3878314797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d78108c40 0x7f7d7810b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.056+0000 7f7d7e000700 1 --2- 172.21.15.116:0/3878314797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d78073810 0x7f7d78073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.056+0000 7f7d7e000700 1 -- 172.21.15.116:0/3878314797 >> 172.21.15.116:0/3878314797 conn(0x7f7d780fc2f0 msgr2=0x7f7d780fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.056+0000 7f7d7e000700 1 -- 172.21.15.116:0/3878314797 shutdown_connections 2024-03-30T00:57:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.056+0000 7f7d7e000700 1 -- 172.21.15.116:0/3878314797 wait complete. 2024-03-30T00:57:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.057+0000 7f7d7e000700 1 Processor -- start 2024-03-30T00:57:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.057+0000 7f7d7e000700 1 -- start start 2024-03-30T00:57:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.057+0000 7f7d7e000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d78073810 0x7f7d78198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.057+0000 7f7d7e000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d78074df0 0x7f7d78199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.057+0000 7f7d7e000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d78108c40 0x7f7d781a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.057+0000 7f7d7e000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d781a09f0 con 0x7f7d78108c40 2024-03-30T00:57:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.057+0000 7f7d7e000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d781a0b30 con 0x7f7d78073810 2024-03-30T00:57:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.057+0000 7f7d7e000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d781a0cc0 con 0x7f7d78074df0 2024-03-30T00:57:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.058+0000 7f7d77400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d78073810 0x7f7d78198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.058+0000 7f7d77e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d78108c40 0x7f7d781a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.058+0000 7f7d76a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d78074df0 0x7f7d78199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.058+0000 7f7d77400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d78073810 0x7f7d78198f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:52580/0 (socket says 172.21.15.116:52580) 2024-03-30T00:57:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.058+0000 7f7d77400700 1 -- 172.21.15.116:0/3685312391 learned_addr learned my addr 172.21.15.116:0/3685312391 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.058+0000 7f7d77e00700 1 -- 172.21.15.116:0/3685312391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d78074df0 msgr2=0x7f7d78199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.058+0000 7f7d77e00700 1 --2- 172.21.15.116:0/3685312391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d78074df0 0x7f7d78199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.058+0000 7f7d77e00700 1 -- 172.21.15.116:0/3685312391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d78073810 msgr2=0x7f7d78198f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.058+0000 7f7d77e00700 1 --2- 172.21.15.116:0/3685312391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d78073810 0x7f7d78198f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.058+0000 7f7d77e00700 1 -- 172.21.15.116:0/3685312391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d680095f0 con 0x7f7d78108c40 2024-03-30T00:57:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.058+0000 7f7d77e00700 1 --2- 172.21.15.116:0/3685312391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d78108c40 0x7f7d781a0420 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000efc0 tx=0x7f7d7000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.058+0000 7f7d6fe00700 1 -- 172.21.15.116:0/3685312391 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d70018070 con 0x7f7d78108c40 2024-03-30T00:57:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.059+0000 7f7d6fe00700 1 -- 172.21.15.116:0/3685312391 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d7000f040 con 0x7f7d78108c40 2024-03-30T00:57:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.059+0000 7f7d7e000700 1 -- 172.21.15.116:0/3685312391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d781a0f40 con 0x7f7d78108c40 2024-03-30T00:57:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.059+0000 7f7d6fe00700 1 -- 172.21.15.116:0/3685312391 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7000cd40 con 0x7f7d78108c40 2024-03-30T00:57:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.059+0000 7f7d7e000700 1 -- 172.21.15.116:0/3685312391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d781a13c0 con 0x7f7d78108c40 2024-03-30T00:57:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.060+0000 7f7d7e000700 1 -- 172.21.15.116:0/3685312391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d78107a00 con 0x7f7d78108c40 2024-03-30T00:57:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.061+0000 7f7d6fe00700 1 -- 172.21.15.116:0/3685312391 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f7d70003a00 con 0x7f7d78108c40 2024-03-30T00:57:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.063+0000 7f7d6fe00700 1 --2- 172.21.15.116:0/3685312391 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7d5c0a1120 0x7f7d5c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.063+0000 7f7d6fe00700 1 -- 172.21.15.116:0/3685312391 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7d700cc7c0 con 0x7f7d78108c40 2024-03-30T00:57:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.063+0000 7f7d77400700 1 --2- 172.21.15.116:0/3685312391 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7d5c0a1120 0x7f7d5c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.064+0000 7f7d77400700 1 --2- 172.21.15.116:0/3685312391 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7d5c0a1120 0x7f7d5c0a35e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800b2e0 tx=0x7f7d6800b750 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:42.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.067+0000 7f7d6fe00700 1 -- 172.21.15.116:0/3685312391 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7d70095c70 con 0x7f7d78108c40 2024-03-30T00:57:42.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.277+0000 7f7d7e000700 1 -- 172.21.15.116:0/3685312391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7d7802ccb0 con 0x7f7d78108c40 2024-03-30T00:57:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.277+0000 7f7d6fe00700 1 -- 172.21.15.116:0/3685312391 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7d70099290 con 0x7f7d78108c40 2024-03-30T00:57:42.284 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-30T00:57:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.280+0000 7f7d7e000700 1 -- 172.21.15.116:0/3685312391 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7d5c0a1120 msgr2=0x7f7d5c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.280+0000 7f7d7e000700 1 --2- 172.21.15.116:0/3685312391 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7d5c0a1120 0x7f7d5c0a35e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800b2e0 tx=0x7f7d6800b750 comp rx=0 tx=0).stop 2024-03-30T00:57:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.280+0000 7f7d7e000700 1 -- 172.21.15.116:0/3685312391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d78108c40 msgr2=0x7f7d781a0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.280+0000 7f7d7e000700 1 --2- 172.21.15.116:0/3685312391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d78108c40 0x7f7d781a0420 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000efc0 tx=0x7f7d7000c3a0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.280+0000 7f7d7e000700 1 -- 172.21.15.116:0/3685312391 shutdown_connections 2024-03-30T00:57:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.280+0000 7f7d7e000700 1 --2- 172.21.15.116:0/3685312391 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f7d5c0a1120 0x7f7d5c0a35e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.280+0000 7f7d7e000700 1 --2- 172.21.15.116:0/3685312391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d78108c40 0x7f7d781a0420 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.280+0000 7f7d7e000700 1 --2- 172.21.15.116:0/3685312391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d78073810 0x7f7d78198f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.280+0000 7f7d7e000700 1 --2- 172.21.15.116:0/3685312391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d78074df0 0x7f7d78199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.280+0000 7f7d7e000700 1 -- 172.21.15.116:0/3685312391 >> 172.21.15.116:0/3685312391 conn(0x7f7d780fc2f0 msgr2=0x7f7d78104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:42.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.281+0000 7f7d7e000700 1 -- 172.21.15.116:0/3685312391 shutdown_connections 2024-03-30T00:57:42.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.281+0000 7f7d7e000700 1 -- 172.21.15.116:0/3685312391 wait complete. 2024-03-30T00:57:42.306 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T00:57:42.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.495+0000 7f0f20e00700 1 -- 172.21.15.116:0/2683897079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f1c101f70 msgr2=0x7f0f1c1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:42.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.495+0000 7f0f20e00700 1 --2- 172.21.15.116:0/2683897079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f1c101f70 0x7f0f1c1023d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009920 tx=0x7f0f08009c30 comp rx=0 tx=0).stop 2024-03-30T00:57:42.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.495+0000 7f0f20e00700 1 -- 172.21.15.116:0/2683897079 shutdown_connections 2024-03-30T00:57:42.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.495+0000 7f0f20e00700 1 --2- 172.21.15.116:0/2683897079 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f1c102910 0x7f0f1c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.495+0000 7f0f20e00700 1 --2- 172.21.15.116:0/2683897079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f1c100d80 0x7f0f1c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.495+0000 7f0f20e00700 1 --2- 172.21.15.116:0/2683897079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f1c101f70 0x7f0f1c1023d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.495+0000 7f0f20e00700 1 -- 172.21.15.116:0/2683897079 >> 172.21.15.116:0/2683897079 conn(0x7f0f1c0fc2f0 msgr2=0x7f0f1c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:42.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.495+0000 7f0f20e00700 1 -- 172.21.15.116:0/2683897079 shutdown_connections 2024-03-30T00:57:42.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.495+0000 7f0f20e00700 1 -- 172.21.15.116:0/2683897079 wait complete. 2024-03-30T00:57:42.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.496+0000 7f0f20e00700 1 Processor -- start 2024-03-30T00:57:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.496+0000 7f0f20e00700 1 -- start start 2024-03-30T00:57:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f20e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f1c100d80 0x7f0f1c199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f20e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f1c101f70 0x7f0f1c199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f20e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f1c102910 0x7f0f1c1a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f20e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c1a0b40 con 0x7f0f1c102910 2024-03-30T00:57:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f20e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c1a0c80 con 0x7f0f1c100d80 2024-03-30T00:57:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f20e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c1a0e10 con 0x7f0f1c101f70 2024-03-30T00:57:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f1a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f1c102910 0x7f0f1c1a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f1a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f1c102910 0x7f0f1c1a0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53186/0 (socket says 172.21.15.116:53186) 2024-03-30T00:57:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f1a800700 1 -- 172.21.15.116:0/1727532494 learned_addr learned my addr 172.21.15.116:0/1727532494 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f19e00700 1 --2- 172.21.15.116:0/1727532494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f1c100d80 0x7f0f1c199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f19400700 1 --2- 172.21.15.116:0/1727532494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f1c101f70 0x7f0f1c199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f1a800700 1 -- 172.21.15.116:0/1727532494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f1c101f70 msgr2=0x7f0f1c199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f1a800700 1 --2- 172.21.15.116:0/1727532494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f1c101f70 0x7f0f1c199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f1a800700 1 -- 172.21.15.116:0/1727532494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f1c100d80 msgr2=0x7f0f1c199010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f1a800700 1 --2- 172.21.15.116:0/1727532494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f1c100d80 0x7f0f1c199010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.497+0000 7f0f1a800700 1 -- 172.21.15.116:0/1727532494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f08009520 con 0x7f0f1c102910 2024-03-30T00:57:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.498+0000 7f0f1a800700 1 --2- 172.21.15.116:0/1727532494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f1c102910 0x7f0f1c1a0570 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400ef30 tx=0x7f0f1400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.498+0000 7f0f12a00700 1 -- 172.21.15.116:0/1727532494 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14018070 con 0x7f0f1c102910 2024-03-30T00:57:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.498+0000 7f0f20e00700 1 -- 172.21.15.116:0/1727532494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1c1a1090 con 0x7f0f1c102910 2024-03-30T00:57:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.498+0000 7f0f12a00700 1 -- 172.21.15.116:0/1727532494 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f1400f040 con 0x7f0f1c102910 2024-03-30T00:57:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.498+0000 7f0f12a00700 1 -- 172.21.15.116:0/1727532494 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1400ccf0 con 0x7f0f1c102910 2024-03-30T00:57:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.498+0000 7f0f20e00700 1 -- 172.21.15.116:0/1727532494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1c1a15a0 con 0x7f0f1c102910 2024-03-30T00:57:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.499+0000 7f0f20e00700 1 -- 172.21.15.116:0/1727532494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1c067af0 con 0x7f0f1c102910 2024-03-30T00:57:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.501+0000 7f0f12a00700 1 -- 172.21.15.116:0/1727532494 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f0f14003a00 con 0x7f0f1c102910 2024-03-30T00:57:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.502+0000 7f0f12a00700 1 --2- 172.21.15.116:0/1727532494 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0f000a1120 0x7f0f000a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.502+0000 7f0f12a00700 1 -- 172.21.15.116:0/1727532494 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0f140cd010 con 0x7f0f1c102910 2024-03-30T00:57:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.503+0000 7f0f19e00700 1 --2- 172.21.15.116:0/1727532494 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0f000a1120 0x7f0f000a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.503+0000 7f0f19e00700 1 --2- 172.21.15.116:0/1727532494 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0f000a1120 0x7f0f000a35e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c101dd0 tx=0x7f0f0c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:42.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.505+0000 7f0f12a00700 1 -- 172.21.15.116:0/1727532494 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0f14096520 con 0x7f0f1c102910 2024-03-30T00:57:42.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:42 smithi116 ceph-mon[38086]: pgmap v286: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-30T00:57:42.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:42 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2308959374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:42.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:42 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3685312391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:42.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.709+0000 7f0f20e00700 1 -- 172.21.15.116:0/1727532494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0f1c10c770 con 0x7f0f1c102910 2024-03-30T00:57:42.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.710+0000 7f0f12a00700 1 -- 172.21.15.116:0/1727532494 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v83) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0f14099b40 con 0x7f0f1c102910 2024-03-30T00:57:42.716 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:57:42.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.712+0000 7f0f20e00700 1 -- 172.21.15.116:0/1727532494 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0f000a1120 msgr2=0x7f0f000a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.712+0000 7f0f20e00700 1 --2- 172.21.15.116:0/1727532494 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0f000a1120 0x7f0f000a35e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c101dd0 tx=0x7f0f0c009310 comp rx=0 tx=0).stop 2024-03-30T00:57:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.712+0000 7f0f20e00700 1 -- 172.21.15.116:0/1727532494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f1c102910 msgr2=0x7f0f1c1a0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.712+0000 7f0f20e00700 1 --2- 172.21.15.116:0/1727532494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f1c102910 0x7f0f1c1a0570 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400ef30 tx=0x7f0f1400c3a0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.712+0000 7f0f20e00700 1 -- 172.21.15.116:0/1727532494 shutdown_connections 2024-03-30T00:57:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.712+0000 7f0f20e00700 1 --2- 172.21.15.116:0/1727532494 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0f000a1120 0x7f0f000a35e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.712+0000 7f0f20e00700 1 --2- 172.21.15.116:0/1727532494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f1c102910 0x7f0f1c1a0570 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.712+0000 7f0f20e00700 1 --2- 172.21.15.116:0/1727532494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f1c100d80 0x7f0f1c199010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.712+0000 7f0f20e00700 1 --2- 172.21.15.116:0/1727532494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f1c101f70 0x7f0f1c199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.712+0000 7f0f20e00700 1 -- 172.21.15.116:0/1727532494 >> 172.21.15.116:0/1727532494 conn(0x7f0f1c0fc2f0 msgr2=0x7f0f1c1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.712+0000 7f0f20e00700 1 -- 172.21.15.116:0/1727532494 shutdown_connections 2024-03-30T00:57:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.712+0000 7f0f20e00700 1 -- 172.21.15.116:0/1727532494 wait complete. 2024-03-30T00:57:42.738 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:57:42.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:42 smithi181 ceph-mon[47622]: pgmap v286: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-30T00:57:42.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:42 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2308959374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:42.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:42 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3685312391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:42.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:42 smithi130 ceph-mon[47626]: pgmap v286: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-30T00:57:42.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:42 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2308959374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:42.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:42 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3685312391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:42.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.924+0000 7fa22fa00700 1 -- 172.21.15.116:0/4176585104 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa228100c80 msgr2=0x7fa2281010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:42.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.924+0000 7fa22fa00700 1 --2- 172.21.15.116:0/4176585104 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa228100c80 0x7fa2281010a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa21c0098f0 tx=0x7fa21c009c00 comp rx=0 tx=0).stop 2024-03-30T00:57:42.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.924+0000 7fa22fa00700 1 -- 172.21.15.116:0/4176585104 shutdown_connections 2024-03-30T00:57:42.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.924+0000 7fa22fa00700 1 --2- 172.21.15.116:0/4176585104 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa228100c80 0x7fa2281010a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.924+0000 7fa22fa00700 1 --2- 172.21.15.116:0/4176585104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228102810 0x7fa2281090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.924+0000 7fa22fa00700 1 --2- 172.21.15.116:0/4176585104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa228101e70 0x7fa2281022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.924+0000 7fa22fa00700 1 -- 172.21.15.116:0/4176585104 >> 172.21.15.116:0/4176585104 conn(0x7fa2280fc1f0 msgr2=0x7fa2280fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:42.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.924+0000 7fa22fa00700 1 -- 172.21.15.116:0/4176585104 shutdown_connections 2024-03-30T00:57:42.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.924+0000 7fa22fa00700 1 -- 172.21.15.116:0/4176585104 wait complete. 2024-03-30T00:57:42.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.925+0000 7fa22fa00700 1 Processor -- start 2024-03-30T00:57:42.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.925+0000 7fa22fa00700 1 -- start start 2024-03-30T00:57:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.925+0000 7fa22fa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228100c80 0x7fa228074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.925+0000 7fa22fa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa228101e70 0x7fa2280731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22fa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa228102810 0x7fa2280736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22fa00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa228073e80 con 0x7fa228100c80 2024-03-30T00:57:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22fa00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa228073ff0 con 0x7fa228101e70 2024-03-30T00:57:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22fa00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2281a0ce0 con 0x7fa228102810 2024-03-30T00:57:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22d000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228100c80 0x7fa228074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22d000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228100c80 0x7fa228074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53200/0 (socket says 172.21.15.116:53200) 2024-03-30T00:57:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22d000700 1 -- 172.21.15.116:0/1690775821 learned_addr learned my addr 172.21.15.116:0/1690775821 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa227e00700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa228101e70 0x7fa2280731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22da00700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa228102810 0x7fa2280736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22d000700 1 -- 172.21.15.116:0/1690775821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa228102810 msgr2=0x7fa2280736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22d000700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa228102810 0x7fa2280736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22d000700 1 -- 172.21.15.116:0/1690775821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa228101e70 msgr2=0x7fa2280731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22d000700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa228101e70 0x7fa2280731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22d000700 1 -- 172.21.15.116:0/1690775821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c009520 con 0x7fa228100c80 2024-03-30T00:57:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa227e00700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa228101e70 0x7fa2280731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22da00700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa228102810 0x7fa2280736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T00:57:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.926+0000 7fa22d000700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228100c80 0x7fa228074b10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa21c009810 tx=0x7fa21c00b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.927+0000 7fa225600700 1 -- 172.21.15.116:0/1690775821 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c01c070 con 0x7fa228100c80 2024-03-30T00:57:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.927+0000 7fa225600700 1 -- 172.21.15.116:0/1690775821 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa21c005040 con 0x7fa228100c80 2024-03-30T00:57:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.927+0000 7fa22fa00700 1 -- 172.21.15.116:0/1690775821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2281a0f10 con 0x7fa228100c80 2024-03-30T00:57:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.927+0000 7fa225600700 1 -- 172.21.15.116:0/1690775821 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c003ac0 con 0x7fa228100c80 2024-03-30T00:57:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.927+0000 7fa22fa00700 1 -- 172.21.15.116:0/1690775821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2281a1280 con 0x7fa228100c80 2024-03-30T00:57:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.928+0000 7fa22fa00700 1 -- 172.21.15.116:0/1690775821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2281081a0 con 0x7fa228100c80 2024-03-30T00:57:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.930+0000 7fa225600700 1 -- 172.21.15.116:0/1690775821 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fa21c020030 con 0x7fa228100c80 2024-03-30T00:57:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.931+0000 7fa225600700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa2100a1120 0x7fa2100a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.931+0000 7fa225600700 1 -- 172.21.15.116:0/1690775821 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa21c0cdd00 con 0x7fa228100c80 2024-03-30T00:57:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.931+0000 7fa227e00700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa2100a1120 0x7fa2100a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.932+0000 7fa227e00700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa2100a1120 0x7fa2100a35e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa214009cb0 tx=0x7fa214009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:42.935+0000 7fa225600700 1 -- 172.21.15.116:0/1690775821 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa21c0971b0 con 0x7fa228100c80 2024-03-30T00:57:43.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.177+0000 7fa22fa00700 1 -- 172.21.15.116:0/1690775821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa228067a10 con 0x7fa228100c80 2024-03-30T00:57:43.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.179+0000 7fa225600700 1 -- 172.21.15.116:0/1690775821 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+24556 (secure 0 0 0) 0x7fa21c09a7d0 con 0x7fa228100c80 2024-03-30T00:57:43.187 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:43.187 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:56:36.456069+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:57:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.181+0000 7fa22fa00700 1 -- 172.21.15.116:0/1690775821 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa2100a1120 msgr2=0x7fa2100a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.181+0000 7fa22fa00700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa2100a1120 0x7fa2100a35e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa214009cb0 tx=0x7fa214009260 comp rx=0 tx=0).stop 2024-03-30T00:57:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.181+0000 7fa22fa00700 1 -- 172.21.15.116:0/1690775821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228100c80 msgr2=0x7fa228074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.181+0000 7fa22fa00700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228100c80 0x7fa228074b10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa21c009810 tx=0x7fa21c00b560 comp rx=0 tx=0).stop 2024-03-30T00:57:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.181+0000 7fa22fa00700 1 -- 172.21.15.116:0/1690775821 shutdown_connections 2024-03-30T00:57:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.181+0000 7fa22fa00700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa2100a1120 0x7fa2100a35e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.181+0000 7fa22fa00700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa228102810 0x7fa2280736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.181+0000 7fa22fa00700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228100c80 0x7fa228074b10 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.181+0000 7fa22fa00700 1 --2- 172.21.15.116:0/1690775821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa228101e70 0x7fa2280731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.181+0000 7fa22fa00700 1 -- 172.21.15.116:0/1690775821 >> 172.21.15.116:0/1690775821 conn(0x7fa2280fc1f0 msgr2=0x7fa2281052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.181+0000 7fa22fa00700 1 -- 172.21.15.116:0/1690775821 shutdown_connections 2024-03-30T00:57:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.181+0000 7fa22fa00700 1 -- 172.21.15.116:0/1690775821 wait complete. 2024-03-30T00:57:43.207 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T00:52:29.602428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T00:55:50.220543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T00:55:51.274498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T00:55:54.301591+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T00:57:43.208 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T00:57:43.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.400+0000 7fc029800700 1 -- 172.21.15.116:0/952609364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc024102910 msgr2=0x7fc0241091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:43.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.400+0000 7fc029800700 1 --2- 172.21.15.116:0/952609364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc024102910 0x7fc0241091c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc01c00b0d0 tx=0x7fc01c00b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.400+0000 7fc029800700 1 -- 172.21.15.116:0/952609364 shutdown_connections 2024-03-30T00:57:43.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.400+0000 7fc029800700 1 --2- 172.21.15.116:0/952609364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc024100d80 0x7fc0241011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.400+0000 7fc029800700 1 --2- 172.21.15.116:0/952609364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc024102910 0x7fc0241091c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.400+0000 7fc029800700 1 --2- 172.21.15.116:0/952609364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc024101f70 0x7fc0241023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.400+0000 7fc029800700 1 -- 172.21.15.116:0/952609364 >> 172.21.15.116:0/952609364 conn(0x7fc0240fc2f0 msgr2=0x7fc0240fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:43.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.400+0000 7fc029800700 1 -- 172.21.15.116:0/952609364 shutdown_connections 2024-03-30T00:57:43.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.400+0000 7fc029800700 1 -- 172.21.15.116:0/952609364 wait complete. 2024-03-30T00:57:43.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.401+0000 7fc029800700 1 Processor -- start 2024-03-30T00:57:43.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.402+0000 7fc029800700 1 -- start start 2024-03-30T00:57:43.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.402+0000 7fc029800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc024100d80 0x7fc024074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:43.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.402+0000 7fc029800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc024101f70 0x7fc0240731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:43.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.402+0000 7fc029800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc024102910 0x7fc0240736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:43.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.402+0000 7fc029800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc024073e80 con 0x7fc024101f70 2024-03-30T00:57:43.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.402+0000 7fc029800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc024073ff0 con 0x7fc024100d80 2024-03-30T00:57:43.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.402+0000 7fc029800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0241a0e60 con 0x7fc024102910 2024-03-30T00:57:43.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.402+0000 7fc022000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc024101f70 0x7fc0240731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:43.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.402+0000 7fc022000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc024101f70 0x7fc0240731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53210/0 (socket says 172.21.15.116:53210) 2024-03-30T00:57:43.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.402+0000 7fc022000700 1 -- 172.21.15.116:0/1311635324 learned_addr learned my addr 172.21.15.116:0/1311635324 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:43.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.402+0000 7fc023400700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc024102910 0x7fc0240736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:43.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.402+0000 7fc022a00700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc024100d80 0x7fc024074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:43.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.403+0000 7fc022000700 1 -- 172.21.15.116:0/1311635324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc024102910 msgr2=0x7fc0240736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:43.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.403+0000 7fc022000700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc024102910 0x7fc0240736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.403+0000 7fc022000700 1 -- 172.21.15.116:0/1311635324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc024100d80 msgr2=0x7fc024074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:43.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.403+0000 7fc022000700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc024100d80 0x7fc024074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.403+0000 7fc022000700 1 -- 172.21.15.116:0/1311635324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c009d00 con 0x7fc024101f70 2024-03-30T00:57:43.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.403+0000 7fc022a00700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc024100d80 0x7fc024074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:43.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.403+0000 7fc022000700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc024101f70 0x7fc0240731a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc01000e9a0 tx=0x7fc01000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:43.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.403+0000 7fc01b400700 1 -- 172.21.15.116:0/1311635324 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc010009790 con 0x7fc024101f70 2024-03-30T00:57:43.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.403+0000 7fc01b400700 1 -- 172.21.15.116:0/1311635324 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc01000cb70 con 0x7fc024101f70 2024-03-30T00:57:43.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.403+0000 7fc01b400700 1 -- 172.21.15.116:0/1311635324 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc010010ea0 con 0x7fc024101f70 2024-03-30T00:57:43.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.403+0000 7fc029800700 1 -- 172.21.15.116:0/1311635324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0241a10f0 con 0x7fc024101f70 2024-03-30T00:57:43.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.403+0000 7fc029800700 1 -- 172.21.15.116:0/1311635324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0241a15b0 con 0x7fc024101f70 2024-03-30T00:57:43.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.404+0000 7fc029800700 1 -- 172.21.15.116:0/1311635324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc024108130 con 0x7fc024101f70 2024-03-30T00:57:43.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.406+0000 7fc01b400700 1 -- 172.21.15.116:0/1311635324 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fc01000cce0 con 0x7fc024101f70 2024-03-30T00:57:43.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.407+0000 7fc01b400700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc0080a1120 0x7fc0080a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:43.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.407+0000 7fc01b400700 1 -- 172.21.15.116:0/1311635324 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc010014070 con 0x7fc024101f70 2024-03-30T00:57:43.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.408+0000 7fc022a00700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc0080a1120 0x7fc0080a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:43.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.408+0000 7fc022a00700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc0080a1120 0x7fc0080a35e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc024101dd0 tx=0x7fc014009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:43.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.411+0000 7fc01b400700 1 -- 172.21.15.116:0/1311635324 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc0100529f0 con 0x7fc024101f70 2024-03-30T00:57:43.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.618+0000 7fc029800700 1 -- 172.21.15.116:0/1311635324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc02410c870 con 0x7fc024101f70 2024-03-30T00:57:43.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.619+0000 7fc01b400700 1 -- 172.21.15.116:0/1311635324 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc0100962c0 con 0x7fc024101f70 2024-03-30T00:57:43.626 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-30T00:57:43.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.621+0000 7fc029800700 1 -- 172.21.15.116:0/1311635324 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc0080a1120 msgr2=0x7fc0080a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:43.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.621+0000 7fc029800700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc0080a1120 0x7fc0080a35e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc024101dd0 tx=0x7fc014009260 comp rx=0 tx=0).stop 2024-03-30T00:57:43.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.621+0000 7fc029800700 1 -- 172.21.15.116:0/1311635324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc024101f70 msgr2=0x7fc0240731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:43.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.621+0000 7fc029800700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc024101f70 0x7fc0240731a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc01000e9a0 tx=0x7fc01000ecb0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.621+0000 7fc029800700 1 -- 172.21.15.116:0/1311635324 shutdown_connections 2024-03-30T00:57:43.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.621+0000 7fc029800700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc0080a1120 0x7fc0080a35e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.621+0000 7fc029800700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc024102910 0x7fc0240736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.621+0000 7fc029800700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc024100d80 0x7fc024074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.621+0000 7fc029800700 1 --2- 172.21.15.116:0/1311635324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc024101f70 0x7fc0240731a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.621+0000 7fc029800700 1 -- 172.21.15.116:0/1311635324 >> 172.21.15.116:0/1311635324 conn(0x7fc0240fc2f0 msgr2=0x7fc024105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:43.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.621+0000 7fc029800700 1 -- 172.21.15.116:0/1311635324 shutdown_connections 2024-03-30T00:57:43.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.621+0000 7fc029800700 1 -- 172.21.15.116:0/1311635324 wait complete. 2024-03-30T00:57:43.647 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T00:57:43.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:43 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1727532494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:43.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:43 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1690775821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:43.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:43 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1727532494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:43.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:43 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1690775821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:43.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.840+0000 7f5541600700 1 -- 172.21.15.116:0/3132391838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553c101f70 msgr2=0x7f553c1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:43.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.840+0000 7f5541600700 1 --2- 172.21.15.116:0/3132391838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553c101f70 0x7f553c1023d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f55280099c0 tx=0x7f5528009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.840+0000 7f5541600700 1 -- 172.21.15.116:0/3132391838 shutdown_connections 2024-03-30T00:57:43.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.840+0000 7f5541600700 1 --2- 172.21.15.116:0/3132391838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f553c102910 0x7f553c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.840+0000 7f5541600700 1 --2- 172.21.15.116:0/3132391838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f553c100d80 0x7f553c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.840+0000 7f5541600700 1 --2- 172.21.15.116:0/3132391838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553c101f70 0x7f553c1023d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.840+0000 7f5541600700 1 -- 172.21.15.116:0/3132391838 >> 172.21.15.116:0/3132391838 conn(0x7f553c0fc2f0 msgr2=0x7f553c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:43.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.841+0000 7f5541600700 1 -- 172.21.15.116:0/3132391838 shutdown_connections 2024-03-30T00:57:43.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.841+0000 7f5541600700 1 -- 172.21.15.116:0/3132391838 wait complete. 2024-03-30T00:57:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.842+0000 7f5541600700 1 Processor -- start 2024-03-30T00:57:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.842+0000 7f5541600700 1 -- start start 2024-03-30T00:57:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.842+0000 7f5541600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f553c100d80 0x7f553c198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.842+0000 7f5541600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553c101f70 0x7f553c199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.842+0000 7f5541600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f553c102910 0x7f553c1a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.842+0000 7f5541600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c1a0a20 con 0x7f553c101f70 2024-03-30T00:57:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.842+0000 7f5541600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c1a0b60 con 0x7f553c102910 2024-03-30T00:57:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.842+0000 7f5541600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c1a0cf0 con 0x7f553c100d80 2024-03-30T00:57:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.842+0000 7f553aa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f553c100d80 0x7f553c198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.842+0000 7f553a000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553c101f70 0x7f553c199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.842+0000 7f553aa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f553c100d80 0x7f553c198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:43344/0 (socket says 172.21.15.116:43344) 2024-03-30T00:57:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.842+0000 7f553aa00700 1 -- 172.21.15.116:0/4205100762 learned_addr learned my addr 172.21.15.116:0/4205100762 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.842+0000 7f553b400700 1 --2- 172.21.15.116:0/4205100762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f553c102910 0x7f553c1a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.843+0000 7f553aa00700 1 -- 172.21.15.116:0/4205100762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f553c102910 msgr2=0x7f553c1a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.843+0000 7f553aa00700 1 --2- 172.21.15.116:0/4205100762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f553c102910 0x7f553c1a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.843+0000 7f553aa00700 1 -- 172.21.15.116:0/4205100762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553c101f70 msgr2=0x7f553c199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.843+0000 7f553aa00700 1 --2- 172.21.15.116:0/4205100762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553c101f70 0x7f553c199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.843+0000 7f553aa00700 1 -- 172.21.15.116:0/4205100762 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55280095f0 con 0x7f553c100d80 2024-03-30T00:57:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.843+0000 7f553a000700 1 --2- 172.21.15.116:0/4205100762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553c101f70 0x7f553c199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.843+0000 7f553aa00700 1 --2- 172.21.15.116:0/4205100762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f553c100d80 0x7f553c198ef0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f552c00e970 tx=0x7f552c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.843+0000 7f5533400700 1 -- 172.21.15.116:0/4205100762 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c009790 con 0x7f553c100d80 2024-03-30T00:57:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.843+0000 7f5541600700 1 -- 172.21.15.116:0/4205100762 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f553c1a0fd0 con 0x7f553c100d80 2024-03-30T00:57:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.843+0000 7f5533400700 1 -- 172.21.15.116:0/4205100762 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f552c00c7e0 con 0x7f553c100d80 2024-03-30T00:57:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.843+0000 7f5533400700 1 -- 172.21.15.116:0/4205100762 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c003c70 con 0x7f553c100d80 2024-03-30T00:57:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.844+0000 7f5541600700 1 -- 172.21.15.116:0/4205100762 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f553c1a1400 con 0x7f553c100d80 2024-03-30T00:57:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.844+0000 7f5541600700 1 -- 172.21.15.116:0/4205100762 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f553c067af0 con 0x7f553c100d80 2024-03-30T00:57:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.847+0000 7f5533400700 1 -- 172.21.15.116:0/4205100762 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f552c00cc50 con 0x7f553c100d80 2024-03-30T00:57:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.848+0000 7f5533400700 1 --2- 172.21.15.116:0/4205100762 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f55200a1120 0x7f55200a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.848+0000 7f5533400700 1 -- 172.21.15.116:0/4205100762 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f552c014070 con 0x7f553c100d80 2024-03-30T00:57:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.848+0000 7f553a000700 1 --2- 172.21.15.116:0/4205100762 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f55200a1120 0x7f55200a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.849+0000 7f553a000700 1 --2- 172.21.15.116:0/4205100762 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f55200a1120 0x7f55200a35e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5528006010 tx=0x7f552800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:43.850+0000 7f5533400700 1 -- 172.21.15.116:0/4205100762 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f552c096350 con 0x7f553c100d80 2024-03-30T00:57:43.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:43 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1727532494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:43.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:43 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1690775821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:44.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.056+0000 7f5541600700 1 -- 172.21.15.116:0/4205100762 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f553c10c790 con 0x7f553c100d80 2024-03-30T00:57:44.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.057+0000 7f5533400700 1 -- 172.21.15.116:0/4205100762 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f552c003dd0 con 0x7f553c100d80 2024-03-30T00:57:44.064 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:57:44.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.059+0000 7f5541600700 1 -- 172.21.15.116:0/4205100762 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f55200a1120 msgr2=0x7f55200a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:44.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.059+0000 7f5541600700 1 --2- 172.21.15.116:0/4205100762 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f55200a1120 0x7f55200a35e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5528006010 tx=0x7f552800b540 comp rx=0 tx=0).stop 2024-03-30T00:57:44.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.059+0000 7f5541600700 1 -- 172.21.15.116:0/4205100762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f553c100d80 msgr2=0x7f553c198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:44.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.059+0000 7f5541600700 1 --2- 172.21.15.116:0/4205100762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f553c100d80 0x7f553c198ef0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f552c00e970 tx=0x7f552c00ec80 comp rx=0 tx=0).stop 2024-03-30T00:57:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.060+0000 7f5541600700 1 -- 172.21.15.116:0/4205100762 shutdown_connections 2024-03-30T00:57:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.060+0000 7f5541600700 1 --2- 172.21.15.116:0/4205100762 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f55200a1120 0x7f55200a35e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.060+0000 7f5541600700 1 --2- 172.21.15.116:0/4205100762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f553c100d80 0x7f553c198ef0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.060+0000 7f5541600700 1 --2- 172.21.15.116:0/4205100762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f553c102910 0x7f553c1a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.060+0000 7f5541600700 1 --2- 172.21.15.116:0/4205100762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553c101f70 0x7f553c199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.060+0000 7f5541600700 1 -- 172.21.15.116:0/4205100762 >> 172.21.15.116:0/4205100762 conn(0x7f553c0fc2f0 msgr2=0x7f553c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.060+0000 7f5541600700 1 -- 172.21.15.116:0/4205100762 shutdown_connections 2024-03-30T00:57:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.060+0000 7f5541600700 1 -- 172.21.15.116:0/4205100762 wait complete. 2024-03-30T00:57:44.086 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T00:57:44.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.278+0000 7f08f8600700 1 -- 172.21.15.116:0/3701831184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08f0102910 msgr2=0x7f08f01091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:44.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.278+0000 7f08f8600700 1 --2- 172.21.15.116:0/3701831184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08f0102910 0x7f08f01091c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f08e800b0d0 tx=0x7f08e800b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.278+0000 7f08f8600700 1 -- 172.21.15.116:0/3701831184 shutdown_connections 2024-03-30T00:57:44.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.278+0000 7f08f8600700 1 --2- 172.21.15.116:0/3701831184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08f0102910 0x7f08f01091c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.278+0000 7f08f8600700 1 --2- 172.21.15.116:0/3701831184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08f0100d80 0x7f08f01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.278+0000 7f08f8600700 1 --2- 172.21.15.116:0/3701831184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08f0101f70 0x7f08f01023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.278+0000 7f08f8600700 1 -- 172.21.15.116:0/3701831184 >> 172.21.15.116:0/3701831184 conn(0x7f08f00fc2f0 msgr2=0x7f08f00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:44.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.278+0000 7f08f8600700 1 -- 172.21.15.116:0/3701831184 shutdown_connections 2024-03-30T00:57:44.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.278+0000 7f08f8600700 1 -- 172.21.15.116:0/3701831184 wait complete. 2024-03-30T00:57:44.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.279+0000 7f08f8600700 1 Processor -- start 2024-03-30T00:57:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.279+0000 7f08f8600700 1 -- start start 2024-03-30T00:57:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.279+0000 7f08f8600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08f0100d80 0x7f08f0198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.279+0000 7f08f8600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08f0101f70 0x7f08f0199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.279+0000 7f08f8600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08f0102910 0x7f08f01a0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.279+0000 7f08f8600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f01a0b20 con 0x7f08f0102910 2024-03-30T00:57:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.279+0000 7f08f8600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f01a0c60 con 0x7f08f0100d80 2024-03-30T00:57:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.279+0000 7f08f8600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f01a0df0 con 0x7f08f0101f70 2024-03-30T00:57:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08f6600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08f0102910 0x7f08f01a0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08f5c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08f0100d80 0x7f08f0198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08f5200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08f0101f70 0x7f08f0199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08f5c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08f0100d80 0x7f08f0198ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:52662/0 (socket says 172.21.15.116:52662) 2024-03-30T00:57:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08f5c00700 1 -- 172.21.15.116:0/2499373540 learned_addr learned my addr 172.21.15.116:0/2499373540 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08f6600700 1 -- 172.21.15.116:0/2499373540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08f0101f70 msgr2=0x7f08f0199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08f6600700 1 --2- 172.21.15.116:0/2499373540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08f0101f70 0x7f08f0199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08f6600700 1 -- 172.21.15.116:0/2499373540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08f0100d80 msgr2=0x7f08f0198ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08f6600700 1 --2- 172.21.15.116:0/2499373540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08f0100d80 0x7f08f0198ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08f6600700 1 -- 172.21.15.116:0/2499373540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e8009d00 con 0x7f08f0102910 2024-03-30T00:57:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08f6600700 1 --2- 172.21.15.116:0/2499373540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08f0102910 0x7f08f01a0550 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f08e8007b60 tx=0x7f08e8008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08ee000700 1 -- 172.21.15.116:0/2499373540 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e800e040 con 0x7f08f0102910 2024-03-30T00:57:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08ee000700 1 -- 172.21.15.116:0/2499373540 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08e8005070 con 0x7f08f0102910 2024-03-30T00:57:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08ee000700 1 -- 172.21.15.116:0/2499373540 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e801d6d0 con 0x7f08f0102910 2024-03-30T00:57:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.280+0000 7f08f8600700 1 -- 172.21.15.116:0/2499373540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f01a1070 con 0x7f08f0102910 2024-03-30T00:57:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.281+0000 7f08f8600700 1 -- 172.21.15.116:0/2499373540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f01a1440 con 0x7f08f0102910 2024-03-30T00:57:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.281+0000 7f08f8600700 1 -- 172.21.15.116:0/2499373540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f0067af0 con 0x7f08f0102910 2024-03-30T00:57:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.283+0000 7f08ee000700 1 -- 172.21.15.116:0/2499373540 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f08e80051e0 con 0x7f08f0102910 2024-03-30T00:57:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.285+0000 7f08ee000700 1 --2- 172.21.15.116:0/2499373540 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f08d80a11a0 0x7f08d80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.285+0000 7f08ee000700 1 -- 172.21.15.116:0/2499373540 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f08e80ce080 con 0x7f08f0102910 2024-03-30T00:57:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.285+0000 7f08f5c00700 1 --2- 172.21.15.116:0/2499373540 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f08d80a11a0 0x7f08d80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:44.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.286+0000 7f08f5c00700 1 --2- 172.21.15.116:0/2499373540 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f08d80a11a0 0x7f08d80a3660 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f08e4005950 tx=0x7f08e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.287+0000 7f08ee000700 1 -- 172.21.15.116:0/2499373540 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f08e8097530 con 0x7f08f0102910 2024-03-30T00:57:44.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.496+0000 7f08f8600700 1 -- 172.21.15.116:0/2499373540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f08f010c820 con 0x7f08f0102910 2024-03-30T00:57:44.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.496+0000 7f08ee000700 1 -- 172.21.15.116:0/2499373540 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f08e809ab50 con 0x7f08f0102910 2024-03-30T00:57:44.503 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-30T00:57:44.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.498+0000 7f08f8600700 1 -- 172.21.15.116:0/2499373540 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f08d80a11a0 msgr2=0x7f08d80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:44.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.498+0000 7f08f8600700 1 --2- 172.21.15.116:0/2499373540 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f08d80a11a0 0x7f08d80a3660 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f08e4005950 tx=0x7f08e4009260 comp rx=0 tx=0).stop 2024-03-30T00:57:44.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.498+0000 7f08f8600700 1 -- 172.21.15.116:0/2499373540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08f0102910 msgr2=0x7f08f01a0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:44.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.498+0000 7f08f8600700 1 --2- 172.21.15.116:0/2499373540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08f0102910 0x7f08f01a0550 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f08e8007b60 tx=0x7f08e8008e40 comp rx=0 tx=0).stop 2024-03-30T00:57:44.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.499+0000 7f08f8600700 1 -- 172.21.15.116:0/2499373540 shutdown_connections 2024-03-30T00:57:44.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.499+0000 7f08f8600700 1 --2- 172.21.15.116:0/2499373540 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f08d80a11a0 0x7f08d80a3660 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.499+0000 7f08f8600700 1 --2- 172.21.15.116:0/2499373540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08f0102910 0x7f08f01a0550 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.499+0000 7f08f8600700 1 --2- 172.21.15.116:0/2499373540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08f0100d80 0x7f08f0198ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.499+0000 7f08f8600700 1 --2- 172.21.15.116:0/2499373540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08f0101f70 0x7f08f0199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.499+0000 7f08f8600700 1 -- 172.21.15.116:0/2499373540 >> 172.21.15.116:0/2499373540 conn(0x7f08f00fc2f0 msgr2=0x7f08f0072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:44.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.499+0000 7f08f8600700 1 -- 172.21.15.116:0/2499373540 shutdown_connections 2024-03-30T00:57:44.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.499+0000 7f08f8600700 1 -- 172.21.15.116:0/2499373540 wait complete. 2024-03-30T00:57:44.525 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T00:57:44.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.713+0000 7f5b03200700 1 -- 172.21.15.116:0/1566487246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc074df0 msgr2=0x7f5afc073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.713+0000 7f5b03200700 1 --2- 172.21.15.116:0/1566487246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc074df0 0x7f5afc073240 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5aec009990 tx=0x7f5aec009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.713+0000 7f5b03200700 1 -- 172.21.15.116:0/1566487246 shutdown_connections 2024-03-30T00:57:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.713+0000 7f5b03200700 1 --2- 172.21.15.116:0/1566487246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5afc108c40 0x7f5afc10b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.713+0000 7f5b03200700 1 --2- 172.21.15.116:0/1566487246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc074df0 0x7f5afc073240 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.713+0000 7f5b03200700 1 --2- 172.21.15.116:0/1566487246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5afc073810 0x7f5afc073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.713+0000 7f5b03200700 1 -- 172.21.15.116:0/1566487246 >> 172.21.15.116:0/1566487246 conn(0x7f5afc0fc2f0 msgr2=0x7f5afc0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.713+0000 7f5b03200700 1 -- 172.21.15.116:0/1566487246 shutdown_connections 2024-03-30T00:57:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.714+0000 7f5b03200700 1 -- 172.21.15.116:0/1566487246 wait complete. 2024-03-30T00:57:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.714+0000 7f5b03200700 1 Processor -- start 2024-03-30T00:57:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.715+0000 7f5b03200700 1 -- start start 2024-03-30T00:57:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.715+0000 7f5b03200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5afc073810 0x7f5afc198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.715+0000 7f5b03200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5afc074df0 0x7f5afc1994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.715+0000 7f5b03200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc108c40 0x7f5afc1a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.715+0000 7f5b03200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc1a0a60 con 0x7f5afc108c40 2024-03-30T00:57:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.715+0000 7f5b03200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc1a0ba0 con 0x7f5afc073810 2024-03-30T00:57:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.715+0000 7f5b03200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc1a0d30 con 0x7f5afc074df0 2024-03-30T00:57:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.715+0000 7f5b01200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc108c40 0x7f5afc1a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.715+0000 7f5b01200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc108c40 0x7f5afc1a0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53262/0 (socket says 172.21.15.116:53262) 2024-03-30T00:57:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.715+0000 7f5afb400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5afc074df0 0x7f5afc1994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.715+0000 7f5b01200700 1 -- 172.21.15.116:0/2064642982 learned_addr learned my addr 172.21.15.116:0/2064642982 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.715+0000 7f5afbe00700 1 --2- 172.21.15.116:0/2064642982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5afc073810 0x7f5afc198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.716+0000 7f5b01200700 1 -- 172.21.15.116:0/2064642982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5afc074df0 msgr2=0x7f5afc1994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.716+0000 7f5b01200700 1 --2- 172.21.15.116:0/2064642982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5afc074df0 0x7f5afc1994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.716+0000 7f5b01200700 1 -- 172.21.15.116:0/2064642982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5afc073810 msgr2=0x7f5afc198f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.716+0000 7f5b01200700 1 --2- 172.21.15.116:0/2064642982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5afc073810 0x7f5afc198f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.716+0000 7f5b01200700 1 -- 172.21.15.116:0/2064642982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec0095f0 con 0x7f5afc108c40 2024-03-30T00:57:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.716+0000 7f5b01200700 1 --2- 172.21.15.116:0/2064642982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc108c40 0x7f5afc1a0490 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5af4007ee0 tx=0x7f5af400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.716+0000 7f5af8c00700 1 -- 172.21.15.116:0/2064642982 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af4017070 con 0x7f5afc108c40 2024-03-30T00:57:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.716+0000 7f5af8c00700 1 -- 172.21.15.116:0/2064642982 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5af400f040 con 0x7f5afc108c40 2024-03-30T00:57:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.716+0000 7f5b03200700 1 -- 172.21.15.116:0/2064642982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc1a0fb0 con 0x7f5afc108c40 2024-03-30T00:57:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.716+0000 7f5af8c00700 1 -- 172.21.15.116:0/2064642982 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af4005550 con 0x7f5afc108c40 2024-03-30T00:57:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.716+0000 7f5b03200700 1 -- 172.21.15.116:0/2064642982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc1a1490 con 0x7f5afc108c40 2024-03-30T00:57:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.717+0000 7f5b03200700 1 -- 172.21.15.116:0/2064642982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5afc107a00 con 0x7f5afc108c40 2024-03-30T00:57:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.719+0000 7f5af8c00700 1 -- 172.21.15.116:0/2064642982 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f5af400db20 con 0x7f5afc108c40 2024-03-30T00:57:44.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.721+0000 7f5af8c00700 1 --2- 172.21.15.116:0/2064642982 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ae00a1120 0x7f5ae00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:44.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.721+0000 7f5af8c00700 1 -- 172.21.15.116:0/2064642982 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5af40cc2d0 con 0x7f5afc108c40 2024-03-30T00:57:44.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.721+0000 7f5afbe00700 1 --2- 172.21.15.116:0/2064642982 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ae00a1120 0x7f5ae00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:44.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.722+0000 7f5afbe00700 1 --2- 172.21.15.116:0/2064642982 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ae00a1120 0x7f5ae00a35e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00b2e0 tx=0x7f5aec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:44.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.724+0000 7f5af8c00700 1 -- 172.21.15.116:0/2064642982 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5af4095750 con 0x7f5afc108c40 2024-03-30T00:57:44.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:44 smithi181 ceph-mon[47622]: pgmap v287: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-30T00:57:44.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:44 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1311635324' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:44.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:44 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/4205100762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:44.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:44 smithi130 ceph-mon[47626]: pgmap v287: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-30T00:57:44.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:44 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1311635324' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:44.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:44 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/4205100762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:44.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:44 smithi116 ceph-mon[38086]: pgmap v287: 129 pgs: 129 active+clean; 641 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-30T00:57:44.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:44 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1311635324' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:44.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:44 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/4205100762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:44.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.930+0000 7f5b03200700 1 -- 172.21.15.116:0/2064642982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5afc02cd70 con 0x7f5afc108c40 2024-03-30T00:57:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.930+0000 7f5af8c00700 1 -- 172.21.15.116:0/2064642982 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v83) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5af4098d70 con 0x7f5afc108c40 2024-03-30T00:57:44.937 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:57:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.933+0000 7f5b03200700 1 -- 172.21.15.116:0/2064642982 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ae00a1120 msgr2=0x7f5ae00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.933+0000 7f5b03200700 1 --2- 172.21.15.116:0/2064642982 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ae00a1120 0x7f5ae00a35e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00b2e0 tx=0x7f5aec005fd0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.933+0000 7f5b03200700 1 -- 172.21.15.116:0/2064642982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc108c40 msgr2=0x7f5afc1a0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.933+0000 7f5b03200700 1 --2- 172.21.15.116:0/2064642982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc108c40 0x7f5afc1a0490 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5af4007ee0 tx=0x7f5af400d3b0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.933+0000 7f5b03200700 1 -- 172.21.15.116:0/2064642982 shutdown_connections 2024-03-30T00:57:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.933+0000 7f5b03200700 1 --2- 172.21.15.116:0/2064642982 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ae00a1120 0x7f5ae00a35e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.933+0000 7f5b03200700 1 --2- 172.21.15.116:0/2064642982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc108c40 0x7f5afc1a0490 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.933+0000 7f5b03200700 1 --2- 172.21.15.116:0/2064642982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5afc073810 0x7f5afc198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.933+0000 7f5b03200700 1 --2- 172.21.15.116:0/2064642982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5afc074df0 0x7f5afc1994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.933+0000 7f5b03200700 1 -- 172.21.15.116:0/2064642982 >> 172.21.15.116:0/2064642982 conn(0x7f5afc0fc2f0 msgr2=0x7f5afc104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.933+0000 7f5b03200700 1 -- 172.21.15.116:0/2064642982 shutdown_connections 2024-03-30T00:57:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:44.933+0000 7f5b03200700 1 -- 172.21.15.116:0/2064642982 wait complete. 2024-03-30T00:57:44.959 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-30T00:57:44.959 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:57:45.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.147+0000 7fb802c00700 1 -- 172.21.15.116:0/2490166236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7fc102810 msgr2=0x7fb7fc1090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:45.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.147+0000 7fb802c00700 1 --2- 172.21.15.116:0/2490166236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7fc102810 0x7fb7fc1090c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb7f400b0a0 tx=0x7fb7f400b3b0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.147+0000 7fb802c00700 1 -- 172.21.15.116:0/2490166236 shutdown_connections 2024-03-30T00:57:45.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.147+0000 7fb802c00700 1 --2- 172.21.15.116:0/2490166236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7fc102810 0x7fb7fc1090c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.147+0000 7fb802c00700 1 --2- 172.21.15.116:0/2490166236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7fc100c80 0x7fb7fc1010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.147+0000 7fb802c00700 1 --2- 172.21.15.116:0/2490166236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7fc101e70 0x7fb7fc1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.147+0000 7fb802c00700 1 -- 172.21.15.116:0/2490166236 >> 172.21.15.116:0/2490166236 conn(0x7fb7fc0fc1f0 msgr2=0x7fb7fc0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:45.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.147+0000 7fb802c00700 1 -- 172.21.15.116:0/2490166236 shutdown_connections 2024-03-30T00:57:45.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.147+0000 7fb802c00700 1 -- 172.21.15.116:0/2490166236 wait complete. 2024-03-30T00:57:45.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.148+0000 7fb802c00700 1 Processor -- start 2024-03-30T00:57:45.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.148+0000 7fb802c00700 1 -- start start 2024-03-30T00:57:45.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.148+0000 7fb802c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7fc100c80 0x7fb7fc196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:45.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.148+0000 7fb802c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7fc101e70 0x7fb7fc1972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:45.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.148+0000 7fb802c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7fc102810 0x7fb7fc19e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:45.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.148+0000 7fb802c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc19e860 con 0x7fb7fc101e70 2024-03-30T00:57:45.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.148+0000 7fb802c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc19e9d0 con 0x7fb7fc100c80 2024-03-30T00:57:45.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.148+0000 7fb802c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc19eb90 con 0x7fb7fc102810 2024-03-30T00:57:45.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.148+0000 7fb7fb400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7fc101e70 0x7fb7fc1972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:45.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.148+0000 7fb7fb400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7fc101e70 0x7fb7fc1972b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58498/0 (socket says 172.21.15.116:58498) 2024-03-30T00:57:45.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.149+0000 7fb7fb400700 1 -- 172.21.15.116:0/1156078971 learned_addr learned my addr 172.21.15.116:0/1156078971 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:45.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.149+0000 7fb7fbe00700 1 --2- 172.21.15.116:0/1156078971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7fc100c80 0x7fb7fc196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:45.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.149+0000 7fb800c00700 1 --2- 172.21.15.116:0/1156078971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7fc102810 0x7fb7fc19e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:45.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.149+0000 7fb7fb400700 1 -- 172.21.15.116:0/1156078971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7fc102810 msgr2=0x7fb7fc19e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:45.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.149+0000 7fb7fb400700 1 --2- 172.21.15.116:0/1156078971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7fc102810 0x7fb7fc19e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.149+0000 7fb7fb400700 1 -- 172.21.15.116:0/1156078971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7fc100c80 msgr2=0x7fb7fc196d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:45.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.149+0000 7fb7fb400700 1 --2- 172.21.15.116:0/1156078971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7fc100c80 0x7fb7fc196d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.149+0000 7fb7fb400700 1 -- 172.21.15.116:0/1156078971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f4009d00 con 0x7fb7fc101e70 2024-03-30T00:57:45.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.149+0000 7fb7fb400700 1 --2- 172.21.15.116:0/1156078971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7fc101e70 0x7fb7fc1972b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800e9a0 tx=0x7fb7e800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:45.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.149+0000 7fb7f8c00700 1 -- 172.21.15.116:0/1156078971 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8009790 con 0x7fb7fc101e70 2024-03-30T00:57:45.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.149+0000 7fb7f8c00700 1 -- 172.21.15.116:0/1156078971 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7e8010430 con 0x7fb7fc101e70 2024-03-30T00:57:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.149+0000 7fb802c00700 1 -- 172.21.15.116:0/1156078971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7fc19ee70 con 0x7fb7fc101e70 2024-03-30T00:57:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.149+0000 7fb7f8c00700 1 -- 172.21.15.116:0/1156078971 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8018680 con 0x7fb7fc101e70 2024-03-30T00:57:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.150+0000 7fb802c00700 1 -- 172.21.15.116:0/1156078971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7fc19f290 con 0x7fb7fc101e70 2024-03-30T00:57:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.151+0000 7fb802c00700 1 -- 172.21.15.116:0/1156078971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7fc108080 con 0x7fb7fc101e70 2024-03-30T00:57:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.153+0000 7fb7f8c00700 1 -- 172.21.15.116:0/1156078971 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fb7e80105a0 con 0x7fb7fc101e70 2024-03-30T00:57:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.154+0000 7fb7f8c00700 1 --2- 172.21.15.116:0/1156078971 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb7e00a1120 0x7fb7e00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.154+0000 7fb7f8c00700 1 -- 172.21.15.116:0/1156078971 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb7e8014070 con 0x7fb7fc101e70 2024-03-30T00:57:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.154+0000 7fb7fbe00700 1 --2- 172.21.15.116:0/1156078971 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb7e00a1120 0x7fb7e00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.155+0000 7fb7fbe00700 1 --2- 172.21.15.116:0/1156078971 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb7e00a1120 0x7fb7e00a35e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc1981f0 tx=0x7fb7ec007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:45.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.158+0000 7fb7f8c00700 1 -- 172.21.15.116:0/1156078971 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb7e80529f0 con 0x7fb7fc101e70 2024-03-30T00:57:45.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.411+0000 7fb802c00700 1 -- 172.21.15.116:0/1156078971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7fc067a10 con 0x7fb7fc101e70 2024-03-30T00:57:45.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.412+0000 7fb7f8c00700 1 -- 172.21.15.116:0/1156078971 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10886 (secure 0 0 0) 0x7fb7e80961c0 con 0x7fb7fc101e70 2024-03-30T00:57:45.419 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:45.419 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:57:40.442321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T00:57:45.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.415+0000 7fb802c00700 1 -- 172.21.15.116:0/1156078971 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb7e00a1120 msgr2=0x7fb7e00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:45.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.415+0000 7fb802c00700 1 --2- 172.21.15.116:0/1156078971 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb7e00a1120 0x7fb7e00a35e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc1981f0 tx=0x7fb7ec007400 comp rx=0 tx=0).stop 2024-03-30T00:57:45.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.415+0000 7fb802c00700 1 -- 172.21.15.116:0/1156078971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7fc101e70 msgr2=0x7fb7fc1972b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:45.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.415+0000 7fb802c00700 1 --2- 172.21.15.116:0/1156078971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7fc101e70 0x7fb7fc1972b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800e9a0 tx=0x7fb7e800ecb0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.415+0000 7fb802c00700 1 -- 172.21.15.116:0/1156078971 shutdown_connections 2024-03-30T00:57:45.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.415+0000 7fb802c00700 1 --2- 172.21.15.116:0/1156078971 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb7e00a1120 0x7fb7e00a35e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.415+0000 7fb802c00700 1 --2- 172.21.15.116:0/1156078971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7fc102810 0x7fb7fc19e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.415+0000 7fb802c00700 1 --2- 172.21.15.116:0/1156078971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7fc100c80 0x7fb7fc196d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.415+0000 7fb802c00700 1 --2- 172.21.15.116:0/1156078971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7fc101e70 0x7fb7fc1972b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.415+0000 7fb802c00700 1 -- 172.21.15.116:0/1156078971 >> 172.21.15.116:0/1156078971 conn(0x7fb7fc0fc1f0 msgr2=0x7fb7fc1051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:45.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.415+0000 7fb802c00700 1 -- 172.21.15.116:0/1156078971 shutdown_connections 2024-03-30T00:57:45.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.415+0000 7fb802c00700 1 -- 172.21.15.116:0/1156078971 wait complete. 2024-03-30T00:57:45.423 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 23 2024-03-30T00:57:45.441 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-30T00:57:45.441 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-30T00:57:45.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T00:57:45.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.631+0000 7fa379200700 1 -- 172.21.15.116:0/479345510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa374101e70 msgr2=0x7fa3741022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:45.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.631+0000 7fa379200700 1 --2- 172.21.15.116:0/479345510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa374101e70 0x7fa3741022d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa3600099c0 tx=0x7fa360009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.631+0000 7fa379200700 1 -- 172.21.15.116:0/479345510 shutdown_connections 2024-03-30T00:57:45.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.631+0000 7fa379200700 1 --2- 172.21.15.116:0/479345510 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa374100c80 0x7fa3741010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.631+0000 7fa379200700 1 --2- 172.21.15.116:0/479345510 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa374102810 0x7fa3741090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.631+0000 7fa379200700 1 --2- 172.21.15.116:0/479345510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa374101e70 0x7fa3741022d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.631+0000 7fa379200700 1 -- 172.21.15.116:0/479345510 >> 172.21.15.116:0/479345510 conn(0x7fa3740fc1f0 msgr2=0x7fa3740fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:45.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.631+0000 7fa379200700 1 -- 172.21.15.116:0/479345510 shutdown_connections 2024-03-30T00:57:45.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.631+0000 7fa379200700 1 -- 172.21.15.116:0/479345510 wait complete. 2024-03-30T00:57:45.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.632+0000 7fa379200700 1 Processor -- start 2024-03-30T00:57:45.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.632+0000 7fa379200700 1 -- start start 2024-03-30T00:57:45.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.632+0000 7fa379200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa374100c80 0x7fa374198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:45.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.632+0000 7fa379200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa374101e70 0x7fa374199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:45.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.632+0000 7fa379200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa374102810 0x7fa3741a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:45.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.632+0000 7fa379200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3741a08e0 con 0x7fa374100c80 2024-03-30T00:57:45.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.632+0000 7fa379200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3741a0a50 con 0x7fa374102810 2024-03-30T00:57:45.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.632+0000 7fa379200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3741a0c10 con 0x7fa374101e70 2024-03-30T00:57:45.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.633+0000 7fa371e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa374100c80 0x7fa374198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:45.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.633+0000 7fa371e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa374100c80 0x7fa374198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58512/0 (socket says 172.21.15.116:58512) 2024-03-30T00:57:45.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.633+0000 7fa371e00700 1 -- 172.21.15.116:0/1558739458 learned_addr learned my addr 172.21.15.116:0/1558739458 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:45.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.633+0000 7fa371400700 1 --2- 172.21.15.116:0/1558739458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa374101e70 0x7fa374199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:45.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.633+0000 7fa372800700 1 --2- 172.21.15.116:0/1558739458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa374102810 0x7fa3741a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:45.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.633+0000 7fa371e00700 1 -- 172.21.15.116:0/1558739458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa374101e70 msgr2=0x7fa374199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:45.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.633+0000 7fa371e00700 1 --2- 172.21.15.116:0/1558739458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa374101e70 0x7fa374199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.633+0000 7fa371e00700 1 -- 172.21.15.116:0/1558739458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa374102810 msgr2=0x7fa3741a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:45.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.633+0000 7fa371e00700 1 --2- 172.21.15.116:0/1558739458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa374102810 0x7fa3741a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.633+0000 7fa371e00700 1 -- 172.21.15.116:0/1558739458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3600095f0 con 0x7fa374100c80 2024-03-30T00:57:45.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.633+0000 7fa371e00700 1 --2- 172.21.15.116:0/1558739458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa374100c80 0x7fa374198df0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa36400e970 tx=0x7fa36400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:45.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.633+0000 7fa36aa00700 1 -- 172.21.15.116:0/1558739458 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364009790 con 0x7fa374100c80 2024-03-30T00:57:45.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.634+0000 7fa36aa00700 1 -- 172.21.15.116:0/1558739458 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa36400eec0 con 0x7fa374100c80 2024-03-30T00:57:45.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.634+0000 7fa379200700 1 -- 172.21.15.116:0/1558739458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3741a0ef0 con 0x7fa374100c80 2024-03-30T00:57:45.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.634+0000 7fa36aa00700 1 -- 172.21.15.116:0/1558739458 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364003cf0 con 0x7fa374100c80 2024-03-30T00:57:45.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.634+0000 7fa379200700 1 -- 172.21.15.116:0/1558739458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3741a1310 con 0x7fa374100c80 2024-03-30T00:57:45.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.635+0000 7fa379200700 1 -- 172.21.15.116:0/1558739458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa374067a10 con 0x7fa374100c80 2024-03-30T00:57:45.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.637+0000 7fa36aa00700 1 -- 172.21.15.116:0/1558739458 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fa36400ccd0 con 0x7fa374100c80 2024-03-30T00:57:45.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.638+0000 7fa36aa00700 1 --2- 172.21.15.116:0/1558739458 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3580a1120 0x7fa3580a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:45.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.638+0000 7fa36aa00700 1 -- 172.21.15.116:0/1558739458 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa364014070 con 0x7fa374100c80 2024-03-30T00:57:45.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.638+0000 7fa371400700 1 --2- 172.21.15.116:0/1558739458 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3580a1120 0x7fa3580a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:45.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.639+0000 7fa371400700 1 --2- 172.21.15.116:0/1558739458 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3580a1120 0x7fa3580a35e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa360006010 tx=0x7fa36000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:45.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.640+0000 7fa36aa00700 1 -- 172.21.15.116:0/1558739458 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa364096420 con 0x7fa374100c80 2024-03-30T00:57:45.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:45 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2499373540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:45.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:45 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2064642982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:45.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:45 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1156078971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:45.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:45 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2499373540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:45.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:45 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2064642982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:45.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:45 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1156078971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:45.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.843+0000 7fa379200700 1 -- 172.21.15.116:0/1558739458 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa374074700 con 0x7fa3580a1120 2024-03-30T00:57:45.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.856+0000 7fa36aa00700 1 -- 172.21.15.116:0/1558739458 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427819 (secure 0 0 0) 0x7fa374074700 con 0x7fa3580a1120 2024-03-30T00:57:45.873 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:45.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.862+0000 7fa379200700 1 -- 172.21.15.116:0/1558739458 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3580a1120 msgr2=0x7fa3580a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:45.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.862+0000 7fa379200700 1 --2- 172.21.15.116:0/1558739458 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3580a1120 0x7fa3580a35e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa360006010 tx=0x7fa36000b540 comp rx=0 tx=0).stop 2024-03-30T00:57:45.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.862+0000 7fa379200700 1 -- 172.21.15.116:0/1558739458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa374100c80 msgr2=0x7fa374198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:45.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.862+0000 7fa379200700 1 --2- 172.21.15.116:0/1558739458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa374100c80 0x7fa374198df0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa36400e970 tx=0x7fa36400ec80 comp rx=0 tx=0).stop 2024-03-30T00:57:45.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.862+0000 7fa379200700 1 -- 172.21.15.116:0/1558739458 shutdown_connections 2024-03-30T00:57:45.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.862+0000 7fa379200700 1 --2- 172.21.15.116:0/1558739458 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa3580a1120 0x7fa3580a35e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.862+0000 7fa379200700 1 --2- 172.21.15.116:0/1558739458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa374102810 0x7fa3741a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.862+0000 7fa379200700 1 --2- 172.21.15.116:0/1558739458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa374100c80 0x7fa374198df0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.862+0000 7fa379200700 1 --2- 172.21.15.116:0/1558739458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa374101e70 0x7fa374199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:45.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.862+0000 7fa379200700 1 -- 172.21.15.116:0/1558739458 >> 172.21.15.116:0/1558739458 conn(0x7fa3740fc1f0 msgr2=0x7fa374072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:45.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.862+0000 7fa379200700 1 -- 172.21.15.116:0/1558739458 shutdown_connections 2024-03-30T00:57:45.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:45.862+0000 7fa379200700 1 -- 172.21.15.116:0/1558739458 wait complete. 2024-03-30T00:57:45.882 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-30T00:57:45.885 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:45 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2499373540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:45.885 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:45 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2064642982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:45.886 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:45 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1156078971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:45.888 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-03-30T00:57:45.130320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662450,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":648,"num_read_kb":448,"num_write":351,"num_write_kb":3068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180887,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":788,"ondisk_log_size":788,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":351704,"kb_used_data":11128,"kb_used_omap":18,"kb_used_meta":340461,"kb_avail":1124590120,"statfs":{"total":1151940427776,"available":1151580282880,"internally_reserved":0,"allocated":11395072,"data_stored":7508472,"data_compressed":320673,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19128,"internal_metadata":348632392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6516,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":86,"num_read_kb":51,"num_write":12,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003660"},"pg_stats":[{"pgid":"3.3f","version":"83'8","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.167286+0000","last_change":"2024-03-30T00:55:51.809659+0000","last_active":"2024-03-30T00:57:35.167286+0000","last_peered":"2024-03-30T00:57:35.167286+0000","last_clean":"2024-03-30T00:57:35.167286+0000","last_became_active":"2024-03-30T00:55:51.809337+0000","last_became_peered":"2024-03-30T00:55:51.809337+0000","last_unstale":"2024-03-30T00:57:35.167286+0000","last_undegraded":"2024-03-30T00:57:35.167286+0000","last_fullsized":"2024-03-30T00:57:35.167286+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:20:50.640033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"83'4","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162969+0000","last_change":"2024-03-30T00:55:51.813778+0000","last_active":"2024-03-30T00:57:35.162969+0000","last_peered":"2024-03-30T00:57:35.162969+0000","last_clean":"2024-03-30T00:57:35.162969+0000","last_became_active":"2024-03-30T00:55:51.810347+0000","last_became_peered":"2024-03-30T00:55:51.810347+0000","last_unstale":"2024-03-30T00:57:35.162969+0000","last_undegraded":"2024-03-30T00:57:35.162969+0000","last_fullsized":"2024-03-30T00:57:35.162969+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:49:07.031477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166821+0000","last_change":"2024-03-30T00:55:51.817647+0000","last_active":"2024-03-30T00:57:35.166821+0000","last_peered":"2024-03-30T00:57:35.166821+0000","last_clean":"2024-03-30T00:57:35.166821+0000","last_became_active":"2024-03-30T00:55:51.816495+0000","last_became_peered":"2024-03-30T00:55:51.816495+0000","last_unstale":"2024-03-30T00:57:35.166821+0000","last_undegraded":"2024-03-30T00:57:35.166821+0000","last_fullsized":"2024-03-30T00:57:35.166821+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:59:42.965346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"83'8","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153424+0000","last_change":"2024-03-30T00:55:51.817581+0000","last_active":"2024-03-30T00:57:35.153424+0000","last_peered":"2024-03-30T00:57:35.153424+0000","last_clean":"2024-03-30T00:57:35.153424+0000","last_became_active":"2024-03-30T00:55:51.816223+0000","last_became_peered":"2024-03-30T00:55:51.816223+0000","last_unstale":"2024-03-30T00:57:35.153424+0000","last_undegraded":"2024-03-30T00:57:35.153424+0000","last_fullsized":"2024-03-30T00:57:35.153424+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:48:32.203975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"83'4","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162997+0000","last_change":"2024-03-30T00:55:51.811707+0000","last_active":"2024-03-30T00:57:35.162997+0000","last_peered":"2024-03-30T00:57:35.162997+0000","last_clean":"2024-03-30T00:57:35.162997+0000","last_became_active":"2024-03-30T00:55:51.811366+0000","last_became_peered":"2024-03-30T00:55:51.811366+0000","last_unstale":"2024-03-30T00:57:35.162997+0000","last_undegraded":"2024-03-30T00:57:35.162997+0000","last_fullsized":"2024-03-30T00:57:35.162997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:20.438505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153163+0000","last_change":"2024-03-30T00:55:51.826849+0000","last_active":"2024-03-30T00:57:35.153163+0000","last_peered":"2024-03-30T00:57:35.153163+0000","last_clean":"2024-03-30T00:57:35.153163+0000","last_became_active":"2024-03-30T00:55:51.826520+0000","last_became_peered":"2024-03-30T00:55:51.826520+0000","last_unstale":"2024-03-30T00:57:35.153163+0000","last_undegraded":"2024-03-30T00:57:35.153163+0000","last_fullsized":"2024-03-30T00:57:35.153163+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:54:03.423725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.149197+0000","last_change":"2024-03-30T00:55:51.812880+0000","last_active":"2024-03-30T00:57:35.149197+0000","last_peered":"2024-03-30T00:57:35.149197+0000","last_clean":"2024-03-30T00:57:35.149197+0000","last_became_active":"2024-03-30T00:55:51.812572+0000","last_became_peered":"2024-03-30T00:55:51.812572+0000","last_unstale":"2024-03-30T00:57:35.149197+0000","last_undegraded":"2024-03-30T00:57:35.149197+0000","last_fullsized":"2024-03-30T00:57:35.149197+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:44:49.670641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163852+0000","last_change":"2024-03-30T00:55:51.815480+0000","last_active":"2024-03-30T00:57:35.163852+0000","last_peered":"2024-03-30T00:57:35.163852+0000","last_clean":"2024-03-30T00:57:35.163852+0000","last_became_active":"2024-03-30T00:55:51.814052+0000","last_became_peered":"2024-03-30T00:55:51.814052+0000","last_unstale":"2024-03-30T00:57:35.163852+0000","last_undegraded":"2024-03-30T00:57:35.163852+0000","last_fullsized":"2024-03-30T00:57:35.163852+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:27:49.930928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"83'11","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153313+0000","last_change":"2024-03-30T00:55:51.814127+0000","last_active":"2024-03-30T00:57:35.153313+0000","last_peered":"2024-03-30T00:57:35.153313+0000","last_clean":"2024-03-30T00:57:35.153313+0000","last_became_active":"2024-03-30T00:55:51.813820+0000","last_became_peered":"2024-03-30T00:55:51.813820+0000","last_unstale":"2024-03-30T00:57:35.153313+0000","last_undegraded":"2024-03-30T00:57:35.153313+0000","last_fullsized":"2024-03-30T00:57:35.153313+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:21:31.605935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"83'16","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.154813+0000","last_change":"2024-03-30T00:55:51.815198+0000","last_active":"2024-03-30T00:57:35.154813+0000","last_peered":"2024-03-30T00:57:35.154813+0000","last_clean":"2024-03-30T00:57:35.154813+0000","last_became_active":"2024-03-30T00:55:51.814901+0000","last_became_peered":"2024-03-30T00:55:51.814901+0000","last_unstale":"2024-03-30T00:57:35.154813+0000","last_undegraded":"2024-03-30T00:57:35.154813+0000","last_fullsized":"2024-03-30T00:57:35.154813+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:45:51.224346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.148870+0000","last_change":"2024-03-30T00:55:51.809884+0000","last_active":"2024-03-30T00:57:35.148870+0000","last_peered":"2024-03-30T00:57:35.148870+0000","last_clean":"2024-03-30T00:57:35.148870+0000","last_became_active":"2024-03-30T00:55:51.809576+0000","last_became_peered":"2024-03-30T00:55:51.809576+0000","last_unstale":"2024-03-30T00:57:35.148870+0000","last_undegraded":"2024-03-30T00:57:35.148870+0000","last_fullsized":"2024-03-30T00:57:35.148870+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:39:51.767744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153546+0000","last_change":"2024-03-30T00:55:51.817384+0000","last_active":"2024-03-30T00:57:35.153546+0000","last_peered":"2024-03-30T00:57:35.153546+0000","last_clean":"2024-03-30T00:57:35.153546+0000","last_became_active":"2024-03-30T00:55:51.816564+0000","last_became_peered":"2024-03-30T00:55:51.816564+0000","last_unstale":"2024-03-30T00:57:35.153546+0000","last_undegraded":"2024-03-30T00:57:35.153546+0000","last_fullsized":"2024-03-30T00:57:35.153546+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:47:07.195650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"83'10","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166481+0000","last_change":"2024-03-30T00:55:51.814310+0000","last_active":"2024-03-30T00:57:35.166481+0000","last_peered":"2024-03-30T00:57:35.166481+0000","last_clean":"2024-03-30T00:57:35.166481+0000","last_became_active":"2024-03-30T00:55:51.813968+0000","last_became_peered":"2024-03-30T00:55:51.813968+0000","last_unstale":"2024-03-30T00:57:35.166481+0000","last_undegraded":"2024-03-30T00:57:35.166481+0000","last_fullsized":"2024-03-30T00:57:35.166481+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:59:55.955502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.161903+0000","last_change":"2024-03-30T00:55:51.815655+0000","last_active":"2024-03-30T00:57:35.161903+0000","last_peered":"2024-03-30T00:57:35.161903+0000","last_clean":"2024-03-30T00:57:35.161903+0000","last_became_active":"2024-03-30T00:55:51.814099+0000","last_became_peered":"2024-03-30T00:55:51.814099+0000","last_unstale":"2024-03-30T00:57:35.161903+0000","last_undegraded":"2024-03-30T00:57:35.161903+0000","last_fullsized":"2024-03-30T00:57:35.161903+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:42.958153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.157146+0000","last_change":"2024-03-30T00:56:36.472491+0000","last_active":"2024-03-30T00:57:35.157146+0000","last_peered":"2024-03-30T00:57:35.157146+0000","last_clean":"2024-03-30T00:57:35.157146+0000","last_became_active":"2024-03-30T00:56:36.472037+0000","last_became_peered":"2024-03-30T00:56:36.472037+0000","last_unstale":"2024-03-30T00:57:35.157146+0000","last_undegraded":"2024-03-30T00:57:35.157146+0000","last_fullsized":"2024-03-30T00:57:35.157146+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:52:08.402775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.164993+0000","last_change":"2024-03-30T00:55:51.822278+0000","last_active":"2024-03-30T00:57:35.164993+0000","last_peered":"2024-03-30T00:57:35.164993+0000","last_clean":"2024-03-30T00:57:35.164993+0000","last_became_active":"2024-03-30T00:55:51.821996+0000","last_became_peered":"2024-03-30T00:55:51.821996+0000","last_unstale":"2024-03-30T00:57:35.164993+0000","last_undegraded":"2024-03-30T00:57:35.164993+0000","last_fullsized":"2024-03-30T00:57:35.164993+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:37:45.841853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"83'8","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162074+0000","last_change":"2024-03-30T00:55:51.809547+0000","last_active":"2024-03-30T00:57:35.162074+0000","last_peered":"2024-03-30T00:57:35.162074+0000","last_clean":"2024-03-30T00:57:35.162074+0000","last_became_active":"2024-03-30T00:55:51.809314+0000","last_became_peered":"2024-03-30T00:55:51.809314+0000","last_unstale":"2024-03-30T00:57:35.162074+0000","last_undegraded":"2024-03-30T00:57:35.162074+0000","last_fullsized":"2024-03-30T00:57:35.162074+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:39:31.661362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"83'10","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.154392+0000","last_change":"2024-03-30T00:55:51.810857+0000","last_active":"2024-03-30T00:57:35.154392+0000","last_peered":"2024-03-30T00:57:35.154392+0000","last_clean":"2024-03-30T00:57:35.154392+0000","last_became_active":"2024-03-30T00:55:51.809420+0000","last_became_peered":"2024-03-30T00:55:51.809420+0000","last_unstale":"2024-03-30T00:57:35.154392+0000","last_undegraded":"2024-03-30T00:57:35.154392+0000","last_fullsized":"2024-03-30T00:57:35.154392+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:36:48.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"83'7","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163497+0000","last_change":"2024-03-30T00:55:51.814693+0000","last_active":"2024-03-30T00:57:35.163497+0000","last_peered":"2024-03-30T00:57:35.163497+0000","last_clean":"2024-03-30T00:57:35.163497+0000","last_became_active":"2024-03-30T00:55:51.814359+0000","last_became_peered":"2024-03-30T00:55:51.814359+0000","last_unstale":"2024-03-30T00:57:35.163497+0000","last_undegraded":"2024-03-30T00:57:35.163497+0000","last_fullsized":"2024-03-30T00:57:35.163497+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:20:50.856313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470709+0000","last_change":"2024-03-30T00:55:55.839089+0000","last_active":"2024-03-30T00:56:35.470709+0000","last_peered":"2024-03-30T00:56:35.470709+0000","last_clean":"2024-03-30T00:56:35.470709+0000","last_became_active":"2024-03-30T00:55:55.838705+0000","last_became_peered":"2024-03-30T00:55:55.838705+0000","last_unstale":"2024-03-30T00:56:35.470709+0000","last_undegraded":"2024-03-30T00:56:35.470709+0000","last_fullsized":"2024-03-30T00:56:35.470709+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:33:56.010272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'9","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.487878+0000","last_change":"2024-03-30T00:55:51.298577+0000","last_active":"2024-03-30T00:56:35.487878+0000","last_peered":"2024-03-30T00:56:35.487878+0000","last_clean":"2024-03-30T00:56:35.487878+0000","last_became_active":"2024-03-30T00:55:51.298269+0000","last_became_peered":"2024-03-30T00:55:51.298269+0000","last_unstale":"2024-03-30T00:56:35.487878+0000","last_undegraded":"2024-03-30T00:56:35.487878+0000","last_fullsized":"2024-03-30T00:56:35.487878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:10:04.998621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.154700+0000","last_change":"2024-03-30T00:55:51.818517+0000","last_active":"2024-03-30T00:57:35.154700+0000","last_peered":"2024-03-30T00:57:35.154700+0000","last_clean":"2024-03-30T00:57:35.154700+0000","last_became_active":"2024-03-30T00:55:51.818174+0000","last_became_peered":"2024-03-30T00:55:51.818174+0000","last_unstale":"2024-03-30T00:57:35.154700+0000","last_undegraded":"2024-03-30T00:57:35.154700+0000","last_fullsized":"2024-03-30T00:57:35.154700+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:03:01.604135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.623912+0000","last_change":"2024-03-30T00:56:36.623912+0000","last_active":"2024-03-30T00:56:36.623912+0000","last_peered":"2024-03-30T00:56:36.623912+0000","last_clean":"2024-03-30T00:56:36.623912+0000","last_became_active":"2024-03-30T00:56:36.623266+0000","last_became_peered":"2024-03-30T00:56:36.623266+0000","last_unstale":"2024-03-30T00:56:36.623912+0000","last_undegraded":"2024-03-30T00:56:36.623912+0000","last_fullsized":"2024-03-30T00:56:36.623912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:46:47.166079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.471579+0000","last_change":"2024-03-30T00:55:51.312686+0000","last_active":"2024-03-30T00:56:35.471579+0000","last_peered":"2024-03-30T00:56:35.471579+0000","last_clean":"2024-03-30T00:56:35.471579+0000","last_became_active":"2024-03-30T00:55:51.312296+0000","last_became_peered":"2024-03-30T00:55:51.312296+0000","last_unstale":"2024-03-30T00:56:35.471579+0000","last_undegraded":"2024-03-30T00:56:35.471579+0000","last_fullsized":"2024-03-30T00:56:35.471579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:28:58.398577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165549+0000","last_change":"2024-03-30T00:55:51.812197+0000","last_active":"2024-03-30T00:57:35.165549+0000","last_peered":"2024-03-30T00:57:35.165549+0000","last_clean":"2024-03-30T00:57:35.165549+0000","last_became_active":"2024-03-30T00:55:51.811892+0000","last_became_peered":"2024-03-30T00:55:51.811892+0000","last_unstale":"2024-03-30T00:57:35.165549+0000","last_undegraded":"2024-03-30T00:57:35.165549+0000","last_fullsized":"2024-03-30T00:57:35.165549+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:06:53.459760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462318+0000","last_change":"2024-03-30T00:55:55.839015+0000","last_active":"2024-03-30T00:56:35.462318+0000","last_peered":"2024-03-30T00:56:35.462318+0000","last_clean":"2024-03-30T00:56:35.462318+0000","last_became_active":"2024-03-30T00:55:55.838608+0000","last_became_peered":"2024-03-30T00:55:55.838608+0000","last_unstale":"2024-03-30T00:56:35.462318+0000","last_undegraded":"2024-03-30T00:56:35.462318+0000","last_fullsized":"2024-03-30T00:56:35.462318+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:24:23.804754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'4","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463392+0000","last_change":"2024-03-30T00:55:51.304199+0000","last_active":"2024-03-30T00:56:35.463392+0000","last_peered":"2024-03-30T00:56:35.463392+0000","last_clean":"2024-03-30T00:56:35.463392+0000","last_became_active":"2024-03-30T00:55:51.303837+0000","last_became_peered":"2024-03-30T00:55:51.303837+0000","last_unstale":"2024-03-30T00:56:35.463392+0000","last_undegraded":"2024-03-30T00:56:35.463392+0000","last_fullsized":"2024-03-30T00:56:35.463392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:19.154957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.164480+0000","last_change":"2024-03-30T00:55:51.817984+0000","last_active":"2024-03-30T00:57:35.164480+0000","last_peered":"2024-03-30T00:57:35.164480+0000","last_clean":"2024-03-30T00:57:35.164480+0000","last_became_active":"2024-03-30T00:55:51.817767+0000","last_became_peered":"2024-03-30T00:55:51.817767+0000","last_unstale":"2024-03-30T00:57:35.164480+0000","last_undegraded":"2024-03-30T00:57:35.164480+0000","last_fullsized":"2024-03-30T00:57:35.164480+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:12:48.558024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.623581+0000","last_change":"2024-03-30T00:56:36.623581+0000","last_active":"2024-03-30T00:56:36.623581+0000","last_peered":"2024-03-30T00:56:36.623581+0000","last_clean":"2024-03-30T00:56:36.623581+0000","last_became_active":"2024-03-30T00:56:36.623168+0000","last_became_peered":"2024-03-30T00:56:36.623168+0000","last_unstale":"2024-03-30T00:56:36.623581+0000","last_undegraded":"2024-03-30T00:56:36.623581+0000","last_fullsized":"2024-03-30T00:56:36.623581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:41.448935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"76'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.480921+0000","last_change":"2024-03-30T00:56:36.480478+0000","last_active":"2024-03-30T00:56:36.480921+0000","last_peered":"2024-03-30T00:56:36.480921+0000","last_clean":"2024-03-30T00:56:36.480921+0000","last_became_active":"2024-03-30T00:56:36.473297+0000","last_became_peered":"2024-03-30T00:56:36.473297+0000","last_unstale":"2024-03-30T00:56:36.480921+0000","last_undegraded":"2024-03-30T00:56:36.480921+0000","last_fullsized":"2024-03-30T00:56:36.480921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:37:08.090281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"83'15","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.151958+0000","last_change":"2024-03-30T00:55:51.827985+0000","last_active":"2024-03-30T00:57:35.151958+0000","last_peered":"2024-03-30T00:57:35.151958+0000","last_clean":"2024-03-30T00:57:35.151958+0000","last_became_active":"2024-03-30T00:55:51.827678+0000","last_became_peered":"2024-03-30T00:55:51.827678+0000","last_unstale":"2024-03-30T00:57:35.151958+0000","last_undegraded":"2024-03-30T00:57:35.151958+0000","last_fullsized":"2024-03-30T00:57:35.151958+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:47.088009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470870+0000","last_change":"2024-03-30T00:55:55.827267+0000","last_active":"2024-03-30T00:56:35.470870+0000","last_peered":"2024-03-30T00:56:35.470870+0000","last_clean":"2024-03-30T00:56:35.470870+0000","last_became_active":"2024-03-30T00:55:55.826821+0000","last_became_peered":"2024-03-30T00:55:55.826821+0000","last_unstale":"2024-03-30T00:56:35.470870+0000","last_undegraded":"2024-03-30T00:56:35.470870+0000","last_fullsized":"2024-03-30T00:56:35.470870+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:15:49.084359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"83'7","reported_seq":106,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:38.260458+0000","last_change":"2024-03-30T00:55:51.305177+0000","last_active":"2024-03-30T00:57:38.260458+0000","last_peered":"2024-03-30T00:57:38.260458+0000","last_clean":"2024-03-30T00:57:38.260458+0000","last_became_active":"2024-03-30T00:55:51.304813+0000","last_became_peered":"2024-03-30T00:55:51.304813+0000","last_unstale":"2024-03-30T00:57:38.260458+0000","last_undegraded":"2024-03-30T00:57:38.260458+0000","last_fullsized":"2024-03-30T00:57:38.260458+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:05:20.578897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1108,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"83'7","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.140612+0000","last_change":"2024-03-30T00:55:51.812714+0000","last_active":"2024-03-30T00:57:35.140612+0000","last_peered":"2024-03-30T00:57:35.140612+0000","last_clean":"2024-03-30T00:57:35.140612+0000","last_became_active":"2024-03-30T00:55:51.812406+0000","last_became_peered":"2024-03-30T00:55:51.812406+0000","last_unstale":"2024-03-30T00:57:35.140612+0000","last_undegraded":"2024-03-30T00:57:35.140612+0000","last_fullsized":"2024-03-30T00:57:35.140612+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:47:01.691412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470799+0000","last_change":"2024-03-30T00:55:55.831994+0000","last_active":"2024-03-30T00:56:35.470799+0000","last_peered":"2024-03-30T00:56:35.470799+0000","last_clean":"2024-03-30T00:56:35.470799+0000","last_became_active":"2024-03-30T00:55:55.831537+0000","last_became_peered":"2024-03-30T00:55:55.831537+0000","last_unstale":"2024-03-30T00:56:35.470799+0000","last_undegraded":"2024-03-30T00:56:35.470799+0000","last_fullsized":"2024-03-30T00:56:35.470799+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:44:47.240981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467196+0000","last_change":"2024-03-30T00:55:51.303468+0000","last_active":"2024-03-30T00:56:35.467196+0000","last_peered":"2024-03-30T00:56:35.467196+0000","last_clean":"2024-03-30T00:56:35.467196+0000","last_became_active":"2024-03-30T00:55:51.303068+0000","last_became_peered":"2024-03-30T00:55:51.303068+0000","last_unstale":"2024-03-30T00:56:35.467196+0000","last_undegraded":"2024-03-30T00:56:35.467196+0000","last_fullsized":"2024-03-30T00:56:35.467196+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:45:25.278196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"83'8","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.159899+0000","last_change":"2024-03-30T00:55:51.825099+0000","last_active":"2024-03-30T00:57:35.159899+0000","last_peered":"2024-03-30T00:57:35.159899+0000","last_clean":"2024-03-30T00:57:35.159899+0000","last_became_active":"2024-03-30T00:55:51.824717+0000","last_became_peered":"2024-03-30T00:55:51.824717+0000","last_unstale":"2024-03-30T00:57:35.159899+0000","last_undegraded":"2024-03-30T00:57:35.159899+0000","last_fullsized":"2024-03-30T00:57:35.159899+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:49:31.542430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460499+0000","last_change":"2024-03-30T00:55:55.826106+0000","last_active":"2024-03-30T00:56:35.460499+0000","last_peered":"2024-03-30T00:56:35.460499+0000","last_clean":"2024-03-30T00:56:35.460499+0000","last_became_active":"2024-03-30T00:55:55.825597+0000","last_became_peered":"2024-03-30T00:55:55.825597+0000","last_unstale":"2024-03-30T00:56:35.460499+0000","last_undegraded":"2024-03-30T00:56:35.460499+0000","last_fullsized":"2024-03-30T00:56:35.460499+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:00:18.991732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.473981+0000","last_change":"2024-03-30T00:55:51.294090+0000","last_active":"2024-03-30T00:56:35.473981+0000","last_peered":"2024-03-30T00:56:35.473981+0000","last_clean":"2024-03-30T00:56:35.473981+0000","last_became_active":"2024-03-30T00:55:51.293668+0000","last_became_peered":"2024-03-30T00:55:51.293668+0000","last_unstale":"2024-03-30T00:56:35.473981+0000","last_undegraded":"2024-03-30T00:56:35.473981+0000","last_fullsized":"2024-03-30T00:56:35.473981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:33:36.661760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166304+0000","last_change":"2024-03-30T00:55:51.823428+0000","last_active":"2024-03-30T00:57:35.166304+0000","last_peered":"2024-03-30T00:57:35.166304+0000","last_clean":"2024-03-30T00:57:35.166304+0000","last_became_active":"2024-03-30T00:55:51.822222+0000","last_became_peered":"2024-03-30T00:55:51.822222+0000","last_unstale":"2024-03-30T00:57:35.166304+0000","last_undegraded":"2024-03-30T00:57:35.166304+0000","last_fullsized":"2024-03-30T00:57:35.166304+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:05:34.798940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495876+0000","last_change":"2024-03-30T00:55:55.828115+0000","last_active":"2024-03-30T00:56:35.495876+0000","last_peered":"2024-03-30T00:56:35.495876+0000","last_clean":"2024-03-30T00:56:35.495876+0000","last_became_active":"2024-03-30T00:55:55.827681+0000","last_became_peered":"2024-03-30T00:55:55.827681+0000","last_unstale":"2024-03-30T00:56:35.495876+0000","last_undegraded":"2024-03-30T00:56:35.495876+0000","last_fullsized":"2024-03-30T00:56:35.495876+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:18:25.751350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'1","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466298+0000","last_change":"2024-03-30T00:55:51.305163+0000","last_active":"2024-03-30T00:56:35.466298+0000","last_peered":"2024-03-30T00:56:35.466298+0000","last_clean":"2024-03-30T00:56:35.466298+0000","last_became_active":"2024-03-30T00:55:51.304860+0000","last_became_peered":"2024-03-30T00:55:51.304860+0000","last_unstale":"2024-03-30T00:56:35.466298+0000","last_undegraded":"2024-03-30T00:56:35.466298+0000","last_fullsized":"2024-03-30T00:56:35.466298+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:44:41.850998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"83'11","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.159021+0000","last_change":"2024-03-30T00:55:51.819401+0000","last_active":"2024-03-30T00:57:35.159021+0000","last_peered":"2024-03-30T00:57:35.159021+0000","last_clean":"2024-03-30T00:57:35.159021+0000","last_became_active":"2024-03-30T00:55:51.819021+0000","last_became_peered":"2024-03-30T00:55:51.819021+0000","last_unstale":"2024-03-30T00:57:35.159021+0000","last_undegraded":"2024-03-30T00:57:35.159021+0000","last_fullsized":"2024-03-30T00:57:35.159021+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:01:45.195719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.469445+0000","last_change":"2024-03-30T00:55:55.836806+0000","last_active":"2024-03-30T00:56:35.469445+0000","last_peered":"2024-03-30T00:56:35.469445+0000","last_clean":"2024-03-30T00:56:35.469445+0000","last_became_active":"2024-03-30T00:55:55.836403+0000","last_became_peered":"2024-03-30T00:55:55.836403+0000","last_unstale":"2024-03-30T00:56:35.469445+0000","last_undegraded":"2024-03-30T00:56:35.469445+0000","last_fullsized":"2024-03-30T00:56:35.469445+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:17:13.913675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"83'4","reported_seq":104,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:43.267468+0000","last_change":"2024-03-30T00:55:51.300833+0000","last_active":"2024-03-30T00:57:43.267468+0000","last_peered":"2024-03-30T00:57:43.267468+0000","last_clean":"2024-03-30T00:57:43.267468+0000","last_became_active":"2024-03-30T00:55:51.300473+0000","last_became_peered":"2024-03-30T00:55:51.300473+0000","last_unstale":"2024-03-30T00:57:43.267468+0000","last_undegraded":"2024-03-30T00:57:43.267468+0000","last_fullsized":"2024-03-30T00:57:43.267468+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:00:39.211981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"83'10","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.155649+0000","last_change":"2024-03-30T00:55:51.810781+0000","last_active":"2024-03-30T00:57:35.155649+0000","last_peered":"2024-03-30T00:57:35.155649+0000","last_clean":"2024-03-30T00:57:35.155649+0000","last_became_active":"2024-03-30T00:55:51.810215+0000","last_became_peered":"2024-03-30T00:55:51.810215+0000","last_unstale":"2024-03-30T00:57:35.155649+0000","last_undegraded":"2024-03-30T00:57:35.155649+0000","last_fullsized":"2024-03-30T00:57:35.155649+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:53:50.283209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.484181+0000","last_change":"2024-03-30T00:55:55.834505+0000","last_active":"2024-03-30T00:56:35.484181+0000","last_peered":"2024-03-30T00:56:35.484181+0000","last_clean":"2024-03-30T00:56:35.484181+0000","last_became_active":"2024-03-30T00:55:55.829319+0000","last_became_peered":"2024-03-30T00:55:55.829319+0000","last_unstale":"2024-03-30T00:56:35.484181+0000","last_undegraded":"2024-03-30T00:56:35.484181+0000","last_fullsized":"2024-03-30T00:56:35.484181+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:51:37.595523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466067+0000","last_change":"2024-03-30T00:55:51.302070+0000","last_active":"2024-03-30T00:56:35.466067+0000","last_peered":"2024-03-30T00:56:35.466067+0000","last_clean":"2024-03-30T00:56:35.466067+0000","last_became_active":"2024-03-30T00:55:51.298940+0000","last_became_peered":"2024-03-30T00:55:51.298940+0000","last_unstale":"2024-03-30T00:56:35.466067+0000","last_undegraded":"2024-03-30T00:56:35.466067+0000","last_fullsized":"2024-03-30T00:56:35.466067+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:07:47.915506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"83'9","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166604+0000","last_change":"2024-03-30T00:55:51.819918+0000","last_active":"2024-03-30T00:57:35.166604+0000","last_peered":"2024-03-30T00:57:35.166604+0000","last_clean":"2024-03-30T00:57:35.166604+0000","last_became_active":"2024-03-30T00:55:51.819493+0000","last_became_peered":"2024-03-30T00:55:51.819493+0000","last_unstale":"2024-03-30T00:57:35.166604+0000","last_undegraded":"2024-03-30T00:57:35.166604+0000","last_fullsized":"2024-03-30T00:57:35.166604+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:43:43.067031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462710+0000","last_change":"2024-03-30T00:55:55.827176+0000","last_active":"2024-03-30T00:56:35.462710+0000","last_peered":"2024-03-30T00:56:35.462710+0000","last_clean":"2024-03-30T00:56:35.462710+0000","last_became_active":"2024-03-30T00:55:55.826772+0000","last_became_peered":"2024-03-30T00:55:55.826772+0000","last_unstale":"2024-03-30T00:56:35.462710+0000","last_undegraded":"2024-03-30T00:56:35.462710+0000","last_fullsized":"2024-03-30T00:56:35.462710+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:02:04.512050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'4","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467237+0000","last_change":"2024-03-30T00:55:51.306688+0000","last_active":"2024-03-30T00:56:35.467237+0000","last_peered":"2024-03-30T00:56:35.467237+0000","last_clean":"2024-03-30T00:56:35.467237+0000","last_became_active":"2024-03-30T00:55:51.306333+0000","last_became_peered":"2024-03-30T00:55:51.306333+0000","last_unstale":"2024-03-30T00:56:35.467237+0000","last_undegraded":"2024-03-30T00:56:35.467237+0000","last_fullsized":"2024-03-30T00:56:35.467237+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:21:13.719947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"83'11","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.152507+0000","last_change":"2024-03-30T00:55:51.824879+0000","last_active":"2024-03-30T00:57:35.152507+0000","last_peered":"2024-03-30T00:57:35.152507+0000","last_clean":"2024-03-30T00:57:35.152507+0000","last_became_active":"2024-03-30T00:55:51.824566+0000","last_became_peered":"2024-03-30T00:55:51.824566+0000","last_unstale":"2024-03-30T00:57:35.152507+0000","last_undegraded":"2024-03-30T00:57:35.152507+0000","last_fullsized":"2024-03-30T00:57:35.152507+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:04:28.505693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.484132+0000","last_change":"2024-03-30T00:55:55.826387+0000","last_active":"2024-03-30T00:56:35.484132+0000","last_peered":"2024-03-30T00:56:35.484132+0000","last_clean":"2024-03-30T00:56:35.484132+0000","last_became_active":"2024-03-30T00:55:55.825941+0000","last_became_peered":"2024-03-30T00:55:55.825941+0000","last_unstale":"2024-03-30T00:56:35.484132+0000","last_undegraded":"2024-03-30T00:56:35.484132+0000","last_fullsized":"2024-03-30T00:56:35.484132+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:34:43.432712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"83'38","reported_seq":232,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:44.268981+0000","last_change":"2024-03-30T00:55:51.304592+0000","last_active":"2024-03-30T00:57:44.268981+0000","last_peered":"2024-03-30T00:57:44.268981+0000","last_clean":"2024-03-30T00:57:44.268981+0000","last_became_active":"2024-03-30T00:55:51.303931+0000","last_became_peered":"2024-03-30T00:55:51.303931+0000","last_unstale":"2024-03-30T00:57:44.268981+0000","last_undegraded":"2024-03-30T00:57:44.268981+0000","last_fullsized":"2024-03-30T00:57:44.268981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:53:40.976289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66960,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":177,"num_read_kb":154,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.148546+0000","last_change":"2024-03-30T00:55:51.815196+0000","last_active":"2024-03-30T00:57:35.148546+0000","last_peered":"2024-03-30T00:57:35.148546+0000","last_clean":"2024-03-30T00:57:35.148546+0000","last_became_active":"2024-03-30T00:55:51.813376+0000","last_became_peered":"2024-03-30T00:55:51.813376+0000","last_unstale":"2024-03-30T00:57:35.148546+0000","last_undegraded":"2024-03-30T00:57:35.148546+0000","last_fullsized":"2024-03-30T00:57:35.148546+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:40:26.350248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467447+0000","last_change":"2024-03-30T00:55:55.836906+0000","last_active":"2024-03-30T00:56:35.467447+0000","last_peered":"2024-03-30T00:56:35.467447+0000","last_clean":"2024-03-30T00:56:35.467447+0000","last_became_active":"2024-03-30T00:55:55.836539+0000","last_became_peered":"2024-03-30T00:55:55.836539+0000","last_unstale":"2024-03-30T00:56:35.467447+0000","last_undegraded":"2024-03-30T00:56:35.467447+0000","last_fullsized":"2024-03-30T00:56:35.467447+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:21:39.320192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'6","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.478499+0000","last_change":"2024-03-30T00:56:36.478132+0000","last_active":"2024-03-30T00:56:36.478499+0000","last_peered":"2024-03-30T00:56:36.478499+0000","last_clean":"2024-03-30T00:56:36.478499+0000","last_became_active":"2024-03-30T00:56:36.467449+0000","last_became_peered":"2024-03-30T00:56:36.467449+0000","last_unstale":"2024-03-30T00:56:36.478499+0000","last_undegraded":"2024-03-30T00:56:36.478499+0000","last_fullsized":"2024-03-30T00:56:36.478499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:27:51.986112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165118+0000","last_change":"2024-03-30T00:55:51.819825+0000","last_active":"2024-03-30T00:57:35.165118+0000","last_peered":"2024-03-30T00:57:35.165118+0000","last_clean":"2024-03-30T00:57:35.165118+0000","last_became_active":"2024-03-30T00:55:51.819481+0000","last_became_peered":"2024-03-30T00:55:51.819481+0000","last_unstale":"2024-03-30T00:57:35.165118+0000","last_undegraded":"2024-03-30T00:57:35.165118+0000","last_fullsized":"2024-03-30T00:57:35.165118+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:10:17.138541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460246+0000","last_change":"2024-03-30T00:55:55.832675+0000","last_active":"2024-03-30T00:56:35.460246+0000","last_peered":"2024-03-30T00:56:35.460246+0000","last_clean":"2024-03-30T00:56:35.460246+0000","last_became_active":"2024-03-30T00:55:55.832325+0000","last_became_peered":"2024-03-30T00:55:55.832325+0000","last_unstale":"2024-03-30T00:56:35.460246+0000","last_undegraded":"2024-03-30T00:56:35.460246+0000","last_fullsized":"2024-03-30T00:56:35.460246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:38:46.765473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467208+0000","last_change":"2024-03-30T00:55:51.313700+0000","last_active":"2024-03-30T00:56:35.467208+0000","last_peered":"2024-03-30T00:56:35.467208+0000","last_clean":"2024-03-30T00:56:35.467208+0000","last_became_active":"2024-03-30T00:55:51.313355+0000","last_became_peered":"2024-03-30T00:55:51.313355+0000","last_unstale":"2024-03-30T00:56:35.467208+0000","last_undegraded":"2024-03-30T00:56:35.467208+0000","last_fullsized":"2024-03-30T00:56:35.467208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:43:04.461398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"83'15","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163894+0000","last_change":"2024-03-30T00:55:51.819939+0000","last_active":"2024-03-30T00:57:35.163894+0000","last_peered":"2024-03-30T00:57:35.163894+0000","last_clean":"2024-03-30T00:57:35.163894+0000","last_became_active":"2024-03-30T00:55:51.819614+0000","last_became_peered":"2024-03-30T00:55:51.819614+0000","last_unstale":"2024-03-30T00:57:35.163894+0000","last_undegraded":"2024-03-30T00:57:35.163894+0000","last_fullsized":"2024-03-30T00:57:35.163894+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:20:37.362779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.469703+0000","last_change":"2024-03-30T00:56:36.469703+0000","last_active":"2024-03-30T00:56:36.469703+0000","last_peered":"2024-03-30T00:56:36.469703+0000","last_clean":"2024-03-30T00:56:36.469703+0000","last_became_active":"2024-03-30T00:56:36.469195+0000","last_became_peered":"2024-03-30T00:56:36.469195+0000","last_unstale":"2024-03-30T00:56:36.469703+0000","last_undegraded":"2024-03-30T00:56:36.469703+0000","last_fullsized":"2024-03-30T00:56:36.469703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:10:10.421447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"83'6","reported_seq":110,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:42.266577+0000","last_change":"2024-03-30T00:55:51.299246+0000","last_active":"2024-03-30T00:57:42.266577+0000","last_peered":"2024-03-30T00:57:42.266577+0000","last_clean":"2024-03-30T00:57:42.266577+0000","last_became_active":"2024-03-30T00:55:51.298924+0000","last_became_peered":"2024-03-30T00:55:51.298924+0000","last_unstale":"2024-03-30T00:57:42.266577+0000","last_undegraded":"2024-03-30T00:57:42.266577+0000","last_fullsized":"2024-03-30T00:57:42.266577+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:50:05.897612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1380,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.149967+0000","last_change":"2024-03-30T00:55:51.813804+0000","last_active":"2024-03-30T00:57:35.149967+0000","last_peered":"2024-03-30T00:57:35.149967+0000","last_clean":"2024-03-30T00:57:35.149967+0000","last_became_active":"2024-03-30T00:55:51.813500+0000","last_became_peered":"2024-03-30T00:55:51.813500+0000","last_unstale":"2024-03-30T00:57:35.149967+0000","last_undegraded":"2024-03-30T00:57:35.149967+0000","last_fullsized":"2024-03-30T00:57:35.149967+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:42:21.614900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.477003+0000","last_change":"2024-03-30T00:55:55.837737+0000","last_active":"2024-03-30T00:56:35.477003+0000","last_peered":"2024-03-30T00:56:35.477003+0000","last_clean":"2024-03-30T00:56:35.477003+0000","last_became_active":"2024-03-30T00:55:55.837312+0000","last_became_peered":"2024-03-30T00:55:55.837312+0000","last_unstale":"2024-03-30T00:56:35.477003+0000","last_undegraded":"2024-03-30T00:56:35.477003+0000","last_fullsized":"2024-03-30T00:56:35.477003+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:11:46.139979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'15","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462932+0000","last_change":"2024-03-30T00:55:51.306398+0000","last_active":"2024-03-30T00:56:35.462932+0000","last_peered":"2024-03-30T00:56:35.462932+0000","last_clean":"2024-03-30T00:56:35.462932+0000","last_became_active":"2024-03-30T00:55:51.305878+0000","last_became_peered":"2024-03-30T00:55:51.305878+0000","last_unstale":"2024-03-30T00:56:35.462932+0000","last_undegraded":"2024-03-30T00:56:35.462932+0000","last_fullsized":"2024-03-30T00:56:35.462932+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:20:32.742423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.151564+0000","last_change":"2024-03-30T00:55:51.824294+0000","last_active":"2024-03-30T00:57:35.151564+0000","last_peered":"2024-03-30T00:57:35.151564+0000","last_clean":"2024-03-30T00:57:35.151564+0000","last_became_active":"2024-03-30T00:55:51.823957+0000","last_became_peered":"2024-03-30T00:55:51.823957+0000","last_unstale":"2024-03-30T00:57:35.151564+0000","last_undegraded":"2024-03-30T00:57:35.151564+0000","last_fullsized":"2024-03-30T00:57:35.151564+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:23:16.873140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466253+0000","last_change":"2024-03-30T00:55:55.837023+0000","last_active":"2024-03-30T00:56:35.466253+0000","last_peered":"2024-03-30T00:56:35.466253+0000","last_clean":"2024-03-30T00:56:35.466253+0000","last_became_active":"2024-03-30T00:55:55.836541+0000","last_became_peered":"2024-03-30T00:55:55.836541+0000","last_unstale":"2024-03-30T00:56:35.466253+0000","last_undegraded":"2024-03-30T00:56:35.466253+0000","last_fullsized":"2024-03-30T00:56:35.466253+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:58:40.323953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463352+0000","last_change":"2024-03-30T00:55:51.300883+0000","last_active":"2024-03-30T00:56:35.463352+0000","last_peered":"2024-03-30T00:56:35.463352+0000","last_clean":"2024-03-30T00:56:35.463352+0000","last_became_active":"2024-03-30T00:55:51.300531+0000","last_became_peered":"2024-03-30T00:55:51.300531+0000","last_unstale":"2024-03-30T00:56:35.463352+0000","last_undegraded":"2024-03-30T00:56:35.463352+0000","last_fullsized":"2024-03-30T00:56:35.463352+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:38:38.874075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"83'5","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.151560+0000","last_change":"2024-03-30T00:55:51.822970+0000","last_active":"2024-03-30T00:57:35.151560+0000","last_peered":"2024-03-30T00:57:35.151560+0000","last_clean":"2024-03-30T00:57:35.151560+0000","last_became_active":"2024-03-30T00:55:51.822653+0000","last_became_peered":"2024-03-30T00:55:51.822653+0000","last_unstale":"2024-03-30T00:57:35.151560+0000","last_undegraded":"2024-03-30T00:57:35.151560+0000","last_fullsized":"2024-03-30T00:57:35.151560+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:59:26.306672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463256+0000","last_change":"2024-03-30T00:55:55.856605+0000","last_active":"2024-03-30T00:56:35.463256+0000","last_peered":"2024-03-30T00:56:35.463256+0000","last_clean":"2024-03-30T00:56:35.463256+0000","last_became_active":"2024-03-30T00:55:55.856061+0000","last_became_peered":"2024-03-30T00:55:55.856061+0000","last_unstale":"2024-03-30T00:56:35.463256+0000","last_undegraded":"2024-03-30T00:56:35.463256+0000","last_fullsized":"2024-03-30T00:56:35.463256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:58:41.010206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.488831+0000","last_change":"2024-03-30T00:55:51.301961+0000","last_active":"2024-03-30T00:56:35.488831+0000","last_peered":"2024-03-30T00:56:35.488831+0000","last_clean":"2024-03-30T00:56:35.488831+0000","last_became_active":"2024-03-30T00:55:51.301618+0000","last_became_peered":"2024-03-30T00:55:51.301618+0000","last_unstale":"2024-03-30T00:56:35.488831+0000","last_undegraded":"2024-03-30T00:56:35.488831+0000","last_fullsized":"2024-03-30T00:56:35.488831+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:38:46.681490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"83'4","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162610+0000","last_change":"2024-03-30T00:55:51.821331+0000","last_active":"2024-03-30T00:57:35.162610+0000","last_peered":"2024-03-30T00:57:35.162610+0000","last_clean":"2024-03-30T00:57:35.162610+0000","last_became_active":"2024-03-30T00:55:51.820988+0000","last_became_peered":"2024-03-30T00:55:51.820988+0000","last_unstale":"2024-03-30T00:57:35.162610+0000","last_undegraded":"2024-03-30T00:57:35.162610+0000","last_fullsized":"2024-03-30T00:57:35.162610+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:25:20.150789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.475451+0000","last_change":"2024-03-30T00:55:55.826251+0000","last_active":"2024-03-30T00:56:35.475451+0000","last_peered":"2024-03-30T00:56:35.475451+0000","last_clean":"2024-03-30T00:56:35.475451+0000","last_became_active":"2024-03-30T00:55:55.822695+0000","last_became_peered":"2024-03-30T00:55:55.822695+0000","last_unstale":"2024-03-30T00:56:35.475451+0000","last_undegraded":"2024-03-30T00:56:35.475451+0000","last_fullsized":"2024-03-30T00:56:35.475451+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:32:19.011454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466393+0000","last_change":"2024-03-30T00:55:51.309412+0000","last_active":"2024-03-30T00:56:35.466393+0000","last_peered":"2024-03-30T00:56:35.466393+0000","last_clean":"2024-03-30T00:56:35.466393+0000","last_became_active":"2024-03-30T00:55:51.309028+0000","last_became_peered":"2024-03-30T00:55:51.309028+0000","last_unstale":"2024-03-30T00:56:35.466393+0000","last_undegraded":"2024-03-30T00:56:35.466393+0000","last_fullsized":"2024-03-30T00:56:35.466393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:00:49.022772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.152521+0000","last_change":"2024-03-30T00:56:36.467986+0000","last_active":"2024-03-30T00:57:35.152521+0000","last_peered":"2024-03-30T00:57:35.152521+0000","last_clean":"2024-03-30T00:57:35.152521+0000","last_became_active":"2024-03-30T00:56:36.467601+0000","last_became_peered":"2024-03-30T00:56:36.467601+0000","last_unstale":"2024-03-30T00:57:35.152521+0000","last_undegraded":"2024-03-30T00:57:35.152521+0000","last_fullsized":"2024-03-30T00:57:35.152521+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:37:32.979094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.479139+0000","last_change":"2024-03-30T00:55:55.856377+0000","last_active":"2024-03-30T00:56:35.479139+0000","last_peered":"2024-03-30T00:56:35.479139+0000","last_clean":"2024-03-30T00:56:35.479139+0000","last_became_active":"2024-03-30T00:55:55.855986+0000","last_became_peered":"2024-03-30T00:55:55.855986+0000","last_unstale":"2024-03-30T00:56:35.479139+0000","last_undegraded":"2024-03-30T00:56:35.479139+0000","last_fullsized":"2024-03-30T00:56:35.479139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:45:47.303446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"83'8","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.147047+0000","last_change":"2024-03-30T00:56:36.464863+0000","last_active":"2024-03-30T00:57:35.147047+0000","last_peered":"2024-03-30T00:57:35.147047+0000","last_clean":"2024-03-30T00:57:35.147047+0000","last_became_active":"2024-03-30T00:56:36.464323+0000","last_became_peered":"2024-03-30T00:56:36.464323+0000","last_unstale":"2024-03-30T00:57:35.147047+0000","last_undegraded":"2024-03-30T00:57:35.147047+0000","last_fullsized":"2024-03-30T00:57:35.147047+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:43:10.899726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"83'5","reported_seq":102,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:39.261685+0000","last_change":"2024-03-30T00:55:51.314834+0000","last_active":"2024-03-30T00:57:39.261685+0000","last_peered":"2024-03-30T00:57:39.261685+0000","last_clean":"2024-03-30T00:57:39.261685+0000","last_became_active":"2024-03-30T00:55:51.314429+0000","last_became_peered":"2024-03-30T00:55:51.314429+0000","last_unstale":"2024-03-30T00:57:39.261685+0000","last_undegraded":"2024-03-30T00:57:39.261685+0000","last_fullsized":"2024-03-30T00:57:39.261685+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:17:13.987180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":79,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466483+0000","last_change":"2024-03-30T00:55:55.840721+0000","last_active":"2024-03-30T00:56:35.466483+0000","last_peered":"2024-03-30T00:56:35.466483+0000","last_clean":"2024-03-30T00:56:35.466483+0000","last_became_active":"2024-03-30T00:55:55.840269+0000","last_became_peered":"2024-03-30T00:55:55.840269+0000","last_unstale":"2024-03-30T00:56:35.466483+0000","last_undegraded":"2024-03-30T00:56:35.466483+0000","last_fullsized":"2024-03-30T00:56:35.466483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:28:55.366411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.160257+0000","last_change":"2024-03-30T00:55:51.812609+0000","last_active":"2024-03-30T00:57:35.160257+0000","last_peered":"2024-03-30T00:57:35.160257+0000","last_clean":"2024-03-30T00:57:35.160257+0000","last_became_active":"2024-03-30T00:55:51.812325+0000","last_became_peered":"2024-03-30T00:55:51.812325+0000","last_unstale":"2024-03-30T00:57:35.160257+0000","last_undegraded":"2024-03-30T00:57:35.160257+0000","last_fullsized":"2024-03-30T00:57:35.160257+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:22:22.074405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'3","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463363+0000","last_change":"2024-03-30T00:55:51.296173+0000","last_active":"2024-03-30T00:56:35.463363+0000","last_peered":"2024-03-30T00:56:35.463363+0000","last_clean":"2024-03-30T00:56:35.463363+0000","last_became_active":"2024-03-30T00:55:51.295331+0000","last_became_peered":"2024-03-30T00:55:51.295331+0000","last_unstale":"2024-03-30T00:56:35.463363+0000","last_undegraded":"2024-03-30T00:56:35.463363+0000","last_fullsized":"2024-03-30T00:56:35.463363+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:51:58.343765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":326,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466805+0000","last_change":"2024-03-30T00:55:02.487656+0000","last_active":"2024-03-30T00:56:35.466805+0000","last_peered":"2024-03-30T00:56:35.466805+0000","last_clean":"2024-03-30T00:56:35.466805+0000","last_became_active":"2024-03-30T00:55:02.487263+0000","last_became_peered":"2024-03-30T00:55:02.487263+0000","last_unstale":"2024-03-30T00:56:35.466805+0000","last_undegraded":"2024-03-30T00:56:35.466805+0000","last_fullsized":"2024-03-30T00:56:35.466805+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:52:29.691461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:52:29.691461+0000","last_clean_scrub_stamp":"2024-03-30T00:52:29.691461+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:19:20.825038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466944+0000","last_change":"2024-03-30T00:55:55.856487+0000","last_active":"2024-03-30T00:56:35.466944+0000","last_peered":"2024-03-30T00:56:35.466944+0000","last_clean":"2024-03-30T00:56:35.466944+0000","last_became_active":"2024-03-30T00:55:55.856106+0000","last_became_peered":"2024-03-30T00:55:55.856106+0000","last_unstale":"2024-03-30T00:56:35.466944+0000","last_undegraded":"2024-03-30T00:56:35.466944+0000","last_fullsized":"2024-03-30T00:56:35.466944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:14.504218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.164966+0000","last_change":"2024-03-30T00:55:51.821519+0000","last_active":"2024-03-30T00:57:35.164966+0000","last_peered":"2024-03-30T00:57:35.164966+0000","last_clean":"2024-03-30T00:57:35.164966+0000","last_became_active":"2024-03-30T00:55:51.821095+0000","last_became_peered":"2024-03-30T00:55:51.821095+0000","last_unstale":"2024-03-30T00:57:35.164966+0000","last_undegraded":"2024-03-30T00:57:35.164966+0000","last_fullsized":"2024-03-30T00:57:35.164966+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:50:22.297450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'3","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466994+0000","last_change":"2024-03-30T00:55:51.303047+0000","last_active":"2024-03-30T00:56:35.466994+0000","last_peered":"2024-03-30T00:56:35.466994+0000","last_clean":"2024-03-30T00:56:35.466994+0000","last_became_active":"2024-03-30T00:55:51.302712+0000","last_became_peered":"2024-03-30T00:55:51.302712+0000","last_unstale":"2024-03-30T00:56:35.466994+0000","last_undegraded":"2024-03-30T00:56:35.466994+0000","last_fullsized":"2024-03-30T00:56:35.466994+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:07:59.451674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460262+0000","last_change":"2024-03-30T00:55:55.823692+0000","last_active":"2024-03-30T00:56:35.460262+0000","last_peered":"2024-03-30T00:56:35.460262+0000","last_clean":"2024-03-30T00:56:35.460262+0000","last_became_active":"2024-03-30T00:55:55.823188+0000","last_became_peered":"2024-03-30T00:55:55.823188+0000","last_unstale":"2024-03-30T00:56:35.460262+0000","last_undegraded":"2024-03-30T00:56:35.460262+0000","last_fullsized":"2024-03-30T00:56:35.460262+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:32:32.089166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'1","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463201+0000","last_change":"2024-03-30T00:55:51.306935+0000","last_active":"2024-03-30T00:56:35.463201+0000","last_peered":"2024-03-30T00:56:35.463201+0000","last_clean":"2024-03-30T00:56:35.463201+0000","last_became_active":"2024-03-30T00:55:51.306595+0000","last_became_peered":"2024-03-30T00:55:51.306595+0000","last_unstale":"2024-03-30T00:56:35.463201+0000","last_undegraded":"2024-03-30T00:56:35.463201+0000","last_fullsized":"2024-03-30T00:56:35.463201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:50:30.396762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166210+0000","last_change":"2024-03-30T00:55:51.807084+0000","last_active":"2024-03-30T00:57:35.166210+0000","last_peered":"2024-03-30T00:57:35.166210+0000","last_clean":"2024-03-30T00:57:35.166210+0000","last_became_active":"2024-03-30T00:55:51.806505+0000","last_became_peered":"2024-03-30T00:55:51.806505+0000","last_unstale":"2024-03-30T00:57:35.166210+0000","last_undegraded":"2024-03-30T00:57:35.166210+0000","last_fullsized":"2024-03-30T00:57:35.166210+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:38:44.759863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470455+0000","last_change":"2024-03-30T00:55:55.856457+0000","last_active":"2024-03-30T00:56:35.470455+0000","last_peered":"2024-03-30T00:56:35.470455+0000","last_clean":"2024-03-30T00:56:35.470455+0000","last_became_active":"2024-03-30T00:55:55.855982+0000","last_became_peered":"2024-03-30T00:55:55.855982+0000","last_unstale":"2024-03-30T00:56:35.470455+0000","last_undegraded":"2024-03-30T00:56:35.470455+0000","last_fullsized":"2024-03-30T00:56:35.470455+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:18:25.154634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"83'6","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.460763+0000","last_change":"2024-03-30T00:55:51.312581+0000","last_active":"2024-03-30T00:56:36.460763+0000","last_peered":"2024-03-30T00:56:36.460763+0000","last_clean":"2024-03-30T00:56:36.460763+0000","last_became_active":"2024-03-30T00:55:51.312225+0000","last_became_peered":"2024-03-30T00:55:51.312225+0000","last_unstale":"2024-03-30T00:56:36.460763+0000","last_undegraded":"2024-03-30T00:56:36.460763+0000","last_fullsized":"2024-03-30T00:56:36.460763+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:27:12.615250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.150369+0000","last_change":"2024-03-30T00:55:51.821527+0000","last_active":"2024-03-30T00:57:35.150369+0000","last_peered":"2024-03-30T00:57:35.150369+0000","last_clean":"2024-03-30T00:57:35.150369+0000","last_became_active":"2024-03-30T00:55:51.820689+0000","last_became_peered":"2024-03-30T00:55:51.820689+0000","last_unstale":"2024-03-30T00:57:35.150369+0000","last_undegraded":"2024-03-30T00:57:35.150369+0000","last_fullsized":"2024-03-30T00:57:35.150369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:14:35.735985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470468+0000","last_change":"2024-03-30T00:55:55.824307+0000","last_active":"2024-03-30T00:56:35.470468+0000","last_peered":"2024-03-30T00:56:35.470468+0000","last_clean":"2024-03-30T00:56:35.470468+0000","last_became_active":"2024-03-30T00:55:55.823900+0000","last_became_peered":"2024-03-30T00:55:55.823900+0000","last_unstale":"2024-03-30T00:56:35.470468+0000","last_undegraded":"2024-03-30T00:56:35.470468+0000","last_fullsized":"2024-03-30T00:56:35.470468+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:44:22.139120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'4","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463230+0000","last_change":"2024-03-30T00:55:51.296965+0000","last_active":"2024-03-30T00:56:35.463230+0000","last_peered":"2024-03-30T00:56:35.463230+0000","last_clean":"2024-03-30T00:56:35.463230+0000","last_became_active":"2024-03-30T00:55:51.296645+0000","last_became_peered":"2024-03-30T00:55:51.296645+0000","last_unstale":"2024-03-30T00:56:35.463230+0000","last_undegraded":"2024-03-30T00:56:35.463230+0000","last_fullsized":"2024-03-30T00:56:35.463230+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:53:46.519972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163410+0000","last_change":"2024-03-30T00:55:51.826282+0000","last_active":"2024-03-30T00:57:35.163410+0000","last_peered":"2024-03-30T00:57:35.163410+0000","last_clean":"2024-03-30T00:57:35.163410+0000","last_became_active":"2024-03-30T00:55:51.823910+0000","last_became_peered":"2024-03-30T00:55:51.823910+0000","last_unstale":"2024-03-30T00:57:35.163410+0000","last_undegraded":"2024-03-30T00:57:35.163410+0000","last_fullsized":"2024-03-30T00:57:35.163410+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:47.454280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462851+0000","last_change":"2024-03-30T00:55:55.856750+0000","last_active":"2024-03-30T00:56:35.462851+0000","last_peered":"2024-03-30T00:56:35.462851+0000","last_clean":"2024-03-30T00:56:35.462851+0000","last_became_active":"2024-03-30T00:55:55.856288+0000","last_became_peered":"2024-03-30T00:55:55.856288+0000","last_unstale":"2024-03-30T00:56:35.462851+0000","last_undegraded":"2024-03-30T00:56:35.462851+0000","last_fullsized":"2024-03-30T00:56:35.462851+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:43:28.085910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467887+0000","last_change":"2024-03-30T00:55:51.306357+0000","last_active":"2024-03-30T00:56:35.467887+0000","last_peered":"2024-03-30T00:56:35.467887+0000","last_clean":"2024-03-30T00:56:35.467887+0000","last_became_active":"2024-03-30T00:55:51.305815+0000","last_became_peered":"2024-03-30T00:55:51.305815+0000","last_unstale":"2024-03-30T00:56:35.467887+0000","last_undegraded":"2024-03-30T00:56:35.467887+0000","last_fullsized":"2024-03-30T00:56:35.467887+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:16:21.753446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"83'2","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.117501+0000","last_change":"2024-03-30T00:55:51.810686+0000","last_active":"2024-03-30T00:57:35.117501+0000","last_peered":"2024-03-30T00:57:35.117501+0000","last_clean":"2024-03-30T00:57:35.117501+0000","last_became_active":"2024-03-30T00:55:51.810331+0000","last_became_peered":"2024-03-30T00:55:51.810331+0000","last_unstale":"2024-03-30T00:57:35.117501+0000","last_undegraded":"2024-03-30T00:57:35.117501+0000","last_fullsized":"2024-03-30T00:57:35.117501+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:24:13.750905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.487787+0000","last_change":"2024-03-30T00:55:55.842894+0000","last_active":"2024-03-30T00:56:35.487787+0000","last_peered":"2024-03-30T00:56:35.487787+0000","last_clean":"2024-03-30T00:56:35.487787+0000","last_became_active":"2024-03-30T00:55:55.842118+0000","last_became_peered":"2024-03-30T00:55:55.842118+0000","last_unstale":"2024-03-30T00:56:35.487787+0000","last_undegraded":"2024-03-30T00:56:35.487787+0000","last_fullsized":"2024-03-30T00:56:35.487787+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:39.246228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'3","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470956+0000","last_change":"2024-03-30T00:55:51.313188+0000","last_active":"2024-03-30T00:56:35.470956+0000","last_peered":"2024-03-30T00:56:35.470956+0000","last_clean":"2024-03-30T00:56:35.470956+0000","last_became_active":"2024-03-30T00:55:51.312879+0000","last_became_peered":"2024-03-30T00:55:51.312879+0000","last_unstale":"2024-03-30T00:56:35.470956+0000","last_undegraded":"2024-03-30T00:56:35.470956+0000","last_fullsized":"2024-03-30T00:56:35.470956+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:41:23.420427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"83'9","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165088+0000","last_change":"2024-03-30T00:55:51.808998+0000","last_active":"2024-03-30T00:57:35.165088+0000","last_peered":"2024-03-30T00:57:35.165088+0000","last_clean":"2024-03-30T00:57:35.165088+0000","last_became_active":"2024-03-30T00:55:51.808636+0000","last_became_peered":"2024-03-30T00:55:51.808636+0000","last_unstale":"2024-03-30T00:57:35.165088+0000","last_undegraded":"2024-03-30T00:57:35.165088+0000","last_fullsized":"2024-03-30T00:57:35.165088+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:02.379294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.480854+0000","last_change":"2024-03-30T00:56:36.480854+0000","last_active":"2024-03-30T00:56:36.480854+0000","last_peered":"2024-03-30T00:56:36.480854+0000","last_clean":"2024-03-30T00:56:36.480854+0000","last_became_active":"2024-03-30T00:56:36.467064+0000","last_became_peered":"2024-03-30T00:56:36.467064+0000","last_unstale":"2024-03-30T00:56:36.480854+0000","last_undegraded":"2024-03-30T00:56:36.480854+0000","last_fullsized":"2024-03-30T00:56:36.480854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:12:55.154544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470669+0000","last_change":"2024-03-30T00:55:51.313708+0000","last_active":"2024-03-30T00:56:35.470669+0000","last_peered":"2024-03-30T00:56:35.470669+0000","last_clean":"2024-03-30T00:56:35.470669+0000","last_became_active":"2024-03-30T00:55:51.313362+0000","last_became_peered":"2024-03-30T00:55:51.313362+0000","last_unstale":"2024-03-30T00:56:35.470669+0000","last_undegraded":"2024-03-30T00:56:35.470669+0000","last_fullsized":"2024-03-30T00:56:35.470669+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:03:17.859640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165069+0000","last_change":"2024-03-30T00:55:51.819638+0000","last_active":"2024-03-30T00:57:35.165069+0000","last_peered":"2024-03-30T00:57:35.165069+0000","last_clean":"2024-03-30T00:57:35.165069+0000","last_became_active":"2024-03-30T00:55:51.819282+0000","last_became_peered":"2024-03-30T00:55:51.819282+0000","last_unstale":"2024-03-30T00:57:35.165069+0000","last_undegraded":"2024-03-30T00:57:35.165069+0000","last_fullsized":"2024-03-30T00:57:35.165069+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:13:39.249550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462909+0000","last_change":"2024-03-30T00:55:55.856498+0000","last_active":"2024-03-30T00:56:35.462909+0000","last_peered":"2024-03-30T00:56:35.462909+0000","last_clean":"2024-03-30T00:56:35.462909+0000","last_became_active":"2024-03-30T00:55:55.856015+0000","last_became_peered":"2024-03-30T00:55:55.856015+0000","last_unstale":"2024-03-30T00:56:35.462909+0000","last_undegraded":"2024-03-30T00:56:35.462909+0000","last_fullsized":"2024-03-30T00:56:35.462909+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:14:00.248812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462909+0000","last_change":"2024-03-30T00:55:51.302995+0000","last_active":"2024-03-30T00:56:35.462909+0000","last_peered":"2024-03-30T00:56:35.462909+0000","last_clean":"2024-03-30T00:56:35.462909+0000","last_became_active":"2024-03-30T00:55:51.302650+0000","last_became_peered":"2024-03-30T00:55:51.302650+0000","last_unstale":"2024-03-30T00:56:35.462909+0000","last_undegraded":"2024-03-30T00:56:35.462909+0000","last_fullsized":"2024-03-30T00:56:35.462909+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:17:32.556550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162368+0000","last_change":"2024-03-30T00:55:51.808969+0000","last_active":"2024-03-30T00:57:35.162368+0000","last_peered":"2024-03-30T00:57:35.162368+0000","last_clean":"2024-03-30T00:57:35.162368+0000","last_became_active":"2024-03-30T00:55:51.808633+0000","last_became_peered":"2024-03-30T00:55:51.808633+0000","last_unstale":"2024-03-30T00:57:35.162368+0000","last_undegraded":"2024-03-30T00:57:35.162368+0000","last_fullsized":"2024-03-30T00:57:35.162368+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:16:46.128066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468267+0000","last_change":"2024-03-30T00:55:55.824964+0000","last_active":"2024-03-30T00:56:35.468267+0000","last_peered":"2024-03-30T00:56:35.468267+0000","last_clean":"2024-03-30T00:56:35.468267+0000","last_became_active":"2024-03-30T00:55:55.824559+0000","last_became_peered":"2024-03-30T00:55:55.824559+0000","last_unstale":"2024-03-30T00:56:35.468267+0000","last_undegraded":"2024-03-30T00:56:35.468267+0000","last_fullsized":"2024-03-30T00:56:35.468267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:19:24.746963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'3","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470912+0000","last_change":"2024-03-30T00:55:51.301602+0000","last_active":"2024-03-30T00:56:35.470912+0000","last_peered":"2024-03-30T00:56:35.470912+0000","last_clean":"2024-03-30T00:56:35.470912+0000","last_became_active":"2024-03-30T00:55:51.301221+0000","last_became_peered":"2024-03-30T00:55:51.301221+0000","last_unstale":"2024-03-30T00:56:35.470912+0000","last_undegraded":"2024-03-30T00:56:35.470912+0000","last_fullsized":"2024-03-30T00:56:35.470912+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:37:36.727603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"83'7","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153553+0000","last_change":"2024-03-30T00:55:51.810197+0000","last_active":"2024-03-30T00:57:35.153553+0000","last_peered":"2024-03-30T00:57:35.153553+0000","last_clean":"2024-03-30T00:57:35.153553+0000","last_became_active":"2024-03-30T00:55:51.809193+0000","last_became_peered":"2024-03-30T00:55:51.809193+0000","last_unstale":"2024-03-30T00:57:35.153553+0000","last_undegraded":"2024-03-30T00:57:35.153553+0000","last_fullsized":"2024-03-30T00:57:35.153553+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:52:41.078425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.494873+0000","last_change":"2024-03-30T00:55:55.830549+0000","last_active":"2024-03-30T00:56:35.494873+0000","last_peered":"2024-03-30T00:56:35.494873+0000","last_clean":"2024-03-30T00:56:35.494873+0000","last_became_active":"2024-03-30T00:55:55.827966+0000","last_became_peered":"2024-03-30T00:55:55.827966+0000","last_unstale":"2024-03-30T00:56:35.494873+0000","last_undegraded":"2024-03-30T00:56:35.494873+0000","last_fullsized":"2024-03-30T00:56:35.494873+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:30:36.531960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"76'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467092+0000","last_change":"2024-03-30T00:55:51.311998+0000","last_active":"2024-03-30T00:56:35.467092+0000","last_peered":"2024-03-30T00:56:35.467092+0000","last_clean":"2024-03-30T00:56:35.467092+0000","last_became_active":"2024-03-30T00:55:51.311627+0000","last_became_peered":"2024-03-30T00:55:51.311627+0000","last_unstale":"2024-03-30T00:56:35.467092+0000","last_undegraded":"2024-03-30T00:56:35.467092+0000","last_fullsized":"2024-03-30T00:56:35.467092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:23:06.939919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.158955+0000","last_change":"2024-03-30T00:55:51.819457+0000","last_active":"2024-03-30T00:57:35.158955+0000","last_peered":"2024-03-30T00:57:35.158955+0000","last_clean":"2024-03-30T00:57:35.158955+0000","last_became_active":"2024-03-30T00:55:51.819099+0000","last_became_peered":"2024-03-30T00:55:51.819099+0000","last_unstale":"2024-03-30T00:57:35.158955+0000","last_undegraded":"2024-03-30T00:57:35.158955+0000","last_fullsized":"2024-03-30T00:57:35.158955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:31:57.363328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466097+0000","last_change":"2024-03-30T00:55:55.824882+0000","last_active":"2024-03-30T00:56:35.466097+0000","last_peered":"2024-03-30T00:56:35.466097+0000","last_clean":"2024-03-30T00:56:35.466097+0000","last_became_active":"2024-03-30T00:55:55.824493+0000","last_became_peered":"2024-03-30T00:55:55.824493+0000","last_unstale":"2024-03-30T00:56:35.466097+0000","last_undegraded":"2024-03-30T00:56:35.466097+0000","last_fullsized":"2024-03-30T00:56:35.466097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:24:48.825776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"76'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.476919+0000","last_change":"2024-03-30T00:55:51.311787+0000","last_active":"2024-03-30T00:56:35.476919+0000","last_peered":"2024-03-30T00:56:35.476919+0000","last_clean":"2024-03-30T00:56:35.476919+0000","last_became_active":"2024-03-30T00:55:51.311402+0000","last_became_peered":"2024-03-30T00:55:51.311402+0000","last_unstale":"2024-03-30T00:56:35.476919+0000","last_undegraded":"2024-03-30T00:56:35.476919+0000","last_fullsized":"2024-03-30T00:56:35.476919+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:37:13.960453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.151904+0000","last_change":"2024-03-30T00:55:51.809742+0000","last_active":"2024-03-30T00:57:35.151904+0000","last_peered":"2024-03-30T00:57:35.151904+0000","last_clean":"2024-03-30T00:57:35.151904+0000","last_became_active":"2024-03-30T00:55:51.809396+0000","last_became_peered":"2024-03-30T00:55:51.809396+0000","last_unstale":"2024-03-30T00:57:35.151904+0000","last_undegraded":"2024-03-30T00:57:35.151904+0000","last_fullsized":"2024-03-30T00:57:35.151904+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:22.535472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"83'9","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.145644+0000","last_change":"2024-03-30T00:55:51.814429+0000","last_active":"2024-03-30T00:57:35.145644+0000","last_peered":"2024-03-30T00:57:35.145644+0000","last_clean":"2024-03-30T00:57:35.145644+0000","last_became_active":"2024-03-30T00:55:51.814044+0000","last_became_peered":"2024-03-30T00:55:51.814044+0000","last_unstale":"2024-03-30T00:57:35.145644+0000","last_undegraded":"2024-03-30T00:57:35.145644+0000","last_fullsized":"2024-03-30T00:57:35.145644+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:24:58.830738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"83'12","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.146416+0000","last_change":"2024-03-30T00:55:51.828977+0000","last_active":"2024-03-30T00:57:35.146416+0000","last_peered":"2024-03-30T00:57:35.146416+0000","last_clean":"2024-03-30T00:57:35.146416+0000","last_became_active":"2024-03-30T00:55:51.828656+0000","last_became_peered":"2024-03-30T00:55:51.828656+0000","last_unstale":"2024-03-30T00:57:35.146416+0000","last_undegraded":"2024-03-30T00:57:35.146416+0000","last_fullsized":"2024-03-30T00:57:35.146416+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:10:23.463770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.161205+0000","last_change":"2024-03-30T00:55:51.821511+0000","last_active":"2024-03-30T00:57:35.161205+0000","last_peered":"2024-03-30T00:57:35.161205+0000","last_clean":"2024-03-30T00:57:35.161205+0000","last_became_active":"2024-03-30T00:55:51.820876+0000","last_became_peered":"2024-03-30T00:55:51.820876+0000","last_unstale":"2024-03-30T00:57:35.161205+0000","last_undegraded":"2024-03-30T00:57:35.161205+0000","last_fullsized":"2024-03-30T00:57:35.161205+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:14:48.299376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.167065+0000","last_change":"2024-03-30T00:55:51.823867+0000","last_active":"2024-03-30T00:57:35.167065+0000","last_peered":"2024-03-30T00:57:35.167065+0000","last_clean":"2024-03-30T00:57:35.167065+0000","last_became_active":"2024-03-30T00:55:51.820230+0000","last_became_peered":"2024-03-30T00:55:51.820230+0000","last_unstale":"2024-03-30T00:57:35.167065+0000","last_undegraded":"2024-03-30T00:57:35.167065+0000","last_fullsized":"2024-03-30T00:57:35.167065+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:38:51.257855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"83'4","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165868+0000","last_change":"2024-03-30T00:55:51.821413+0000","last_active":"2024-03-30T00:57:35.165868+0000","last_peered":"2024-03-30T00:57:35.165868+0000","last_clean":"2024-03-30T00:57:35.165868+0000","last_became_active":"2024-03-30T00:55:51.820833+0000","last_became_peered":"2024-03-30T00:55:51.820833+0000","last_unstale":"2024-03-30T00:57:35.165868+0000","last_undegraded":"2024-03-30T00:57:35.165868+0000","last_fullsized":"2024-03-30T00:57:35.165868+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:34:11.969849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.149312+0000","last_change":"2024-03-30T00:55:51.813674+0000","last_active":"2024-03-30T00:57:35.149312+0000","last_peered":"2024-03-30T00:57:35.149312+0000","last_clean":"2024-03-30T00:57:35.149312+0000","last_became_active":"2024-03-30T00:55:51.813341+0000","last_became_peered":"2024-03-30T00:55:51.813341+0000","last_unstale":"2024-03-30T00:57:35.149312+0000","last_undegraded":"2024-03-30T00:57:35.149312+0000","last_fullsized":"2024-03-30T00:57:35.149312+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:54:17.285997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"83'15","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.148932+0000","last_change":"2024-03-30T00:55:51.812472+0000","last_active":"2024-03-30T00:57:35.148932+0000","last_peered":"2024-03-30T00:57:35.148932+0000","last_clean":"2024-03-30T00:57:35.148932+0000","last_became_active":"2024-03-30T00:55:51.812061+0000","last_became_peered":"2024-03-30T00:55:51.812061+0000","last_unstale":"2024-03-30T00:57:35.148932+0000","last_undegraded":"2024-03-30T00:57:35.148932+0000","last_fullsized":"2024-03-30T00:57:35.148932+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:46.550201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"83'2","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.134547+0000","last_change":"2024-03-30T00:55:51.829179+0000","last_active":"2024-03-30T00:57:35.134547+0000","last_peered":"2024-03-30T00:57:35.134547+0000","last_clean":"2024-03-30T00:57:35.134547+0000","last_became_active":"2024-03-30T00:55:51.828805+0000","last_became_peered":"2024-03-30T00:55:51.828805+0000","last_unstale":"2024-03-30T00:57:35.134547+0000","last_undegraded":"2024-03-30T00:57:35.134547+0000","last_fullsized":"2024-03-30T00:57:35.134547+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:23.576125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.155774+0000","last_change":"2024-03-30T00:55:51.823170+0000","last_active":"2024-03-30T00:57:35.155774+0000","last_peered":"2024-03-30T00:57:35.155774+0000","last_clean":"2024-03-30T00:57:35.155774+0000","last_became_active":"2024-03-30T00:55:51.822785+0000","last_became_peered":"2024-03-30T00:55:51.822785+0000","last_unstale":"2024-03-30T00:57:35.155774+0000","last_undegraded":"2024-03-30T00:57:35.155774+0000","last_fullsized":"2024-03-30T00:57:35.155774+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:34:43.393540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153749+0000","last_change":"2024-03-30T00:55:51.817160+0000","last_active":"2024-03-30T00:57:35.153749+0000","last_peered":"2024-03-30T00:57:35.153749+0000","last_clean":"2024-03-30T00:57:35.153749+0000","last_became_active":"2024-03-30T00:55:51.816881+0000","last_became_peered":"2024-03-30T00:55:51.816881+0000","last_unstale":"2024-03-30T00:57:35.153749+0000","last_undegraded":"2024-03-30T00:57:35.153749+0000","last_fullsized":"2024-03-30T00:57:35.153749+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:09:05.211242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"83'10","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163651+0000","last_change":"2024-03-30T00:55:51.817670+0000","last_active":"2024-03-30T00:57:35.163651+0000","last_peered":"2024-03-30T00:57:35.163651+0000","last_clean":"2024-03-30T00:57:35.163651+0000","last_became_active":"2024-03-30T00:55:51.817380+0000","last_became_peered":"2024-03-30T00:55:51.817380+0000","last_unstale":"2024-03-30T00:57:35.163651+0000","last_undegraded":"2024-03-30T00:57:35.163651+0000","last_fullsized":"2024-03-30T00:57:35.163651+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:24.582557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"83'11","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.167761+0000","last_change":"2024-03-30T00:55:51.813071+0000","last_active":"2024-03-30T00:57:35.167761+0000","last_peered":"2024-03-30T00:57:35.167761+0000","last_clean":"2024-03-30T00:57:35.167761+0000","last_became_active":"2024-03-30T00:55:51.811681+0000","last_became_peered":"2024-03-30T00:55:51.811681+0000","last_unstale":"2024-03-30T00:57:35.167761+0000","last_undegraded":"2024-03-30T00:57:35.167761+0000","last_fullsized":"2024-03-30T00:57:35.167761+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:10:53.840055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.154929+0000","last_change":"2024-03-30T00:56:36.472592+0000","last_active":"2024-03-30T00:57:35.154929+0000","last_peered":"2024-03-30T00:57:35.154929+0000","last_clean":"2024-03-30T00:57:35.154929+0000","last_became_active":"2024-03-30T00:56:36.472173+0000","last_became_peered":"2024-03-30T00:56:36.472173+0000","last_unstale":"2024-03-30T00:57:35.154929+0000","last_undegraded":"2024-03-30T00:57:35.154929+0000","last_fullsized":"2024-03-30T00:57:35.154929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:22:27.915984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":212960,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71848,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":522,"num_read_kb":339,"num_write":168,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":215544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645345,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":462418,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.41499999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.73099999999999998,"5min":1.399,"15min":1.399},"last":0.44},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.622,"15min":0.626},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.83399999999999996,"5min":1.282,"15min":1.282},"last":0.57799999999999996}]},{"osd":1,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.61199999999999999,"15min":0.60699999999999998},"min":{"1min":0.46800000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.45900000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.69599999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.61199999999999999}]},{"osd":2,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.46500000000000002,"15min":0.45400000000000001},"min":{"1min":0.36799999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.46200000000000002,"15min":0.45800000000000002},"min":{"1min":0.373,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.41399999999999998}]},{"osd":3,"last update":"Sat Mar 30 00:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.505}]},{"osd":4,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.62,"15min":0.61199999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64000000000000001,"15min":0.63800000000000001},"min":{"1min":0.49399999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.76100000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.72299999999999998}]},{"osd":5,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.58399999999999996,"15min":0.57499999999999996},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.876,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.47199999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.54400000000000004}]},{"osd":6,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.51700000000000002,"5min":0.436,"15min":0.436},"max":{"1min":1.0149999999999999,"5min":1.056,"15min":1.056},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72699999999999998,"15min":0.71799999999999997},"min":{"1min":0.622,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97499999999999998,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.76100000000000001}]},{"osd":7,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71099999999999997,"15min":0.70299999999999996},"min":{"1min":0.66000000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.748},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72499999999999998,"15min":0.71799999999999997},"min":{"1min":0.68000000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.96499999999999997,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.68500000000000005}]},{"osd":8,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.78700000000000003}]},{"osd":9,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.80800000000000005,"15min":0.79700000000000004},"min":{"1min":0.70099999999999996,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.052,"5min":1.353,"15min":1.353},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.75700000000000001,"15min":0.751},"min":{"1min":0.69399999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98599999999999999,"5min":1.083,"15min":1.083},"last":0.71299999999999997}]},{"osd":10,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77400000000000002,"15min":0.77000000000000002},"min":{"1min":0.72499999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.90200000000000002,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.83399999999999996,"15min":0.82899999999999996},"min":{"1min":0.749,"5min":0.623,"15min":0.623},"max":{"1min":1.0580000000000001,"5min":1.52,"15min":1.52},"last":0.82799999999999996}]}]},{"osd":10,"up_from":65,"seq":279172874276,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":1121169,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.58399999999999996}]},{"osd":1,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.439,"15min":0.441},"min":{"1min":0.29299999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.53900000000000003,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.33800000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56599999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.42099999999999999}]},{"osd":2,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.54700000000000004,"15min":0.54000000000000004},"min":{"1min":0.38200000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.626},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.58499999999999996},"min":{"1min":0.42199999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.75700000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.504}]},{"osd":3,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82699999999999996,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63200000000000001,"15min":0.63900000000000001},"min":{"1min":0.495,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.60199999999999998}]},{"osd":4,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35999999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.65200000000000002,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.45000000000000001}]},{"osd":5,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.61599999999999999,"15min":0.61299999999999999},"min":{"1min":0.45600000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.46800000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.52900000000000003}]},{"osd":6,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.876,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68799999999999994,"15min":0.68999999999999995},"min":{"1min":0.52300000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78500000000000003,"5min":1.131,"15min":1.131},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":8,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.69599999999999995,"15min":0.69799999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.80800000000000005,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.55400000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.83399999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.69199999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.75900000000000001,"15min":0.76500000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.85999999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.748,"15min":0.752},"min":{"1min":0.52400000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.85999999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.72299999999999998}]},{"osd":11,"last update":"Sat Mar 30 00:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73099999999999998,"15min":0.73299999999999998},"min":{"1min":0.60299999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.83099999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.749,"15min":0.749},"min":{"1min":0.59499999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.85699999999999998,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.81699999999999995}]}]},{"osd":9,"up_from":60,"seq":257698037798,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":462620,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.628,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.316}]},{"osd":1,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.94199999999999995,"5min":1.274,"15min":1.274},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.73399999999999999,"15min":0.74299999999999999},"min":{"1min":0.51600000000000001,"5min":0.504,"15min":0.504},"max":{"1min":0.90600000000000003,"5min":1.411,"15min":1.411},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.73399999999999999,"15min":0.74299999999999999},"min":{"1min":0.55400000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.84699999999999998,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.501,"5min":0.435,"15min":0.435},"max":{"1min":0.83999999999999997,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.83899999999999997}]},{"osd":3,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.105,"5min":1.391,"15min":1.391},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.73999999999999999,"15min":0.75900000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95899999999999996,"5min":1.302,"15min":1.302},"last":0.434}]},{"osd":4,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69199999999999995,"15min":0.69799999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.88700000000000001,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.877},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.81100000000000005,"15min":0.82299999999999995},"min":{"1min":0.56200000000000006,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98299999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.63}]},{"osd":5,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.71899999999999997,"15min":0.72899999999999998},"min":{"1min":0.45800000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.88300000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.83799999999999997,"15min":0.85799999999999998},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0129999999999999,"5min":1.853,"15min":1.853},"last":0.5}]},{"osd":6,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.623}]},{"osd":7,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.86399999999999999,"15min":0.88},"min":{"1min":0.54100000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.94099999999999995,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.83299999999999996,"15min":0.84399999999999997},"min":{"1min":0.58699999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94599999999999995,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.91200000000000003}]},{"osd":8,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.90100000000000002,"15min":0.92000000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.98099999999999998,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.90000000000000002,"15min":0.92200000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0720000000000001,"5min":1.968,"15min":1.968},"last":0.52000000000000002}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.88700000000000001,"15min":0.88100000000000001},"min":{"1min":0.65000000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.8170000000000002,"5min":2.8170000000000002,"15min":2.8170000000000002},"last":0.622},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.84999999999999998,"15min":0.84099999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":2.9889999999999999,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":0.94199999999999995}]},{"osd":11,"last update":"Sat Mar 30 00:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.53600000000000003}]}]},{"osd":8,"up_from":53,"seq":227633266731,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":921600,"data_stored":462209,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56499999999999995,"15min":0.56100000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77500000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54900000000000004,"15min":0.54400000000000004},"min":{"1min":0.40999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.80400000000000005,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.505}]},{"osd":1,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.59199999999999997,"15min":0.57299999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78700000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59299999999999997,"15min":0.57999999999999996},"min":{"1min":0.47999999999999998,"5min":0.436,"15min":0.436},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.44600000000000001,"15min":0.42599999999999999},"min":{"1min":0.35899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64100000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.45000000000000001},"min":{"1min":0.38800000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.68999999999999995,"5min":0.751,"15min":0.751},"last":0.64500000000000002}]},{"osd":3,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.68899999999999995,"15min":0.72399999999999998},"min":{"1min":0.434,"5min":0.372,"15min":0.372},"max":{"1min":0.79700000000000004,"5min":2.883,"15min":2.883},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.59999999999999998,"15min":0.58399999999999996},"min":{"1min":0.47599999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.62}]},{"osd":4,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.61099999999999999,"15min":0.60599999999999998},"min":{"1min":0.51000000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.85599999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.629,"15min":0.60699999999999998},"min":{"1min":0.53100000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.57599999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.52300000000000002,"15min":0.498},"min":{"1min":0.41299999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.55700000000000005,"15min":0.53900000000000003},"min":{"1min":0.45100000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84199999999999997,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.40200000000000002}]},{"osd":6,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.66200000000000003,"15min":0.63800000000000001},"min":{"1min":0.56100000000000005,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.77000000000000002,"15min":0.78700000000000003},"min":{"1min":0.59999999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.92600000000000005,"5min":2.8940000000000001,"15min":2.8940000000000001},"last":0.60699999999999998}]},{"osd":7,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.67600000000000005,"15min":0.66000000000000003},"min":{"1min":0.51600000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88500000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.54400000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.88500000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.58999999999999997}]},{"osd":9,"last update":"Sat Mar 30 00:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.70999999999999996,"15min":0.69899999999999995},"min":{"1min":0.58999999999999997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93500000000000005,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.74299999999999999,"15min":0.72599999999999998},"min":{"1min":0.68600000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.65800000000000003}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.53800000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91300000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.75700000000000001,"15min":0.74399999999999999},"min":{"1min":0.66100000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.94899999999999995,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.74099999999999999}]},{"osd":11,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.78700000000000003}]}]},{"osd":7,"up_from":46,"seq":197568495663,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":1059005,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60199999999999998}]},{"osd":1,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.44800000000000001,"15min":0.44700000000000001},"min":{"1min":0.32500000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45000000000000001,"15min":0.442},"min":{"1min":0.33700000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68700000000000006,"5min":0.747,"15min":0.747},"last":0.379}]},{"osd":2,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83299999999999996,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.621},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58199999999999996,"15min":0.56399999999999995},"min":{"1min":0.40600000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78900000000000003,"5min":0.877,"15min":0.877},"last":0.50600000000000001}]},{"osd":3,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60599999999999998,"15min":0.60199999999999998},"min":{"1min":0.443,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.85299999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.66100000000000003,"15min":0.625},"min":{"1min":0.45800000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":5.7430000000000003,"5min":5.7430000000000003,"15min":5.7430000000000003},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53100000000000003,"15min":0.52600000000000002},"min":{"1min":0.39500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66400000000000003,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.314,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.69399999999999995,"5min":1.008,"15min":1.008},"last":0.40899999999999997}]},{"osd":5,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63300000000000001,"15min":0.62},"min":{"1min":0.47399999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82699999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.63300000000000001,"15min":0.62},"min":{"1min":0.49399999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80600000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.66300000000000003}]},{"osd":6,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66700000000000004,"15min":0.65500000000000003},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.372},"max":{"1min":0.86799999999999999,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.73899999999999999,"15min":0.71899999999999997},"min":{"1min":0.51000000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89200000000000002,"5min":4.117,"15min":4.117},"last":0.52700000000000002}]},{"osd":8,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.621,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86599999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.752,"15min":0.751},"min":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.88900000000000001,"5min":1.014,"15min":1.014},"last":0.65600000000000003}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.74399999999999999,"15min":0.73999999999999999},"min":{"1min":0.60299999999999998,"5min":0.495,"15min":0.495},"max":{"1min":0.94199999999999995,"5min":1.018,"15min":1.018},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.91400000000000003,"15min":0.91900000000000004},"min":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":3.8839999999999999,"5min":4.0519999999999996,"15min":4.0519999999999996},"last":0.67600000000000005}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.755,"15min":0.76400000000000001},"min":{"1min":0.51900000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91300000000000003,"5min":1.032,"15min":1.032},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.71299999999999997,"15min":0.71699999999999997},"min":{"1min":0.59499999999999997,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55500000000000005}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.71599999999999997}]}]},{"osd":6,"up_from":42,"seq":180388626483,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":905216,"data_stored":457796,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.308,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.499,"15min":0.48699999999999999},"min":{"1min":0.373,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.878}]},{"osd":1,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.66300000000000003,"15min":0.63500000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60599999999999998,"15min":0.59599999999999997},"min":{"1min":0.39300000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.68100000000000005}]},{"osd":2,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70899999999999996,"15min":0.68300000000000005},"min":{"1min":0.51000000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.98699999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.66500000000000004,"15min":0.64300000000000002},"min":{"1min":0.47299999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":1.004}]},{"osd":3,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.65000000000000002},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.97999999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67600000000000005,"15min":0.67200000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.76300000000000001,"15min":0.751},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.105,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69399999999999995,"15min":0.68200000000000005},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.93899999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.74099999999999999,"15min":0.73099999999999998},"min":{"1min":0.51600000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":1.069}]},{"osd":7,"last update":"Sat Mar 30 00:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.81399999999999995},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.046,"5min":1.127,"15min":1.127},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76800000000000002,"15min":0.76900000000000002},"min":{"1min":0.57699999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.2330000000000001,"5min":1.335,"15min":1.335},"last":0.65900000000000003}]},{"osd":8,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.82399999999999995,"15min":0.84499999999999997},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.2709999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.626},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.876,"15min":0.88800000000000001},"min":{"1min":0.628,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0009999999999999,"5min":1.206,"15min":1.206},"last":0.76700000000000002}]},{"osd":9,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.94299999999999995,"15min":0.95199999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.244,"5min":1.268,"15min":1.268},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.347,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":1.103}]},{"osd":10,"last update":"Sat Mar 30 00:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85499999999999998,"15min":0.85699999999999998},"min":{"1min":0.69199999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":1.204},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.92500000000000004,"15min":0.93000000000000005},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.0760000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.82599999999999996}]},{"osd":11,"last update":"Sat Mar 30 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.71499999999999997}]}]},{"osd":1,"up_from":12,"seq":51539607621,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":524393,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54900000000000004,"15min":0.53300000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65800000000000003,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.58599999999999997,"15min":0.55500000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.74399999999999999,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.45700000000000002}]},{"osd":2,"last update":"Sat Mar 30 00:57:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.56999999999999995,"15min":0.50700000000000001},"min":{"1min":0.47299999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.81100000000000005,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.57599999999999996,"15min":0.51400000000000001},"min":{"1min":0.42599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.70799999999999996}]},{"osd":3,"last update":"Sat Mar 30 00:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56200000000000006,"15min":0.52800000000000002},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.69199999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.503},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57399999999999995,"15min":0.55000000000000004},"min":{"1min":0.501,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.66900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.626}]},{"osd":4,"last update":"Sat Mar 30 00:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.439,"15min":0.41499999999999998},"min":{"1min":0.377,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.50600000000000001,"15min":0.47399999999999998},"min":{"1min":0.38200000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.57899999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.61099999999999999,"15min":0.61299999999999999},"min":{"1min":0.52900000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78600000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.66900000000000004,"15min":0.64500000000000002},"min":{"1min":0.625,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.748}]},{"osd":6,"last update":"Sat Mar 30 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.55700000000000005,"5min":0.435,"15min":0.435},"max":{"1min":0.81699999999999995,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.54800000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.84799999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.68300000000000005}]},{"osd":7,"last update":"Sat Mar 30 00:57:08 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.57799999999999996,"15min":0.55900000000000005},"min":{"1min":0.48899999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.59399999999999997,"15min":0.58699999999999997},"min":{"1min":0.5,"5min":0.313,"15min":0.313},"max":{"1min":0.91800000000000004,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.73599999999999999}]},{"osd":8,"last update":"Sat Mar 30 00:57:31 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.67200000000000004,"15min":0.66000000000000003},"min":{"1min":0.58999999999999997,"5min":0.38,"15min":0.38},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.63900000000000001,"15min":0.63},"min":{"1min":0.52100000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.69899999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.746,"15min":0.73699999999999999},"min":{"1min":0.56899999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.91600000000000004,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72999999999999998,"15min":0.72399999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.81100000000000005}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.66200000000000003,"15min":0.65400000000000003},"min":{"1min":0.50800000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.74099999999999999,"15min":0.72899999999999998},"min":{"1min":0.71999999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.95499999999999996,"5min":1.385,"15min":1.385},"last":0.82799999999999996}]},{"osd":11,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.79700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32016,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713136,"statfs":{"total":95995035648,"available":95962251264,"internally_reserved":0,"allocated":925696,"data_stored":458911,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 00:57:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.67800000000000005,"15min":0.60199999999999998},"min":{"1min":0.39300000000000002,"5min":0.372,"15min":0.372},"max":{"1min":9.266,"5min":9.266,"15min":9.266},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.624,"15min":0.57799999999999996},"min":{"1min":0.42299999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.6899999999999999,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":0.65400000000000003}]},{"osd":2,"last update":"Sat Mar 30 00:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.59599999999999997,"15min":0.54300000000000004},"min":{"1min":0.41399999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71999999999999997,"15min":0.60499999999999998},"min":{"1min":0.46300000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.071,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":0.52600000000000002}]},{"osd":3,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.9139999999999997,"5min":7.9139999999999997,"15min":7.9139999999999997},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.96,"5min":7.96,"15min":7.96},"last":0.503}]},{"osd":4,"last update":"Sat Mar 30 00:57:16 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":0.75800000000000001,"15min":0.65900000000000003},"min":{"1min":0.48199999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":9.2349999999999994,"5min":9.2349999999999994,"15min":9.2349999999999994},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.71199999999999997,"15min":0.63400000000000001},"min":{"1min":0.626,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":3.73,"5min":3.73,"15min":3.73},"last":0.70199999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76400000000000001,"15min":0.67300000000000004},"min":{"1min":0.52300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.02,"5min":9.0579999999999998,"15min":9.0579999999999998},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.72399999999999998,"15min":0.64900000000000002},"min":{"1min":0.53200000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.157,"5min":3.8039999999999998,"15min":3.8039999999999998},"last":0.67800000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":0.72099999999999997,"15min":0.65900000000000003},"min":{"1min":0.42399999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":8.0050000000000008,"5min":8.0050000000000008,"15min":8.0050000000000008},"last":0.628},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.66800000000000004,"15min":0.65800000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.71399999999999997}]},{"osd":7,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.4140000000000001,"5min":2.4140000000000001,"15min":2.4140000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":9.2959999999999994,"5min":9.2959999999999994,"15min":9.2959999999999994},"last":0.78700000000000003}]},{"osd":8,"last update":"Sat Mar 30 00:57:29 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.78200000000000003,"15min":0.78000000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.375,"5min":2.375,"15min":2.375},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.226,"5min":0.91200000000000003,"15min":0.876},"min":{"1min":0.57299999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":9.2040000000000006,"5min":9.2040000000000006,"15min":9.2040000000000006},"last":0.76900000000000002}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":0.88700000000000001,"15min":0.83699999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":8.0389999999999997,"5min":8.0389999999999997,"15min":8.0389999999999997},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.78400000000000003,"15min":0.76600000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":0.83599999999999997}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":0.90800000000000003,"15min":0.88200000000000001},"min":{"1min":0.64000000000000001,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":3.8519999999999999,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":0.752},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.82799999999999996,"15min":0.81200000000000006},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.4470000000000001,"5min":2.4470000000000001,"15min":2.4470000000000001},"last":0.96399999999999997}]},{"osd":11,"last update":"Sat Mar 30 00:57:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.86099999999999999,"15min":0.84799999999999998},"min":{"1min":0.73699999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":0.995},{"interface":"front","average":{"1min":1.3140000000000001,"5min":0.92400000000000004,"15min":0.85899999999999999},"min":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":9.1709999999999994,"5min":9.1709999999999994,"15min":9.1709999999999994},"last":0.83199999999999996}]}]},{"osd":2,"up_from":17,"seq":73014444099,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":525799,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54100000000000004,"15min":0.52900000000000003},"min":{"1min":0.41399999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.70999999999999996,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54600000000000004,"15min":0.52700000000000002},"min":{"1min":0.38800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72899999999999998,"5min":1.097,"15min":1.097},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Mar 30 00:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.56699999999999995,"15min":0.52400000000000002},"min":{"1min":0.46400000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82599999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56000000000000005,"15min":0.53400000000000003},"min":{"1min":0.46200000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.84099999999999997,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.70699999999999996}]},{"osd":3,"last update":"Sat Mar 30 00:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.59499999999999997,"15min":0.56499999999999995},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.80300000000000005,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59799999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.72499999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.65800000000000003}]},{"osd":4,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.95799999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:57:31 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.45800000000000002,"15min":0.436},"min":{"1min":0.378,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.78800000000000003,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.48399999999999999,"15min":0.46200000000000002},"min":{"1min":0.38300000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73199999999999998,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.76700000000000002}]},{"osd":6,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65100000000000002,"15min":0.65000000000000002},"min":{"1min":0.44600000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86399999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66500000000000004,"15min":0.66100000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.85999999999999999}]},{"osd":7,"last update":"Sat Mar 30 00:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.88},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.79000000000000004}]},{"osd":8,"last update":"Sat Mar 30 00:57:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60199999999999998,"15min":0.59199999999999997},"min":{"1min":0.48199999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.61299999999999999,"15min":0.60099999999999998},"min":{"1min":0.497,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.84499999999999997}]},{"osd":9,"last update":"Sat Mar 30 00:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.72199999999999998,"15min":0.70699999999999996},"min":{"1min":0.58599999999999997,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.75700000000000001,"15min":0.73899999999999999},"min":{"1min":0.65600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.81799999999999995}]},{"osd":10,"last update":"Sat Mar 30 00:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67300000000000004,"15min":0.66700000000000004},"min":{"1min":0.55000000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91000000000000003,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.68100000000000005,"15min":0.66700000000000004},"min":{"1min":0.66700000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.97899999999999998}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":1.073}]}]},{"osd":3,"up_from":24,"seq":103079215166,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":457717,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.46000000000000002,"15min":0.41399999999999998},"min":{"1min":0.32900000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46400000000000002,"15min":0.432},"min":{"1min":0.35199999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.997,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.30599999999999999}]},{"osd":1,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65100000000000002,"15min":0.60299999999999998},"min":{"1min":0.45000000000000001,"5min":0.375,"15min":0.375},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.439,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.58099999999999996}]},{"osd":2,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.67200000000000004,"15min":0.629},"min":{"1min":0.51400000000000001,"5min":0.371,"15min":0.371},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64800000000000002,"15min":0.622},"min":{"1min":0.47599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.68799999999999994}]},{"osd":4,"last update":"Sat Mar 30 00:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70799999999999996,"15min":0.66800000000000004},"min":{"1min":0.501,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67900000000000005,"15min":0.66500000000000004},"min":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65100000000000002}]},{"osd":5,"last update":"Sat Mar 30 00:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.67400000000000004,"15min":0.65200000000000002},"min":{"1min":0.51000000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.73399999999999999,"15min":0.67600000000000005},"min":{"1min":0.48799999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":0.53800000000000003}]},{"osd":6,"last update":"Sat Mar 30 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.65000000000000002,"15min":0.63800000000000001},"min":{"1min":0.46800000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.44,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.34799999999999998}]},{"osd":7,"last update":"Sat Mar 30 00:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.74399999999999999,"15min":0.75},"min":{"1min":0.51200000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83699999999999997,"15min":0.83999999999999997},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.73399999999999999}]},{"osd":8,"last update":"Sat Mar 30 00:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.57399999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.92400000000000004,"15min":0.94399999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.2410000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.502},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.84499999999999997,"15min":0.85899999999999999},"min":{"1min":0.54700000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.224,"5min":1.347,"15min":1.347},"last":0.52500000000000002}]},{"osd":10,"last update":"Sat Mar 30 00:57:03 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77600000000000002,"15min":0.77100000000000002},"min":{"1min":0.53800000000000003,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.92000000000000004,"15min":0.92200000000000004},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.61499999999999999}]},{"osd":11,"last update":"Sat Mar 30 00:56:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.755},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.60599999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986235,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":909312,"data_stored":462634,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.66700000000000004,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.79300000000000004,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73099999999999998,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.41099999999999998,"15min":0.39600000000000002},"min":{"1min":0.34899999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.65300000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.41899999999999998,"15min":0.40699999999999997},"min":{"1min":0.36199999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71799999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.45200000000000001}]},{"osd":2,"last update":"Sat Mar 30 00:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.56799999999999995,"15min":0.55000000000000004},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83299999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.56699999999999995,"15min":0.54300000000000004},"min":{"1min":0.45700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71699999999999997,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.51200000000000001}]},{"osd":3,"last update":"Sat Mar 30 00:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.63700000000000001,"15min":0.69099999999999995},"min":{"1min":0.48899999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.74399999999999999,"5min":3.887,"15min":3.887},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.45000000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.53800000000000003}]},{"osd":5,"last update":"Sat Mar 30 00:57:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.58499999999999996,"15min":0.56699999999999995},"min":{"1min":0.38600000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.81000000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.59599999999999997,"15min":0.57499999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55400000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.623,"15min":0.625},"min":{"1min":0.42699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.64800000000000002,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82599999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.63100000000000001}]},{"osd":7,"last update":"Sat Mar 30 00:57:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55100000000000005,"15min":0.54200000000000004},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.79400000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.54800000000000004,"15min":0.52800000000000002},"min":{"1min":0.46100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.73899999999999999}]},{"osd":8,"last update":"Sat Mar 30 00:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.66500000000000004,"15min":0.65600000000000003},"min":{"1min":0.51800000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94299999999999995,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.68799999999999994,"15min":0.68000000000000005},"min":{"1min":0.53800000000000003,"5min":0.432,"15min":0.432},"max":{"1min":0.84099999999999997,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.64300000000000002}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.69899999999999995},"min":{"1min":0.53500000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.73299999999999998,"15min":0.73699999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.98999999999999999,"5min":1.012,"15min":1.012},"last":0.71099999999999997}]},{"osd":10,"last update":"Sat Mar 30 00:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.53600000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.97499999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.81599999999999995}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.751},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.79400000000000004}]}]},{"osd":5,"up_from":35,"seq":150323855414,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":978944,"data_stored":1053801,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.56899999999999995,"15min":0.52000000000000002},"min":{"1min":0.38,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":5.3970000000000002,"5min":5.3970000000000002,"15min":5.3970000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.499},"min":{"1min":0.41499999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.69599999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.45600000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57399999999999995,"15min":0.56599999999999995},"min":{"1min":0.47399999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.74299999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.56999999999999995,"15min":0.54600000000000004},"min":{"1min":0.51000000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.51000000000000001}]},{"osd":2,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.34999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.52700000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.40600000000000003,"15min":0.39900000000000002},"min":{"1min":0.32100000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.68600000000000005}]},{"osd":3,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.42599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.74399999999999999,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54800000000000004,"15min":0.53200000000000003},"min":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.68400000000000005,"5min":0.877,"15min":0.877},"last":0.54800000000000004}]},{"osd":4,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.56499999999999995},"min":{"1min":0.495,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.71799999999999997,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.60599999999999998,"15min":0.58599999999999997},"min":{"1min":0.41599999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.83699999999999997,"5min":1.038,"15min":1.038},"last":0.73299999999999998}]},{"osd":6,"last update":"Sat Mar 30 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59899999999999998,"15min":0.58099999999999996},"min":{"1min":0.42799999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81200000000000006,"5min":0.997,"15min":0.997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60499999999999998,"15min":0.59799999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.72999999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.60799999999999998}]},{"osd":7,"last update":"Sat Mar 30 00:57:09 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61899999999999999,"15min":0.60699999999999998},"min":{"1min":0.505,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.64000000000000001},"min":{"1min":0.55600000000000005,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.85399999999999998,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.58799999999999997}]},{"osd":8,"last update":"Sat Mar 30 00:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.55300000000000005,"15min":0.53600000000000003},"min":{"1min":0.373,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82799999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.56399999999999995,"15min":0.54800000000000004},"min":{"1min":0.45800000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77700000000000002,"5min":0.872,"15min":0.872},"last":0.67000000000000004}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.67300000000000004,"15min":0.66700000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68400000000000005,"15min":0.68100000000000005},"min":{"1min":0.58099999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63300000000000001}]},{"osd":10,"last update":"Sat Mar 30 00:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.70599999999999996,"15min":0.70399999999999996},"min":{"1min":0.54700000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.92200000000000004,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.71799999999999997,"15min":0.71699999999999997},"min":{"1min":0.60499999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.89800000000000002,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.66000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":69330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T00:57:45.898 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T00:57:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.114+0000 7f6d81e00700 1 -- 172.21.15.116:0/362710874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d7c100c80 msgr2=0x7f6d7c1010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:46.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.114+0000 7f6d81e00700 1 --2- 172.21.15.116:0/362710874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d7c100c80 0x7f6d7c1010a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c0098f0 tx=0x7f6d6c009c00 comp rx=0 tx=0).stop 2024-03-30T00:57:46.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.114+0000 7f6d81e00700 1 -- 172.21.15.116:0/362710874 shutdown_connections 2024-03-30T00:57:46.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.114+0000 7f6d81e00700 1 --2- 172.21.15.116:0/362710874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d7c100c80 0x7f6d7c1010a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.114+0000 7f6d81e00700 1 --2- 172.21.15.116:0/362710874 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d7c102810 0x7f6d7c1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.114+0000 7f6d81e00700 1 --2- 172.21.15.116:0/362710874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d7c101e70 0x7f6d7c1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.114+0000 7f6d81e00700 1 -- 172.21.15.116:0/362710874 >> 172.21.15.116:0/362710874 conn(0x7f6d7c0fc1f0 msgr2=0x7f6d7c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:46.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.114+0000 7f6d81e00700 1 -- 172.21.15.116:0/362710874 shutdown_connections 2024-03-30T00:57:46.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.114+0000 7f6d81e00700 1 -- 172.21.15.116:0/362710874 wait complete. 2024-03-30T00:57:46.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.115+0000 7f6d81e00700 1 Processor -- start 2024-03-30T00:57:46.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.115+0000 7f6d81e00700 1 -- start start 2024-03-30T00:57:46.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.115+0000 7f6d81e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d7c100c80 0x7f6d7c198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:46.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.115+0000 7f6d81e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d7c101e70 0x7f6d7c199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:46.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.115+0000 7f6d81e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d7c102810 0x7f6d7c1a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:46.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.115+0000 7f6d81e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c1a09b0 con 0x7f6d7c101e70 2024-03-30T00:57:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.115+0000 7f6d81e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c1a0af0 con 0x7f6d7c100c80 2024-03-30T00:57:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.115+0000 7f6d81e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c1a0c80 con 0x7f6d7c102810 2024-03-30T00:57:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.116+0000 7f6d7aa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d7c100c80 0x7f6d7c198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.116+0000 7f6d7b400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d7c102810 0x7f6d7c1a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.116+0000 7f6d7aa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d7c100c80 0x7f6d7c198f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:60410/0 (socket says 172.21.15.116:60410) 2024-03-30T00:57:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.116+0000 7f6d7aa00700 1 -- 172.21.15.116:0/2698010160 learned_addr learned my addr 172.21.15.116:0/2698010160 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.116+0000 7f6d7a000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d7c101e70 0x7f6d7c199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.116+0000 7f6d7aa00700 1 -- 172.21.15.116:0/2698010160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d7c102810 msgr2=0x7f6d7c1a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.116+0000 7f6d7aa00700 1 --2- 172.21.15.116:0/2698010160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d7c102810 0x7f6d7c1a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.116+0000 7f6d7aa00700 1 -- 172.21.15.116:0/2698010160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d7c101e70 msgr2=0x7f6d7c199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.116+0000 7f6d7aa00700 1 --2- 172.21.15.116:0/2698010160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d7c101e70 0x7f6d7c199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.116+0000 7f6d7aa00700 1 -- 172.21.15.116:0/2698010160 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c009520 con 0x7f6d7c100c80 2024-03-30T00:57:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.116+0000 7f6d7aa00700 1 --2- 172.21.15.116:0/2698010160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d7c100c80 0x7f6d7c198f20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c009810 tx=0x7f6d6c00bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.116+0000 7f6d80a00700 1 -- 172.21.15.116:0/2698010160 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c01c070 con 0x7f6d7c100c80 2024-03-30T00:57:46.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.117+0000 7f6d80a00700 1 -- 172.21.15.116:0/2698010160 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d6c005040 con 0x7f6d7c100c80 2024-03-30T00:57:46.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.117+0000 7f6d81e00700 1 -- 172.21.15.116:0/2698010160 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7c1a0f00 con 0x7f6d7c100c80 2024-03-30T00:57:46.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.117+0000 7f6d80a00700 1 -- 172.21.15.116:0/2698010160 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c0038c0 con 0x7f6d7c100c80 2024-03-30T00:57:46.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.117+0000 7f6d81e00700 1 -- 172.21.15.116:0/2698010160 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d7c1a1270 con 0x7f6d7c100c80 2024-03-30T00:57:46.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.118+0000 7f6d81e00700 1 -- 172.21.15.116:0/2698010160 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d7c067a80 con 0x7f6d7c100c80 2024-03-30T00:57:46.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.120+0000 7f6d80a00700 1 -- 172.21.15.116:0/2698010160 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f6d6c020030 con 0x7f6d7c100c80 2024-03-30T00:57:46.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.121+0000 7f6d80a00700 1 --2- 172.21.15.116:0/2698010160 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6d600a1120 0x7f6d600a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:46.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.121+0000 7f6d80a00700 1 -- 172.21.15.116:0/2698010160 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6d6c0ce0c0 con 0x7f6d7c100c80 2024-03-30T00:57:46.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.121+0000 7f6d7a000700 1 --2- 172.21.15.116:0/2698010160 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6d600a1120 0x7f6d600a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:46.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.122+0000 7f6d7a000700 1 --2- 172.21.15.116:0/2698010160 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6d600a1120 0x7f6d600a35e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6d68005fd0 tx=0x7f6d68009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:46.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.124+0000 7f6d80a00700 1 -- 172.21.15.116:0/2698010160 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6d6c097570 con 0x7f6d7c100c80 2024-03-30T00:57:46.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.329+0000 7f6d81e00700 1 -- 172.21.15.116:0/2698010160 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6d7c067a10 con 0x7f6d600a1120 2024-03-30T00:57:46.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.341+0000 7f6d80a00700 1 -- 172.21.15.116:0/2698010160 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427819 (secure 0 0 0) 0x7f6d7c067a10 con 0x7f6d600a1120 2024-03-30T00:57:46.354 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:46.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.346+0000 7f6d81e00700 1 -- 172.21.15.116:0/2698010160 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6d600a1120 msgr2=0x7f6d600a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:46.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.346+0000 7f6d81e00700 1 --2- 172.21.15.116:0/2698010160 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6d600a1120 0x7f6d600a35e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6d68005fd0 tx=0x7f6d68009310 comp rx=0 tx=0).stop 2024-03-30T00:57:46.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.346+0000 7f6d81e00700 1 -- 172.21.15.116:0/2698010160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d7c100c80 msgr2=0x7f6d7c198f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:46.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.346+0000 7f6d81e00700 1 --2- 172.21.15.116:0/2698010160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d7c100c80 0x7f6d7c198f20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c009810 tx=0x7f6d6c00bf60 comp rx=0 tx=0).stop 2024-03-30T00:57:46.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.347+0000 7f6d81e00700 1 -- 172.21.15.116:0/2698010160 shutdown_connections 2024-03-30T00:57:46.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.347+0000 7f6d81e00700 1 --2- 172.21.15.116:0/2698010160 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6d600a1120 0x7f6d600a35e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.347+0000 7f6d81e00700 1 --2- 172.21.15.116:0/2698010160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6d7c102810 0x7f6d7c1a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.347+0000 7f6d81e00700 1 --2- 172.21.15.116:0/2698010160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d7c100c80 0x7f6d7c198f20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.347+0000 7f6d81e00700 1 --2- 172.21.15.116:0/2698010160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d7c101e70 0x7f6d7c199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.347+0000 7f6d81e00700 1 -- 172.21.15.116:0/2698010160 >> 172.21.15.116:0/2698010160 conn(0x7f6d7c0fc1f0 msgr2=0x7f6d7c1076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:46.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.347+0000 7f6d81e00700 1 -- 172.21.15.116:0/2698010160 shutdown_connections 2024-03-30T00:57:46.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.347+0000 7f6d81e00700 1 -- 172.21.15.116:0/2698010160 wait complete. 2024-03-30T00:57:46.359 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-30T00:57:46.372 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-03-30T00:57:45.130320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662450,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":648,"num_read_kb":448,"num_write":351,"num_write_kb":3068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180887,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":788,"ondisk_log_size":788,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":351704,"kb_used_data":11128,"kb_used_omap":18,"kb_used_meta":340461,"kb_avail":1124590120,"statfs":{"total":1151940427776,"available":1151580282880,"internally_reserved":0,"allocated":11395072,"data_stored":7508472,"data_compressed":320673,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19128,"internal_metadata":348632392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6516,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":86,"num_read_kb":51,"num_write":12,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003660"},"pg_stats":[{"pgid":"3.3f","version":"83'8","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.167286+0000","last_change":"2024-03-30T00:55:51.809659+0000","last_active":"2024-03-30T00:57:35.167286+0000","last_peered":"2024-03-30T00:57:35.167286+0000","last_clean":"2024-03-30T00:57:35.167286+0000","last_became_active":"2024-03-30T00:55:51.809337+0000","last_became_peered":"2024-03-30T00:55:51.809337+0000","last_unstale":"2024-03-30T00:57:35.167286+0000","last_undegraded":"2024-03-30T00:57:35.167286+0000","last_fullsized":"2024-03-30T00:57:35.167286+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:20:50.640033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"83'4","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162969+0000","last_change":"2024-03-30T00:55:51.813778+0000","last_active":"2024-03-30T00:57:35.162969+0000","last_peered":"2024-03-30T00:57:35.162969+0000","last_clean":"2024-03-30T00:57:35.162969+0000","last_became_active":"2024-03-30T00:55:51.810347+0000","last_became_peered":"2024-03-30T00:55:51.810347+0000","last_unstale":"2024-03-30T00:57:35.162969+0000","last_undegraded":"2024-03-30T00:57:35.162969+0000","last_fullsized":"2024-03-30T00:57:35.162969+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:49:07.031477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166821+0000","last_change":"2024-03-30T00:55:51.817647+0000","last_active":"2024-03-30T00:57:35.166821+0000","last_peered":"2024-03-30T00:57:35.166821+0000","last_clean":"2024-03-30T00:57:35.166821+0000","last_became_active":"2024-03-30T00:55:51.816495+0000","last_became_peered":"2024-03-30T00:55:51.816495+0000","last_unstale":"2024-03-30T00:57:35.166821+0000","last_undegraded":"2024-03-30T00:57:35.166821+0000","last_fullsized":"2024-03-30T00:57:35.166821+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:59:42.965346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"83'8","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153424+0000","last_change":"2024-03-30T00:55:51.817581+0000","last_active":"2024-03-30T00:57:35.153424+0000","last_peered":"2024-03-30T00:57:35.153424+0000","last_clean":"2024-03-30T00:57:35.153424+0000","last_became_active":"2024-03-30T00:55:51.816223+0000","last_became_peered":"2024-03-30T00:55:51.816223+0000","last_unstale":"2024-03-30T00:57:35.153424+0000","last_undegraded":"2024-03-30T00:57:35.153424+0000","last_fullsized":"2024-03-30T00:57:35.153424+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:48:32.203975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"83'4","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162997+0000","last_change":"2024-03-30T00:55:51.811707+0000","last_active":"2024-03-30T00:57:35.162997+0000","last_peered":"2024-03-30T00:57:35.162997+0000","last_clean":"2024-03-30T00:57:35.162997+0000","last_became_active":"2024-03-30T00:55:51.811366+0000","last_became_peered":"2024-03-30T00:55:51.811366+0000","last_unstale":"2024-03-30T00:57:35.162997+0000","last_undegraded":"2024-03-30T00:57:35.162997+0000","last_fullsized":"2024-03-30T00:57:35.162997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:20.438505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153163+0000","last_change":"2024-03-30T00:55:51.826849+0000","last_active":"2024-03-30T00:57:35.153163+0000","last_peered":"2024-03-30T00:57:35.153163+0000","last_clean":"2024-03-30T00:57:35.153163+0000","last_became_active":"2024-03-30T00:55:51.826520+0000","last_became_peered":"2024-03-30T00:55:51.826520+0000","last_unstale":"2024-03-30T00:57:35.153163+0000","last_undegraded":"2024-03-30T00:57:35.153163+0000","last_fullsized":"2024-03-30T00:57:35.153163+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:54:03.423725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.149197+0000","last_change":"2024-03-30T00:55:51.812880+0000","last_active":"2024-03-30T00:57:35.149197+0000","last_peered":"2024-03-30T00:57:35.149197+0000","last_clean":"2024-03-30T00:57:35.149197+0000","last_became_active":"2024-03-30T00:55:51.812572+0000","last_became_peered":"2024-03-30T00:55:51.812572+0000","last_unstale":"2024-03-30T00:57:35.149197+0000","last_undegraded":"2024-03-30T00:57:35.149197+0000","last_fullsized":"2024-03-30T00:57:35.149197+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:44:49.670641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163852+0000","last_change":"2024-03-30T00:55:51.815480+0000","last_active":"2024-03-30T00:57:35.163852+0000","last_peered":"2024-03-30T00:57:35.163852+0000","last_clean":"2024-03-30T00:57:35.163852+0000","last_became_active":"2024-03-30T00:55:51.814052+0000","last_became_peered":"2024-03-30T00:55:51.814052+0000","last_unstale":"2024-03-30T00:57:35.163852+0000","last_undegraded":"2024-03-30T00:57:35.163852+0000","last_fullsized":"2024-03-30T00:57:35.163852+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:27:49.930928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"83'11","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153313+0000","last_change":"2024-03-30T00:55:51.814127+0000","last_active":"2024-03-30T00:57:35.153313+0000","last_peered":"2024-03-30T00:57:35.153313+0000","last_clean":"2024-03-30T00:57:35.153313+0000","last_became_active":"2024-03-30T00:55:51.813820+0000","last_became_peered":"2024-03-30T00:55:51.813820+0000","last_unstale":"2024-03-30T00:57:35.153313+0000","last_undegraded":"2024-03-30T00:57:35.153313+0000","last_fullsized":"2024-03-30T00:57:35.153313+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:21:31.605935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"83'16","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.154813+0000","last_change":"2024-03-30T00:55:51.815198+0000","last_active":"2024-03-30T00:57:35.154813+0000","last_peered":"2024-03-30T00:57:35.154813+0000","last_clean":"2024-03-30T00:57:35.154813+0000","last_became_active":"2024-03-30T00:55:51.814901+0000","last_became_peered":"2024-03-30T00:55:51.814901+0000","last_unstale":"2024-03-30T00:57:35.154813+0000","last_undegraded":"2024-03-30T00:57:35.154813+0000","last_fullsized":"2024-03-30T00:57:35.154813+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:45:51.224346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.148870+0000","last_change":"2024-03-30T00:55:51.809884+0000","last_active":"2024-03-30T00:57:35.148870+0000","last_peered":"2024-03-30T00:57:35.148870+0000","last_clean":"2024-03-30T00:57:35.148870+0000","last_became_active":"2024-03-30T00:55:51.809576+0000","last_became_peered":"2024-03-30T00:55:51.809576+0000","last_unstale":"2024-03-30T00:57:35.148870+0000","last_undegraded":"2024-03-30T00:57:35.148870+0000","last_fullsized":"2024-03-30T00:57:35.148870+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:39:51.767744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153546+0000","last_change":"2024-03-30T00:55:51.817384+0000","last_active":"2024-03-30T00:57:35.153546+0000","last_peered":"2024-03-30T00:57:35.153546+0000","last_clean":"2024-03-30T00:57:35.153546+0000","last_became_active":"2024-03-30T00:55:51.816564+0000","last_became_peered":"2024-03-30T00:55:51.816564+0000","last_unstale":"2024-03-30T00:57:35.153546+0000","last_undegraded":"2024-03-30T00:57:35.153546+0000","last_fullsized":"2024-03-30T00:57:35.153546+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:47:07.195650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"83'10","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166481+0000","last_change":"2024-03-30T00:55:51.814310+0000","last_active":"2024-03-30T00:57:35.166481+0000","last_peered":"2024-03-30T00:57:35.166481+0000","last_clean":"2024-03-30T00:57:35.166481+0000","last_became_active":"2024-03-30T00:55:51.813968+0000","last_became_peered":"2024-03-30T00:55:51.813968+0000","last_unstale":"2024-03-30T00:57:35.166481+0000","last_undegraded":"2024-03-30T00:57:35.166481+0000","last_fullsized":"2024-03-30T00:57:35.166481+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:59:55.955502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.161903+0000","last_change":"2024-03-30T00:55:51.815655+0000","last_active":"2024-03-30T00:57:35.161903+0000","last_peered":"2024-03-30T00:57:35.161903+0000","last_clean":"2024-03-30T00:57:35.161903+0000","last_became_active":"2024-03-30T00:55:51.814099+0000","last_became_peered":"2024-03-30T00:55:51.814099+0000","last_unstale":"2024-03-30T00:57:35.161903+0000","last_undegraded":"2024-03-30T00:57:35.161903+0000","last_fullsized":"2024-03-30T00:57:35.161903+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:42.958153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.157146+0000","last_change":"2024-03-30T00:56:36.472491+0000","last_active":"2024-03-30T00:57:35.157146+0000","last_peered":"2024-03-30T00:57:35.157146+0000","last_clean":"2024-03-30T00:57:35.157146+0000","last_became_active":"2024-03-30T00:56:36.472037+0000","last_became_peered":"2024-03-30T00:56:36.472037+0000","last_unstale":"2024-03-30T00:57:35.157146+0000","last_undegraded":"2024-03-30T00:57:35.157146+0000","last_fullsized":"2024-03-30T00:57:35.157146+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:52:08.402775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.164993+0000","last_change":"2024-03-30T00:55:51.822278+0000","last_active":"2024-03-30T00:57:35.164993+0000","last_peered":"2024-03-30T00:57:35.164993+0000","last_clean":"2024-03-30T00:57:35.164993+0000","last_became_active":"2024-03-30T00:55:51.821996+0000","last_became_peered":"2024-03-30T00:55:51.821996+0000","last_unstale":"2024-03-30T00:57:35.164993+0000","last_undegraded":"2024-03-30T00:57:35.164993+0000","last_fullsized":"2024-03-30T00:57:35.164993+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:37:45.841853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"83'8","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162074+0000","last_change":"2024-03-30T00:55:51.809547+0000","last_active":"2024-03-30T00:57:35.162074+0000","last_peered":"2024-03-30T00:57:35.162074+0000","last_clean":"2024-03-30T00:57:35.162074+0000","last_became_active":"2024-03-30T00:55:51.809314+0000","last_became_peered":"2024-03-30T00:55:51.809314+0000","last_unstale":"2024-03-30T00:57:35.162074+0000","last_undegraded":"2024-03-30T00:57:35.162074+0000","last_fullsized":"2024-03-30T00:57:35.162074+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:39:31.661362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"83'10","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.154392+0000","last_change":"2024-03-30T00:55:51.810857+0000","last_active":"2024-03-30T00:57:35.154392+0000","last_peered":"2024-03-30T00:57:35.154392+0000","last_clean":"2024-03-30T00:57:35.154392+0000","last_became_active":"2024-03-30T00:55:51.809420+0000","last_became_peered":"2024-03-30T00:55:51.809420+0000","last_unstale":"2024-03-30T00:57:35.154392+0000","last_undegraded":"2024-03-30T00:57:35.154392+0000","last_fullsized":"2024-03-30T00:57:35.154392+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:36:48.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"83'7","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163497+0000","last_change":"2024-03-30T00:55:51.814693+0000","last_active":"2024-03-30T00:57:35.163497+0000","last_peered":"2024-03-30T00:57:35.163497+0000","last_clean":"2024-03-30T00:57:35.163497+0000","last_became_active":"2024-03-30T00:55:51.814359+0000","last_became_peered":"2024-03-30T00:55:51.814359+0000","last_unstale":"2024-03-30T00:57:35.163497+0000","last_undegraded":"2024-03-30T00:57:35.163497+0000","last_fullsized":"2024-03-30T00:57:35.163497+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:20:50.856313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470709+0000","last_change":"2024-03-30T00:55:55.839089+0000","last_active":"2024-03-30T00:56:35.470709+0000","last_peered":"2024-03-30T00:56:35.470709+0000","last_clean":"2024-03-30T00:56:35.470709+0000","last_became_active":"2024-03-30T00:55:55.838705+0000","last_became_peered":"2024-03-30T00:55:55.838705+0000","last_unstale":"2024-03-30T00:56:35.470709+0000","last_undegraded":"2024-03-30T00:56:35.470709+0000","last_fullsized":"2024-03-30T00:56:35.470709+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:33:56.010272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'9","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.487878+0000","last_change":"2024-03-30T00:55:51.298577+0000","last_active":"2024-03-30T00:56:35.487878+0000","last_peered":"2024-03-30T00:56:35.487878+0000","last_clean":"2024-03-30T00:56:35.487878+0000","last_became_active":"2024-03-30T00:55:51.298269+0000","last_became_peered":"2024-03-30T00:55:51.298269+0000","last_unstale":"2024-03-30T00:56:35.487878+0000","last_undegraded":"2024-03-30T00:56:35.487878+0000","last_fullsized":"2024-03-30T00:56:35.487878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:10:04.998621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.154700+0000","last_change":"2024-03-30T00:55:51.818517+0000","last_active":"2024-03-30T00:57:35.154700+0000","last_peered":"2024-03-30T00:57:35.154700+0000","last_clean":"2024-03-30T00:57:35.154700+0000","last_became_active":"2024-03-30T00:55:51.818174+0000","last_became_peered":"2024-03-30T00:55:51.818174+0000","last_unstale":"2024-03-30T00:57:35.154700+0000","last_undegraded":"2024-03-30T00:57:35.154700+0000","last_fullsized":"2024-03-30T00:57:35.154700+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:03:01.604135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.623912+0000","last_change":"2024-03-30T00:56:36.623912+0000","last_active":"2024-03-30T00:56:36.623912+0000","last_peered":"2024-03-30T00:56:36.623912+0000","last_clean":"2024-03-30T00:56:36.623912+0000","last_became_active":"2024-03-30T00:56:36.623266+0000","last_became_peered":"2024-03-30T00:56:36.623266+0000","last_unstale":"2024-03-30T00:56:36.623912+0000","last_undegraded":"2024-03-30T00:56:36.623912+0000","last_fullsized":"2024-03-30T00:56:36.623912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:46:47.166079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.471579+0000","last_change":"2024-03-30T00:55:51.312686+0000","last_active":"2024-03-30T00:56:35.471579+0000","last_peered":"2024-03-30T00:56:35.471579+0000","last_clean":"2024-03-30T00:56:35.471579+0000","last_became_active":"2024-03-30T00:55:51.312296+0000","last_became_peered":"2024-03-30T00:55:51.312296+0000","last_unstale":"2024-03-30T00:56:35.471579+0000","last_undegraded":"2024-03-30T00:56:35.471579+0000","last_fullsized":"2024-03-30T00:56:35.471579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:28:58.398577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165549+0000","last_change":"2024-03-30T00:55:51.812197+0000","last_active":"2024-03-30T00:57:35.165549+0000","last_peered":"2024-03-30T00:57:35.165549+0000","last_clean":"2024-03-30T00:57:35.165549+0000","last_became_active":"2024-03-30T00:55:51.811892+0000","last_became_peered":"2024-03-30T00:55:51.811892+0000","last_unstale":"2024-03-30T00:57:35.165549+0000","last_undegraded":"2024-03-30T00:57:35.165549+0000","last_fullsized":"2024-03-30T00:57:35.165549+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:06:53.459760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462318+0000","last_change":"2024-03-30T00:55:55.839015+0000","last_active":"2024-03-30T00:56:35.462318+0000","last_peered":"2024-03-30T00:56:35.462318+0000","last_clean":"2024-03-30T00:56:35.462318+0000","last_became_active":"2024-03-30T00:55:55.838608+0000","last_became_peered":"2024-03-30T00:55:55.838608+0000","last_unstale":"2024-03-30T00:56:35.462318+0000","last_undegraded":"2024-03-30T00:56:35.462318+0000","last_fullsized":"2024-03-30T00:56:35.462318+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:24:23.804754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'4","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463392+0000","last_change":"2024-03-30T00:55:51.304199+0000","last_active":"2024-03-30T00:56:35.463392+0000","last_peered":"2024-03-30T00:56:35.463392+0000","last_clean":"2024-03-30T00:56:35.463392+0000","last_became_active":"2024-03-30T00:55:51.303837+0000","last_became_peered":"2024-03-30T00:55:51.303837+0000","last_unstale":"2024-03-30T00:56:35.463392+0000","last_undegraded":"2024-03-30T00:56:35.463392+0000","last_fullsized":"2024-03-30T00:56:35.463392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:19.154957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.164480+0000","last_change":"2024-03-30T00:55:51.817984+0000","last_active":"2024-03-30T00:57:35.164480+0000","last_peered":"2024-03-30T00:57:35.164480+0000","last_clean":"2024-03-30T00:57:35.164480+0000","last_became_active":"2024-03-30T00:55:51.817767+0000","last_became_peered":"2024-03-30T00:55:51.817767+0000","last_unstale":"2024-03-30T00:57:35.164480+0000","last_undegraded":"2024-03-30T00:57:35.164480+0000","last_fullsized":"2024-03-30T00:57:35.164480+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:12:48.558024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.623581+0000","last_change":"2024-03-30T00:56:36.623581+0000","last_active":"2024-03-30T00:56:36.623581+0000","last_peered":"2024-03-30T00:56:36.623581+0000","last_clean":"2024-03-30T00:56:36.623581+0000","last_became_active":"2024-03-30T00:56:36.623168+0000","last_became_peered":"2024-03-30T00:56:36.623168+0000","last_unstale":"2024-03-30T00:56:36.623581+0000","last_undegraded":"2024-03-30T00:56:36.623581+0000","last_fullsized":"2024-03-30T00:56:36.623581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:41.448935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"76'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.480921+0000","last_change":"2024-03-30T00:56:36.480478+0000","last_active":"2024-03-30T00:56:36.480921+0000","last_peered":"2024-03-30T00:56:36.480921+0000","last_clean":"2024-03-30T00:56:36.480921+0000","last_became_active":"2024-03-30T00:56:36.473297+0000","last_became_peered":"2024-03-30T00:56:36.473297+0000","last_unstale":"2024-03-30T00:56:36.480921+0000","last_undegraded":"2024-03-30T00:56:36.480921+0000","last_fullsized":"2024-03-30T00:56:36.480921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:37:08.090281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"83'15","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.151958+0000","last_change":"2024-03-30T00:55:51.827985+0000","last_active":"2024-03-30T00:57:35.151958+0000","last_peered":"2024-03-30T00:57:35.151958+0000","last_clean":"2024-03-30T00:57:35.151958+0000","last_became_active":"2024-03-30T00:55:51.827678+0000","last_became_peered":"2024-03-30T00:55:51.827678+0000","last_unstale":"2024-03-30T00:57:35.151958+0000","last_undegraded":"2024-03-30T00:57:35.151958+0000","last_fullsized":"2024-03-30T00:57:35.151958+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:47.088009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470870+0000","last_change":"2024-03-30T00:55:55.827267+0000","last_active":"2024-03-30T00:56:35.470870+0000","last_peered":"2024-03-30T00:56:35.470870+0000","last_clean":"2024-03-30T00:56:35.470870+0000","last_became_active":"2024-03-30T00:55:55.826821+0000","last_became_peered":"2024-03-30T00:55:55.826821+0000","last_unstale":"2024-03-30T00:56:35.470870+0000","last_undegraded":"2024-03-30T00:56:35.470870+0000","last_fullsized":"2024-03-30T00:56:35.470870+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:15:49.084359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"83'7","reported_seq":106,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:38.260458+0000","last_change":"2024-03-30T00:55:51.305177+0000","last_active":"2024-03-30T00:57:38.260458+0000","last_peered":"2024-03-30T00:57:38.260458+0000","last_clean":"2024-03-30T00:57:38.260458+0000","last_became_active":"2024-03-30T00:55:51.304813+0000","last_became_peered":"2024-03-30T00:55:51.304813+0000","last_unstale":"2024-03-30T00:57:38.260458+0000","last_undegraded":"2024-03-30T00:57:38.260458+0000","last_fullsized":"2024-03-30T00:57:38.260458+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:05:20.578897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1108,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"83'7","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.140612+0000","last_change":"2024-03-30T00:55:51.812714+0000","last_active":"2024-03-30T00:57:35.140612+0000","last_peered":"2024-03-30T00:57:35.140612+0000","last_clean":"2024-03-30T00:57:35.140612+0000","last_became_active":"2024-03-30T00:55:51.812406+0000","last_became_peered":"2024-03-30T00:55:51.812406+0000","last_unstale":"2024-03-30T00:57:35.140612+0000","last_undegraded":"2024-03-30T00:57:35.140612+0000","last_fullsized":"2024-03-30T00:57:35.140612+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:47:01.691412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470799+0000","last_change":"2024-03-30T00:55:55.831994+0000","last_active":"2024-03-30T00:56:35.470799+0000","last_peered":"2024-03-30T00:56:35.470799+0000","last_clean":"2024-03-30T00:56:35.470799+0000","last_became_active":"2024-03-30T00:55:55.831537+0000","last_became_peered":"2024-03-30T00:55:55.831537+0000","last_unstale":"2024-03-30T00:56:35.470799+0000","last_undegraded":"2024-03-30T00:56:35.470799+0000","last_fullsized":"2024-03-30T00:56:35.470799+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:44:47.240981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467196+0000","last_change":"2024-03-30T00:55:51.303468+0000","last_active":"2024-03-30T00:56:35.467196+0000","last_peered":"2024-03-30T00:56:35.467196+0000","last_clean":"2024-03-30T00:56:35.467196+0000","last_became_active":"2024-03-30T00:55:51.303068+0000","last_became_peered":"2024-03-30T00:55:51.303068+0000","last_unstale":"2024-03-30T00:56:35.467196+0000","last_undegraded":"2024-03-30T00:56:35.467196+0000","last_fullsized":"2024-03-30T00:56:35.467196+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:45:25.278196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"83'8","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.159899+0000","last_change":"2024-03-30T00:55:51.825099+0000","last_active":"2024-03-30T00:57:35.159899+0000","last_peered":"2024-03-30T00:57:35.159899+0000","last_clean":"2024-03-30T00:57:35.159899+0000","last_became_active":"2024-03-30T00:55:51.824717+0000","last_became_peered":"2024-03-30T00:55:51.824717+0000","last_unstale":"2024-03-30T00:57:35.159899+0000","last_undegraded":"2024-03-30T00:57:35.159899+0000","last_fullsized":"2024-03-30T00:57:35.159899+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:49:31.542430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460499+0000","last_change":"2024-03-30T00:55:55.826106+0000","last_active":"2024-03-30T00:56:35.460499+0000","last_peered":"2024-03-30T00:56:35.460499+0000","last_clean":"2024-03-30T00:56:35.460499+0000","last_became_active":"2024-03-30T00:55:55.825597+0000","last_became_peered":"2024-03-30T00:55:55.825597+0000","last_unstale":"2024-03-30T00:56:35.460499+0000","last_undegraded":"2024-03-30T00:56:35.460499+0000","last_fullsized":"2024-03-30T00:56:35.460499+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:00:18.991732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.473981+0000","last_change":"2024-03-30T00:55:51.294090+0000","last_active":"2024-03-30T00:56:35.473981+0000","last_peered":"2024-03-30T00:56:35.473981+0000","last_clean":"2024-03-30T00:56:35.473981+0000","last_became_active":"2024-03-30T00:55:51.293668+0000","last_became_peered":"2024-03-30T00:55:51.293668+0000","last_unstale":"2024-03-30T00:56:35.473981+0000","last_undegraded":"2024-03-30T00:56:35.473981+0000","last_fullsized":"2024-03-30T00:56:35.473981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:33:36.661760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166304+0000","last_change":"2024-03-30T00:55:51.823428+0000","last_active":"2024-03-30T00:57:35.166304+0000","last_peered":"2024-03-30T00:57:35.166304+0000","last_clean":"2024-03-30T00:57:35.166304+0000","last_became_active":"2024-03-30T00:55:51.822222+0000","last_became_peered":"2024-03-30T00:55:51.822222+0000","last_unstale":"2024-03-30T00:57:35.166304+0000","last_undegraded":"2024-03-30T00:57:35.166304+0000","last_fullsized":"2024-03-30T00:57:35.166304+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:05:34.798940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495876+0000","last_change":"2024-03-30T00:55:55.828115+0000","last_active":"2024-03-30T00:56:35.495876+0000","last_peered":"2024-03-30T00:56:35.495876+0000","last_clean":"2024-03-30T00:56:35.495876+0000","last_became_active":"2024-03-30T00:55:55.827681+0000","last_became_peered":"2024-03-30T00:55:55.827681+0000","last_unstale":"2024-03-30T00:56:35.495876+0000","last_undegraded":"2024-03-30T00:56:35.495876+0000","last_fullsized":"2024-03-30T00:56:35.495876+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:18:25.751350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'1","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466298+0000","last_change":"2024-03-30T00:55:51.305163+0000","last_active":"2024-03-30T00:56:35.466298+0000","last_peered":"2024-03-30T00:56:35.466298+0000","last_clean":"2024-03-30T00:56:35.466298+0000","last_became_active":"2024-03-30T00:55:51.304860+0000","last_became_peered":"2024-03-30T00:55:51.304860+0000","last_unstale":"2024-03-30T00:56:35.466298+0000","last_undegraded":"2024-03-30T00:56:35.466298+0000","last_fullsized":"2024-03-30T00:56:35.466298+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:44:41.850998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"83'11","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.159021+0000","last_change":"2024-03-30T00:55:51.819401+0000","last_active":"2024-03-30T00:57:35.159021+0000","last_peered":"2024-03-30T00:57:35.159021+0000","last_clean":"2024-03-30T00:57:35.159021+0000","last_became_active":"2024-03-30T00:55:51.819021+0000","last_became_peered":"2024-03-30T00:55:51.819021+0000","last_unstale":"2024-03-30T00:57:35.159021+0000","last_undegraded":"2024-03-30T00:57:35.159021+0000","last_fullsized":"2024-03-30T00:57:35.159021+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:01:45.195719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.469445+0000","last_change":"2024-03-30T00:55:55.836806+0000","last_active":"2024-03-30T00:56:35.469445+0000","last_peered":"2024-03-30T00:56:35.469445+0000","last_clean":"2024-03-30T00:56:35.469445+0000","last_became_active":"2024-03-30T00:55:55.836403+0000","last_became_peered":"2024-03-30T00:55:55.836403+0000","last_unstale":"2024-03-30T00:56:35.469445+0000","last_undegraded":"2024-03-30T00:56:35.469445+0000","last_fullsized":"2024-03-30T00:56:35.469445+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:17:13.913675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"83'4","reported_seq":104,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:43.267468+0000","last_change":"2024-03-30T00:55:51.300833+0000","last_active":"2024-03-30T00:57:43.267468+0000","last_peered":"2024-03-30T00:57:43.267468+0000","last_clean":"2024-03-30T00:57:43.267468+0000","last_became_active":"2024-03-30T00:55:51.300473+0000","last_became_peered":"2024-03-30T00:55:51.300473+0000","last_unstale":"2024-03-30T00:57:43.267468+0000","last_undegraded":"2024-03-30T00:57:43.267468+0000","last_fullsized":"2024-03-30T00:57:43.267468+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:00:39.211981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"83'10","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.155649+0000","last_change":"2024-03-30T00:55:51.810781+0000","last_active":"2024-03-30T00:57:35.155649+0000","last_peered":"2024-03-30T00:57:35.155649+0000","last_clean":"2024-03-30T00:57:35.155649+0000","last_became_active":"2024-03-30T00:55:51.810215+0000","last_became_peered":"2024-03-30T00:55:51.810215+0000","last_unstale":"2024-03-30T00:57:35.155649+0000","last_undegraded":"2024-03-30T00:57:35.155649+0000","last_fullsized":"2024-03-30T00:57:35.155649+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:53:50.283209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.484181+0000","last_change":"2024-03-30T00:55:55.834505+0000","last_active":"2024-03-30T00:56:35.484181+0000","last_peered":"2024-03-30T00:56:35.484181+0000","last_clean":"2024-03-30T00:56:35.484181+0000","last_became_active":"2024-03-30T00:55:55.829319+0000","last_became_peered":"2024-03-30T00:55:55.829319+0000","last_unstale":"2024-03-30T00:56:35.484181+0000","last_undegraded":"2024-03-30T00:56:35.484181+0000","last_fullsized":"2024-03-30T00:56:35.484181+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:51:37.595523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466067+0000","last_change":"2024-03-30T00:55:51.302070+0000","last_active":"2024-03-30T00:56:35.466067+0000","last_peered":"2024-03-30T00:56:35.466067+0000","last_clean":"2024-03-30T00:56:35.466067+0000","last_became_active":"2024-03-30T00:55:51.298940+0000","last_became_peered":"2024-03-30T00:55:51.298940+0000","last_unstale":"2024-03-30T00:56:35.466067+0000","last_undegraded":"2024-03-30T00:56:35.466067+0000","last_fullsized":"2024-03-30T00:56:35.466067+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:07:47.915506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"83'9","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166604+0000","last_change":"2024-03-30T00:55:51.819918+0000","last_active":"2024-03-30T00:57:35.166604+0000","last_peered":"2024-03-30T00:57:35.166604+0000","last_clean":"2024-03-30T00:57:35.166604+0000","last_became_active":"2024-03-30T00:55:51.819493+0000","last_became_peered":"2024-03-30T00:55:51.819493+0000","last_unstale":"2024-03-30T00:57:35.166604+0000","last_undegraded":"2024-03-30T00:57:35.166604+0000","last_fullsized":"2024-03-30T00:57:35.166604+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:43:43.067031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462710+0000","last_change":"2024-03-30T00:55:55.827176+0000","last_active":"2024-03-30T00:56:35.462710+0000","last_peered":"2024-03-30T00:56:35.462710+0000","last_clean":"2024-03-30T00:56:35.462710+0000","last_became_active":"2024-03-30T00:55:55.826772+0000","last_became_peered":"2024-03-30T00:55:55.826772+0000","last_unstale":"2024-03-30T00:56:35.462710+0000","last_undegraded":"2024-03-30T00:56:35.462710+0000","last_fullsized":"2024-03-30T00:56:35.462710+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:02:04.512050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'4","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467237+0000","last_change":"2024-03-30T00:55:51.306688+0000","last_active":"2024-03-30T00:56:35.467237+0000","last_peered":"2024-03-30T00:56:35.467237+0000","last_clean":"2024-03-30T00:56:35.467237+0000","last_became_active":"2024-03-30T00:55:51.306333+0000","last_became_peered":"2024-03-30T00:55:51.306333+0000","last_unstale":"2024-03-30T00:56:35.467237+0000","last_undegraded":"2024-03-30T00:56:35.467237+0000","last_fullsized":"2024-03-30T00:56:35.467237+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:21:13.719947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"83'11","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.152507+0000","last_change":"2024-03-30T00:55:51.824879+0000","last_active":"2024-03-30T00:57:35.152507+0000","last_peered":"2024-03-30T00:57:35.152507+0000","last_clean":"2024-03-30T00:57:35.152507+0000","last_became_active":"2024-03-30T00:55:51.824566+0000","last_became_peered":"2024-03-30T00:55:51.824566+0000","last_unstale":"2024-03-30T00:57:35.152507+0000","last_undegraded":"2024-03-30T00:57:35.152507+0000","last_fullsized":"2024-03-30T00:57:35.152507+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:04:28.505693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.484132+0000","last_change":"2024-03-30T00:55:55.826387+0000","last_active":"2024-03-30T00:56:35.484132+0000","last_peered":"2024-03-30T00:56:35.484132+0000","last_clean":"2024-03-30T00:56:35.484132+0000","last_became_active":"2024-03-30T00:55:55.825941+0000","last_became_peered":"2024-03-30T00:55:55.825941+0000","last_unstale":"2024-03-30T00:56:35.484132+0000","last_undegraded":"2024-03-30T00:56:35.484132+0000","last_fullsized":"2024-03-30T00:56:35.484132+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:34:43.432712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"83'38","reported_seq":232,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:44.268981+0000","last_change":"2024-03-30T00:55:51.304592+0000","last_active":"2024-03-30T00:57:44.268981+0000","last_peered":"2024-03-30T00:57:44.268981+0000","last_clean":"2024-03-30T00:57:44.268981+0000","last_became_active":"2024-03-30T00:55:51.303931+0000","last_became_peered":"2024-03-30T00:55:51.303931+0000","last_unstale":"2024-03-30T00:57:44.268981+0000","last_undegraded":"2024-03-30T00:57:44.268981+0000","last_fullsized":"2024-03-30T00:57:44.268981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:53:40.976289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66960,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":177,"num_read_kb":154,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.148546+0000","last_change":"2024-03-30T00:55:51.815196+0000","last_active":"2024-03-30T00:57:35.148546+0000","last_peered":"2024-03-30T00:57:35.148546+0000","last_clean":"2024-03-30T00:57:35.148546+0000","last_became_active":"2024-03-30T00:55:51.813376+0000","last_became_peered":"2024-03-30T00:55:51.813376+0000","last_unstale":"2024-03-30T00:57:35.148546+0000","last_undegraded":"2024-03-30T00:57:35.148546+0000","last_fullsized":"2024-03-30T00:57:35.148546+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:40:26.350248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467447+0000","last_change":"2024-03-30T00:55:55.836906+0000","last_active":"2024-03-30T00:56:35.467447+0000","last_peered":"2024-03-30T00:56:35.467447+0000","last_clean":"2024-03-30T00:56:35.467447+0000","last_became_active":"2024-03-30T00:55:55.836539+0000","last_became_peered":"2024-03-30T00:55:55.836539+0000","last_unstale":"2024-03-30T00:56:35.467447+0000","last_undegraded":"2024-03-30T00:56:35.467447+0000","last_fullsized":"2024-03-30T00:56:35.467447+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:21:39.320192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'6","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.478499+0000","last_change":"2024-03-30T00:56:36.478132+0000","last_active":"2024-03-30T00:56:36.478499+0000","last_peered":"2024-03-30T00:56:36.478499+0000","last_clean":"2024-03-30T00:56:36.478499+0000","last_became_active":"2024-03-30T00:56:36.467449+0000","last_became_peered":"2024-03-30T00:56:36.467449+0000","last_unstale":"2024-03-30T00:56:36.478499+0000","last_undegraded":"2024-03-30T00:56:36.478499+0000","last_fullsized":"2024-03-30T00:56:36.478499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:27:51.986112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165118+0000","last_change":"2024-03-30T00:55:51.819825+0000","last_active":"2024-03-30T00:57:35.165118+0000","last_peered":"2024-03-30T00:57:35.165118+0000","last_clean":"2024-03-30T00:57:35.165118+0000","last_became_active":"2024-03-30T00:55:51.819481+0000","last_became_peered":"2024-03-30T00:55:51.819481+0000","last_unstale":"2024-03-30T00:57:35.165118+0000","last_undegraded":"2024-03-30T00:57:35.165118+0000","last_fullsized":"2024-03-30T00:57:35.165118+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:10:17.138541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460246+0000","last_change":"2024-03-30T00:55:55.832675+0000","last_active":"2024-03-30T00:56:35.460246+0000","last_peered":"2024-03-30T00:56:35.460246+0000","last_clean":"2024-03-30T00:56:35.460246+0000","last_became_active":"2024-03-30T00:55:55.832325+0000","last_became_peered":"2024-03-30T00:55:55.832325+0000","last_unstale":"2024-03-30T00:56:35.460246+0000","last_undegraded":"2024-03-30T00:56:35.460246+0000","last_fullsized":"2024-03-30T00:56:35.460246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:38:46.765473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467208+0000","last_change":"2024-03-30T00:55:51.313700+0000","last_active":"2024-03-30T00:56:35.467208+0000","last_peered":"2024-03-30T00:56:35.467208+0000","last_clean":"2024-03-30T00:56:35.467208+0000","last_became_active":"2024-03-30T00:55:51.313355+0000","last_became_peered":"2024-03-30T00:55:51.313355+0000","last_unstale":"2024-03-30T00:56:35.467208+0000","last_undegraded":"2024-03-30T00:56:35.467208+0000","last_fullsized":"2024-03-30T00:56:35.467208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:43:04.461398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"83'15","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163894+0000","last_change":"2024-03-30T00:55:51.819939+0000","last_active":"2024-03-30T00:57:35.163894+0000","last_peered":"2024-03-30T00:57:35.163894+0000","last_clean":"2024-03-30T00:57:35.163894+0000","last_became_active":"2024-03-30T00:55:51.819614+0000","last_became_peered":"2024-03-30T00:55:51.819614+0000","last_unstale":"2024-03-30T00:57:35.163894+0000","last_undegraded":"2024-03-30T00:57:35.163894+0000","last_fullsized":"2024-03-30T00:57:35.163894+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:20:37.362779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.469703+0000","last_change":"2024-03-30T00:56:36.469703+0000","last_active":"2024-03-30T00:56:36.469703+0000","last_peered":"2024-03-30T00:56:36.469703+0000","last_clean":"2024-03-30T00:56:36.469703+0000","last_became_active":"2024-03-30T00:56:36.469195+0000","last_became_peered":"2024-03-30T00:56:36.469195+0000","last_unstale":"2024-03-30T00:56:36.469703+0000","last_undegraded":"2024-03-30T00:56:36.469703+0000","last_fullsized":"2024-03-30T00:56:36.469703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:10:10.421447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"83'6","reported_seq":110,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:42.266577+0000","last_change":"2024-03-30T00:55:51.299246+0000","last_active":"2024-03-30T00:57:42.266577+0000","last_peered":"2024-03-30T00:57:42.266577+0000","last_clean":"2024-03-30T00:57:42.266577+0000","last_became_active":"2024-03-30T00:55:51.298924+0000","last_became_peered":"2024-03-30T00:55:51.298924+0000","last_unstale":"2024-03-30T00:57:42.266577+0000","last_undegraded":"2024-03-30T00:57:42.266577+0000","last_fullsized":"2024-03-30T00:57:42.266577+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:50:05.897612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1380,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.149967+0000","last_change":"2024-03-30T00:55:51.813804+0000","last_active":"2024-03-30T00:57:35.149967+0000","last_peered":"2024-03-30T00:57:35.149967+0000","last_clean":"2024-03-30T00:57:35.149967+0000","last_became_active":"2024-03-30T00:55:51.813500+0000","last_became_peered":"2024-03-30T00:55:51.813500+0000","last_unstale":"2024-03-30T00:57:35.149967+0000","last_undegraded":"2024-03-30T00:57:35.149967+0000","last_fullsized":"2024-03-30T00:57:35.149967+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:42:21.614900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.477003+0000","last_change":"2024-03-30T00:55:55.837737+0000","last_active":"2024-03-30T00:56:35.477003+0000","last_peered":"2024-03-30T00:56:35.477003+0000","last_clean":"2024-03-30T00:56:35.477003+0000","last_became_active":"2024-03-30T00:55:55.837312+0000","last_became_peered":"2024-03-30T00:55:55.837312+0000","last_unstale":"2024-03-30T00:56:35.477003+0000","last_undegraded":"2024-03-30T00:56:35.477003+0000","last_fullsized":"2024-03-30T00:56:35.477003+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:11:46.139979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'15","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462932+0000","last_change":"2024-03-30T00:55:51.306398+0000","last_active":"2024-03-30T00:56:35.462932+0000","last_peered":"2024-03-30T00:56:35.462932+0000","last_clean":"2024-03-30T00:56:35.462932+0000","last_became_active":"2024-03-30T00:55:51.305878+0000","last_became_peered":"2024-03-30T00:55:51.305878+0000","last_unstale":"2024-03-30T00:56:35.462932+0000","last_undegraded":"2024-03-30T00:56:35.462932+0000","last_fullsized":"2024-03-30T00:56:35.462932+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:20:32.742423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.151564+0000","last_change":"2024-03-30T00:55:51.824294+0000","last_active":"2024-03-30T00:57:35.151564+0000","last_peered":"2024-03-30T00:57:35.151564+0000","last_clean":"2024-03-30T00:57:35.151564+0000","last_became_active":"2024-03-30T00:55:51.823957+0000","last_became_peered":"2024-03-30T00:55:51.823957+0000","last_unstale":"2024-03-30T00:57:35.151564+0000","last_undegraded":"2024-03-30T00:57:35.151564+0000","last_fullsized":"2024-03-30T00:57:35.151564+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:23:16.873140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466253+0000","last_change":"2024-03-30T00:55:55.837023+0000","last_active":"2024-03-30T00:56:35.466253+0000","last_peered":"2024-03-30T00:56:35.466253+0000","last_clean":"2024-03-30T00:56:35.466253+0000","last_became_active":"2024-03-30T00:55:55.836541+0000","last_became_peered":"2024-03-30T00:55:55.836541+0000","last_unstale":"2024-03-30T00:56:35.466253+0000","last_undegraded":"2024-03-30T00:56:35.466253+0000","last_fullsized":"2024-03-30T00:56:35.466253+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:58:40.323953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463352+0000","last_change":"2024-03-30T00:55:51.300883+0000","last_active":"2024-03-30T00:56:35.463352+0000","last_peered":"2024-03-30T00:56:35.463352+0000","last_clean":"2024-03-30T00:56:35.463352+0000","last_became_active":"2024-03-30T00:55:51.300531+0000","last_became_peered":"2024-03-30T00:55:51.300531+0000","last_unstale":"2024-03-30T00:56:35.463352+0000","last_undegraded":"2024-03-30T00:56:35.463352+0000","last_fullsized":"2024-03-30T00:56:35.463352+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:38:38.874075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"83'5","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.151560+0000","last_change":"2024-03-30T00:55:51.822970+0000","last_active":"2024-03-30T00:57:35.151560+0000","last_peered":"2024-03-30T00:57:35.151560+0000","last_clean":"2024-03-30T00:57:35.151560+0000","last_became_active":"2024-03-30T00:55:51.822653+0000","last_became_peered":"2024-03-30T00:55:51.822653+0000","last_unstale":"2024-03-30T00:57:35.151560+0000","last_undegraded":"2024-03-30T00:57:35.151560+0000","last_fullsized":"2024-03-30T00:57:35.151560+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:59:26.306672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463256+0000","last_change":"2024-03-30T00:55:55.856605+0000","last_active":"2024-03-30T00:56:35.463256+0000","last_peered":"2024-03-30T00:56:35.463256+0000","last_clean":"2024-03-30T00:56:35.463256+0000","last_became_active":"2024-03-30T00:55:55.856061+0000","last_became_peered":"2024-03-30T00:55:55.856061+0000","last_unstale":"2024-03-30T00:56:35.463256+0000","last_undegraded":"2024-03-30T00:56:35.463256+0000","last_fullsized":"2024-03-30T00:56:35.463256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:58:41.010206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.488831+0000","last_change":"2024-03-30T00:55:51.301961+0000","last_active":"2024-03-30T00:56:35.488831+0000","last_peered":"2024-03-30T00:56:35.488831+0000","last_clean":"2024-03-30T00:56:35.488831+0000","last_became_active":"2024-03-30T00:55:51.301618+0000","last_became_peered":"2024-03-30T00:55:51.301618+0000","last_unstale":"2024-03-30T00:56:35.488831+0000","last_undegraded":"2024-03-30T00:56:35.488831+0000","last_fullsized":"2024-03-30T00:56:35.488831+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:38:46.681490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"83'4","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162610+0000","last_change":"2024-03-30T00:55:51.821331+0000","last_active":"2024-03-30T00:57:35.162610+0000","last_peered":"2024-03-30T00:57:35.162610+0000","last_clean":"2024-03-30T00:57:35.162610+0000","last_became_active":"2024-03-30T00:55:51.820988+0000","last_became_peered":"2024-03-30T00:55:51.820988+0000","last_unstale":"2024-03-30T00:57:35.162610+0000","last_undegraded":"2024-03-30T00:57:35.162610+0000","last_fullsized":"2024-03-30T00:57:35.162610+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:25:20.150789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.475451+0000","last_change":"2024-03-30T00:55:55.826251+0000","last_active":"2024-03-30T00:56:35.475451+0000","last_peered":"2024-03-30T00:56:35.475451+0000","last_clean":"2024-03-30T00:56:35.475451+0000","last_became_active":"2024-03-30T00:55:55.822695+0000","last_became_peered":"2024-03-30T00:55:55.822695+0000","last_unstale":"2024-03-30T00:56:35.475451+0000","last_undegraded":"2024-03-30T00:56:35.475451+0000","last_fullsized":"2024-03-30T00:56:35.475451+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:32:19.011454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466393+0000","last_change":"2024-03-30T00:55:51.309412+0000","last_active":"2024-03-30T00:56:35.466393+0000","last_peered":"2024-03-30T00:56:35.466393+0000","last_clean":"2024-03-30T00:56:35.466393+0000","last_became_active":"2024-03-30T00:55:51.309028+0000","last_became_peered":"2024-03-30T00:55:51.309028+0000","last_unstale":"2024-03-30T00:56:35.466393+0000","last_undegraded":"2024-03-30T00:56:35.466393+0000","last_fullsized":"2024-03-30T00:56:35.466393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:00:49.022772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.152521+0000","last_change":"2024-03-30T00:56:36.467986+0000","last_active":"2024-03-30T00:57:35.152521+0000","last_peered":"2024-03-30T00:57:35.152521+0000","last_clean":"2024-03-30T00:57:35.152521+0000","last_became_active":"2024-03-30T00:56:36.467601+0000","last_became_peered":"2024-03-30T00:56:36.467601+0000","last_unstale":"2024-03-30T00:57:35.152521+0000","last_undegraded":"2024-03-30T00:57:35.152521+0000","last_fullsized":"2024-03-30T00:57:35.152521+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:37:32.979094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.479139+0000","last_change":"2024-03-30T00:55:55.856377+0000","last_active":"2024-03-30T00:56:35.479139+0000","last_peered":"2024-03-30T00:56:35.479139+0000","last_clean":"2024-03-30T00:56:35.479139+0000","last_became_active":"2024-03-30T00:55:55.855986+0000","last_became_peered":"2024-03-30T00:55:55.855986+0000","last_unstale":"2024-03-30T00:56:35.479139+0000","last_undegraded":"2024-03-30T00:56:35.479139+0000","last_fullsized":"2024-03-30T00:56:35.479139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:45:47.303446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"83'8","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.147047+0000","last_change":"2024-03-30T00:56:36.464863+0000","last_active":"2024-03-30T00:57:35.147047+0000","last_peered":"2024-03-30T00:57:35.147047+0000","last_clean":"2024-03-30T00:57:35.147047+0000","last_became_active":"2024-03-30T00:56:36.464323+0000","last_became_peered":"2024-03-30T00:56:36.464323+0000","last_unstale":"2024-03-30T00:57:35.147047+0000","last_undegraded":"2024-03-30T00:57:35.147047+0000","last_fullsized":"2024-03-30T00:57:35.147047+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:43:10.899726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"83'5","reported_seq":102,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:39.261685+0000","last_change":"2024-03-30T00:55:51.314834+0000","last_active":"2024-03-30T00:57:39.261685+0000","last_peered":"2024-03-30T00:57:39.261685+0000","last_clean":"2024-03-30T00:57:39.261685+0000","last_became_active":"2024-03-30T00:55:51.314429+0000","last_became_peered":"2024-03-30T00:55:51.314429+0000","last_unstale":"2024-03-30T00:57:39.261685+0000","last_undegraded":"2024-03-30T00:57:39.261685+0000","last_fullsized":"2024-03-30T00:57:39.261685+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:17:13.987180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":79,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466483+0000","last_change":"2024-03-30T00:55:55.840721+0000","last_active":"2024-03-30T00:56:35.466483+0000","last_peered":"2024-03-30T00:56:35.466483+0000","last_clean":"2024-03-30T00:56:35.466483+0000","last_became_active":"2024-03-30T00:55:55.840269+0000","last_became_peered":"2024-03-30T00:55:55.840269+0000","last_unstale":"2024-03-30T00:56:35.466483+0000","last_undegraded":"2024-03-30T00:56:35.466483+0000","last_fullsized":"2024-03-30T00:56:35.466483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:28:55.366411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.160257+0000","last_change":"2024-03-30T00:55:51.812609+0000","last_active":"2024-03-30T00:57:35.160257+0000","last_peered":"2024-03-30T00:57:35.160257+0000","last_clean":"2024-03-30T00:57:35.160257+0000","last_became_active":"2024-03-30T00:55:51.812325+0000","last_became_peered":"2024-03-30T00:55:51.812325+0000","last_unstale":"2024-03-30T00:57:35.160257+0000","last_undegraded":"2024-03-30T00:57:35.160257+0000","last_fullsized":"2024-03-30T00:57:35.160257+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:22:22.074405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'3","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463363+0000","last_change":"2024-03-30T00:55:51.296173+0000","last_active":"2024-03-30T00:56:35.463363+0000","last_peered":"2024-03-30T00:56:35.463363+0000","last_clean":"2024-03-30T00:56:35.463363+0000","last_became_active":"2024-03-30T00:55:51.295331+0000","last_became_peered":"2024-03-30T00:55:51.295331+0000","last_unstale":"2024-03-30T00:56:35.463363+0000","last_undegraded":"2024-03-30T00:56:35.463363+0000","last_fullsized":"2024-03-30T00:56:35.463363+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:51:58.343765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":326,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466805+0000","last_change":"2024-03-30T00:55:02.487656+0000","last_active":"2024-03-30T00:56:35.466805+0000","last_peered":"2024-03-30T00:56:35.466805+0000","last_clean":"2024-03-30T00:56:35.466805+0000","last_became_active":"2024-03-30T00:55:02.487263+0000","last_became_peered":"2024-03-30T00:55:02.487263+0000","last_unstale":"2024-03-30T00:56:35.466805+0000","last_undegraded":"2024-03-30T00:56:35.466805+0000","last_fullsized":"2024-03-30T00:56:35.466805+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:52:29.691461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:52:29.691461+0000","last_clean_scrub_stamp":"2024-03-30T00:52:29.691461+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:19:20.825038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466944+0000","last_change":"2024-03-30T00:55:55.856487+0000","last_active":"2024-03-30T00:56:35.466944+0000","last_peered":"2024-03-30T00:56:35.466944+0000","last_clean":"2024-03-30T00:56:35.466944+0000","last_became_active":"2024-03-30T00:55:55.856106+0000","last_became_peered":"2024-03-30T00:55:55.856106+0000","last_unstale":"2024-03-30T00:56:35.466944+0000","last_undegraded":"2024-03-30T00:56:35.466944+0000","last_fullsized":"2024-03-30T00:56:35.466944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:14.504218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.164966+0000","last_change":"2024-03-30T00:55:51.821519+0000","last_active":"2024-03-30T00:57:35.164966+0000","last_peered":"2024-03-30T00:57:35.164966+0000","last_clean":"2024-03-30T00:57:35.164966+0000","last_became_active":"2024-03-30T00:55:51.821095+0000","last_became_peered":"2024-03-30T00:55:51.821095+0000","last_unstale":"2024-03-30T00:57:35.164966+0000","last_undegraded":"2024-03-30T00:57:35.164966+0000","last_fullsized":"2024-03-30T00:57:35.164966+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:50:22.297450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'3","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466994+0000","last_change":"2024-03-30T00:55:51.303047+0000","last_active":"2024-03-30T00:56:35.466994+0000","last_peered":"2024-03-30T00:56:35.466994+0000","last_clean":"2024-03-30T00:56:35.466994+0000","last_became_active":"2024-03-30T00:55:51.302712+0000","last_became_peered":"2024-03-30T00:55:51.302712+0000","last_unstale":"2024-03-30T00:56:35.466994+0000","last_undegraded":"2024-03-30T00:56:35.466994+0000","last_fullsized":"2024-03-30T00:56:35.466994+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:07:59.451674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460262+0000","last_change":"2024-03-30T00:55:55.823692+0000","last_active":"2024-03-30T00:56:35.460262+0000","last_peered":"2024-03-30T00:56:35.460262+0000","last_clean":"2024-03-30T00:56:35.460262+0000","last_became_active":"2024-03-30T00:55:55.823188+0000","last_became_peered":"2024-03-30T00:55:55.823188+0000","last_unstale":"2024-03-30T00:56:35.460262+0000","last_undegraded":"2024-03-30T00:56:35.460262+0000","last_fullsized":"2024-03-30T00:56:35.460262+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:32:32.089166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'1","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463201+0000","last_change":"2024-03-30T00:55:51.306935+0000","last_active":"2024-03-30T00:56:35.463201+0000","last_peered":"2024-03-30T00:56:35.463201+0000","last_clean":"2024-03-30T00:56:35.463201+0000","last_became_active":"2024-03-30T00:55:51.306595+0000","last_became_peered":"2024-03-30T00:55:51.306595+0000","last_unstale":"2024-03-30T00:56:35.463201+0000","last_undegraded":"2024-03-30T00:56:35.463201+0000","last_fullsized":"2024-03-30T00:56:35.463201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:50:30.396762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166210+0000","last_change":"2024-03-30T00:55:51.807084+0000","last_active":"2024-03-30T00:57:35.166210+0000","last_peered":"2024-03-30T00:57:35.166210+0000","last_clean":"2024-03-30T00:57:35.166210+0000","last_became_active":"2024-03-30T00:55:51.806505+0000","last_became_peered":"2024-03-30T00:55:51.806505+0000","last_unstale":"2024-03-30T00:57:35.166210+0000","last_undegraded":"2024-03-30T00:57:35.166210+0000","last_fullsized":"2024-03-30T00:57:35.166210+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:38:44.759863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470455+0000","last_change":"2024-03-30T00:55:55.856457+0000","last_active":"2024-03-30T00:56:35.470455+0000","last_peered":"2024-03-30T00:56:35.470455+0000","last_clean":"2024-03-30T00:56:35.470455+0000","last_became_active":"2024-03-30T00:55:55.855982+0000","last_became_peered":"2024-03-30T00:55:55.855982+0000","last_unstale":"2024-03-30T00:56:35.470455+0000","last_undegraded":"2024-03-30T00:56:35.470455+0000","last_fullsized":"2024-03-30T00:56:35.470455+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:18:25.154634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"83'6","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.460763+0000","last_change":"2024-03-30T00:55:51.312581+0000","last_active":"2024-03-30T00:56:36.460763+0000","last_peered":"2024-03-30T00:56:36.460763+0000","last_clean":"2024-03-30T00:56:36.460763+0000","last_became_active":"2024-03-30T00:55:51.312225+0000","last_became_peered":"2024-03-30T00:55:51.312225+0000","last_unstale":"2024-03-30T00:56:36.460763+0000","last_undegraded":"2024-03-30T00:56:36.460763+0000","last_fullsized":"2024-03-30T00:56:36.460763+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:27:12.615250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.150369+0000","last_change":"2024-03-30T00:55:51.821527+0000","last_active":"2024-03-30T00:57:35.150369+0000","last_peered":"2024-03-30T00:57:35.150369+0000","last_clean":"2024-03-30T00:57:35.150369+0000","last_became_active":"2024-03-30T00:55:51.820689+0000","last_became_peered":"2024-03-30T00:55:51.820689+0000","last_unstale":"2024-03-30T00:57:35.150369+0000","last_undegraded":"2024-03-30T00:57:35.150369+0000","last_fullsized":"2024-03-30T00:57:35.150369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:14:35.735985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470468+0000","last_change":"2024-03-30T00:55:55.824307+0000","last_active":"2024-03-30T00:56:35.470468+0000","last_peered":"2024-03-30T00:56:35.470468+0000","last_clean":"2024-03-30T00:56:35.470468+0000","last_became_active":"2024-03-30T00:55:55.823900+0000","last_became_peered":"2024-03-30T00:55:55.823900+0000","last_unstale":"2024-03-30T00:56:35.470468+0000","last_undegraded":"2024-03-30T00:56:35.470468+0000","last_fullsized":"2024-03-30T00:56:35.470468+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:44:22.139120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'4","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463230+0000","last_change":"2024-03-30T00:55:51.296965+0000","last_active":"2024-03-30T00:56:35.463230+0000","last_peered":"2024-03-30T00:56:35.463230+0000","last_clean":"2024-03-30T00:56:35.463230+0000","last_became_active":"2024-03-30T00:55:51.296645+0000","last_became_peered":"2024-03-30T00:55:51.296645+0000","last_unstale":"2024-03-30T00:56:35.463230+0000","last_undegraded":"2024-03-30T00:56:35.463230+0000","last_fullsized":"2024-03-30T00:56:35.463230+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:53:46.519972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163410+0000","last_change":"2024-03-30T00:55:51.826282+0000","last_active":"2024-03-30T00:57:35.163410+0000","last_peered":"2024-03-30T00:57:35.163410+0000","last_clean":"2024-03-30T00:57:35.163410+0000","last_became_active":"2024-03-30T00:55:51.823910+0000","last_became_peered":"2024-03-30T00:55:51.823910+0000","last_unstale":"2024-03-30T00:57:35.163410+0000","last_undegraded":"2024-03-30T00:57:35.163410+0000","last_fullsized":"2024-03-30T00:57:35.163410+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:47.454280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462851+0000","last_change":"2024-03-30T00:55:55.856750+0000","last_active":"2024-03-30T00:56:35.462851+0000","last_peered":"2024-03-30T00:56:35.462851+0000","last_clean":"2024-03-30T00:56:35.462851+0000","last_became_active":"2024-03-30T00:55:55.856288+0000","last_became_peered":"2024-03-30T00:55:55.856288+0000","last_unstale":"2024-03-30T00:56:35.462851+0000","last_undegraded":"2024-03-30T00:56:35.462851+0000","last_fullsized":"2024-03-30T00:56:35.462851+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:43:28.085910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467887+0000","last_change":"2024-03-30T00:55:51.306357+0000","last_active":"2024-03-30T00:56:35.467887+0000","last_peered":"2024-03-30T00:56:35.467887+0000","last_clean":"2024-03-30T00:56:35.467887+0000","last_became_active":"2024-03-30T00:55:51.305815+0000","last_became_peered":"2024-03-30T00:55:51.305815+0000","last_unstale":"2024-03-30T00:56:35.467887+0000","last_undegraded":"2024-03-30T00:56:35.467887+0000","last_fullsized":"2024-03-30T00:56:35.467887+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:16:21.753446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"83'2","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.117501+0000","last_change":"2024-03-30T00:55:51.810686+0000","last_active":"2024-03-30T00:57:35.117501+0000","last_peered":"2024-03-30T00:57:35.117501+0000","last_clean":"2024-03-30T00:57:35.117501+0000","last_became_active":"2024-03-30T00:55:51.810331+0000","last_became_peered":"2024-03-30T00:55:51.810331+0000","last_unstale":"2024-03-30T00:57:35.117501+0000","last_undegraded":"2024-03-30T00:57:35.117501+0000","last_fullsized":"2024-03-30T00:57:35.117501+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:24:13.750905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.487787+0000","last_change":"2024-03-30T00:55:55.842894+0000","last_active":"2024-03-30T00:56:35.487787+0000","last_peered":"2024-03-30T00:56:35.487787+0000","last_clean":"2024-03-30T00:56:35.487787+0000","last_became_active":"2024-03-30T00:55:55.842118+0000","last_became_peered":"2024-03-30T00:55:55.842118+0000","last_unstale":"2024-03-30T00:56:35.487787+0000","last_undegraded":"2024-03-30T00:56:35.487787+0000","last_fullsized":"2024-03-30T00:56:35.487787+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:39.246228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'3","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470956+0000","last_change":"2024-03-30T00:55:51.313188+0000","last_active":"2024-03-30T00:56:35.470956+0000","last_peered":"2024-03-30T00:56:35.470956+0000","last_clean":"2024-03-30T00:56:35.470956+0000","last_became_active":"2024-03-30T00:55:51.312879+0000","last_became_peered":"2024-03-30T00:55:51.312879+0000","last_unstale":"2024-03-30T00:56:35.470956+0000","last_undegraded":"2024-03-30T00:56:35.470956+0000","last_fullsized":"2024-03-30T00:56:35.470956+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:41:23.420427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"83'9","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165088+0000","last_change":"2024-03-30T00:55:51.808998+0000","last_active":"2024-03-30T00:57:35.165088+0000","last_peered":"2024-03-30T00:57:35.165088+0000","last_clean":"2024-03-30T00:57:35.165088+0000","last_became_active":"2024-03-30T00:55:51.808636+0000","last_became_peered":"2024-03-30T00:55:51.808636+0000","last_unstale":"2024-03-30T00:57:35.165088+0000","last_undegraded":"2024-03-30T00:57:35.165088+0000","last_fullsized":"2024-03-30T00:57:35.165088+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:02.379294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.480854+0000","last_change":"2024-03-30T00:56:36.480854+0000","last_active":"2024-03-30T00:56:36.480854+0000","last_peered":"2024-03-30T00:56:36.480854+0000","last_clean":"2024-03-30T00:56:36.480854+0000","last_became_active":"2024-03-30T00:56:36.467064+0000","last_became_peered":"2024-03-30T00:56:36.467064+0000","last_unstale":"2024-03-30T00:56:36.480854+0000","last_undegraded":"2024-03-30T00:56:36.480854+0000","last_fullsized":"2024-03-30T00:56:36.480854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:12:55.154544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470669+0000","last_change":"2024-03-30T00:55:51.313708+0000","last_active":"2024-03-30T00:56:35.470669+0000","last_peered":"2024-03-30T00:56:35.470669+0000","last_clean":"2024-03-30T00:56:35.470669+0000","last_became_active":"2024-03-30T00:55:51.313362+0000","last_became_peered":"2024-03-30T00:55:51.313362+0000","last_unstale":"2024-03-30T00:56:35.470669+0000","last_undegraded":"2024-03-30T00:56:35.470669+0000","last_fullsized":"2024-03-30T00:56:35.470669+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:03:17.859640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165069+0000","last_change":"2024-03-30T00:55:51.819638+0000","last_active":"2024-03-30T00:57:35.165069+0000","last_peered":"2024-03-30T00:57:35.165069+0000","last_clean":"2024-03-30T00:57:35.165069+0000","last_became_active":"2024-03-30T00:55:51.819282+0000","last_became_peered":"2024-03-30T00:55:51.819282+0000","last_unstale":"2024-03-30T00:57:35.165069+0000","last_undegraded":"2024-03-30T00:57:35.165069+0000","last_fullsized":"2024-03-30T00:57:35.165069+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:13:39.249550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462909+0000","last_change":"2024-03-30T00:55:55.856498+0000","last_active":"2024-03-30T00:56:35.462909+0000","last_peered":"2024-03-30T00:56:35.462909+0000","last_clean":"2024-03-30T00:56:35.462909+0000","last_became_active":"2024-03-30T00:55:55.856015+0000","last_became_peered":"2024-03-30T00:55:55.856015+0000","last_unstale":"2024-03-30T00:56:35.462909+0000","last_undegraded":"2024-03-30T00:56:35.462909+0000","last_fullsized":"2024-03-30T00:56:35.462909+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:14:00.248812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462909+0000","last_change":"2024-03-30T00:55:51.302995+0000","last_active":"2024-03-30T00:56:35.462909+0000","last_peered":"2024-03-30T00:56:35.462909+0000","last_clean":"2024-03-30T00:56:35.462909+0000","last_became_active":"2024-03-30T00:55:51.302650+0000","last_became_peered":"2024-03-30T00:55:51.302650+0000","last_unstale":"2024-03-30T00:56:35.462909+0000","last_undegraded":"2024-03-30T00:56:35.462909+0000","last_fullsized":"2024-03-30T00:56:35.462909+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:17:32.556550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162368+0000","last_change":"2024-03-30T00:55:51.808969+0000","last_active":"2024-03-30T00:57:35.162368+0000","last_peered":"2024-03-30T00:57:35.162368+0000","last_clean":"2024-03-30T00:57:35.162368+0000","last_became_active":"2024-03-30T00:55:51.808633+0000","last_became_peered":"2024-03-30T00:55:51.808633+0000","last_unstale":"2024-03-30T00:57:35.162368+0000","last_undegraded":"2024-03-30T00:57:35.162368+0000","last_fullsized":"2024-03-30T00:57:35.162368+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:16:46.128066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468267+0000","last_change":"2024-03-30T00:55:55.824964+0000","last_active":"2024-03-30T00:56:35.468267+0000","last_peered":"2024-03-30T00:56:35.468267+0000","last_clean":"2024-03-30T00:56:35.468267+0000","last_became_active":"2024-03-30T00:55:55.824559+0000","last_became_peered":"2024-03-30T00:55:55.824559+0000","last_unstale":"2024-03-30T00:56:35.468267+0000","last_undegraded":"2024-03-30T00:56:35.468267+0000","last_fullsized":"2024-03-30T00:56:35.468267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:19:24.746963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'3","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470912+0000","last_change":"2024-03-30T00:55:51.301602+0000","last_active":"2024-03-30T00:56:35.470912+0000","last_peered":"2024-03-30T00:56:35.470912+0000","last_clean":"2024-03-30T00:56:35.470912+0000","last_became_active":"2024-03-30T00:55:51.301221+0000","last_became_peered":"2024-03-30T00:55:51.301221+0000","last_unstale":"2024-03-30T00:56:35.470912+0000","last_undegraded":"2024-03-30T00:56:35.470912+0000","last_fullsized":"2024-03-30T00:56:35.470912+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:37:36.727603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"83'7","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153553+0000","last_change":"2024-03-30T00:55:51.810197+0000","last_active":"2024-03-30T00:57:35.153553+0000","last_peered":"2024-03-30T00:57:35.153553+0000","last_clean":"2024-03-30T00:57:35.153553+0000","last_became_active":"2024-03-30T00:55:51.809193+0000","last_became_peered":"2024-03-30T00:55:51.809193+0000","last_unstale":"2024-03-30T00:57:35.153553+0000","last_undegraded":"2024-03-30T00:57:35.153553+0000","last_fullsized":"2024-03-30T00:57:35.153553+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:52:41.078425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.494873+0000","last_change":"2024-03-30T00:55:55.830549+0000","last_active":"2024-03-30T00:56:35.494873+0000","last_peered":"2024-03-30T00:56:35.494873+0000","last_clean":"2024-03-30T00:56:35.494873+0000","last_became_active":"2024-03-30T00:55:55.827966+0000","last_became_peered":"2024-03-30T00:55:55.827966+0000","last_unstale":"2024-03-30T00:56:35.494873+0000","last_undegraded":"2024-03-30T00:56:35.494873+0000","last_fullsized":"2024-03-30T00:56:35.494873+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:30:36.531960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"76'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467092+0000","last_change":"2024-03-30T00:55:51.311998+0000","last_active":"2024-03-30T00:56:35.467092+0000","last_peered":"2024-03-30T00:56:35.467092+0000","last_clean":"2024-03-30T00:56:35.467092+0000","last_became_active":"2024-03-30T00:55:51.311627+0000","last_became_peered":"2024-03-30T00:55:51.311627+0000","last_unstale":"2024-03-30T00:56:35.467092+0000","last_undegraded":"2024-03-30T00:56:35.467092+0000","last_fullsized":"2024-03-30T00:56:35.467092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:23:06.939919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.158955+0000","last_change":"2024-03-30T00:55:51.819457+0000","last_active":"2024-03-30T00:57:35.158955+0000","last_peered":"2024-03-30T00:57:35.158955+0000","last_clean":"2024-03-30T00:57:35.158955+0000","last_became_active":"2024-03-30T00:55:51.819099+0000","last_became_peered":"2024-03-30T00:55:51.819099+0000","last_unstale":"2024-03-30T00:57:35.158955+0000","last_undegraded":"2024-03-30T00:57:35.158955+0000","last_fullsized":"2024-03-30T00:57:35.158955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:31:57.363328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466097+0000","last_change":"2024-03-30T00:55:55.824882+0000","last_active":"2024-03-30T00:56:35.466097+0000","last_peered":"2024-03-30T00:56:35.466097+0000","last_clean":"2024-03-30T00:56:35.466097+0000","last_became_active":"2024-03-30T00:55:55.824493+0000","last_became_peered":"2024-03-30T00:55:55.824493+0000","last_unstale":"2024-03-30T00:56:35.466097+0000","last_undegraded":"2024-03-30T00:56:35.466097+0000","last_fullsized":"2024-03-30T00:56:35.466097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:24:48.825776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"76'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.476919+0000","last_change":"2024-03-30T00:55:51.311787+0000","last_active":"2024-03-30T00:56:35.476919+0000","last_peered":"2024-03-30T00:56:35.476919+0000","last_clean":"2024-03-30T00:56:35.476919+0000","last_became_active":"2024-03-30T00:55:51.311402+0000","last_became_peered":"2024-03-30T00:55:51.311402+0000","last_unstale":"2024-03-30T00:56:35.476919+0000","last_undegraded":"2024-03-30T00:56:35.476919+0000","last_fullsized":"2024-03-30T00:56:35.476919+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:37:13.960453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.151904+0000","last_change":"2024-03-30T00:55:51.809742+0000","last_active":"2024-03-30T00:57:35.151904+0000","last_peered":"2024-03-30T00:57:35.151904+0000","last_clean":"2024-03-30T00:57:35.151904+0000","last_became_active":"2024-03-30T00:55:51.809396+0000","last_became_peered":"2024-03-30T00:55:51.809396+0000","last_unstale":"2024-03-30T00:57:35.151904+0000","last_undegraded":"2024-03-30T00:57:35.151904+0000","last_fullsized":"2024-03-30T00:57:35.151904+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:22.535472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"83'9","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.145644+0000","last_change":"2024-03-30T00:55:51.814429+0000","last_active":"2024-03-30T00:57:35.145644+0000","last_peered":"2024-03-30T00:57:35.145644+0000","last_clean":"2024-03-30T00:57:35.145644+0000","last_became_active":"2024-03-30T00:55:51.814044+0000","last_became_peered":"2024-03-30T00:55:51.814044+0000","last_unstale":"2024-03-30T00:57:35.145644+0000","last_undegraded":"2024-03-30T00:57:35.145644+0000","last_fullsized":"2024-03-30T00:57:35.145644+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:24:58.830738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"83'12","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.146416+0000","last_change":"2024-03-30T00:55:51.828977+0000","last_active":"2024-03-30T00:57:35.146416+0000","last_peered":"2024-03-30T00:57:35.146416+0000","last_clean":"2024-03-30T00:57:35.146416+0000","last_became_active":"2024-03-30T00:55:51.828656+0000","last_became_peered":"2024-03-30T00:55:51.828656+0000","last_unstale":"2024-03-30T00:57:35.146416+0000","last_undegraded":"2024-03-30T00:57:35.146416+0000","last_fullsized":"2024-03-30T00:57:35.146416+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:10:23.463770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.161205+0000","last_change":"2024-03-30T00:55:51.821511+0000","last_active":"2024-03-30T00:57:35.161205+0000","last_peered":"2024-03-30T00:57:35.161205+0000","last_clean":"2024-03-30T00:57:35.161205+0000","last_became_active":"2024-03-30T00:55:51.820876+0000","last_became_peered":"2024-03-30T00:55:51.820876+0000","last_unstale":"2024-03-30T00:57:35.161205+0000","last_undegraded":"2024-03-30T00:57:35.161205+0000","last_fullsized":"2024-03-30T00:57:35.161205+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:14:48.299376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.167065+0000","last_change":"2024-03-30T00:55:51.823867+0000","last_active":"2024-03-30T00:57:35.167065+0000","last_peered":"2024-03-30T00:57:35.167065+0000","last_clean":"2024-03-30T00:57:35.167065+0000","last_became_active":"2024-03-30T00:55:51.820230+0000","last_became_peered":"2024-03-30T00:55:51.820230+0000","last_unstale":"2024-03-30T00:57:35.167065+0000","last_undegraded":"2024-03-30T00:57:35.167065+0000","last_fullsized":"2024-03-30T00:57:35.167065+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:38:51.257855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"83'4","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165868+0000","last_change":"2024-03-30T00:55:51.821413+0000","last_active":"2024-03-30T00:57:35.165868+0000","last_peered":"2024-03-30T00:57:35.165868+0000","last_clean":"2024-03-30T00:57:35.165868+0000","last_became_active":"2024-03-30T00:55:51.820833+0000","last_became_peered":"2024-03-30T00:55:51.820833+0000","last_unstale":"2024-03-30T00:57:35.165868+0000","last_undegraded":"2024-03-30T00:57:35.165868+0000","last_fullsized":"2024-03-30T00:57:35.165868+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:34:11.969849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.149312+0000","last_change":"2024-03-30T00:55:51.813674+0000","last_active":"2024-03-30T00:57:35.149312+0000","last_peered":"2024-03-30T00:57:35.149312+0000","last_clean":"2024-03-30T00:57:35.149312+0000","last_became_active":"2024-03-30T00:55:51.813341+0000","last_became_peered":"2024-03-30T00:55:51.813341+0000","last_unstale":"2024-03-30T00:57:35.149312+0000","last_undegraded":"2024-03-30T00:57:35.149312+0000","last_fullsized":"2024-03-30T00:57:35.149312+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:54:17.285997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"83'15","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.148932+0000","last_change":"2024-03-30T00:55:51.812472+0000","last_active":"2024-03-30T00:57:35.148932+0000","last_peered":"2024-03-30T00:57:35.148932+0000","last_clean":"2024-03-30T00:57:35.148932+0000","last_became_active":"2024-03-30T00:55:51.812061+0000","last_became_peered":"2024-03-30T00:55:51.812061+0000","last_unstale":"2024-03-30T00:57:35.148932+0000","last_undegraded":"2024-03-30T00:57:35.148932+0000","last_fullsized":"2024-03-30T00:57:35.148932+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:46.550201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"83'2","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.134547+0000","last_change":"2024-03-30T00:55:51.829179+0000","last_active":"2024-03-30T00:57:35.134547+0000","last_peered":"2024-03-30T00:57:35.134547+0000","last_clean":"2024-03-30T00:57:35.134547+0000","last_became_active":"2024-03-30T00:55:51.828805+0000","last_became_peered":"2024-03-30T00:55:51.828805+0000","last_unstale":"2024-03-30T00:57:35.134547+0000","last_undegraded":"2024-03-30T00:57:35.134547+0000","last_fullsized":"2024-03-30T00:57:35.134547+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:23.576125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.155774+0000","last_change":"2024-03-30T00:55:51.823170+0000","last_active":"2024-03-30T00:57:35.155774+0000","last_peered":"2024-03-30T00:57:35.155774+0000","last_clean":"2024-03-30T00:57:35.155774+0000","last_became_active":"2024-03-30T00:55:51.822785+0000","last_became_peered":"2024-03-30T00:55:51.822785+0000","last_unstale":"2024-03-30T00:57:35.155774+0000","last_undegraded":"2024-03-30T00:57:35.155774+0000","last_fullsized":"2024-03-30T00:57:35.155774+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:34:43.393540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153749+0000","last_change":"2024-03-30T00:55:51.817160+0000","last_active":"2024-03-30T00:57:35.153749+0000","last_peered":"2024-03-30T00:57:35.153749+0000","last_clean":"2024-03-30T00:57:35.153749+0000","last_became_active":"2024-03-30T00:55:51.816881+0000","last_became_peered":"2024-03-30T00:55:51.816881+0000","last_unstale":"2024-03-30T00:57:35.153749+0000","last_undegraded":"2024-03-30T00:57:35.153749+0000","last_fullsized":"2024-03-30T00:57:35.153749+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:09:05.211242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"83'10","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163651+0000","last_change":"2024-03-30T00:55:51.817670+0000","last_active":"2024-03-30T00:57:35.163651+0000","last_peered":"2024-03-30T00:57:35.163651+0000","last_clean":"2024-03-30T00:57:35.163651+0000","last_became_active":"2024-03-30T00:55:51.817380+0000","last_became_peered":"2024-03-30T00:55:51.817380+0000","last_unstale":"2024-03-30T00:57:35.163651+0000","last_undegraded":"2024-03-30T00:57:35.163651+0000","last_fullsized":"2024-03-30T00:57:35.163651+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:24.582557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"83'11","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.167761+0000","last_change":"2024-03-30T00:55:51.813071+0000","last_active":"2024-03-30T00:57:35.167761+0000","last_peered":"2024-03-30T00:57:35.167761+0000","last_clean":"2024-03-30T00:57:35.167761+0000","last_became_active":"2024-03-30T00:55:51.811681+0000","last_became_peered":"2024-03-30T00:55:51.811681+0000","last_unstale":"2024-03-30T00:57:35.167761+0000","last_undegraded":"2024-03-30T00:57:35.167761+0000","last_fullsized":"2024-03-30T00:57:35.167761+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:10:53.840055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.154929+0000","last_change":"2024-03-30T00:56:36.472592+0000","last_active":"2024-03-30T00:57:35.154929+0000","last_peered":"2024-03-30T00:57:35.154929+0000","last_clean":"2024-03-30T00:57:35.154929+0000","last_became_active":"2024-03-30T00:56:36.472173+0000","last_became_peered":"2024-03-30T00:56:36.472173+0000","last_unstale":"2024-03-30T00:57:35.154929+0000","last_undegraded":"2024-03-30T00:57:35.154929+0000","last_fullsized":"2024-03-30T00:57:35.154929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:22:27.915984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":212960,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71848,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":522,"num_read_kb":339,"num_write":168,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":215544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645345,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":462418,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.41499999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.73099999999999998,"5min":1.399,"15min":1.399},"last":0.44},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.622,"15min":0.626},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.83399999999999996,"5min":1.282,"15min":1.282},"last":0.57799999999999996}]},{"osd":1,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.61199999999999999,"15min":0.60699999999999998},"min":{"1min":0.46800000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.45900000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.69599999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.61199999999999999}]},{"osd":2,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.46500000000000002,"15min":0.45400000000000001},"min":{"1min":0.36799999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.46200000000000002,"15min":0.45800000000000002},"min":{"1min":0.373,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.41399999999999998}]},{"osd":3,"last update":"Sat Mar 30 00:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.505}]},{"osd":4,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.62,"15min":0.61199999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64000000000000001,"15min":0.63800000000000001},"min":{"1min":0.49399999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.76100000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.72299999999999998}]},{"osd":5,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.58399999999999996,"15min":0.57499999999999996},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.876,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.47199999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.54400000000000004}]},{"osd":6,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.51700000000000002,"5min":0.436,"15min":0.436},"max":{"1min":1.0149999999999999,"5min":1.056,"15min":1.056},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72699999999999998,"15min":0.71799999999999997},"min":{"1min":0.622,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97499999999999998,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.76100000000000001}]},{"osd":7,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71099999999999997,"15min":0.70299999999999996},"min":{"1min":0.66000000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.748},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72499999999999998,"15min":0.71799999999999997},"min":{"1min":0.68000000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.96499999999999997,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.68500000000000005}]},{"osd":8,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.78700000000000003}]},{"osd":9,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.80800000000000005,"15min":0.79700000000000004},"min":{"1min":0.70099999999999996,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.052,"5min":1.353,"15min":1.353},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.75700000000000001,"15min":0.751},"min":{"1min":0.69399999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98599999999999999,"5min":1.083,"15min":1.083},"last":0.71299999999999997}]},{"osd":10,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77400000000000002,"15min":0.77000000000000002},"min":{"1min":0.72499999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.90200000000000002,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.83399999999999996,"15min":0.82899999999999996},"min":{"1min":0.749,"5min":0.623,"15min":0.623},"max":{"1min":1.0580000000000001,"5min":1.52,"15min":1.52},"last":0.82799999999999996}]}]},{"osd":10,"up_from":65,"seq":279172874276,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":1121169,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.58399999999999996}]},{"osd":1,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.439,"15min":0.441},"min":{"1min":0.29299999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.53900000000000003,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.33800000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56599999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.42099999999999999}]},{"osd":2,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.54700000000000004,"15min":0.54000000000000004},"min":{"1min":0.38200000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.626},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.58499999999999996},"min":{"1min":0.42199999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.75700000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.504}]},{"osd":3,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82699999999999996,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63200000000000001,"15min":0.63900000000000001},"min":{"1min":0.495,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.60199999999999998}]},{"osd":4,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35999999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.65200000000000002,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.45000000000000001}]},{"osd":5,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.61599999999999999,"15min":0.61299999999999999},"min":{"1min":0.45600000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.46800000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.52900000000000003}]},{"osd":6,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.876,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68799999999999994,"15min":0.68999999999999995},"min":{"1min":0.52300000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78500000000000003,"5min":1.131,"15min":1.131},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":8,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.69599999999999995,"15min":0.69799999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.80800000000000005,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.55400000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.83399999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.69199999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.75900000000000001,"15min":0.76500000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.85999999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.748,"15min":0.752},"min":{"1min":0.52400000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.85999999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.72299999999999998}]},{"osd":11,"last update":"Sat Mar 30 00:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73099999999999998,"15min":0.73299999999999998},"min":{"1min":0.60299999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.83099999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.749,"15min":0.749},"min":{"1min":0.59499999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.85699999999999998,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.81699999999999995}]}]},{"osd":9,"up_from":60,"seq":257698037798,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":462620,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.628,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.316}]},{"osd":1,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.94199999999999995,"5min":1.274,"15min":1.274},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.73399999999999999,"15min":0.74299999999999999},"min":{"1min":0.51600000000000001,"5min":0.504,"15min":0.504},"max":{"1min":0.90600000000000003,"5min":1.411,"15min":1.411},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.73399999999999999,"15min":0.74299999999999999},"min":{"1min":0.55400000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.84699999999999998,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.501,"5min":0.435,"15min":0.435},"max":{"1min":0.83999999999999997,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.83899999999999997}]},{"osd":3,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.105,"5min":1.391,"15min":1.391},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.73999999999999999,"15min":0.75900000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95899999999999996,"5min":1.302,"15min":1.302},"last":0.434}]},{"osd":4,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69199999999999995,"15min":0.69799999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.88700000000000001,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.877},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.81100000000000005,"15min":0.82299999999999995},"min":{"1min":0.56200000000000006,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98299999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.63}]},{"osd":5,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.71899999999999997,"15min":0.72899999999999998},"min":{"1min":0.45800000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.88300000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.83799999999999997,"15min":0.85799999999999998},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0129999999999999,"5min":1.853,"15min":1.853},"last":0.5}]},{"osd":6,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.623}]},{"osd":7,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.86399999999999999,"15min":0.88},"min":{"1min":0.54100000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.94099999999999995,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.83299999999999996,"15min":0.84399999999999997},"min":{"1min":0.58699999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94599999999999995,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.91200000000000003}]},{"osd":8,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.90100000000000002,"15min":0.92000000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.98099999999999998,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.90000000000000002,"15min":0.92200000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0720000000000001,"5min":1.968,"15min":1.968},"last":0.52000000000000002}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.88700000000000001,"15min":0.88100000000000001},"min":{"1min":0.65000000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.8170000000000002,"5min":2.8170000000000002,"15min":2.8170000000000002},"last":0.622},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.84999999999999998,"15min":0.84099999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":2.9889999999999999,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":0.94199999999999995}]},{"osd":11,"last update":"Sat Mar 30 00:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.53600000000000003}]}]},{"osd":8,"up_from":53,"seq":227633266731,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":921600,"data_stored":462209,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56499999999999995,"15min":0.56100000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77500000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54900000000000004,"15min":0.54400000000000004},"min":{"1min":0.40999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.80400000000000005,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.505}]},{"osd":1,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.59199999999999997,"15min":0.57299999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78700000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59299999999999997,"15min":0.57999999999999996},"min":{"1min":0.47999999999999998,"5min":0.436,"15min":0.436},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.44600000000000001,"15min":0.42599999999999999},"min":{"1min":0.35899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64100000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.45000000000000001},"min":{"1min":0.38800000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.68999999999999995,"5min":0.751,"15min":0.751},"last":0.64500000000000002}]},{"osd":3,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.68899999999999995,"15min":0.72399999999999998},"min":{"1min":0.434,"5min":0.372,"15min":0.372},"max":{"1min":0.79700000000000004,"5min":2.883,"15min":2.883},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.59999999999999998,"15min":0.58399999999999996},"min":{"1min":0.47599999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.62}]},{"osd":4,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.61099999999999999,"15min":0.60599999999999998},"min":{"1min":0.51000000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.85599999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.629,"15min":0.60699999999999998},"min":{"1min":0.53100000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.57599999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.52300000000000002,"15min":0.498},"min":{"1min":0.41299999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.55700000000000005,"15min":0.53900000000000003},"min":{"1min":0.45100000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84199999999999997,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.40200000000000002}]},{"osd":6,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.66200000000000003,"15min":0.63800000000000001},"min":{"1min":0.56100000000000005,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.77000000000000002,"15min":0.78700000000000003},"min":{"1min":0.59999999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.92600000000000005,"5min":2.8940000000000001,"15min":2.8940000000000001},"last":0.60699999999999998}]},{"osd":7,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.67600000000000005,"15min":0.66000000000000003},"min":{"1min":0.51600000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88500000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.54400000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.88500000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.58999999999999997}]},{"osd":9,"last update":"Sat Mar 30 00:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.70999999999999996,"15min":0.69899999999999995},"min":{"1min":0.58999999999999997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93500000000000005,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.74299999999999999,"15min":0.72599999999999998},"min":{"1min":0.68600000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.65800000000000003}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.53800000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91300000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.75700000000000001,"15min":0.74399999999999999},"min":{"1min":0.66100000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.94899999999999995,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.74099999999999999}]},{"osd":11,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.78700000000000003}]}]},{"osd":7,"up_from":46,"seq":197568495663,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":1059005,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60199999999999998}]},{"osd":1,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.44800000000000001,"15min":0.44700000000000001},"min":{"1min":0.32500000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45000000000000001,"15min":0.442},"min":{"1min":0.33700000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68700000000000006,"5min":0.747,"15min":0.747},"last":0.379}]},{"osd":2,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83299999999999996,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.621},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58199999999999996,"15min":0.56399999999999995},"min":{"1min":0.40600000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78900000000000003,"5min":0.877,"15min":0.877},"last":0.50600000000000001}]},{"osd":3,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60599999999999998,"15min":0.60199999999999998},"min":{"1min":0.443,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.85299999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.66100000000000003,"15min":0.625},"min":{"1min":0.45800000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":5.7430000000000003,"5min":5.7430000000000003,"15min":5.7430000000000003},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53100000000000003,"15min":0.52600000000000002},"min":{"1min":0.39500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66400000000000003,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.314,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.69399999999999995,"5min":1.008,"15min":1.008},"last":0.40899999999999997}]},{"osd":5,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63300000000000001,"15min":0.62},"min":{"1min":0.47399999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82699999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.63300000000000001,"15min":0.62},"min":{"1min":0.49399999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80600000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.66300000000000003}]},{"osd":6,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66700000000000004,"15min":0.65500000000000003},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.372},"max":{"1min":0.86799999999999999,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.73899999999999999,"15min":0.71899999999999997},"min":{"1min":0.51000000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89200000000000002,"5min":4.117,"15min":4.117},"last":0.52700000000000002}]},{"osd":8,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.621,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86599999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.752,"15min":0.751},"min":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.88900000000000001,"5min":1.014,"15min":1.014},"last":0.65600000000000003}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.74399999999999999,"15min":0.73999999999999999},"min":{"1min":0.60299999999999998,"5min":0.495,"15min":0.495},"max":{"1min":0.94199999999999995,"5min":1.018,"15min":1.018},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.91400000000000003,"15min":0.91900000000000004},"min":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":3.8839999999999999,"5min":4.0519999999999996,"15min":4.0519999999999996},"last":0.67600000000000005}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.755,"15min":0.76400000000000001},"min":{"1min":0.51900000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91300000000000003,"5min":1.032,"15min":1.032},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.71299999999999997,"15min":0.71699999999999997},"min":{"1min":0.59499999999999997,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55500000000000005}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.71599999999999997}]}]},{"osd":6,"up_from":42,"seq":180388626483,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":905216,"data_stored":457796,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.308,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.499,"15min":0.48699999999999999},"min":{"1min":0.373,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.878}]},{"osd":1,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.66300000000000003,"15min":0.63500000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60599999999999998,"15min":0.59599999999999997},"min":{"1min":0.39300000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.68100000000000005}]},{"osd":2,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70899999999999996,"15min":0.68300000000000005},"min":{"1min":0.51000000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.98699999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.66500000000000004,"15min":0.64300000000000002},"min":{"1min":0.47299999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":1.004}]},{"osd":3,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.65000000000000002},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.97999999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67600000000000005,"15min":0.67200000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.76300000000000001,"15min":0.751},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.105,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69399999999999995,"15min":0.68200000000000005},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.93899999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.74099999999999999,"15min":0.73099999999999998},"min":{"1min":0.51600000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":1.069}]},{"osd":7,"last update":"Sat Mar 30 00:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.81399999999999995},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.046,"5min":1.127,"15min":1.127},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76800000000000002,"15min":0.76900000000000002},"min":{"1min":0.57699999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.2330000000000001,"5min":1.335,"15min":1.335},"last":0.65900000000000003}]},{"osd":8,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.82399999999999995,"15min":0.84499999999999997},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.2709999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.626},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.876,"15min":0.88800000000000001},"min":{"1min":0.628,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0009999999999999,"5min":1.206,"15min":1.206},"last":0.76700000000000002}]},{"osd":9,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.94299999999999995,"15min":0.95199999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.244,"5min":1.268,"15min":1.268},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.347,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":1.103}]},{"osd":10,"last update":"Sat Mar 30 00:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85499999999999998,"15min":0.85699999999999998},"min":{"1min":0.69199999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":1.204},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.92500000000000004,"15min":0.93000000000000005},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.0760000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.82599999999999996}]},{"osd":11,"last update":"Sat Mar 30 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.71499999999999997}]}]},{"osd":1,"up_from":12,"seq":51539607621,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":524393,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54900000000000004,"15min":0.53300000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65800000000000003,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.58599999999999997,"15min":0.55500000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.74399999999999999,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.45700000000000002}]},{"osd":2,"last update":"Sat Mar 30 00:57:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.56999999999999995,"15min":0.50700000000000001},"min":{"1min":0.47299999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.81100000000000005,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.57599999999999996,"15min":0.51400000000000001},"min":{"1min":0.42599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.70799999999999996}]},{"osd":3,"last update":"Sat Mar 30 00:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56200000000000006,"15min":0.52800000000000002},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.69199999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.503},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57399999999999995,"15min":0.55000000000000004},"min":{"1min":0.501,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.66900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.626}]},{"osd":4,"last update":"Sat Mar 30 00:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.439,"15min":0.41499999999999998},"min":{"1min":0.377,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.50600000000000001,"15min":0.47399999999999998},"min":{"1min":0.38200000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.57899999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.61099999999999999,"15min":0.61299999999999999},"min":{"1min":0.52900000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78600000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.66900000000000004,"15min":0.64500000000000002},"min":{"1min":0.625,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.748}]},{"osd":6,"last update":"Sat Mar 30 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.55700000000000005,"5min":0.435,"15min":0.435},"max":{"1min":0.81699999999999995,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.54800000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.84799999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.68300000000000005}]},{"osd":7,"last update":"Sat Mar 30 00:57:08 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.57799999999999996,"15min":0.55900000000000005},"min":{"1min":0.48899999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.59399999999999997,"15min":0.58699999999999997},"min":{"1min":0.5,"5min":0.313,"15min":0.313},"max":{"1min":0.91800000000000004,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.73599999999999999}]},{"osd":8,"last update":"Sat Mar 30 00:57:31 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.67200000000000004,"15min":0.66000000000000003},"min":{"1min":0.58999999999999997,"5min":0.38,"15min":0.38},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.63900000000000001,"15min":0.63},"min":{"1min":0.52100000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.69899999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.746,"15min":0.73699999999999999},"min":{"1min":0.56899999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.91600000000000004,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72999999999999998,"15min":0.72399999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.81100000000000005}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.66200000000000003,"15min":0.65400000000000003},"min":{"1min":0.50800000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.74099999999999999,"15min":0.72899999999999998},"min":{"1min":0.71999999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.95499999999999996,"5min":1.385,"15min":1.385},"last":0.82799999999999996}]},{"osd":11,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.79700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32016,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713136,"statfs":{"total":95995035648,"available":95962251264,"internally_reserved":0,"allocated":925696,"data_stored":458911,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 00:57:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.67800000000000005,"15min":0.60199999999999998},"min":{"1min":0.39300000000000002,"5min":0.372,"15min":0.372},"max":{"1min":9.266,"5min":9.266,"15min":9.266},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.624,"15min":0.57799999999999996},"min":{"1min":0.42299999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.6899999999999999,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":0.65400000000000003}]},{"osd":2,"last update":"Sat Mar 30 00:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.59599999999999997,"15min":0.54300000000000004},"min":{"1min":0.41399999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71999999999999997,"15min":0.60499999999999998},"min":{"1min":0.46300000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.071,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":0.52600000000000002}]},{"osd":3,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.9139999999999997,"5min":7.9139999999999997,"15min":7.9139999999999997},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.96,"5min":7.96,"15min":7.96},"last":0.503}]},{"osd":4,"last update":"Sat Mar 30 00:57:16 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":0.75800000000000001,"15min":0.65900000000000003},"min":{"1min":0.48199999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":9.2349999999999994,"5min":9.2349999999999994,"15min":9.2349999999999994},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.71199999999999997,"15min":0.63400000000000001},"min":{"1min":0.626,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":3.73,"5min":3.73,"15min":3.73},"last":0.70199999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76400000000000001,"15min":0.67300000000000004},"min":{"1min":0.52300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.02,"5min":9.0579999999999998,"15min":9.0579999999999998},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.72399999999999998,"15min":0.64900000000000002},"min":{"1min":0.53200000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.157,"5min":3.8039999999999998,"15min":3.8039999999999998},"last":0.67800000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":0.72099999999999997,"15min":0.65900000000000003},"min":{"1min":0.42399999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":8.0050000000000008,"5min":8.0050000000000008,"15min":8.0050000000000008},"last":0.628},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.66800000000000004,"15min":0.65800000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.71399999999999997}]},{"osd":7,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.4140000000000001,"5min":2.4140000000000001,"15min":2.4140000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":9.2959999999999994,"5min":9.2959999999999994,"15min":9.2959999999999994},"last":0.78700000000000003}]},{"osd":8,"last update":"Sat Mar 30 00:57:29 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.78200000000000003,"15min":0.78000000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.375,"5min":2.375,"15min":2.375},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.226,"5min":0.91200000000000003,"15min":0.876},"min":{"1min":0.57299999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":9.2040000000000006,"5min":9.2040000000000006,"15min":9.2040000000000006},"last":0.76900000000000002}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":0.88700000000000001,"15min":0.83699999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":8.0389999999999997,"5min":8.0389999999999997,"15min":8.0389999999999997},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.78400000000000003,"15min":0.76600000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":0.83599999999999997}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":0.90800000000000003,"15min":0.88200000000000001},"min":{"1min":0.64000000000000001,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":3.8519999999999999,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":0.752},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.82799999999999996,"15min":0.81200000000000006},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.4470000000000001,"5min":2.4470000000000001,"15min":2.4470000000000001},"last":0.96399999999999997}]},{"osd":11,"last update":"Sat Mar 30 00:57:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.86099999999999999,"15min":0.84799999999999998},"min":{"1min":0.73699999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":0.995},{"interface":"front","average":{"1min":1.3140000000000001,"5min":0.92400000000000004,"15min":0.85899999999999999},"min":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":9.1709999999999994,"5min":9.1709999999999994,"15min":9.1709999999999994},"last":0.83199999999999996}]}]},{"osd":2,"up_from":17,"seq":73014444099,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":525799,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54100000000000004,"15min":0.52900000000000003},"min":{"1min":0.41399999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.70999999999999996,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54600000000000004,"15min":0.52700000000000002},"min":{"1min":0.38800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72899999999999998,"5min":1.097,"15min":1.097},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Mar 30 00:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.56699999999999995,"15min":0.52400000000000002},"min":{"1min":0.46400000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82599999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56000000000000005,"15min":0.53400000000000003},"min":{"1min":0.46200000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.84099999999999997,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.70699999999999996}]},{"osd":3,"last update":"Sat Mar 30 00:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.59499999999999997,"15min":0.56499999999999995},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.80300000000000005,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59799999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.72499999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.65800000000000003}]},{"osd":4,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.95799999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:57:31 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.45800000000000002,"15min":0.436},"min":{"1min":0.378,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.78800000000000003,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.48399999999999999,"15min":0.46200000000000002},"min":{"1min":0.38300000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73199999999999998,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.76700000000000002}]},{"osd":6,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65100000000000002,"15min":0.65000000000000002},"min":{"1min":0.44600000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86399999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66500000000000004,"15min":0.66100000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.85999999999999999}]},{"osd":7,"last update":"Sat Mar 30 00:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.88},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.79000000000000004}]},{"osd":8,"last update":"Sat Mar 30 00:57:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60199999999999998,"15min":0.59199999999999997},"min":{"1min":0.48199999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.61299999999999999,"15min":0.60099999999999998},"min":{"1min":0.497,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.84499999999999997}]},{"osd":9,"last update":"Sat Mar 30 00:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.72199999999999998,"15min":0.70699999999999996},"min":{"1min":0.58599999999999997,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.75700000000000001,"15min":0.73899999999999999},"min":{"1min":0.65600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.81799999999999995}]},{"osd":10,"last update":"Sat Mar 30 00:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67300000000000004,"15min":0.66700000000000004},"min":{"1min":0.55000000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91000000000000003,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.68100000000000005,"15min":0.66700000000000004},"min":{"1min":0.66700000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.97899999999999998}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":1.073}]}]},{"osd":3,"up_from":24,"seq":103079215166,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":457717,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.46000000000000002,"15min":0.41399999999999998},"min":{"1min":0.32900000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46400000000000002,"15min":0.432},"min":{"1min":0.35199999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.997,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.30599999999999999}]},{"osd":1,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65100000000000002,"15min":0.60299999999999998},"min":{"1min":0.45000000000000001,"5min":0.375,"15min":0.375},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.439,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.58099999999999996}]},{"osd":2,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.67200000000000004,"15min":0.629},"min":{"1min":0.51400000000000001,"5min":0.371,"15min":0.371},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64800000000000002,"15min":0.622},"min":{"1min":0.47599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.68799999999999994}]},{"osd":4,"last update":"Sat Mar 30 00:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70799999999999996,"15min":0.66800000000000004},"min":{"1min":0.501,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67900000000000005,"15min":0.66500000000000004},"min":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65100000000000002}]},{"osd":5,"last update":"Sat Mar 30 00:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.67400000000000004,"15min":0.65200000000000002},"min":{"1min":0.51000000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.73399999999999999,"15min":0.67600000000000005},"min":{"1min":0.48799999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":0.53800000000000003}]},{"osd":6,"last update":"Sat Mar 30 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.65000000000000002,"15min":0.63800000000000001},"min":{"1min":0.46800000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.44,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.34799999999999998}]},{"osd":7,"last update":"Sat Mar 30 00:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.74399999999999999,"15min":0.75},"min":{"1min":0.51200000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83699999999999997,"15min":0.83999999999999997},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.73399999999999999}]},{"osd":8,"last update":"Sat Mar 30 00:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.57399999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.92400000000000004,"15min":0.94399999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.2410000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.502},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.84499999999999997,"15min":0.85899999999999999},"min":{"1min":0.54700000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.224,"5min":1.347,"15min":1.347},"last":0.52500000000000002}]},{"osd":10,"last update":"Sat Mar 30 00:57:03 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77600000000000002,"15min":0.77100000000000002},"min":{"1min":0.53800000000000003,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.92000000000000004,"15min":0.92200000000000004},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.61499999999999999}]},{"osd":11,"last update":"Sat Mar 30 00:56:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.755},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.60599999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986235,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":909312,"data_stored":462634,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.66700000000000004,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.79300000000000004,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73099999999999998,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.41099999999999998,"15min":0.39600000000000002},"min":{"1min":0.34899999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.65300000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.41899999999999998,"15min":0.40699999999999997},"min":{"1min":0.36199999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71799999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.45200000000000001}]},{"osd":2,"last update":"Sat Mar 30 00:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.56799999999999995,"15min":0.55000000000000004},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83299999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.56699999999999995,"15min":0.54300000000000004},"min":{"1min":0.45700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71699999999999997,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.51200000000000001}]},{"osd":3,"last update":"Sat Mar 30 00:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.63700000000000001,"15min":0.69099999999999995},"min":{"1min":0.48899999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.74399999999999999,"5min":3.887,"15min":3.887},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.45000000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.53800000000000003}]},{"osd":5,"last update":"Sat Mar 30 00:57:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.58499999999999996,"15min":0.56699999999999995},"min":{"1min":0.38600000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.81000000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.59599999999999997,"15min":0.57499999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55400000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.623,"15min":0.625},"min":{"1min":0.42699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.64800000000000002,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82599999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.63100000000000001}]},{"osd":7,"last update":"Sat Mar 30 00:57:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55100000000000005,"15min":0.54200000000000004},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.79400000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.54800000000000004,"15min":0.52800000000000002},"min":{"1min":0.46100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.73899999999999999}]},{"osd":8,"last update":"Sat Mar 30 00:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.66500000000000004,"15min":0.65600000000000003},"min":{"1min":0.51800000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94299999999999995,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.68799999999999994,"15min":0.68000000000000005},"min":{"1min":0.53800000000000003,"5min":0.432,"15min":0.432},"max":{"1min":0.84099999999999997,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.64300000000000002}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.69899999999999995},"min":{"1min":0.53500000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.73299999999999998,"15min":0.73699999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.98999999999999999,"5min":1.012,"15min":1.012},"last":0.71099999999999997}]},{"osd":10,"last update":"Sat Mar 30 00:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.53600000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.97499999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.81599999999999995}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.751},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.79400000000000004}]}]},{"osd":5,"up_from":35,"seq":150323855414,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":978944,"data_stored":1053801,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.56899999999999995,"15min":0.52000000000000002},"min":{"1min":0.38,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":5.3970000000000002,"5min":5.3970000000000002,"15min":5.3970000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.499},"min":{"1min":0.41499999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.69599999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.45600000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57399999999999995,"15min":0.56599999999999995},"min":{"1min":0.47399999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.74299999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.56999999999999995,"15min":0.54600000000000004},"min":{"1min":0.51000000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.51000000000000001}]},{"osd":2,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.34999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.52700000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.40600000000000003,"15min":0.39900000000000002},"min":{"1min":0.32100000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.68600000000000005}]},{"osd":3,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.42599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.74399999999999999,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54800000000000004,"15min":0.53200000000000003},"min":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.68400000000000005,"5min":0.877,"15min":0.877},"last":0.54800000000000004}]},{"osd":4,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.56499999999999995},"min":{"1min":0.495,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.71799999999999997,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.60599999999999998,"15min":0.58599999999999997},"min":{"1min":0.41599999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.83699999999999997,"5min":1.038,"15min":1.038},"last":0.73299999999999998}]},{"osd":6,"last update":"Sat Mar 30 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59899999999999998,"15min":0.58099999999999996},"min":{"1min":0.42799999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81200000000000006,"5min":0.997,"15min":0.997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60499999999999998,"15min":0.59799999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.72999999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.60799999999999998}]},{"osd":7,"last update":"Sat Mar 30 00:57:09 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61899999999999999,"15min":0.60699999999999998},"min":{"1min":0.505,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.64000000000000001},"min":{"1min":0.55600000000000005,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.85399999999999998,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.58799999999999997}]},{"osd":8,"last update":"Sat Mar 30 00:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.55300000000000005,"15min":0.53600000000000003},"min":{"1min":0.373,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82799999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.56399999999999995,"15min":0.54800000000000004},"min":{"1min":0.45800000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77700000000000002,"5min":0.872,"15min":0.872},"last":0.67000000000000004}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.67300000000000004,"15min":0.66700000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68400000000000005,"15min":0.68100000000000005},"min":{"1min":0.58099999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63300000000000001}]},{"osd":10,"last update":"Sat Mar 30 00:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.70599999999999996,"15min":0.70399999999999996},"min":{"1min":0.54700000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.92200000000000004,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.71799999999999997,"15min":0.71699999999999997},"min":{"1min":0.60499999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.89800000000000002,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.66000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":69330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T00:57:46.418 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-30T00:57:46.418 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T00:57:46.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.606+0000 7f25bf200700 1 -- 172.21.15.116:0/1770749856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8100c80 msgr2=0x7f25b81010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:46.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.606+0000 7f25bf200700 1 --2- 172.21.15.116:0/1770749856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8100c80 0x7f25b81010a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009990 tx=0x7f25a8009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.606+0000 7f25bf200700 1 -- 172.21.15.116:0/1770749856 shutdown_connections 2024-03-30T00:57:46.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.606+0000 7f25bf200700 1 --2- 172.21.15.116:0/1770749856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8100c80 0x7f25b81010a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.606+0000 7f25bf200700 1 --2- 172.21.15.116:0/1770749856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b8102810 0x7f25b81090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.606+0000 7f25bf200700 1 --2- 172.21.15.116:0/1770749856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25b8101e70 0x7f25b81022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.606+0000 7f25bf200700 1 -- 172.21.15.116:0/1770749856 >> 172.21.15.116:0/1770749856 conn(0x7f25b80fc1f0 msgr2=0x7f25b80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:46.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.606+0000 7f25bf200700 1 -- 172.21.15.116:0/1770749856 shutdown_connections 2024-03-30T00:57:46.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.606+0000 7f25bf200700 1 -- 172.21.15.116:0/1770749856 wait complete. 2024-03-30T00:57:46.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.607+0000 7f25bf200700 1 Processor -- start 2024-03-30T00:57:46.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.607+0000 7f25bf200700 1 -- start start 2024-03-30T00:57:46.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.607+0000 7f25bf200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25b8100c80 0x7f25b8196cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:46.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.607+0000 7f25bf200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b8101e70 0x7f25b8197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.607+0000 7f25bf200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8102810 0x7f25b819e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.607+0000 7f25bf200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b819e760 con 0x7f25b8101e70 2024-03-30T00:57:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.607+0000 7f25bf200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b819e8d0 con 0x7f25b8100c80 2024-03-30T00:57:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.607+0000 7f25bf200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b819ea90 con 0x7f25b8102810 2024-03-30T00:57:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.608+0000 7f25b7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b8101e70 0x7f25b8197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.608+0000 7f25bd200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8102810 0x7f25b819e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.608+0000 7f25b7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b8101e70 0x7f25b8197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58554/0 (socket says 172.21.15.116:58554) 2024-03-30T00:57:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.608+0000 7f25b7e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25b8100c80 0x7f25b8196cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.608+0000 7f25b7400700 1 -- 172.21.15.116:0/3611438993 learned_addr learned my addr 172.21.15.116:0/3611438993 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.608+0000 7f25bd200700 1 -- 172.21.15.116:0/3611438993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25b8100c80 msgr2=0x7f25b8196cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.608+0000 7f25bd200700 1 --2- 172.21.15.116:0/3611438993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25b8100c80 0x7f25b8196cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.608+0000 7f25bd200700 1 -- 172.21.15.116:0/3611438993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b8101e70 msgr2=0x7f25b8197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.608+0000 7f25bd200700 1 --2- 172.21.15.116:0/3611438993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b8101e70 0x7f25b8197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.608+0000 7f25bd200700 1 -- 172.21.15.116:0/3611438993 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b0009c40 con 0x7f25b8102810 2024-03-30T00:57:46.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.608+0000 7f25b7400700 1 --2- 172.21.15.116:0/3611438993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b8101e70 0x7f25b8197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:46.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.608+0000 7f25bd200700 1 --2- 172.21.15.116:0/3611438993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8102810 0x7f25b819e190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f25b000efb0 tx=0x7f25b000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:46.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.608+0000 7f25b4c00700 1 -- 172.21.15.116:0/3611438993 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0003a00 con 0x7f25b8102810 2024-03-30T00:57:46.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.609+0000 7f25bf200700 1 -- 172.21.15.116:0/3611438993 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a80095f0 con 0x7f25b8102810 2024-03-30T00:57:46.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.609+0000 7f25b4c00700 1 -- 172.21.15.116:0/3611438993 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25b000f040 con 0x7f25b8102810 2024-03-30T00:57:46.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.609+0000 7f25b4c00700 1 -- 172.21.15.116:0/3611438993 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0004540 con 0x7f25b8102810 2024-03-30T00:57:46.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.609+0000 7f25bf200700 1 -- 172.21.15.116:0/3611438993 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b819f0c0 con 0x7f25b8102810 2024-03-30T00:57:46.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.610+0000 7f25bf200700 1 -- 172.21.15.116:0/3611438993 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b8108150 con 0x7f25b8102810 2024-03-30T00:57:46.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.611+0000 7f25b4c00700 1 -- 172.21.15.116:0/3611438993 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f25b0003bf0 con 0x7f25b8102810 2024-03-30T00:57:46.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.612+0000 7f25b4c00700 1 --2- 172.21.15.116:0/3611438993 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f259c0a1120 0x7f259c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:46.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.613+0000 7f25b4c00700 1 -- 172.21.15.116:0/3611438993 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f25b00cc870 con 0x7f25b8102810 2024-03-30T00:57:46.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.613+0000 7f25b7e00700 1 --2- 172.21.15.116:0/3611438993 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f259c0a1120 0x7f259c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:46.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.613+0000 7f25b7e00700 1 --2- 172.21.15.116:0/3611438993 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f259c0a1120 0x7f259c0a35e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f25a8005950 tx=0x7f25a80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:46.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.617+0000 7f25b4c00700 1 -- 172.21.15.116:0/3611438993 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f25b0095d20 con 0x7f25b8102810 2024-03-30T00:57:46.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:46 smithi116 ceph-mon[38086]: pgmap v288: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-30T00:57:46.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:46 smithi181 ceph-mon[47622]: pgmap v288: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-30T00:57:46.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.819+0000 7f25bf200700 1 -- 172.21.15.116:0/3611438993 --> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f25b802cd20 con 0x7f259c0a1120 2024-03-30T00:57:46.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.831+0000 7f25b4c00700 1 -- 172.21.15.116:0/3611438993 <== mgr.14152 v2:172.21.15.116:6800/3072243062 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427819 (secure 0 0 0) 0x7f25b802cd20 con 0x7f259c0a1120 2024-03-30T00:57:46.844 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.837+0000 7f25bf200700 1 -- 172.21.15.116:0/3611438993 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f259c0a1120 msgr2=0x7f259c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.837+0000 7f25bf200700 1 --2- 172.21.15.116:0/3611438993 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f259c0a1120 0x7f259c0a35e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f25a8005950 tx=0x7f25a80058e0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.837+0000 7f25bf200700 1 -- 172.21.15.116:0/3611438993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8102810 msgr2=0x7f25b819e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.837+0000 7f25bf200700 1 --2- 172.21.15.116:0/3611438993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8102810 0x7f25b819e190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f25b000efb0 tx=0x7f25b000c3a0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.837+0000 7f25bf200700 1 -- 172.21.15.116:0/3611438993 shutdown_connections 2024-03-30T00:57:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.837+0000 7f25bf200700 1 --2- 172.21.15.116:0/3611438993 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f259c0a1120 0x7f259c0a35e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.837+0000 7f25bf200700 1 --2- 172.21.15.116:0/3611438993 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8102810 0x7f25b819e190 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.837+0000 7f25bf200700 1 --2- 172.21.15.116:0/3611438993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25b8100c80 0x7f25b8196cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.837+0000 7f25bf200700 1 --2- 172.21.15.116:0/3611438993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b8101e70 0x7f25b8197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.837+0000 7f25bf200700 1 -- 172.21.15.116:0/3611438993 >> 172.21.15.116:0/3611438993 conn(0x7f25b80fc1f0 msgr2=0x7f25b8105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.837+0000 7f25bf200700 1 -- 172.21.15.116:0/3611438993 shutdown_connections 2024-03-30T00:57:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:46.837+0000 7f25bf200700 1 -- 172.21.15.116:0/3611438993 wait complete. 2024-03-30T00:57:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-30T00:57:46.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:46 smithi130 ceph-mon[47626]: pgmap v288: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-30T00:57:46.863 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-03-30T00:57:45.130320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662450,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":648,"num_read_kb":448,"num_write":351,"num_write_kb":3068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180887,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":788,"ondisk_log_size":788,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":351704,"kb_used_data":11128,"kb_used_omap":18,"kb_used_meta":340461,"kb_avail":1124590120,"statfs":{"total":1151940427776,"available":1151580282880,"internally_reserved":0,"allocated":11395072,"data_stored":7508472,"data_compressed":320673,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19128,"internal_metadata":348632392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6516,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":86,"num_read_kb":51,"num_write":12,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003660"},"pg_stats":[{"pgid":"3.3f","version":"83'8","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.167286+0000","last_change":"2024-03-30T00:55:51.809659+0000","last_active":"2024-03-30T00:57:35.167286+0000","last_peered":"2024-03-30T00:57:35.167286+0000","last_clean":"2024-03-30T00:57:35.167286+0000","last_became_active":"2024-03-30T00:55:51.809337+0000","last_became_peered":"2024-03-30T00:55:51.809337+0000","last_unstale":"2024-03-30T00:57:35.167286+0000","last_undegraded":"2024-03-30T00:57:35.167286+0000","last_fullsized":"2024-03-30T00:57:35.167286+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:20:50.640033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"83'4","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162969+0000","last_change":"2024-03-30T00:55:51.813778+0000","last_active":"2024-03-30T00:57:35.162969+0000","last_peered":"2024-03-30T00:57:35.162969+0000","last_clean":"2024-03-30T00:57:35.162969+0000","last_became_active":"2024-03-30T00:55:51.810347+0000","last_became_peered":"2024-03-30T00:55:51.810347+0000","last_unstale":"2024-03-30T00:57:35.162969+0000","last_undegraded":"2024-03-30T00:57:35.162969+0000","last_fullsized":"2024-03-30T00:57:35.162969+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:49:07.031477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166821+0000","last_change":"2024-03-30T00:55:51.817647+0000","last_active":"2024-03-30T00:57:35.166821+0000","last_peered":"2024-03-30T00:57:35.166821+0000","last_clean":"2024-03-30T00:57:35.166821+0000","last_became_active":"2024-03-30T00:55:51.816495+0000","last_became_peered":"2024-03-30T00:55:51.816495+0000","last_unstale":"2024-03-30T00:57:35.166821+0000","last_undegraded":"2024-03-30T00:57:35.166821+0000","last_fullsized":"2024-03-30T00:57:35.166821+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:59:42.965346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"83'8","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153424+0000","last_change":"2024-03-30T00:55:51.817581+0000","last_active":"2024-03-30T00:57:35.153424+0000","last_peered":"2024-03-30T00:57:35.153424+0000","last_clean":"2024-03-30T00:57:35.153424+0000","last_became_active":"2024-03-30T00:55:51.816223+0000","last_became_peered":"2024-03-30T00:55:51.816223+0000","last_unstale":"2024-03-30T00:57:35.153424+0000","last_undegraded":"2024-03-30T00:57:35.153424+0000","last_fullsized":"2024-03-30T00:57:35.153424+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:48:32.203975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"83'4","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162997+0000","last_change":"2024-03-30T00:55:51.811707+0000","last_active":"2024-03-30T00:57:35.162997+0000","last_peered":"2024-03-30T00:57:35.162997+0000","last_clean":"2024-03-30T00:57:35.162997+0000","last_became_active":"2024-03-30T00:55:51.811366+0000","last_became_peered":"2024-03-30T00:55:51.811366+0000","last_unstale":"2024-03-30T00:57:35.162997+0000","last_undegraded":"2024-03-30T00:57:35.162997+0000","last_fullsized":"2024-03-30T00:57:35.162997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:20.438505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153163+0000","last_change":"2024-03-30T00:55:51.826849+0000","last_active":"2024-03-30T00:57:35.153163+0000","last_peered":"2024-03-30T00:57:35.153163+0000","last_clean":"2024-03-30T00:57:35.153163+0000","last_became_active":"2024-03-30T00:55:51.826520+0000","last_became_peered":"2024-03-30T00:55:51.826520+0000","last_unstale":"2024-03-30T00:57:35.153163+0000","last_undegraded":"2024-03-30T00:57:35.153163+0000","last_fullsized":"2024-03-30T00:57:35.153163+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:54:03.423725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.149197+0000","last_change":"2024-03-30T00:55:51.812880+0000","last_active":"2024-03-30T00:57:35.149197+0000","last_peered":"2024-03-30T00:57:35.149197+0000","last_clean":"2024-03-30T00:57:35.149197+0000","last_became_active":"2024-03-30T00:55:51.812572+0000","last_became_peered":"2024-03-30T00:55:51.812572+0000","last_unstale":"2024-03-30T00:57:35.149197+0000","last_undegraded":"2024-03-30T00:57:35.149197+0000","last_fullsized":"2024-03-30T00:57:35.149197+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:44:49.670641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163852+0000","last_change":"2024-03-30T00:55:51.815480+0000","last_active":"2024-03-30T00:57:35.163852+0000","last_peered":"2024-03-30T00:57:35.163852+0000","last_clean":"2024-03-30T00:57:35.163852+0000","last_became_active":"2024-03-30T00:55:51.814052+0000","last_became_peered":"2024-03-30T00:55:51.814052+0000","last_unstale":"2024-03-30T00:57:35.163852+0000","last_undegraded":"2024-03-30T00:57:35.163852+0000","last_fullsized":"2024-03-30T00:57:35.163852+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:27:49.930928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"83'11","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153313+0000","last_change":"2024-03-30T00:55:51.814127+0000","last_active":"2024-03-30T00:57:35.153313+0000","last_peered":"2024-03-30T00:57:35.153313+0000","last_clean":"2024-03-30T00:57:35.153313+0000","last_became_active":"2024-03-30T00:55:51.813820+0000","last_became_peered":"2024-03-30T00:55:51.813820+0000","last_unstale":"2024-03-30T00:57:35.153313+0000","last_undegraded":"2024-03-30T00:57:35.153313+0000","last_fullsized":"2024-03-30T00:57:35.153313+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:21:31.605935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"83'16","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.154813+0000","last_change":"2024-03-30T00:55:51.815198+0000","last_active":"2024-03-30T00:57:35.154813+0000","last_peered":"2024-03-30T00:57:35.154813+0000","last_clean":"2024-03-30T00:57:35.154813+0000","last_became_active":"2024-03-30T00:55:51.814901+0000","last_became_peered":"2024-03-30T00:55:51.814901+0000","last_unstale":"2024-03-30T00:57:35.154813+0000","last_undegraded":"2024-03-30T00:57:35.154813+0000","last_fullsized":"2024-03-30T00:57:35.154813+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:45:51.224346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.148870+0000","last_change":"2024-03-30T00:55:51.809884+0000","last_active":"2024-03-30T00:57:35.148870+0000","last_peered":"2024-03-30T00:57:35.148870+0000","last_clean":"2024-03-30T00:57:35.148870+0000","last_became_active":"2024-03-30T00:55:51.809576+0000","last_became_peered":"2024-03-30T00:55:51.809576+0000","last_unstale":"2024-03-30T00:57:35.148870+0000","last_undegraded":"2024-03-30T00:57:35.148870+0000","last_fullsized":"2024-03-30T00:57:35.148870+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:39:51.767744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153546+0000","last_change":"2024-03-30T00:55:51.817384+0000","last_active":"2024-03-30T00:57:35.153546+0000","last_peered":"2024-03-30T00:57:35.153546+0000","last_clean":"2024-03-30T00:57:35.153546+0000","last_became_active":"2024-03-30T00:55:51.816564+0000","last_became_peered":"2024-03-30T00:55:51.816564+0000","last_unstale":"2024-03-30T00:57:35.153546+0000","last_undegraded":"2024-03-30T00:57:35.153546+0000","last_fullsized":"2024-03-30T00:57:35.153546+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:47:07.195650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"83'10","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166481+0000","last_change":"2024-03-30T00:55:51.814310+0000","last_active":"2024-03-30T00:57:35.166481+0000","last_peered":"2024-03-30T00:57:35.166481+0000","last_clean":"2024-03-30T00:57:35.166481+0000","last_became_active":"2024-03-30T00:55:51.813968+0000","last_became_peered":"2024-03-30T00:55:51.813968+0000","last_unstale":"2024-03-30T00:57:35.166481+0000","last_undegraded":"2024-03-30T00:57:35.166481+0000","last_fullsized":"2024-03-30T00:57:35.166481+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:59:55.955502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.161903+0000","last_change":"2024-03-30T00:55:51.815655+0000","last_active":"2024-03-30T00:57:35.161903+0000","last_peered":"2024-03-30T00:57:35.161903+0000","last_clean":"2024-03-30T00:57:35.161903+0000","last_became_active":"2024-03-30T00:55:51.814099+0000","last_became_peered":"2024-03-30T00:55:51.814099+0000","last_unstale":"2024-03-30T00:57:35.161903+0000","last_undegraded":"2024-03-30T00:57:35.161903+0000","last_fullsized":"2024-03-30T00:57:35.161903+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:42.958153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.157146+0000","last_change":"2024-03-30T00:56:36.472491+0000","last_active":"2024-03-30T00:57:35.157146+0000","last_peered":"2024-03-30T00:57:35.157146+0000","last_clean":"2024-03-30T00:57:35.157146+0000","last_became_active":"2024-03-30T00:56:36.472037+0000","last_became_peered":"2024-03-30T00:56:36.472037+0000","last_unstale":"2024-03-30T00:57:35.157146+0000","last_undegraded":"2024-03-30T00:57:35.157146+0000","last_fullsized":"2024-03-30T00:57:35.157146+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:52:08.402775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.164993+0000","last_change":"2024-03-30T00:55:51.822278+0000","last_active":"2024-03-30T00:57:35.164993+0000","last_peered":"2024-03-30T00:57:35.164993+0000","last_clean":"2024-03-30T00:57:35.164993+0000","last_became_active":"2024-03-30T00:55:51.821996+0000","last_became_peered":"2024-03-30T00:55:51.821996+0000","last_unstale":"2024-03-30T00:57:35.164993+0000","last_undegraded":"2024-03-30T00:57:35.164993+0000","last_fullsized":"2024-03-30T00:57:35.164993+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:37:45.841853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"83'8","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162074+0000","last_change":"2024-03-30T00:55:51.809547+0000","last_active":"2024-03-30T00:57:35.162074+0000","last_peered":"2024-03-30T00:57:35.162074+0000","last_clean":"2024-03-30T00:57:35.162074+0000","last_became_active":"2024-03-30T00:55:51.809314+0000","last_became_peered":"2024-03-30T00:55:51.809314+0000","last_unstale":"2024-03-30T00:57:35.162074+0000","last_undegraded":"2024-03-30T00:57:35.162074+0000","last_fullsized":"2024-03-30T00:57:35.162074+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:39:31.661362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"83'10","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.154392+0000","last_change":"2024-03-30T00:55:51.810857+0000","last_active":"2024-03-30T00:57:35.154392+0000","last_peered":"2024-03-30T00:57:35.154392+0000","last_clean":"2024-03-30T00:57:35.154392+0000","last_became_active":"2024-03-30T00:55:51.809420+0000","last_became_peered":"2024-03-30T00:55:51.809420+0000","last_unstale":"2024-03-30T00:57:35.154392+0000","last_undegraded":"2024-03-30T00:57:35.154392+0000","last_fullsized":"2024-03-30T00:57:35.154392+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:36:48.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"83'7","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163497+0000","last_change":"2024-03-30T00:55:51.814693+0000","last_active":"2024-03-30T00:57:35.163497+0000","last_peered":"2024-03-30T00:57:35.163497+0000","last_clean":"2024-03-30T00:57:35.163497+0000","last_became_active":"2024-03-30T00:55:51.814359+0000","last_became_peered":"2024-03-30T00:55:51.814359+0000","last_unstale":"2024-03-30T00:57:35.163497+0000","last_undegraded":"2024-03-30T00:57:35.163497+0000","last_fullsized":"2024-03-30T00:57:35.163497+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:20:50.856313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470709+0000","last_change":"2024-03-30T00:55:55.839089+0000","last_active":"2024-03-30T00:56:35.470709+0000","last_peered":"2024-03-30T00:56:35.470709+0000","last_clean":"2024-03-30T00:56:35.470709+0000","last_became_active":"2024-03-30T00:55:55.838705+0000","last_became_peered":"2024-03-30T00:55:55.838705+0000","last_unstale":"2024-03-30T00:56:35.470709+0000","last_undegraded":"2024-03-30T00:56:35.470709+0000","last_fullsized":"2024-03-30T00:56:35.470709+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:33:56.010272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'9","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.487878+0000","last_change":"2024-03-30T00:55:51.298577+0000","last_active":"2024-03-30T00:56:35.487878+0000","last_peered":"2024-03-30T00:56:35.487878+0000","last_clean":"2024-03-30T00:56:35.487878+0000","last_became_active":"2024-03-30T00:55:51.298269+0000","last_became_peered":"2024-03-30T00:55:51.298269+0000","last_unstale":"2024-03-30T00:56:35.487878+0000","last_undegraded":"2024-03-30T00:56:35.487878+0000","last_fullsized":"2024-03-30T00:56:35.487878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:10:04.998621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.154700+0000","last_change":"2024-03-30T00:55:51.818517+0000","last_active":"2024-03-30T00:57:35.154700+0000","last_peered":"2024-03-30T00:57:35.154700+0000","last_clean":"2024-03-30T00:57:35.154700+0000","last_became_active":"2024-03-30T00:55:51.818174+0000","last_became_peered":"2024-03-30T00:55:51.818174+0000","last_unstale":"2024-03-30T00:57:35.154700+0000","last_undegraded":"2024-03-30T00:57:35.154700+0000","last_fullsized":"2024-03-30T00:57:35.154700+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:03:01.604135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.623912+0000","last_change":"2024-03-30T00:56:36.623912+0000","last_active":"2024-03-30T00:56:36.623912+0000","last_peered":"2024-03-30T00:56:36.623912+0000","last_clean":"2024-03-30T00:56:36.623912+0000","last_became_active":"2024-03-30T00:56:36.623266+0000","last_became_peered":"2024-03-30T00:56:36.623266+0000","last_unstale":"2024-03-30T00:56:36.623912+0000","last_undegraded":"2024-03-30T00:56:36.623912+0000","last_fullsized":"2024-03-30T00:56:36.623912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:46:47.166079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.471579+0000","last_change":"2024-03-30T00:55:51.312686+0000","last_active":"2024-03-30T00:56:35.471579+0000","last_peered":"2024-03-30T00:56:35.471579+0000","last_clean":"2024-03-30T00:56:35.471579+0000","last_became_active":"2024-03-30T00:55:51.312296+0000","last_became_peered":"2024-03-30T00:55:51.312296+0000","last_unstale":"2024-03-30T00:56:35.471579+0000","last_undegraded":"2024-03-30T00:56:35.471579+0000","last_fullsized":"2024-03-30T00:56:35.471579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:28:58.398577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165549+0000","last_change":"2024-03-30T00:55:51.812197+0000","last_active":"2024-03-30T00:57:35.165549+0000","last_peered":"2024-03-30T00:57:35.165549+0000","last_clean":"2024-03-30T00:57:35.165549+0000","last_became_active":"2024-03-30T00:55:51.811892+0000","last_became_peered":"2024-03-30T00:55:51.811892+0000","last_unstale":"2024-03-30T00:57:35.165549+0000","last_undegraded":"2024-03-30T00:57:35.165549+0000","last_fullsized":"2024-03-30T00:57:35.165549+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:06:53.459760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462318+0000","last_change":"2024-03-30T00:55:55.839015+0000","last_active":"2024-03-30T00:56:35.462318+0000","last_peered":"2024-03-30T00:56:35.462318+0000","last_clean":"2024-03-30T00:56:35.462318+0000","last_became_active":"2024-03-30T00:55:55.838608+0000","last_became_peered":"2024-03-30T00:55:55.838608+0000","last_unstale":"2024-03-30T00:56:35.462318+0000","last_undegraded":"2024-03-30T00:56:35.462318+0000","last_fullsized":"2024-03-30T00:56:35.462318+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:24:23.804754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'4","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463392+0000","last_change":"2024-03-30T00:55:51.304199+0000","last_active":"2024-03-30T00:56:35.463392+0000","last_peered":"2024-03-30T00:56:35.463392+0000","last_clean":"2024-03-30T00:56:35.463392+0000","last_became_active":"2024-03-30T00:55:51.303837+0000","last_became_peered":"2024-03-30T00:55:51.303837+0000","last_unstale":"2024-03-30T00:56:35.463392+0000","last_undegraded":"2024-03-30T00:56:35.463392+0000","last_fullsized":"2024-03-30T00:56:35.463392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:19.154957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.164480+0000","last_change":"2024-03-30T00:55:51.817984+0000","last_active":"2024-03-30T00:57:35.164480+0000","last_peered":"2024-03-30T00:57:35.164480+0000","last_clean":"2024-03-30T00:57:35.164480+0000","last_became_active":"2024-03-30T00:55:51.817767+0000","last_became_peered":"2024-03-30T00:55:51.817767+0000","last_unstale":"2024-03-30T00:57:35.164480+0000","last_undegraded":"2024-03-30T00:57:35.164480+0000","last_fullsized":"2024-03-30T00:57:35.164480+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:12:48.558024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.623581+0000","last_change":"2024-03-30T00:56:36.623581+0000","last_active":"2024-03-30T00:56:36.623581+0000","last_peered":"2024-03-30T00:56:36.623581+0000","last_clean":"2024-03-30T00:56:36.623581+0000","last_became_active":"2024-03-30T00:56:36.623168+0000","last_became_peered":"2024-03-30T00:56:36.623168+0000","last_unstale":"2024-03-30T00:56:36.623581+0000","last_undegraded":"2024-03-30T00:56:36.623581+0000","last_fullsized":"2024-03-30T00:56:36.623581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:41.448935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"76'2","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.480921+0000","last_change":"2024-03-30T00:56:36.480478+0000","last_active":"2024-03-30T00:56:36.480921+0000","last_peered":"2024-03-30T00:56:36.480921+0000","last_clean":"2024-03-30T00:56:36.480921+0000","last_became_active":"2024-03-30T00:56:36.473297+0000","last_became_peered":"2024-03-30T00:56:36.473297+0000","last_unstale":"2024-03-30T00:56:36.480921+0000","last_undegraded":"2024-03-30T00:56:36.480921+0000","last_fullsized":"2024-03-30T00:56:36.480921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:37:08.090281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"83'15","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.151958+0000","last_change":"2024-03-30T00:55:51.827985+0000","last_active":"2024-03-30T00:57:35.151958+0000","last_peered":"2024-03-30T00:57:35.151958+0000","last_clean":"2024-03-30T00:57:35.151958+0000","last_became_active":"2024-03-30T00:55:51.827678+0000","last_became_peered":"2024-03-30T00:55:51.827678+0000","last_unstale":"2024-03-30T00:57:35.151958+0000","last_undegraded":"2024-03-30T00:57:35.151958+0000","last_fullsized":"2024-03-30T00:57:35.151958+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:47.088009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470870+0000","last_change":"2024-03-30T00:55:55.827267+0000","last_active":"2024-03-30T00:56:35.470870+0000","last_peered":"2024-03-30T00:56:35.470870+0000","last_clean":"2024-03-30T00:56:35.470870+0000","last_became_active":"2024-03-30T00:55:55.826821+0000","last_became_peered":"2024-03-30T00:55:55.826821+0000","last_unstale":"2024-03-30T00:56:35.470870+0000","last_undegraded":"2024-03-30T00:56:35.470870+0000","last_fullsized":"2024-03-30T00:56:35.470870+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:15:49.084359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"83'7","reported_seq":106,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:38.260458+0000","last_change":"2024-03-30T00:55:51.305177+0000","last_active":"2024-03-30T00:57:38.260458+0000","last_peered":"2024-03-30T00:57:38.260458+0000","last_clean":"2024-03-30T00:57:38.260458+0000","last_became_active":"2024-03-30T00:55:51.304813+0000","last_became_peered":"2024-03-30T00:55:51.304813+0000","last_unstale":"2024-03-30T00:57:38.260458+0000","last_undegraded":"2024-03-30T00:57:38.260458+0000","last_fullsized":"2024-03-30T00:57:38.260458+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:05:20.578897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1108,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"83'7","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.140612+0000","last_change":"2024-03-30T00:55:51.812714+0000","last_active":"2024-03-30T00:57:35.140612+0000","last_peered":"2024-03-30T00:57:35.140612+0000","last_clean":"2024-03-30T00:57:35.140612+0000","last_became_active":"2024-03-30T00:55:51.812406+0000","last_became_peered":"2024-03-30T00:55:51.812406+0000","last_unstale":"2024-03-30T00:57:35.140612+0000","last_undegraded":"2024-03-30T00:57:35.140612+0000","last_fullsized":"2024-03-30T00:57:35.140612+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:47:01.691412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470799+0000","last_change":"2024-03-30T00:55:55.831994+0000","last_active":"2024-03-30T00:56:35.470799+0000","last_peered":"2024-03-30T00:56:35.470799+0000","last_clean":"2024-03-30T00:56:35.470799+0000","last_became_active":"2024-03-30T00:55:55.831537+0000","last_became_peered":"2024-03-30T00:55:55.831537+0000","last_unstale":"2024-03-30T00:56:35.470799+0000","last_undegraded":"2024-03-30T00:56:35.470799+0000","last_fullsized":"2024-03-30T00:56:35.470799+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:44:47.240981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467196+0000","last_change":"2024-03-30T00:55:51.303468+0000","last_active":"2024-03-30T00:56:35.467196+0000","last_peered":"2024-03-30T00:56:35.467196+0000","last_clean":"2024-03-30T00:56:35.467196+0000","last_became_active":"2024-03-30T00:55:51.303068+0000","last_became_peered":"2024-03-30T00:55:51.303068+0000","last_unstale":"2024-03-30T00:56:35.467196+0000","last_undegraded":"2024-03-30T00:56:35.467196+0000","last_fullsized":"2024-03-30T00:56:35.467196+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:45:25.278196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"83'8","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.159899+0000","last_change":"2024-03-30T00:55:51.825099+0000","last_active":"2024-03-30T00:57:35.159899+0000","last_peered":"2024-03-30T00:57:35.159899+0000","last_clean":"2024-03-30T00:57:35.159899+0000","last_became_active":"2024-03-30T00:55:51.824717+0000","last_became_peered":"2024-03-30T00:55:51.824717+0000","last_unstale":"2024-03-30T00:57:35.159899+0000","last_undegraded":"2024-03-30T00:57:35.159899+0000","last_fullsized":"2024-03-30T00:57:35.159899+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:49:31.542430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460499+0000","last_change":"2024-03-30T00:55:55.826106+0000","last_active":"2024-03-30T00:56:35.460499+0000","last_peered":"2024-03-30T00:56:35.460499+0000","last_clean":"2024-03-30T00:56:35.460499+0000","last_became_active":"2024-03-30T00:55:55.825597+0000","last_became_peered":"2024-03-30T00:55:55.825597+0000","last_unstale":"2024-03-30T00:56:35.460499+0000","last_undegraded":"2024-03-30T00:56:35.460499+0000","last_fullsized":"2024-03-30T00:56:35.460499+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:00:18.991732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.473981+0000","last_change":"2024-03-30T00:55:51.294090+0000","last_active":"2024-03-30T00:56:35.473981+0000","last_peered":"2024-03-30T00:56:35.473981+0000","last_clean":"2024-03-30T00:56:35.473981+0000","last_became_active":"2024-03-30T00:55:51.293668+0000","last_became_peered":"2024-03-30T00:55:51.293668+0000","last_unstale":"2024-03-30T00:56:35.473981+0000","last_undegraded":"2024-03-30T00:56:35.473981+0000","last_fullsized":"2024-03-30T00:56:35.473981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:33:36.661760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166304+0000","last_change":"2024-03-30T00:55:51.823428+0000","last_active":"2024-03-30T00:57:35.166304+0000","last_peered":"2024-03-30T00:57:35.166304+0000","last_clean":"2024-03-30T00:57:35.166304+0000","last_became_active":"2024-03-30T00:55:51.822222+0000","last_became_peered":"2024-03-30T00:55:51.822222+0000","last_unstale":"2024-03-30T00:57:35.166304+0000","last_undegraded":"2024-03-30T00:57:35.166304+0000","last_fullsized":"2024-03-30T00:57:35.166304+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:05:34.798940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.495876+0000","last_change":"2024-03-30T00:55:55.828115+0000","last_active":"2024-03-30T00:56:35.495876+0000","last_peered":"2024-03-30T00:56:35.495876+0000","last_clean":"2024-03-30T00:56:35.495876+0000","last_became_active":"2024-03-30T00:55:55.827681+0000","last_became_peered":"2024-03-30T00:55:55.827681+0000","last_unstale":"2024-03-30T00:56:35.495876+0000","last_undegraded":"2024-03-30T00:56:35.495876+0000","last_fullsized":"2024-03-30T00:56:35.495876+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:18:25.751350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'1","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466298+0000","last_change":"2024-03-30T00:55:51.305163+0000","last_active":"2024-03-30T00:56:35.466298+0000","last_peered":"2024-03-30T00:56:35.466298+0000","last_clean":"2024-03-30T00:56:35.466298+0000","last_became_active":"2024-03-30T00:55:51.304860+0000","last_became_peered":"2024-03-30T00:55:51.304860+0000","last_unstale":"2024-03-30T00:56:35.466298+0000","last_undegraded":"2024-03-30T00:56:35.466298+0000","last_fullsized":"2024-03-30T00:56:35.466298+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:44:41.850998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"83'11","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.159021+0000","last_change":"2024-03-30T00:55:51.819401+0000","last_active":"2024-03-30T00:57:35.159021+0000","last_peered":"2024-03-30T00:57:35.159021+0000","last_clean":"2024-03-30T00:57:35.159021+0000","last_became_active":"2024-03-30T00:55:51.819021+0000","last_became_peered":"2024-03-30T00:55:51.819021+0000","last_unstale":"2024-03-30T00:57:35.159021+0000","last_undegraded":"2024-03-30T00:57:35.159021+0000","last_fullsized":"2024-03-30T00:57:35.159021+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:01:45.195719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.469445+0000","last_change":"2024-03-30T00:55:55.836806+0000","last_active":"2024-03-30T00:56:35.469445+0000","last_peered":"2024-03-30T00:56:35.469445+0000","last_clean":"2024-03-30T00:56:35.469445+0000","last_became_active":"2024-03-30T00:55:55.836403+0000","last_became_peered":"2024-03-30T00:55:55.836403+0000","last_unstale":"2024-03-30T00:56:35.469445+0000","last_undegraded":"2024-03-30T00:56:35.469445+0000","last_fullsized":"2024-03-30T00:56:35.469445+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:17:13.913675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"83'4","reported_seq":104,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:43.267468+0000","last_change":"2024-03-30T00:55:51.300833+0000","last_active":"2024-03-30T00:57:43.267468+0000","last_peered":"2024-03-30T00:57:43.267468+0000","last_clean":"2024-03-30T00:57:43.267468+0000","last_became_active":"2024-03-30T00:55:51.300473+0000","last_became_peered":"2024-03-30T00:55:51.300473+0000","last_unstale":"2024-03-30T00:57:43.267468+0000","last_undegraded":"2024-03-30T00:57:43.267468+0000","last_fullsized":"2024-03-30T00:57:43.267468+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:00:39.211981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"83'10","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.155649+0000","last_change":"2024-03-30T00:55:51.810781+0000","last_active":"2024-03-30T00:57:35.155649+0000","last_peered":"2024-03-30T00:57:35.155649+0000","last_clean":"2024-03-30T00:57:35.155649+0000","last_became_active":"2024-03-30T00:55:51.810215+0000","last_became_peered":"2024-03-30T00:55:51.810215+0000","last_unstale":"2024-03-30T00:57:35.155649+0000","last_undegraded":"2024-03-30T00:57:35.155649+0000","last_fullsized":"2024-03-30T00:57:35.155649+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:53:50.283209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.484181+0000","last_change":"2024-03-30T00:55:55.834505+0000","last_active":"2024-03-30T00:56:35.484181+0000","last_peered":"2024-03-30T00:56:35.484181+0000","last_clean":"2024-03-30T00:56:35.484181+0000","last_became_active":"2024-03-30T00:55:55.829319+0000","last_became_peered":"2024-03-30T00:55:55.829319+0000","last_unstale":"2024-03-30T00:56:35.484181+0000","last_undegraded":"2024-03-30T00:56:35.484181+0000","last_fullsized":"2024-03-30T00:56:35.484181+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:51:37.595523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466067+0000","last_change":"2024-03-30T00:55:51.302070+0000","last_active":"2024-03-30T00:56:35.466067+0000","last_peered":"2024-03-30T00:56:35.466067+0000","last_clean":"2024-03-30T00:56:35.466067+0000","last_became_active":"2024-03-30T00:55:51.298940+0000","last_became_peered":"2024-03-30T00:55:51.298940+0000","last_unstale":"2024-03-30T00:56:35.466067+0000","last_undegraded":"2024-03-30T00:56:35.466067+0000","last_fullsized":"2024-03-30T00:56:35.466067+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:07:47.915506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"83'9","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166604+0000","last_change":"2024-03-30T00:55:51.819918+0000","last_active":"2024-03-30T00:57:35.166604+0000","last_peered":"2024-03-30T00:57:35.166604+0000","last_clean":"2024-03-30T00:57:35.166604+0000","last_became_active":"2024-03-30T00:55:51.819493+0000","last_became_peered":"2024-03-30T00:55:51.819493+0000","last_unstale":"2024-03-30T00:57:35.166604+0000","last_undegraded":"2024-03-30T00:57:35.166604+0000","last_fullsized":"2024-03-30T00:57:35.166604+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:43:43.067031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462710+0000","last_change":"2024-03-30T00:55:55.827176+0000","last_active":"2024-03-30T00:56:35.462710+0000","last_peered":"2024-03-30T00:56:35.462710+0000","last_clean":"2024-03-30T00:56:35.462710+0000","last_became_active":"2024-03-30T00:55:55.826772+0000","last_became_peered":"2024-03-30T00:55:55.826772+0000","last_unstale":"2024-03-30T00:56:35.462710+0000","last_undegraded":"2024-03-30T00:56:35.462710+0000","last_fullsized":"2024-03-30T00:56:35.462710+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:02:04.512050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'4","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467237+0000","last_change":"2024-03-30T00:55:51.306688+0000","last_active":"2024-03-30T00:56:35.467237+0000","last_peered":"2024-03-30T00:56:35.467237+0000","last_clean":"2024-03-30T00:56:35.467237+0000","last_became_active":"2024-03-30T00:55:51.306333+0000","last_became_peered":"2024-03-30T00:55:51.306333+0000","last_unstale":"2024-03-30T00:56:35.467237+0000","last_undegraded":"2024-03-30T00:56:35.467237+0000","last_fullsized":"2024-03-30T00:56:35.467237+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:21:13.719947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"83'11","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.152507+0000","last_change":"2024-03-30T00:55:51.824879+0000","last_active":"2024-03-30T00:57:35.152507+0000","last_peered":"2024-03-30T00:57:35.152507+0000","last_clean":"2024-03-30T00:57:35.152507+0000","last_became_active":"2024-03-30T00:55:51.824566+0000","last_became_peered":"2024-03-30T00:55:51.824566+0000","last_unstale":"2024-03-30T00:57:35.152507+0000","last_undegraded":"2024-03-30T00:57:35.152507+0000","last_fullsized":"2024-03-30T00:57:35.152507+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:04:28.505693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.484132+0000","last_change":"2024-03-30T00:55:55.826387+0000","last_active":"2024-03-30T00:56:35.484132+0000","last_peered":"2024-03-30T00:56:35.484132+0000","last_clean":"2024-03-30T00:56:35.484132+0000","last_became_active":"2024-03-30T00:55:55.825941+0000","last_became_peered":"2024-03-30T00:55:55.825941+0000","last_unstale":"2024-03-30T00:56:35.484132+0000","last_undegraded":"2024-03-30T00:56:35.484132+0000","last_fullsized":"2024-03-30T00:56:35.484132+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:34:43.432712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"83'38","reported_seq":232,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:44.268981+0000","last_change":"2024-03-30T00:55:51.304592+0000","last_active":"2024-03-30T00:57:44.268981+0000","last_peered":"2024-03-30T00:57:44.268981+0000","last_clean":"2024-03-30T00:57:44.268981+0000","last_became_active":"2024-03-30T00:55:51.303931+0000","last_became_peered":"2024-03-30T00:55:51.303931+0000","last_unstale":"2024-03-30T00:57:44.268981+0000","last_undegraded":"2024-03-30T00:57:44.268981+0000","last_fullsized":"2024-03-30T00:57:44.268981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:53:40.976289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66960,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":177,"num_read_kb":154,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.148546+0000","last_change":"2024-03-30T00:55:51.815196+0000","last_active":"2024-03-30T00:57:35.148546+0000","last_peered":"2024-03-30T00:57:35.148546+0000","last_clean":"2024-03-30T00:57:35.148546+0000","last_became_active":"2024-03-30T00:55:51.813376+0000","last_became_peered":"2024-03-30T00:55:51.813376+0000","last_unstale":"2024-03-30T00:57:35.148546+0000","last_undegraded":"2024-03-30T00:57:35.148546+0000","last_fullsized":"2024-03-30T00:57:35.148546+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:40:26.350248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467447+0000","last_change":"2024-03-30T00:55:55.836906+0000","last_active":"2024-03-30T00:56:35.467447+0000","last_peered":"2024-03-30T00:56:35.467447+0000","last_clean":"2024-03-30T00:56:35.467447+0000","last_became_active":"2024-03-30T00:55:55.836539+0000","last_became_peered":"2024-03-30T00:55:55.836539+0000","last_unstale":"2024-03-30T00:56:35.467447+0000","last_undegraded":"2024-03-30T00:56:35.467447+0000","last_fullsized":"2024-03-30T00:56:35.467447+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:21:39.320192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'6","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.478499+0000","last_change":"2024-03-30T00:56:36.478132+0000","last_active":"2024-03-30T00:56:36.478499+0000","last_peered":"2024-03-30T00:56:36.478499+0000","last_clean":"2024-03-30T00:56:36.478499+0000","last_became_active":"2024-03-30T00:56:36.467449+0000","last_became_peered":"2024-03-30T00:56:36.467449+0000","last_unstale":"2024-03-30T00:56:36.478499+0000","last_undegraded":"2024-03-30T00:56:36.478499+0000","last_fullsized":"2024-03-30T00:56:36.478499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:27:51.986112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165118+0000","last_change":"2024-03-30T00:55:51.819825+0000","last_active":"2024-03-30T00:57:35.165118+0000","last_peered":"2024-03-30T00:57:35.165118+0000","last_clean":"2024-03-30T00:57:35.165118+0000","last_became_active":"2024-03-30T00:55:51.819481+0000","last_became_peered":"2024-03-30T00:55:51.819481+0000","last_unstale":"2024-03-30T00:57:35.165118+0000","last_undegraded":"2024-03-30T00:57:35.165118+0000","last_fullsized":"2024-03-30T00:57:35.165118+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:10:17.138541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460246+0000","last_change":"2024-03-30T00:55:55.832675+0000","last_active":"2024-03-30T00:56:35.460246+0000","last_peered":"2024-03-30T00:56:35.460246+0000","last_clean":"2024-03-30T00:56:35.460246+0000","last_became_active":"2024-03-30T00:55:55.832325+0000","last_became_peered":"2024-03-30T00:55:55.832325+0000","last_unstale":"2024-03-30T00:56:35.460246+0000","last_undegraded":"2024-03-30T00:56:35.460246+0000","last_fullsized":"2024-03-30T00:56:35.460246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:38:46.765473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'3","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467208+0000","last_change":"2024-03-30T00:55:51.313700+0000","last_active":"2024-03-30T00:56:35.467208+0000","last_peered":"2024-03-30T00:56:35.467208+0000","last_clean":"2024-03-30T00:56:35.467208+0000","last_became_active":"2024-03-30T00:55:51.313355+0000","last_became_peered":"2024-03-30T00:55:51.313355+0000","last_unstale":"2024-03-30T00:56:35.467208+0000","last_undegraded":"2024-03-30T00:56:35.467208+0000","last_fullsized":"2024-03-30T00:56:35.467208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:43:04.461398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"83'15","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163894+0000","last_change":"2024-03-30T00:55:51.819939+0000","last_active":"2024-03-30T00:57:35.163894+0000","last_peered":"2024-03-30T00:57:35.163894+0000","last_clean":"2024-03-30T00:57:35.163894+0000","last_became_active":"2024-03-30T00:55:51.819614+0000","last_became_peered":"2024-03-30T00:55:51.819614+0000","last_unstale":"2024-03-30T00:57:35.163894+0000","last_undegraded":"2024-03-30T00:57:35.163894+0000","last_fullsized":"2024-03-30T00:57:35.163894+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:20:37.362779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.469703+0000","last_change":"2024-03-30T00:56:36.469703+0000","last_active":"2024-03-30T00:56:36.469703+0000","last_peered":"2024-03-30T00:56:36.469703+0000","last_clean":"2024-03-30T00:56:36.469703+0000","last_became_active":"2024-03-30T00:56:36.469195+0000","last_became_peered":"2024-03-30T00:56:36.469195+0000","last_unstale":"2024-03-30T00:56:36.469703+0000","last_undegraded":"2024-03-30T00:56:36.469703+0000","last_fullsized":"2024-03-30T00:56:36.469703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:10:10.421447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"83'6","reported_seq":110,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:42.266577+0000","last_change":"2024-03-30T00:55:51.299246+0000","last_active":"2024-03-30T00:57:42.266577+0000","last_peered":"2024-03-30T00:57:42.266577+0000","last_clean":"2024-03-30T00:57:42.266577+0000","last_became_active":"2024-03-30T00:55:51.298924+0000","last_became_peered":"2024-03-30T00:55:51.298924+0000","last_unstale":"2024-03-30T00:57:42.266577+0000","last_undegraded":"2024-03-30T00:57:42.266577+0000","last_fullsized":"2024-03-30T00:57:42.266577+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:50:05.897612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1380,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.149967+0000","last_change":"2024-03-30T00:55:51.813804+0000","last_active":"2024-03-30T00:57:35.149967+0000","last_peered":"2024-03-30T00:57:35.149967+0000","last_clean":"2024-03-30T00:57:35.149967+0000","last_became_active":"2024-03-30T00:55:51.813500+0000","last_became_peered":"2024-03-30T00:55:51.813500+0000","last_unstale":"2024-03-30T00:57:35.149967+0000","last_undegraded":"2024-03-30T00:57:35.149967+0000","last_fullsized":"2024-03-30T00:57:35.149967+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:42:21.614900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.477003+0000","last_change":"2024-03-30T00:55:55.837737+0000","last_active":"2024-03-30T00:56:35.477003+0000","last_peered":"2024-03-30T00:56:35.477003+0000","last_clean":"2024-03-30T00:56:35.477003+0000","last_became_active":"2024-03-30T00:55:55.837312+0000","last_became_peered":"2024-03-30T00:55:55.837312+0000","last_unstale":"2024-03-30T00:56:35.477003+0000","last_undegraded":"2024-03-30T00:56:35.477003+0000","last_fullsized":"2024-03-30T00:56:35.477003+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:11:46.139979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'15","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462932+0000","last_change":"2024-03-30T00:55:51.306398+0000","last_active":"2024-03-30T00:56:35.462932+0000","last_peered":"2024-03-30T00:56:35.462932+0000","last_clean":"2024-03-30T00:56:35.462932+0000","last_became_active":"2024-03-30T00:55:51.305878+0000","last_became_peered":"2024-03-30T00:55:51.305878+0000","last_unstale":"2024-03-30T00:56:35.462932+0000","last_undegraded":"2024-03-30T00:56:35.462932+0000","last_fullsized":"2024-03-30T00:56:35.462932+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:20:32.742423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.151564+0000","last_change":"2024-03-30T00:55:51.824294+0000","last_active":"2024-03-30T00:57:35.151564+0000","last_peered":"2024-03-30T00:57:35.151564+0000","last_clean":"2024-03-30T00:57:35.151564+0000","last_became_active":"2024-03-30T00:55:51.823957+0000","last_became_peered":"2024-03-30T00:55:51.823957+0000","last_unstale":"2024-03-30T00:57:35.151564+0000","last_undegraded":"2024-03-30T00:57:35.151564+0000","last_fullsized":"2024-03-30T00:57:35.151564+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:23:16.873140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466253+0000","last_change":"2024-03-30T00:55:55.837023+0000","last_active":"2024-03-30T00:56:35.466253+0000","last_peered":"2024-03-30T00:56:35.466253+0000","last_clean":"2024-03-30T00:56:35.466253+0000","last_became_active":"2024-03-30T00:55:55.836541+0000","last_became_peered":"2024-03-30T00:55:55.836541+0000","last_unstale":"2024-03-30T00:56:35.466253+0000","last_undegraded":"2024-03-30T00:56:35.466253+0000","last_fullsized":"2024-03-30T00:56:35.466253+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:58:40.323953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463352+0000","last_change":"2024-03-30T00:55:51.300883+0000","last_active":"2024-03-30T00:56:35.463352+0000","last_peered":"2024-03-30T00:56:35.463352+0000","last_clean":"2024-03-30T00:56:35.463352+0000","last_became_active":"2024-03-30T00:55:51.300531+0000","last_became_peered":"2024-03-30T00:55:51.300531+0000","last_unstale":"2024-03-30T00:56:35.463352+0000","last_undegraded":"2024-03-30T00:56:35.463352+0000","last_fullsized":"2024-03-30T00:56:35.463352+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:38:38.874075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"83'5","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.151560+0000","last_change":"2024-03-30T00:55:51.822970+0000","last_active":"2024-03-30T00:57:35.151560+0000","last_peered":"2024-03-30T00:57:35.151560+0000","last_clean":"2024-03-30T00:57:35.151560+0000","last_became_active":"2024-03-30T00:55:51.822653+0000","last_became_peered":"2024-03-30T00:55:51.822653+0000","last_unstale":"2024-03-30T00:57:35.151560+0000","last_undegraded":"2024-03-30T00:57:35.151560+0000","last_fullsized":"2024-03-30T00:57:35.151560+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:59:26.306672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463256+0000","last_change":"2024-03-30T00:55:55.856605+0000","last_active":"2024-03-30T00:56:35.463256+0000","last_peered":"2024-03-30T00:56:35.463256+0000","last_clean":"2024-03-30T00:56:35.463256+0000","last_became_active":"2024-03-30T00:55:55.856061+0000","last_became_peered":"2024-03-30T00:55:55.856061+0000","last_unstale":"2024-03-30T00:56:35.463256+0000","last_undegraded":"2024-03-30T00:56:35.463256+0000","last_fullsized":"2024-03-30T00:56:35.463256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:58:41.010206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.488831+0000","last_change":"2024-03-30T00:55:51.301961+0000","last_active":"2024-03-30T00:56:35.488831+0000","last_peered":"2024-03-30T00:56:35.488831+0000","last_clean":"2024-03-30T00:56:35.488831+0000","last_became_active":"2024-03-30T00:55:51.301618+0000","last_became_peered":"2024-03-30T00:55:51.301618+0000","last_unstale":"2024-03-30T00:56:35.488831+0000","last_undegraded":"2024-03-30T00:56:35.488831+0000","last_fullsized":"2024-03-30T00:56:35.488831+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:38:46.681490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"83'4","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162610+0000","last_change":"2024-03-30T00:55:51.821331+0000","last_active":"2024-03-30T00:57:35.162610+0000","last_peered":"2024-03-30T00:57:35.162610+0000","last_clean":"2024-03-30T00:57:35.162610+0000","last_became_active":"2024-03-30T00:55:51.820988+0000","last_became_peered":"2024-03-30T00:55:51.820988+0000","last_unstale":"2024-03-30T00:57:35.162610+0000","last_undegraded":"2024-03-30T00:57:35.162610+0000","last_fullsized":"2024-03-30T00:57:35.162610+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:25:20.150789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.475451+0000","last_change":"2024-03-30T00:55:55.826251+0000","last_active":"2024-03-30T00:56:35.475451+0000","last_peered":"2024-03-30T00:56:35.475451+0000","last_clean":"2024-03-30T00:56:35.475451+0000","last_became_active":"2024-03-30T00:55:55.822695+0000","last_became_peered":"2024-03-30T00:55:55.822695+0000","last_unstale":"2024-03-30T00:56:35.475451+0000","last_undegraded":"2024-03-30T00:56:35.475451+0000","last_fullsized":"2024-03-30T00:56:35.475451+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:32:19.011454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466393+0000","last_change":"2024-03-30T00:55:51.309412+0000","last_active":"2024-03-30T00:56:35.466393+0000","last_peered":"2024-03-30T00:56:35.466393+0000","last_clean":"2024-03-30T00:56:35.466393+0000","last_became_active":"2024-03-30T00:55:51.309028+0000","last_became_peered":"2024-03-30T00:55:51.309028+0000","last_unstale":"2024-03-30T00:56:35.466393+0000","last_undegraded":"2024-03-30T00:56:35.466393+0000","last_fullsized":"2024-03-30T00:56:35.466393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:00:49.022772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.152521+0000","last_change":"2024-03-30T00:56:36.467986+0000","last_active":"2024-03-30T00:57:35.152521+0000","last_peered":"2024-03-30T00:57:35.152521+0000","last_clean":"2024-03-30T00:57:35.152521+0000","last_became_active":"2024-03-30T00:56:36.467601+0000","last_became_peered":"2024-03-30T00:56:36.467601+0000","last_unstale":"2024-03-30T00:57:35.152521+0000","last_undegraded":"2024-03-30T00:57:35.152521+0000","last_fullsized":"2024-03-30T00:57:35.152521+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:37:32.979094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.479139+0000","last_change":"2024-03-30T00:55:55.856377+0000","last_active":"2024-03-30T00:56:35.479139+0000","last_peered":"2024-03-30T00:56:35.479139+0000","last_clean":"2024-03-30T00:56:35.479139+0000","last_became_active":"2024-03-30T00:55:55.855986+0000","last_became_peered":"2024-03-30T00:55:55.855986+0000","last_unstale":"2024-03-30T00:56:35.479139+0000","last_undegraded":"2024-03-30T00:56:35.479139+0000","last_fullsized":"2024-03-30T00:56:35.479139+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:45:47.303446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"83'8","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.147047+0000","last_change":"2024-03-30T00:56:36.464863+0000","last_active":"2024-03-30T00:57:35.147047+0000","last_peered":"2024-03-30T00:57:35.147047+0000","last_clean":"2024-03-30T00:57:35.147047+0000","last_became_active":"2024-03-30T00:56:36.464323+0000","last_became_peered":"2024-03-30T00:56:36.464323+0000","last_unstale":"2024-03-30T00:57:35.147047+0000","last_undegraded":"2024-03-30T00:57:35.147047+0000","last_fullsized":"2024-03-30T00:57:35.147047+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:43:10.899726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"83'5","reported_seq":102,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:39.261685+0000","last_change":"2024-03-30T00:55:51.314834+0000","last_active":"2024-03-30T00:57:39.261685+0000","last_peered":"2024-03-30T00:57:39.261685+0000","last_clean":"2024-03-30T00:57:39.261685+0000","last_became_active":"2024-03-30T00:55:51.314429+0000","last_became_peered":"2024-03-30T00:55:51.314429+0000","last_unstale":"2024-03-30T00:57:39.261685+0000","last_undegraded":"2024-03-30T00:57:39.261685+0000","last_fullsized":"2024-03-30T00:57:39.261685+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:17:13.987180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":79,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466483+0000","last_change":"2024-03-30T00:55:55.840721+0000","last_active":"2024-03-30T00:56:35.466483+0000","last_peered":"2024-03-30T00:56:35.466483+0000","last_clean":"2024-03-30T00:56:35.466483+0000","last_became_active":"2024-03-30T00:55:55.840269+0000","last_became_peered":"2024-03-30T00:55:55.840269+0000","last_unstale":"2024-03-30T00:56:35.466483+0000","last_undegraded":"2024-03-30T00:56:35.466483+0000","last_fullsized":"2024-03-30T00:56:35.466483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:28:55.366411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.160257+0000","last_change":"2024-03-30T00:55:51.812609+0000","last_active":"2024-03-30T00:57:35.160257+0000","last_peered":"2024-03-30T00:57:35.160257+0000","last_clean":"2024-03-30T00:57:35.160257+0000","last_became_active":"2024-03-30T00:55:51.812325+0000","last_became_peered":"2024-03-30T00:55:51.812325+0000","last_unstale":"2024-03-30T00:57:35.160257+0000","last_undegraded":"2024-03-30T00:57:35.160257+0000","last_fullsized":"2024-03-30T00:57:35.160257+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:22:22.074405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'3","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463363+0000","last_change":"2024-03-30T00:55:51.296173+0000","last_active":"2024-03-30T00:56:35.463363+0000","last_peered":"2024-03-30T00:56:35.463363+0000","last_clean":"2024-03-30T00:56:35.463363+0000","last_became_active":"2024-03-30T00:55:51.295331+0000","last_became_peered":"2024-03-30T00:55:51.295331+0000","last_unstale":"2024-03-30T00:56:35.463363+0000","last_undegraded":"2024-03-30T00:56:35.463363+0000","last_fullsized":"2024-03-30T00:56:35.463363+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:51:58.343765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":326,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466805+0000","last_change":"2024-03-30T00:55:02.487656+0000","last_active":"2024-03-30T00:56:35.466805+0000","last_peered":"2024-03-30T00:56:35.466805+0000","last_clean":"2024-03-30T00:56:35.466805+0000","last_became_active":"2024-03-30T00:55:02.487263+0000","last_became_peered":"2024-03-30T00:55:02.487263+0000","last_unstale":"2024-03-30T00:56:35.466805+0000","last_undegraded":"2024-03-30T00:56:35.466805+0000","last_fullsized":"2024-03-30T00:56:35.466805+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:52:29.691461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:52:29.691461+0000","last_clean_scrub_stamp":"2024-03-30T00:52:29.691461+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:19:20.825038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466944+0000","last_change":"2024-03-30T00:55:55.856487+0000","last_active":"2024-03-30T00:56:35.466944+0000","last_peered":"2024-03-30T00:56:35.466944+0000","last_clean":"2024-03-30T00:56:35.466944+0000","last_became_active":"2024-03-30T00:55:55.856106+0000","last_became_peered":"2024-03-30T00:55:55.856106+0000","last_unstale":"2024-03-30T00:56:35.466944+0000","last_undegraded":"2024-03-30T00:56:35.466944+0000","last_fullsized":"2024-03-30T00:56:35.466944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:14.504218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.164966+0000","last_change":"2024-03-30T00:55:51.821519+0000","last_active":"2024-03-30T00:57:35.164966+0000","last_peered":"2024-03-30T00:57:35.164966+0000","last_clean":"2024-03-30T00:57:35.164966+0000","last_became_active":"2024-03-30T00:55:51.821095+0000","last_became_peered":"2024-03-30T00:55:51.821095+0000","last_unstale":"2024-03-30T00:57:35.164966+0000","last_undegraded":"2024-03-30T00:57:35.164966+0000","last_fullsized":"2024-03-30T00:57:35.164966+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:50:22.297450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'3","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466994+0000","last_change":"2024-03-30T00:55:51.303047+0000","last_active":"2024-03-30T00:56:35.466994+0000","last_peered":"2024-03-30T00:56:35.466994+0000","last_clean":"2024-03-30T00:56:35.466994+0000","last_became_active":"2024-03-30T00:55:51.302712+0000","last_became_peered":"2024-03-30T00:55:51.302712+0000","last_unstale":"2024-03-30T00:56:35.466994+0000","last_undegraded":"2024-03-30T00:56:35.466994+0000","last_fullsized":"2024-03-30T00:56:35.466994+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:07:59.451674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.460262+0000","last_change":"2024-03-30T00:55:55.823692+0000","last_active":"2024-03-30T00:56:35.460262+0000","last_peered":"2024-03-30T00:56:35.460262+0000","last_clean":"2024-03-30T00:56:35.460262+0000","last_became_active":"2024-03-30T00:55:55.823188+0000","last_became_peered":"2024-03-30T00:55:55.823188+0000","last_unstale":"2024-03-30T00:56:35.460262+0000","last_undegraded":"2024-03-30T00:56:35.460262+0000","last_fullsized":"2024-03-30T00:56:35.460262+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:32:32.089166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'1","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463201+0000","last_change":"2024-03-30T00:55:51.306935+0000","last_active":"2024-03-30T00:56:35.463201+0000","last_peered":"2024-03-30T00:56:35.463201+0000","last_clean":"2024-03-30T00:56:35.463201+0000","last_became_active":"2024-03-30T00:55:51.306595+0000","last_became_peered":"2024-03-30T00:55:51.306595+0000","last_unstale":"2024-03-30T00:56:35.463201+0000","last_undegraded":"2024-03-30T00:56:35.463201+0000","last_fullsized":"2024-03-30T00:56:35.463201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:50:30.396762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.166210+0000","last_change":"2024-03-30T00:55:51.807084+0000","last_active":"2024-03-30T00:57:35.166210+0000","last_peered":"2024-03-30T00:57:35.166210+0000","last_clean":"2024-03-30T00:57:35.166210+0000","last_became_active":"2024-03-30T00:55:51.806505+0000","last_became_peered":"2024-03-30T00:55:51.806505+0000","last_unstale":"2024-03-30T00:57:35.166210+0000","last_undegraded":"2024-03-30T00:57:35.166210+0000","last_fullsized":"2024-03-30T00:57:35.166210+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:38:44.759863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470455+0000","last_change":"2024-03-30T00:55:55.856457+0000","last_active":"2024-03-30T00:56:35.470455+0000","last_peered":"2024-03-30T00:56:35.470455+0000","last_clean":"2024-03-30T00:56:35.470455+0000","last_became_active":"2024-03-30T00:55:55.855982+0000","last_became_peered":"2024-03-30T00:55:55.855982+0000","last_unstale":"2024-03-30T00:56:35.470455+0000","last_undegraded":"2024-03-30T00:56:35.470455+0000","last_fullsized":"2024-03-30T00:56:35.470455+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:18:25.154634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"83'6","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.460763+0000","last_change":"2024-03-30T00:55:51.312581+0000","last_active":"2024-03-30T00:56:36.460763+0000","last_peered":"2024-03-30T00:56:36.460763+0000","last_clean":"2024-03-30T00:56:36.460763+0000","last_became_active":"2024-03-30T00:55:51.312225+0000","last_became_peered":"2024-03-30T00:55:51.312225+0000","last_unstale":"2024-03-30T00:56:36.460763+0000","last_undegraded":"2024-03-30T00:56:36.460763+0000","last_fullsized":"2024-03-30T00:56:36.460763+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:27:12.615250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.150369+0000","last_change":"2024-03-30T00:55:51.821527+0000","last_active":"2024-03-30T00:57:35.150369+0000","last_peered":"2024-03-30T00:57:35.150369+0000","last_clean":"2024-03-30T00:57:35.150369+0000","last_became_active":"2024-03-30T00:55:51.820689+0000","last_became_peered":"2024-03-30T00:55:51.820689+0000","last_unstale":"2024-03-30T00:57:35.150369+0000","last_undegraded":"2024-03-30T00:57:35.150369+0000","last_fullsized":"2024-03-30T00:57:35.150369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:14:35.735985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470468+0000","last_change":"2024-03-30T00:55:55.824307+0000","last_active":"2024-03-30T00:56:35.470468+0000","last_peered":"2024-03-30T00:56:35.470468+0000","last_clean":"2024-03-30T00:56:35.470468+0000","last_became_active":"2024-03-30T00:55:55.823900+0000","last_became_peered":"2024-03-30T00:55:55.823900+0000","last_unstale":"2024-03-30T00:56:35.470468+0000","last_undegraded":"2024-03-30T00:56:35.470468+0000","last_fullsized":"2024-03-30T00:56:35.470468+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:44:22.139120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'4","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.463230+0000","last_change":"2024-03-30T00:55:51.296965+0000","last_active":"2024-03-30T00:56:35.463230+0000","last_peered":"2024-03-30T00:56:35.463230+0000","last_clean":"2024-03-30T00:56:35.463230+0000","last_became_active":"2024-03-30T00:55:51.296645+0000","last_became_peered":"2024-03-30T00:55:51.296645+0000","last_unstale":"2024-03-30T00:56:35.463230+0000","last_undegraded":"2024-03-30T00:56:35.463230+0000","last_fullsized":"2024-03-30T00:56:35.463230+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:53:46.519972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163410+0000","last_change":"2024-03-30T00:55:51.826282+0000","last_active":"2024-03-30T00:57:35.163410+0000","last_peered":"2024-03-30T00:57:35.163410+0000","last_clean":"2024-03-30T00:57:35.163410+0000","last_became_active":"2024-03-30T00:55:51.823910+0000","last_became_peered":"2024-03-30T00:55:51.823910+0000","last_unstale":"2024-03-30T00:57:35.163410+0000","last_undegraded":"2024-03-30T00:57:35.163410+0000","last_fullsized":"2024-03-30T00:57:35.163410+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:47.454280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462851+0000","last_change":"2024-03-30T00:55:55.856750+0000","last_active":"2024-03-30T00:56:35.462851+0000","last_peered":"2024-03-30T00:56:35.462851+0000","last_clean":"2024-03-30T00:56:35.462851+0000","last_became_active":"2024-03-30T00:55:55.856288+0000","last_became_peered":"2024-03-30T00:55:55.856288+0000","last_unstale":"2024-03-30T00:56:35.462851+0000","last_undegraded":"2024-03-30T00:56:35.462851+0000","last_fullsized":"2024-03-30T00:56:35.462851+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:43:28.085910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467887+0000","last_change":"2024-03-30T00:55:51.306357+0000","last_active":"2024-03-30T00:56:35.467887+0000","last_peered":"2024-03-30T00:56:35.467887+0000","last_clean":"2024-03-30T00:56:35.467887+0000","last_became_active":"2024-03-30T00:55:51.305815+0000","last_became_peered":"2024-03-30T00:55:51.305815+0000","last_unstale":"2024-03-30T00:56:35.467887+0000","last_undegraded":"2024-03-30T00:56:35.467887+0000","last_fullsized":"2024-03-30T00:56:35.467887+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T03:16:21.753446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"83'2","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.117501+0000","last_change":"2024-03-30T00:55:51.810686+0000","last_active":"2024-03-30T00:57:35.117501+0000","last_peered":"2024-03-30T00:57:35.117501+0000","last_clean":"2024-03-30T00:57:35.117501+0000","last_became_active":"2024-03-30T00:55:51.810331+0000","last_became_peered":"2024-03-30T00:55:51.810331+0000","last_unstale":"2024-03-30T00:57:35.117501+0000","last_undegraded":"2024-03-30T00:57:35.117501+0000","last_fullsized":"2024-03-30T00:57:35.117501+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:24:13.750905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.487787+0000","last_change":"2024-03-30T00:55:55.842894+0000","last_active":"2024-03-30T00:56:35.487787+0000","last_peered":"2024-03-30T00:56:35.487787+0000","last_clean":"2024-03-30T00:56:35.487787+0000","last_became_active":"2024-03-30T00:55:55.842118+0000","last_became_peered":"2024-03-30T00:55:55.842118+0000","last_unstale":"2024-03-30T00:56:35.487787+0000","last_undegraded":"2024-03-30T00:56:35.487787+0000","last_fullsized":"2024-03-30T00:56:35.487787+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:39.246228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'3","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470956+0000","last_change":"2024-03-30T00:55:51.313188+0000","last_active":"2024-03-30T00:56:35.470956+0000","last_peered":"2024-03-30T00:56:35.470956+0000","last_clean":"2024-03-30T00:56:35.470956+0000","last_became_active":"2024-03-30T00:55:51.312879+0000","last_became_peered":"2024-03-30T00:55:51.312879+0000","last_unstale":"2024-03-30T00:56:35.470956+0000","last_undegraded":"2024-03-30T00:56:35.470956+0000","last_fullsized":"2024-03-30T00:56:35.470956+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:41:23.420427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"83'9","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165088+0000","last_change":"2024-03-30T00:55:51.808998+0000","last_active":"2024-03-30T00:57:35.165088+0000","last_peered":"2024-03-30T00:57:35.165088+0000","last_clean":"2024-03-30T00:57:35.165088+0000","last_became_active":"2024-03-30T00:55:51.808636+0000","last_became_peered":"2024-03-30T00:55:51.808636+0000","last_unstale":"2024-03-30T00:57:35.165088+0000","last_undegraded":"2024-03-30T00:57:35.165088+0000","last_fullsized":"2024-03-30T00:57:35.165088+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:42:02.379294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:56:36.480854+0000","last_change":"2024-03-30T00:56:36.480854+0000","last_active":"2024-03-30T00:56:36.480854+0000","last_peered":"2024-03-30T00:56:36.480854+0000","last_clean":"2024-03-30T00:56:36.480854+0000","last_became_active":"2024-03-30T00:56:36.467064+0000","last_became_peered":"2024-03-30T00:56:36.467064+0000","last_unstale":"2024-03-30T00:56:36.480854+0000","last_undegraded":"2024-03-30T00:56:36.480854+0000","last_fullsized":"2024-03-30T00:56:36.480854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:12:55.154544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470669+0000","last_change":"2024-03-30T00:55:51.313708+0000","last_active":"2024-03-30T00:56:35.470669+0000","last_peered":"2024-03-30T00:56:35.470669+0000","last_clean":"2024-03-30T00:56:35.470669+0000","last_became_active":"2024-03-30T00:55:51.313362+0000","last_became_peered":"2024-03-30T00:55:51.313362+0000","last_unstale":"2024-03-30T00:56:35.470669+0000","last_undegraded":"2024-03-30T00:56:35.470669+0000","last_fullsized":"2024-03-30T00:56:35.470669+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:03:17.859640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165069+0000","last_change":"2024-03-30T00:55:51.819638+0000","last_active":"2024-03-30T00:57:35.165069+0000","last_peered":"2024-03-30T00:57:35.165069+0000","last_clean":"2024-03-30T00:57:35.165069+0000","last_became_active":"2024-03-30T00:55:51.819282+0000","last_became_peered":"2024-03-30T00:55:51.819282+0000","last_unstale":"2024-03-30T00:57:35.165069+0000","last_undegraded":"2024-03-30T00:57:35.165069+0000","last_fullsized":"2024-03-30T00:57:35.165069+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:13:39.249550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462909+0000","last_change":"2024-03-30T00:55:55.856498+0000","last_active":"2024-03-30T00:56:35.462909+0000","last_peered":"2024-03-30T00:56:35.462909+0000","last_clean":"2024-03-30T00:56:35.462909+0000","last_became_active":"2024-03-30T00:55:55.856015+0000","last_became_peered":"2024-03-30T00:55:55.856015+0000","last_unstale":"2024-03-30T00:56:35.462909+0000","last_undegraded":"2024-03-30T00:56:35.462909+0000","last_fullsized":"2024-03-30T00:56:35.462909+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:14:00.248812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.462909+0000","last_change":"2024-03-30T00:55:51.302995+0000","last_active":"2024-03-30T00:56:35.462909+0000","last_peered":"2024-03-30T00:56:35.462909+0000","last_clean":"2024-03-30T00:56:35.462909+0000","last_became_active":"2024-03-30T00:55:51.302650+0000","last_became_peered":"2024-03-30T00:55:51.302650+0000","last_unstale":"2024-03-30T00:56:35.462909+0000","last_undegraded":"2024-03-30T00:56:35.462909+0000","last_fullsized":"2024-03-30T00:56:35.462909+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:17:32.556550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"83'5","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.162368+0000","last_change":"2024-03-30T00:55:51.808969+0000","last_active":"2024-03-30T00:57:35.162368+0000","last_peered":"2024-03-30T00:57:35.162368+0000","last_clean":"2024-03-30T00:57:35.162368+0000","last_became_active":"2024-03-30T00:55:51.808633+0000","last_became_peered":"2024-03-30T00:55:51.808633+0000","last_unstale":"2024-03-30T00:57:35.162368+0000","last_undegraded":"2024-03-30T00:57:35.162368+0000","last_fullsized":"2024-03-30T00:57:35.162368+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:16:46.128066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.468267+0000","last_change":"2024-03-30T00:55:55.824964+0000","last_active":"2024-03-30T00:56:35.468267+0000","last_peered":"2024-03-30T00:56:35.468267+0000","last_clean":"2024-03-30T00:56:35.468267+0000","last_became_active":"2024-03-30T00:55:55.824559+0000","last_became_peered":"2024-03-30T00:55:55.824559+0000","last_unstale":"2024-03-30T00:56:35.468267+0000","last_undegraded":"2024-03-30T00:56:35.468267+0000","last_fullsized":"2024-03-30T00:56:35.468267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:19:24.746963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'3","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.470912+0000","last_change":"2024-03-30T00:55:51.301602+0000","last_active":"2024-03-30T00:56:35.470912+0000","last_peered":"2024-03-30T00:56:35.470912+0000","last_clean":"2024-03-30T00:56:35.470912+0000","last_became_active":"2024-03-30T00:55:51.301221+0000","last_became_peered":"2024-03-30T00:55:51.301221+0000","last_unstale":"2024-03-30T00:56:35.470912+0000","last_undegraded":"2024-03-30T00:56:35.470912+0000","last_fullsized":"2024-03-30T00:56:35.470912+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:37:36.727603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"83'7","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153553+0000","last_change":"2024-03-30T00:55:51.810197+0000","last_active":"2024-03-30T00:57:35.153553+0000","last_peered":"2024-03-30T00:57:35.153553+0000","last_clean":"2024-03-30T00:57:35.153553+0000","last_became_active":"2024-03-30T00:55:51.809193+0000","last_became_peered":"2024-03-30T00:55:51.809193+0000","last_unstale":"2024-03-30T00:57:35.153553+0000","last_undegraded":"2024-03-30T00:57:35.153553+0000","last_fullsized":"2024-03-30T00:57:35.153553+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:52:41.078425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.494873+0000","last_change":"2024-03-30T00:55:55.830549+0000","last_active":"2024-03-30T00:56:35.494873+0000","last_peered":"2024-03-30T00:56:35.494873+0000","last_clean":"2024-03-30T00:56:35.494873+0000","last_became_active":"2024-03-30T00:55:55.827966+0000","last_became_peered":"2024-03-30T00:55:55.827966+0000","last_unstale":"2024-03-30T00:56:35.494873+0000","last_undegraded":"2024-03-30T00:56:35.494873+0000","last_fullsized":"2024-03-30T00:56:35.494873+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:30:36.531960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"76'1","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.467092+0000","last_change":"2024-03-30T00:55:51.311998+0000","last_active":"2024-03-30T00:56:35.467092+0000","last_peered":"2024-03-30T00:56:35.467092+0000","last_clean":"2024-03-30T00:56:35.467092+0000","last_became_active":"2024-03-30T00:55:51.311627+0000","last_became_peered":"2024-03-30T00:55:51.311627+0000","last_unstale":"2024-03-30T00:56:35.467092+0000","last_undegraded":"2024-03-30T00:56:35.467092+0000","last_fullsized":"2024-03-30T00:56:35.467092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:23:06.939919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.158955+0000","last_change":"2024-03-30T00:55:51.819457+0000","last_active":"2024-03-30T00:57:35.158955+0000","last_peered":"2024-03-30T00:57:35.158955+0000","last_clean":"2024-03-30T00:57:35.158955+0000","last_became_active":"2024-03-30T00:55:51.819099+0000","last_became_peered":"2024-03-30T00:55:51.819099+0000","last_unstale":"2024-03-30T00:57:35.158955+0000","last_undegraded":"2024-03-30T00:57:35.158955+0000","last_fullsized":"2024-03-30T00:57:35.158955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:31:57.363328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.466097+0000","last_change":"2024-03-30T00:55:55.824882+0000","last_active":"2024-03-30T00:56:35.466097+0000","last_peered":"2024-03-30T00:56:35.466097+0000","last_clean":"2024-03-30T00:56:35.466097+0000","last_became_active":"2024-03-30T00:55:55.824493+0000","last_became_peered":"2024-03-30T00:55:55.824493+0000","last_unstale":"2024-03-30T00:56:35.466097+0000","last_undegraded":"2024-03-30T00:56:35.466097+0000","last_fullsized":"2024-03-30T00:56:35.466097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:54.799733+0000","last_clean_scrub_stamp":"2024-03-30T00:55:54.799733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:24:48.825776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"76'2","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-30T00:56:35.476919+0000","last_change":"2024-03-30T00:55:51.311787+0000","last_active":"2024-03-30T00:56:35.476919+0000","last_peered":"2024-03-30T00:56:35.476919+0000","last_clean":"2024-03-30T00:56:35.476919+0000","last_became_active":"2024-03-30T00:55:51.311402+0000","last_became_peered":"2024-03-30T00:55:51.311402+0000","last_unstale":"2024-03-30T00:56:35.476919+0000","last_undegraded":"2024-03-30T00:56:35.476919+0000","last_fullsized":"2024-03-30T00:56:35.476919+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:50.270689+0000","last_clean_scrub_stamp":"2024-03-30T00:55:50.270689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:37:13.960453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.151904+0000","last_change":"2024-03-30T00:55:51.809742+0000","last_active":"2024-03-30T00:57:35.151904+0000","last_peered":"2024-03-30T00:57:35.151904+0000","last_clean":"2024-03-30T00:57:35.151904+0000","last_became_active":"2024-03-30T00:55:51.809396+0000","last_became_peered":"2024-03-30T00:55:51.809396+0000","last_unstale":"2024-03-30T00:57:35.151904+0000","last_undegraded":"2024-03-30T00:57:35.151904+0000","last_fullsized":"2024-03-30T00:57:35.151904+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:22.535472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"83'9","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.145644+0000","last_change":"2024-03-30T00:55:51.814429+0000","last_active":"2024-03-30T00:57:35.145644+0000","last_peered":"2024-03-30T00:57:35.145644+0000","last_clean":"2024-03-30T00:57:35.145644+0000","last_became_active":"2024-03-30T00:55:51.814044+0000","last_became_peered":"2024-03-30T00:55:51.814044+0000","last_unstale":"2024-03-30T00:57:35.145644+0000","last_undegraded":"2024-03-30T00:57:35.145644+0000","last_fullsized":"2024-03-30T00:57:35.145644+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:24:58.830738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"83'12","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.146416+0000","last_change":"2024-03-30T00:55:51.828977+0000","last_active":"2024-03-30T00:57:35.146416+0000","last_peered":"2024-03-30T00:57:35.146416+0000","last_clean":"2024-03-30T00:57:35.146416+0000","last_became_active":"2024-03-30T00:55:51.828656+0000","last_became_peered":"2024-03-30T00:55:51.828656+0000","last_unstale":"2024-03-30T00:57:35.146416+0000","last_undegraded":"2024-03-30T00:57:35.146416+0000","last_fullsized":"2024-03-30T00:57:35.146416+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:10:23.463770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.161205+0000","last_change":"2024-03-30T00:55:51.821511+0000","last_active":"2024-03-30T00:57:35.161205+0000","last_peered":"2024-03-30T00:57:35.161205+0000","last_clean":"2024-03-30T00:57:35.161205+0000","last_became_active":"2024-03-30T00:55:51.820876+0000","last_became_peered":"2024-03-30T00:55:51.820876+0000","last_unstale":"2024-03-30T00:57:35.161205+0000","last_undegraded":"2024-03-30T00:57:35.161205+0000","last_fullsized":"2024-03-30T00:57:35.161205+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:14:48.299376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"83'7","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.167065+0000","last_change":"2024-03-30T00:55:51.823867+0000","last_active":"2024-03-30T00:57:35.167065+0000","last_peered":"2024-03-30T00:57:35.167065+0000","last_clean":"2024-03-30T00:57:35.167065+0000","last_became_active":"2024-03-30T00:55:51.820230+0000","last_became_peered":"2024-03-30T00:55:51.820230+0000","last_unstale":"2024-03-30T00:57:35.167065+0000","last_undegraded":"2024-03-30T00:57:35.167065+0000","last_fullsized":"2024-03-30T00:57:35.167065+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:38:51.257855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"83'4","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.165868+0000","last_change":"2024-03-30T00:55:51.821413+0000","last_active":"2024-03-30T00:57:35.165868+0000","last_peered":"2024-03-30T00:57:35.165868+0000","last_clean":"2024-03-30T00:57:35.165868+0000","last_became_active":"2024-03-30T00:55:51.820833+0000","last_became_peered":"2024-03-30T00:55:51.820833+0000","last_unstale":"2024-03-30T00:57:35.165868+0000","last_undegraded":"2024-03-30T00:57:35.165868+0000","last_fullsized":"2024-03-30T00:57:35.165868+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:34:11.969849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.149312+0000","last_change":"2024-03-30T00:55:51.813674+0000","last_active":"2024-03-30T00:57:35.149312+0000","last_peered":"2024-03-30T00:57:35.149312+0000","last_clean":"2024-03-30T00:57:35.149312+0000","last_became_active":"2024-03-30T00:55:51.813341+0000","last_became_peered":"2024-03-30T00:55:51.813341+0000","last_unstale":"2024-03-30T00:57:35.149312+0000","last_undegraded":"2024-03-30T00:57:35.149312+0000","last_fullsized":"2024-03-30T00:57:35.149312+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:54:17.285997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"83'15","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.148932+0000","last_change":"2024-03-30T00:55:51.812472+0000","last_active":"2024-03-30T00:57:35.148932+0000","last_peered":"2024-03-30T00:57:35.148932+0000","last_clean":"2024-03-30T00:57:35.148932+0000","last_became_active":"2024-03-30T00:55:51.812061+0000","last_became_peered":"2024-03-30T00:55:51.812061+0000","last_unstale":"2024-03-30T00:57:35.148932+0000","last_undegraded":"2024-03-30T00:57:35.148932+0000","last_fullsized":"2024-03-30T00:57:35.148932+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:46.550201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"83'2","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.134547+0000","last_change":"2024-03-30T00:55:51.829179+0000","last_active":"2024-03-30T00:57:35.134547+0000","last_peered":"2024-03-30T00:57:35.134547+0000","last_clean":"2024-03-30T00:57:35.134547+0000","last_became_active":"2024-03-30T00:55:51.828805+0000","last_became_peered":"2024-03-30T00:55:51.828805+0000","last_unstale":"2024-03-30T00:57:35.134547+0000","last_undegraded":"2024-03-30T00:57:35.134547+0000","last_fullsized":"2024-03-30T00:57:35.134547+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:23.576125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"83'6","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.155774+0000","last_change":"2024-03-30T00:55:51.823170+0000","last_active":"2024-03-30T00:57:35.155774+0000","last_peered":"2024-03-30T00:57:35.155774+0000","last_clean":"2024-03-30T00:57:35.155774+0000","last_became_active":"2024-03-30T00:55:51.822785+0000","last_became_peered":"2024-03-30T00:55:51.822785+0000","last_unstale":"2024-03-30T00:57:35.155774+0000","last_undegraded":"2024-03-30T00:57:35.155774+0000","last_fullsized":"2024-03-30T00:57:35.155774+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:34:43.393540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"83'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.153749+0000","last_change":"2024-03-30T00:55:51.817160+0000","last_active":"2024-03-30T00:57:35.153749+0000","last_peered":"2024-03-30T00:57:35.153749+0000","last_clean":"2024-03-30T00:57:35.153749+0000","last_became_active":"2024-03-30T00:55:51.816881+0000","last_became_peered":"2024-03-30T00:55:51.816881+0000","last_unstale":"2024-03-30T00:57:35.153749+0000","last_undegraded":"2024-03-30T00:57:35.153749+0000","last_fullsized":"2024-03-30T00:57:35.153749+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T02:09:05.211242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"83'10","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.163651+0000","last_change":"2024-03-30T00:55:51.817670+0000","last_active":"2024-03-30T00:57:35.163651+0000","last_peered":"2024-03-30T00:57:35.163651+0000","last_clean":"2024-03-30T00:57:35.163651+0000","last_became_active":"2024-03-30T00:55:51.817380+0000","last_became_peered":"2024-03-30T00:55:51.817380+0000","last_unstale":"2024-03-30T00:57:35.163651+0000","last_undegraded":"2024-03-30T00:57:35.163651+0000","last_fullsized":"2024-03-30T00:57:35.163651+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:40:24.582557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"83'11","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.167761+0000","last_change":"2024-03-30T00:55:51.813071+0000","last_active":"2024-03-30T00:57:35.167761+0000","last_peered":"2024-03-30T00:57:35.167761+0000","last_clean":"2024-03-30T00:57:35.167761+0000","last_became_active":"2024-03-30T00:55:51.811681+0000","last_became_peered":"2024-03-30T00:55:51.811681+0000","last_unstale":"2024-03-30T00:57:35.167761+0000","last_undegraded":"2024-03-30T00:57:35.167761+0000","last_fullsized":"2024-03-30T00:57:35.167761+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T01:10:53.840055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T00:57:35.154929+0000","last_change":"2024-03-30T00:56:36.472592+0000","last_active":"2024-03-30T00:57:35.154929+0000","last_peered":"2024-03-30T00:57:35.154929+0000","last_clean":"2024-03-30T00:57:35.154929+0000","last_became_active":"2024-03-30T00:56:36.472173+0000","last_became_peered":"2024-03-30T00:56:36.472173+0000","last_unstale":"2024-03-30T00:57:35.154929+0000","last_undegraded":"2024-03-30T00:57:35.154929+0000","last_fullsized":"2024-03-30T00:57:35.154929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T00:55:51.275085+0000","last_clean_scrub_stamp":"2024-03-30T00:55:51.275085+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:22:27.915984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":212960,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71848,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":522,"num_read_kb":339,"num_write":168,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":215544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645345,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":462418,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.41499999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.73099999999999998,"5min":1.399,"15min":1.399},"last":0.44},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.622,"15min":0.626},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.83399999999999996,"5min":1.282,"15min":1.282},"last":0.57799999999999996}]},{"osd":1,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.61199999999999999,"15min":0.60699999999999998},"min":{"1min":0.46800000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.45900000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.69599999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.61199999999999999}]},{"osd":2,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.46500000000000002,"15min":0.45400000000000001},"min":{"1min":0.36799999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.46200000000000002,"15min":0.45800000000000002},"min":{"1min":0.373,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.41399999999999998}]},{"osd":3,"last update":"Sat Mar 30 00:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.505}]},{"osd":4,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.62,"15min":0.61199999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64000000000000001,"15min":0.63800000000000001},"min":{"1min":0.49399999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.76100000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.72299999999999998}]},{"osd":5,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.58399999999999996,"15min":0.57499999999999996},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.876,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.47199999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.54400000000000004}]},{"osd":6,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.51700000000000002,"5min":0.436,"15min":0.436},"max":{"1min":1.0149999999999999,"5min":1.056,"15min":1.056},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72699999999999998,"15min":0.71799999999999997},"min":{"1min":0.622,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97499999999999998,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.76100000000000001}]},{"osd":7,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71099999999999997,"15min":0.70299999999999996},"min":{"1min":0.66000000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.748},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72499999999999998,"15min":0.71799999999999997},"min":{"1min":0.68000000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.96499999999999997,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.68500000000000005}]},{"osd":8,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.78700000000000003}]},{"osd":9,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.80800000000000005,"15min":0.79700000000000004},"min":{"1min":0.70099999999999996,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.052,"5min":1.353,"15min":1.353},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.75700000000000001,"15min":0.751},"min":{"1min":0.69399999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98599999999999999,"5min":1.083,"15min":1.083},"last":0.71299999999999997}]},{"osd":10,"last update":"Sat Mar 30 00:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77400000000000002,"15min":0.77000000000000002},"min":{"1min":0.72499999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.90200000000000002,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.83399999999999996,"15min":0.82899999999999996},"min":{"1min":0.749,"5min":0.623,"15min":0.623},"max":{"1min":1.0580000000000001,"5min":1.52,"15min":1.52},"last":0.82799999999999996}]}]},{"osd":10,"up_from":65,"seq":279172874276,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":1121169,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.58399999999999996}]},{"osd":1,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.439,"15min":0.441},"min":{"1min":0.29299999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.53900000000000003,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.33800000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56599999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.42099999999999999}]},{"osd":2,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.54700000000000004,"15min":0.54000000000000004},"min":{"1min":0.38200000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.626},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.58499999999999996},"min":{"1min":0.42199999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.75700000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.504}]},{"osd":3,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82699999999999996,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63200000000000001,"15min":0.63900000000000001},"min":{"1min":0.495,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.60199999999999998}]},{"osd":4,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35999999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.65200000000000002,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.45000000000000001}]},{"osd":5,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.61599999999999999,"15min":0.61299999999999999},"min":{"1min":0.45600000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.46800000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.52900000000000003}]},{"osd":6,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.876,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68799999999999994,"15min":0.68999999999999995},"min":{"1min":0.52300000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78500000000000003,"5min":1.131,"15min":1.131},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":8,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.69599999999999995,"15min":0.69799999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.80800000000000005,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.55400000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.83399999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.69199999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.75900000000000001,"15min":0.76500000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.85999999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.748,"15min":0.752},"min":{"1min":0.52400000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.85999999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.72299999999999998}]},{"osd":11,"last update":"Sat Mar 30 00:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73099999999999998,"15min":0.73299999999999998},"min":{"1min":0.60299999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.83099999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.749,"15min":0.749},"min":{"1min":0.59499999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.85699999999999998,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.81699999999999995}]}]},{"osd":9,"up_from":60,"seq":257698037798,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":462620,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.55300000000000005,"15min":0.56899999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.628,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.316}]},{"osd":1,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.94199999999999995,"5min":1.274,"15min":1.274},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.73399999999999999,"15min":0.74299999999999999},"min":{"1min":0.51600000000000001,"5min":0.504,"15min":0.504},"max":{"1min":0.90600000000000003,"5min":1.411,"15min":1.411},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.73399999999999999,"15min":0.74299999999999999},"min":{"1min":0.55400000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.84699999999999998,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.501,"5min":0.435,"15min":0.435},"max":{"1min":0.83999999999999997,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.83899999999999997}]},{"osd":3,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.105,"5min":1.391,"15min":1.391},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.73999999999999999,"15min":0.75900000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95899999999999996,"5min":1.302,"15min":1.302},"last":0.434}]},{"osd":4,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69199999999999995,"15min":0.69799999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.88700000000000001,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.877},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.81100000000000005,"15min":0.82299999999999995},"min":{"1min":0.56200000000000006,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98299999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.63}]},{"osd":5,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.71899999999999997,"15min":0.72899999999999998},"min":{"1min":0.45800000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.88300000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.83799999999999997,"15min":0.85799999999999998},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0129999999999999,"5min":1.853,"15min":1.853},"last":0.5}]},{"osd":6,"last update":"Sat Mar 30 00:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.623}]},{"osd":7,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.86399999999999999,"15min":0.88},"min":{"1min":0.54100000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.94099999999999995,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.83299999999999996,"15min":0.84399999999999997},"min":{"1min":0.58699999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.94599999999999995,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.91200000000000003}]},{"osd":8,"last update":"Sat Mar 30 00:56:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.90100000000000002,"15min":0.92000000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.98099999999999998,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.90000000000000002,"15min":0.92200000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0720000000000001,"5min":1.968,"15min":1.968},"last":0.52000000000000002}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.88700000000000001,"15min":0.88100000000000001},"min":{"1min":0.65000000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.8170000000000002,"5min":2.8170000000000002,"15min":2.8170000000000002},"last":0.622},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.84999999999999998,"15min":0.84099999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":2.9889999999999999,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":0.94199999999999995}]},{"osd":11,"last update":"Sat Mar 30 00:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.53600000000000003}]}]},{"osd":8,"up_from":53,"seq":227633266731,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":921600,"data_stored":462209,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56499999999999995,"15min":0.56100000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77500000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54900000000000004,"15min":0.54400000000000004},"min":{"1min":0.40999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.80400000000000005,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.505}]},{"osd":1,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.59199999999999997,"15min":0.57299999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78700000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59299999999999997,"15min":0.57999999999999996},"min":{"1min":0.47999999999999998,"5min":0.436,"15min":0.436},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.44600000000000001,"15min":0.42599999999999999},"min":{"1min":0.35899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64100000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.45000000000000001},"min":{"1min":0.38800000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.68999999999999995,"5min":0.751,"15min":0.751},"last":0.64500000000000002}]},{"osd":3,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.68899999999999995,"15min":0.72399999999999998},"min":{"1min":0.434,"5min":0.372,"15min":0.372},"max":{"1min":0.79700000000000004,"5min":2.883,"15min":2.883},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.59999999999999998,"15min":0.58399999999999996},"min":{"1min":0.47599999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.62}]},{"osd":4,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.61099999999999999,"15min":0.60599999999999998},"min":{"1min":0.51000000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.85599999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.629,"15min":0.60699999999999998},"min":{"1min":0.53100000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.57599999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.52300000000000002,"15min":0.498},"min":{"1min":0.41299999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.55700000000000005,"15min":0.53900000000000003},"min":{"1min":0.45100000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84199999999999997,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.40200000000000002}]},{"osd":6,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.66200000000000003,"15min":0.63800000000000001},"min":{"1min":0.56100000000000005,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.77000000000000002,"15min":0.78700000000000003},"min":{"1min":0.59999999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.92600000000000005,"5min":2.8940000000000001,"15min":2.8940000000000001},"last":0.60699999999999998}]},{"osd":7,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.67600000000000005,"15min":0.66000000000000003},"min":{"1min":0.51600000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88500000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.54400000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.88500000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.58999999999999997}]},{"osd":9,"last update":"Sat Mar 30 00:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.70999999999999996,"15min":0.69899999999999995},"min":{"1min":0.58999999999999997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93500000000000005,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.74299999999999999,"15min":0.72599999999999998},"min":{"1min":0.68600000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.65800000000000003}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.53800000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91300000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.75700000000000001,"15min":0.74399999999999999},"min":{"1min":0.66100000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.94899999999999995,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.74099999999999999}]},{"osd":11,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.78700000000000003}]}]},{"osd":7,"up_from":46,"seq":197568495663,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":1059005,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60199999999999998}]},{"osd":1,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.44800000000000001,"15min":0.44700000000000001},"min":{"1min":0.32500000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45000000000000001,"15min":0.442},"min":{"1min":0.33700000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68700000000000006,"5min":0.747,"15min":0.747},"last":0.379}]},{"osd":2,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83299999999999996,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.621},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58199999999999996,"15min":0.56399999999999995},"min":{"1min":0.40600000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78900000000000003,"5min":0.877,"15min":0.877},"last":0.50600000000000001}]},{"osd":3,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60599999999999998,"15min":0.60199999999999998},"min":{"1min":0.443,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.85299999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.66100000000000003,"15min":0.625},"min":{"1min":0.45800000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":5.7430000000000003,"5min":5.7430000000000003,"15min":5.7430000000000003},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53100000000000003,"15min":0.52600000000000002},"min":{"1min":0.39500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66400000000000003,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.314,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.69399999999999995,"5min":1.008,"15min":1.008},"last":0.40899999999999997}]},{"osd":5,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63300000000000001,"15min":0.62},"min":{"1min":0.47399999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82699999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.63300000000000001,"15min":0.62},"min":{"1min":0.49399999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80600000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.66300000000000003}]},{"osd":6,"last update":"Sat Mar 30 00:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66700000000000004,"15min":0.65500000000000003},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.372},"max":{"1min":0.86799999999999999,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.73899999999999999,"15min":0.71899999999999997},"min":{"1min":0.51000000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89200000000000002,"5min":4.117,"15min":4.117},"last":0.52700000000000002}]},{"osd":8,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.621,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86599999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.752,"15min":0.751},"min":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.88900000000000001,"5min":1.014,"15min":1.014},"last":0.65600000000000003}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.74399999999999999,"15min":0.73999999999999999},"min":{"1min":0.60299999999999998,"5min":0.495,"15min":0.495},"max":{"1min":0.94199999999999995,"5min":1.018,"15min":1.018},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.91400000000000003,"15min":0.91900000000000004},"min":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":3.8839999999999999,"5min":4.0519999999999996,"15min":4.0519999999999996},"last":0.67600000000000005}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.755,"15min":0.76400000000000001},"min":{"1min":0.51900000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91300000000000003,"5min":1.032,"15min":1.032},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.71299999999999997,"15min":0.71699999999999997},"min":{"1min":0.59499999999999997,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55500000000000005}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.71599999999999997}]}]},{"osd":6,"up_from":42,"seq":180388626483,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":905216,"data_stored":457796,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.308,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.499,"15min":0.48699999999999999},"min":{"1min":0.373,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.878}]},{"osd":1,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.66300000000000003,"15min":0.63500000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60599999999999998,"15min":0.59599999999999997},"min":{"1min":0.39300000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.68100000000000005}]},{"osd":2,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70899999999999996,"15min":0.68300000000000005},"min":{"1min":0.51000000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.98699999999999999,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.66500000000000004,"15min":0.64300000000000002},"min":{"1min":0.47299999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":1.004}]},{"osd":3,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.65000000000000002},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.97999999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67600000000000005,"15min":0.67200000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.76300000000000001,"15min":0.751},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.105,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69399999999999995,"15min":0.68200000000000005},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.93899999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.74099999999999999,"15min":0.73099999999999998},"min":{"1min":0.51600000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":1.069}]},{"osd":7,"last update":"Sat Mar 30 00:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.81399999999999995},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.046,"5min":1.127,"15min":1.127},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76800000000000002,"15min":0.76900000000000002},"min":{"1min":0.57699999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.2330000000000001,"5min":1.335,"15min":1.335},"last":0.65900000000000003}]},{"osd":8,"last update":"Sat Mar 30 00:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.82399999999999995,"15min":0.84499999999999997},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.2709999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.626},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.876,"15min":0.88800000000000001},"min":{"1min":0.628,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0009999999999999,"5min":1.206,"15min":1.206},"last":0.76700000000000002}]},{"osd":9,"last update":"Sat Mar 30 00:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.94299999999999995,"15min":0.95199999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.244,"5min":1.268,"15min":1.268},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.347,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":1.103}]},{"osd":10,"last update":"Sat Mar 30 00:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85499999999999998,"15min":0.85699999999999998},"min":{"1min":0.69199999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":1.204},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.92500000000000004,"15min":0.93000000000000005},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.0760000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.82599999999999996}]},{"osd":11,"last update":"Sat Mar 30 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.71499999999999997}]}]},{"osd":1,"up_from":12,"seq":51539607621,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":524393,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54900000000000004,"15min":0.53300000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65800000000000003,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.58599999999999997,"15min":0.55500000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.74399999999999999,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.45700000000000002}]},{"osd":2,"last update":"Sat Mar 30 00:57:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.56999999999999995,"15min":0.50700000000000001},"min":{"1min":0.47299999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.81100000000000005,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.57599999999999996,"15min":0.51400000000000001},"min":{"1min":0.42599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.70799999999999996}]},{"osd":3,"last update":"Sat Mar 30 00:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56200000000000006,"15min":0.52800000000000002},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.69199999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.503},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57399999999999995,"15min":0.55000000000000004},"min":{"1min":0.501,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.66900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.626}]},{"osd":4,"last update":"Sat Mar 30 00:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.439,"15min":0.41499999999999998},"min":{"1min":0.377,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.50600000000000001,"15min":0.47399999999999998},"min":{"1min":0.38200000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.57899999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.61099999999999999,"15min":0.61299999999999999},"min":{"1min":0.52900000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78600000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.66900000000000004,"15min":0.64500000000000002},"min":{"1min":0.625,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.748}]},{"osd":6,"last update":"Sat Mar 30 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.55700000000000005,"5min":0.435,"15min":0.435},"max":{"1min":0.81699999999999995,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.54800000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.84799999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.68300000000000005}]},{"osd":7,"last update":"Sat Mar 30 00:57:08 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.57799999999999996,"15min":0.55900000000000005},"min":{"1min":0.48899999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.59399999999999997,"15min":0.58699999999999997},"min":{"1min":0.5,"5min":0.313,"15min":0.313},"max":{"1min":0.91800000000000004,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.73599999999999999}]},{"osd":8,"last update":"Sat Mar 30 00:57:31 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.67200000000000004,"15min":0.66000000000000003},"min":{"1min":0.58999999999999997,"5min":0.38,"15min":0.38},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.63900000000000001,"15min":0.63},"min":{"1min":0.52100000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.69899999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.746,"15min":0.73699999999999999},"min":{"1min":0.56899999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.91600000000000004,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72999999999999998,"15min":0.72399999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.81100000000000005}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.66200000000000003,"15min":0.65400000000000003},"min":{"1min":0.50800000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.74099999999999999,"15min":0.72899999999999998},"min":{"1min":0.71999999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.95499999999999996,"5min":1.385,"15min":1.385},"last":0.82799999999999996}]},{"osd":11,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.79700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32016,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713136,"statfs":{"total":95995035648,"available":95962251264,"internally_reserved":0,"allocated":925696,"data_stored":458911,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 00:57:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.67800000000000005,"15min":0.60199999999999998},"min":{"1min":0.39300000000000002,"5min":0.372,"15min":0.372},"max":{"1min":9.266,"5min":9.266,"15min":9.266},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.624,"15min":0.57799999999999996},"min":{"1min":0.42299999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.6899999999999999,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":0.65400000000000003}]},{"osd":2,"last update":"Sat Mar 30 00:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.59599999999999997,"15min":0.54300000000000004},"min":{"1min":0.41399999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71999999999999997,"15min":0.60499999999999998},"min":{"1min":0.46300000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.071,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":0.52600000000000002}]},{"osd":3,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.9139999999999997,"5min":7.9139999999999997,"15min":7.9139999999999997},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.96,"5min":7.96,"15min":7.96},"last":0.503}]},{"osd":4,"last update":"Sat Mar 30 00:57:16 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":0.75800000000000001,"15min":0.65900000000000003},"min":{"1min":0.48199999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":9.2349999999999994,"5min":9.2349999999999994,"15min":9.2349999999999994},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.71199999999999997,"15min":0.63400000000000001},"min":{"1min":0.626,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":3.73,"5min":3.73,"15min":3.73},"last":0.70199999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76400000000000001,"15min":0.67300000000000004},"min":{"1min":0.52300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.02,"5min":9.0579999999999998,"15min":9.0579999999999998},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.72399999999999998,"15min":0.64900000000000002},"min":{"1min":0.53200000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.157,"5min":3.8039999999999998,"15min":3.8039999999999998},"last":0.67800000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:56:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":0.72099999999999997,"15min":0.65900000000000003},"min":{"1min":0.42399999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":8.0050000000000008,"5min":8.0050000000000008,"15min":8.0050000000000008},"last":0.628},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.66800000000000004,"15min":0.65800000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.71399999999999997}]},{"osd":7,"last update":"Sat Mar 30 00:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.4140000000000001,"5min":2.4140000000000001,"15min":2.4140000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":9.2959999999999994,"5min":9.2959999999999994,"15min":9.2959999999999994},"last":0.78700000000000003}]},{"osd":8,"last update":"Sat Mar 30 00:57:29 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.78200000000000003,"15min":0.78000000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.375,"5min":2.375,"15min":2.375},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.226,"5min":0.91200000000000003,"15min":0.876},"min":{"1min":0.57299999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":9.2040000000000006,"5min":9.2040000000000006,"15min":9.2040000000000006},"last":0.76900000000000002}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":0.88700000000000001,"15min":0.83699999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":8.0389999999999997,"5min":8.0389999999999997,"15min":8.0389999999999997},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.78400000000000003,"15min":0.76600000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":0.83599999999999997}]},{"osd":10,"last update":"Sat Mar 30 00:57:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":0.90800000000000003,"15min":0.88200000000000001},"min":{"1min":0.64000000000000001,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":3.8519999999999999,"5min":3.8519999999999999,"15min":3.8519999999999999},"last":0.752},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.82799999999999996,"15min":0.81200000000000006},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.4470000000000001,"5min":2.4470000000000001,"15min":2.4470000000000001},"last":0.96399999999999997}]},{"osd":11,"last update":"Sat Mar 30 00:57:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.86099999999999999,"15min":0.84799999999999998},"min":{"1min":0.73699999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":0.995},{"interface":"front","average":{"1min":1.3140000000000001,"5min":0.92400000000000004,"15min":0.85899999999999999},"min":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":9.1709999999999994,"5min":9.1709999999999994,"15min":9.1709999999999994},"last":0.83199999999999996}]}]},{"osd":2,"up_from":17,"seq":73014444099,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":525799,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54100000000000004,"15min":0.52900000000000003},"min":{"1min":0.41399999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.70999999999999996,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54600000000000004,"15min":0.52700000000000002},"min":{"1min":0.38800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72899999999999998,"5min":1.097,"15min":1.097},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Mar 30 00:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.56699999999999995,"15min":0.52400000000000002},"min":{"1min":0.46400000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82599999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56000000000000005,"15min":0.53400000000000003},"min":{"1min":0.46200000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.84099999999999997,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.70699999999999996}]},{"osd":3,"last update":"Sat Mar 30 00:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.59499999999999997,"15min":0.56499999999999995},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.80300000000000005,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59799999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.72499999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.65800000000000003}]},{"osd":4,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.95799999999999996}]},{"osd":5,"last update":"Sat Mar 30 00:57:31 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.45800000000000002,"15min":0.436},"min":{"1min":0.378,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.78800000000000003,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.48399999999999999,"15min":0.46200000000000002},"min":{"1min":0.38300000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.73199999999999998,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.76700000000000002}]},{"osd":6,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65100000000000002,"15min":0.65000000000000002},"min":{"1min":0.44600000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86399999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66500000000000004,"15min":0.66100000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.85999999999999999}]},{"osd":7,"last update":"Sat Mar 30 00:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.88},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.79000000000000004}]},{"osd":8,"last update":"Sat Mar 30 00:57:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60199999999999998,"15min":0.59199999999999997},"min":{"1min":0.48199999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.61299999999999999,"15min":0.60099999999999998},"min":{"1min":0.497,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.84499999999999997}]},{"osd":9,"last update":"Sat Mar 30 00:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.72199999999999998,"15min":0.70699999999999996},"min":{"1min":0.58599999999999997,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.75700000000000001,"15min":0.73899999999999999},"min":{"1min":0.65600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.81799999999999995}]},{"osd":10,"last update":"Sat Mar 30 00:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67300000000000004,"15min":0.66700000000000004},"min":{"1min":0.55000000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91000000000000003,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.68100000000000005,"15min":0.66700000000000004},"min":{"1min":0.66700000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.97899999999999998}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":1.073}]}]},{"osd":3,"up_from":24,"seq":103079215166,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":457717,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.46000000000000002,"15min":0.41399999999999998},"min":{"1min":0.32900000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46400000000000002,"15min":0.432},"min":{"1min":0.35199999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.997,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.30599999999999999}]},{"osd":1,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65100000000000002,"15min":0.60299999999999998},"min":{"1min":0.45000000000000001,"5min":0.375,"15min":0.375},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.439,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.58099999999999996}]},{"osd":2,"last update":"Sat Mar 30 00:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.67200000000000004,"15min":0.629},"min":{"1min":0.51400000000000001,"5min":0.371,"15min":0.371},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64800000000000002,"15min":0.622},"min":{"1min":0.47599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.68799999999999994}]},{"osd":4,"last update":"Sat Mar 30 00:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.70799999999999996,"15min":0.66800000000000004},"min":{"1min":0.501,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67900000000000005,"15min":0.66500000000000004},"min":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65100000000000002}]},{"osd":5,"last update":"Sat Mar 30 00:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.67400000000000004,"15min":0.65200000000000002},"min":{"1min":0.51000000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.73399999999999999,"15min":0.67600000000000005},"min":{"1min":0.48799999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":0.53800000000000003}]},{"osd":6,"last update":"Sat Mar 30 00:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.65000000000000002,"15min":0.63800000000000001},"min":{"1min":0.46800000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.44,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.34799999999999998}]},{"osd":7,"last update":"Sat Mar 30 00:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.74399999999999999,"15min":0.75},"min":{"1min":0.51200000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83699999999999997,"15min":0.83999999999999997},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.73399999999999999}]},{"osd":8,"last update":"Sat Mar 30 00:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.57399999999999995}]},{"osd":9,"last update":"Sat Mar 30 00:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.92400000000000004,"15min":0.94399999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.2410000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.502},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.84499999999999997,"15min":0.85899999999999999},"min":{"1min":0.54700000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.224,"5min":1.347,"15min":1.347},"last":0.52500000000000002}]},{"osd":10,"last update":"Sat Mar 30 00:57:03 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77600000000000002,"15min":0.77100000000000002},"min":{"1min":0.53800000000000003,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.92000000000000004,"15min":0.92200000000000004},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.61499999999999999}]},{"osd":11,"last update":"Sat Mar 30 00:56:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":0.755},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.60599999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986235,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":909312,"data_stored":462634,"data_compressed":17781,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.66700000000000004,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.79300000000000004,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73099999999999998,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.41099999999999998,"15min":0.39600000000000002},"min":{"1min":0.34899999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.65300000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.41899999999999998,"15min":0.40699999999999997},"min":{"1min":0.36199999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71799999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.45200000000000001}]},{"osd":2,"last update":"Sat Mar 30 00:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.56799999999999995,"15min":0.55000000000000004},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83299999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.56699999999999995,"15min":0.54300000000000004},"min":{"1min":0.45700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71699999999999997,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.51200000000000001}]},{"osd":3,"last update":"Sat Mar 30 00:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.63700000000000001,"15min":0.69099999999999995},"min":{"1min":0.48899999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.74399999999999999,"5min":3.887,"15min":3.887},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.45000000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.53800000000000003}]},{"osd":5,"last update":"Sat Mar 30 00:57:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.58499999999999996,"15min":0.56699999999999995},"min":{"1min":0.38600000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.81000000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.59599999999999997,"15min":0.57499999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55400000000000005}]},{"osd":6,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.623,"15min":0.625},"min":{"1min":0.42699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.64800000000000002,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82599999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.63100000000000001}]},{"osd":7,"last update":"Sat Mar 30 00:57:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55100000000000005,"15min":0.54200000000000004},"min":{"1min":0.36499999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.79400000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.54800000000000004,"15min":0.52800000000000002},"min":{"1min":0.46100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.73899999999999999}]},{"osd":8,"last update":"Sat Mar 30 00:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.66500000000000004,"15min":0.65600000000000003},"min":{"1min":0.51800000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94299999999999995,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.68799999999999994,"15min":0.68000000000000005},"min":{"1min":0.53800000000000003,"5min":0.432,"15min":0.432},"max":{"1min":0.84099999999999997,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.64300000000000002}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70399999999999996,"15min":0.69899999999999995},"min":{"1min":0.53500000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.73299999999999998,"15min":0.73699999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.98999999999999999,"5min":1.012,"15min":1.012},"last":0.71099999999999997}]},{"osd":10,"last update":"Sat Mar 30 00:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.53600000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.97499999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.81599999999999995}]},{"osd":11,"last update":"Sat Mar 30 00:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.751},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.79400000000000004}]}]},{"osd":5,"up_from":35,"seq":150323855414,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":978944,"data_stored":1053801,"data_compressed":53548,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.56899999999999995,"15min":0.52000000000000002},"min":{"1min":0.38,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":5.3970000000000002,"5min":5.3970000000000002,"15min":5.3970000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.499},"min":{"1min":0.41499999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.69599999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.45600000000000002}]},{"osd":1,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57399999999999995,"15min":0.56599999999999995},"min":{"1min":0.47399999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.74299999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.56999999999999995,"15min":0.54600000000000004},"min":{"1min":0.51000000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.51000000000000001}]},{"osd":2,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.34999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.52700000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.40600000000000003,"15min":0.39900000000000002},"min":{"1min":0.32100000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.68600000000000005}]},{"osd":3,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.42599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.74399999999999999,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54800000000000004,"15min":0.53200000000000003},"min":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.68400000000000005,"5min":0.877,"15min":0.877},"last":0.54800000000000004}]},{"osd":4,"last update":"Sat Mar 30 00:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.56499999999999995},"min":{"1min":0.495,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.71799999999999997,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.60599999999999998,"15min":0.58599999999999997},"min":{"1min":0.41599999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.83699999999999997,"5min":1.038,"15min":1.038},"last":0.73299999999999998}]},{"osd":6,"last update":"Sat Mar 30 00:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59899999999999998,"15min":0.58099999999999996},"min":{"1min":0.42799999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81200000000000006,"5min":0.997,"15min":0.997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60499999999999998,"15min":0.59799999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.72999999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.60799999999999998}]},{"osd":7,"last update":"Sat Mar 30 00:57:09 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61899999999999999,"15min":0.60699999999999998},"min":{"1min":0.505,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.64000000000000001},"min":{"1min":0.55600000000000005,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.85399999999999998,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.58799999999999997}]},{"osd":8,"last update":"Sat Mar 30 00:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.55300000000000005,"15min":0.53600000000000003},"min":{"1min":0.373,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82799999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.56399999999999995,"15min":0.54800000000000004},"min":{"1min":0.45800000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77700000000000002,"5min":0.872,"15min":0.872},"last":0.67000000000000004}]},{"osd":9,"last update":"Sat Mar 30 00:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.67300000000000004,"15min":0.66700000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68400000000000005,"15min":0.68100000000000005},"min":{"1min":0.58099999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63300000000000001}]},{"osd":10,"last update":"Sat Mar 30 00:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.70599999999999996,"15min":0.70399999999999996},"min":{"1min":0.54700000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.92200000000000004,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.71799999999999997,"15min":0.71699999999999997},"min":{"1min":0.60499999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.89800000000000002,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.66000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35767,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":69330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T00:57:46.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:57:47.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.090+0000 7f914ac00700 1 -- 172.21.15.116:0/1856496503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9144102e90 msgr2=0x7f91441032b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:47.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.090+0000 7f914ac00700 1 --2- 172.21.15.116:0/1856496503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9144102e90 0x7f91441032b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9134009990 tx=0x7f9134009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.090+0000 7f914ac00700 1 -- 172.21.15.116:0/1856496503 shutdown_connections 2024-03-30T00:57:47.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.090+0000 7f914ac00700 1 --2- 172.21.15.116:0/1856496503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f91441049a0 0x7f914410b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.090+0000 7f914ac00700 1 --2- 172.21.15.116:0/1856496503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9144102e90 0x7f91441032b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.090+0000 7f914ac00700 1 --2- 172.21.15.116:0/1856496503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9144103fe0 0x7f9144104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.090+0000 7f914ac00700 1 -- 172.21.15.116:0/1856496503 >> 172.21.15.116:0/1856496503 conn(0x7f91440fe420 msgr2=0x7f9144100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:47.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.090+0000 7f914ac00700 1 -- 172.21.15.116:0/1856496503 shutdown_connections 2024-03-30T00:57:47.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.090+0000 7f914ac00700 1 -- 172.21.15.116:0/1856496503 wait complete. 2024-03-30T00:57:47.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.091+0000 7f914ac00700 1 Processor -- start 2024-03-30T00:57:47.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.092+0000 7f914ac00700 1 -- start start 2024-03-30T00:57:47.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.092+0000 7f914ac00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9144102e90 0x7f9144198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:47.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.092+0000 7f914ac00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9144103fe0 0x7f9144199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:47.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.092+0000 7f914ac00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91441049a0 0x7f91441a03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:47.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.092+0000 7f914ac00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91441a0980 con 0x7f91441049a0 2024-03-30T00:57:47.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.092+0000 7f914ac00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91441a0ac0 con 0x7f9144102e90 2024-03-30T00:57:47.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.092+0000 7f914ac00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91441a0c50 con 0x7f9144103fe0 2024-03-30T00:57:47.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.092+0000 7f9148c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91441049a0 0x7f91441a03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:47.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.092+0000 7f9148c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91441049a0 0x7f91441a03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58578/0 (socket says 172.21.15.116:58578) 2024-03-30T00:57:47.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.092+0000 7f9148c00700 1 -- 172.21.15.116:0/2129463955 learned_addr learned my addr 172.21.15.116:0/2129463955 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:47.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.092+0000 7f9143400700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9144103fe0 0x7f9144199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:47.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.092+0000 7f9143e00700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9144102e90 0x7f9144198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:47.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.093+0000 7f9148c00700 1 -- 172.21.15.116:0/2129463955 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9144103fe0 msgr2=0x7f9144199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:47.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.093+0000 7f9148c00700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9144103fe0 0x7f9144199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.093+0000 7f9148c00700 1 -- 172.21.15.116:0/2129463955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9144102e90 msgr2=0x7f9144198e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:47.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.093+0000 7f9148c00700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9144102e90 0x7f9144198e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.093+0000 7f9148c00700 1 -- 172.21.15.116:0/2129463955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91340095f0 con 0x7f91441049a0 2024-03-30T00:57:47.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.093+0000 7f9143400700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9144103fe0 0x7f9144199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:47.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.093+0000 7f9148c00700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91441049a0 0x7f91441a03b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f913c007ee0 tx=0x7f913c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:47.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.093+0000 7f9140c00700 1 -- 172.21.15.116:0/2129463955 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c00ddd0 con 0x7f91441049a0 2024-03-30T00:57:47.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.093+0000 7f9140c00700 1 -- 172.21.15.116:0/2129463955 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f913c00f040 con 0x7f91441049a0 2024-03-30T00:57:47.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.093+0000 7f9140c00700 1 -- 172.21.15.116:0/2129463955 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c0048e0 con 0x7f91441049a0 2024-03-30T00:57:47.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.093+0000 7f914ac00700 1 -- 172.21.15.116:0/2129463955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91441a0f30 con 0x7f91441049a0 2024-03-30T00:57:47.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.093+0000 7f914ac00700 1 -- 172.21.15.116:0/2129463955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91441a13b0 con 0x7f91441049a0 2024-03-30T00:57:47.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.094+0000 7f914ac00700 1 -- 172.21.15.116:0/2129463955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914410a300 con 0x7f91441049a0 2024-03-30T00:57:47.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.096+0000 7f9140c00700 1 -- 172.21.15.116:0/2129463955 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f913c003a00 con 0x7f91441049a0 2024-03-30T00:57:47.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.098+0000 7f9140c00700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91280a11a0 0x7f91280a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:47.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.098+0000 7f9140c00700 1 -- 172.21.15.116:0/2129463955 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f913c0cc490 con 0x7f91441049a0 2024-03-30T00:57:47.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.098+0000 7f9143e00700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91280a11a0 0x7f91280a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:47.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.099+0000 7f9143e00700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91280a11a0 0x7f91280a3660 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f913400b2e0 tx=0x7f9134005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:47.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.102+0000 7f9140c00700 1 -- 172.21.15.116:0/2129463955 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f913c0958b0 con 0x7f91441049a0 2024-03-30T00:57:47.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.344+0000 7f914ac00700 1 -- 172.21.15.116:0/2129463955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f914410e770 con 0x7f91441049a0 2024-03-30T00:57:47.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.346+0000 7f9140c00700 1 -- 172.21.15.116:0/2129463955 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+24556 (secure 0 0 0) 0x7f913c098ed0 con 0x7f91441049a0 2024-03-30T00:57:47.353 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:47.353 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:56:36.456069+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:57:47.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.348+0000 7f914ac00700 1 -- 172.21.15.116:0/2129463955 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91280a11a0 msgr2=0x7f91280a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:47.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.348+0000 7f914ac00700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91280a11a0 0x7f91280a3660 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f913400b2e0 tx=0x7f9134005fd0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.349+0000 7f914ac00700 1 -- 172.21.15.116:0/2129463955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91441049a0 msgr2=0x7f91441a03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:47.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.349+0000 7f914ac00700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91441049a0 0x7f91441a03b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f913c007ee0 tx=0x7f913c00d3b0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.349+0000 7f914ac00700 1 -- 172.21.15.116:0/2129463955 shutdown_connections 2024-03-30T00:57:47.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.349+0000 7f914ac00700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91280a11a0 0x7f91280a3660 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.349+0000 7f914ac00700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91441049a0 0x7f91441a03b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.349+0000 7f914ac00700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9144102e90 0x7f9144198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.349+0000 7f914ac00700 1 --2- 172.21.15.116:0/2129463955 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9144103fe0 0x7f9144199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.349+0000 7f914ac00700 1 -- 172.21.15.116:0/2129463955 >> 172.21.15.116:0/2129463955 conn(0x7f91440fe420 msgr2=0x7f9144107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:47.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.349+0000 7f914ac00700 1 -- 172.21.15.116:0/2129463955 shutdown_connections 2024-03-30T00:57:47.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.349+0000 7f914ac00700 1 -- 172.21.15.116:0/2129463955 wait complete. 2024-03-30T00:57:47.375 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T00:52:29.602428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T00:55:50.220543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T00:55:51.274498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T00:55:54.301591+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T00:57:47.375 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T00:57:47.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.565+0000 7f6a36a00700 1 -- 172.21.15.116:0/4271362344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30101f70 msgr2=0x7f6a301023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:47.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.565+0000 7f6a36a00700 1 --2- 172.21.15.116:0/4271362344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30101f70 0x7f6a301023d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c0099c0 tx=0x7f6a1c009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.565+0000 7f6a36a00700 1 -- 172.21.15.116:0/4271362344 shutdown_connections 2024-03-30T00:57:47.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.565+0000 7f6a36a00700 1 --2- 172.21.15.116:0/4271362344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a30102910 0x7f6a301091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.565+0000 7f6a36a00700 1 --2- 172.21.15.116:0/4271362344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a30100d80 0x7f6a301011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.565+0000 7f6a36a00700 1 --2- 172.21.15.116:0/4271362344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30101f70 0x7f6a301023d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.565+0000 7f6a36a00700 1 -- 172.21.15.116:0/4271362344 >> 172.21.15.116:0/4271362344 conn(0x7f6a300fc2f0 msgr2=0x7f6a300fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:47.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.565+0000 7f6a36a00700 1 -- 172.21.15.116:0/4271362344 shutdown_connections 2024-03-30T00:57:47.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.565+0000 7f6a36a00700 1 -- 172.21.15.116:0/4271362344 wait complete. 2024-03-30T00:57:47.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.566+0000 7f6a36a00700 1 Processor -- start 2024-03-30T00:57:47.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.566+0000 7f6a36a00700 1 -- start start 2024-03-30T00:57:47.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.566+0000 7f6a36a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30100d80 0x7f6a30198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:47.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.566+0000 7f6a36a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a30101f70 0x7f6a30199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:47.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.566+0000 7f6a36a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a30102910 0x7f6a301a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:47.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.567+0000 7f6a36a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a301a0a20 con 0x7f6a30100d80 2024-03-30T00:57:47.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.567+0000 7f6a36a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a301a0b60 con 0x7f6a30101f70 2024-03-30T00:57:47.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.567+0000 7f6a36a00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a301a0cf0 con 0x7f6a30102910 2024-03-30T00:57:47.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.567+0000 7f6a2fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30100d80 0x7f6a30198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:47.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.567+0000 7f6a2fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30100d80 0x7f6a30198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58592/0 (socket says 172.21.15.116:58592) 2024-03-30T00:57:47.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.567+0000 7f6a2fe00700 1 -- 172.21.15.116:0/2421139783 learned_addr learned my addr 172.21.15.116:0/2421139783 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.567+0000 7f6a2f400700 1 --2- 172.21.15.116:0/2421139783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a30101f70 0x7f6a30199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.567+0000 7f6a34a00700 1 --2- 172.21.15.116:0/2421139783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a30102910 0x7f6a301a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.567+0000 7f6a2fe00700 1 -- 172.21.15.116:0/2421139783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a30102910 msgr2=0x7f6a301a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.567+0000 7f6a2fe00700 1 --2- 172.21.15.116:0/2421139783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a30102910 0x7f6a301a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.567+0000 7f6a2fe00700 1 -- 172.21.15.116:0/2421139783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a30101f70 msgr2=0x7f6a30199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.567+0000 7f6a2fe00700 1 --2- 172.21.15.116:0/2421139783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a30101f70 0x7f6a30199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.567+0000 7f6a2fe00700 1 -- 172.21.15.116:0/2421139783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a1c0095f0 con 0x7f6a30100d80 2024-03-30T00:57:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.567+0000 7f6a2fe00700 1 --2- 172.21.15.116:0/2421139783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30100d80 0x7f6a30198ef0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000e970 tx=0x7f6a2000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.568+0000 7f6a2cc00700 1 -- 172.21.15.116:0/2421139783 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20009790 con 0x7f6a30100d80 2024-03-30T00:57:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.568+0000 7f6a2cc00700 1 -- 172.21.15.116:0/2421139783 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a200044e0 con 0x7f6a30100d80 2024-03-30T00:57:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.568+0000 7f6a36a00700 1 -- 172.21.15.116:0/2421139783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a301a0fd0 con 0x7f6a30100d80 2024-03-30T00:57:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.568+0000 7f6a2cc00700 1 -- 172.21.15.116:0/2421139783 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20003e30 con 0x7f6a30100d80 2024-03-30T00:57:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.568+0000 7f6a36a00700 1 -- 172.21.15.116:0/2421139783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a301a1400 con 0x7f6a30100d80 2024-03-30T00:57:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.569+0000 7f6a36a00700 1 -- 172.21.15.116:0/2421139783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a30067af0 con 0x7f6a30100d80 2024-03-30T00:57:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.571+0000 7f6a2cc00700 1 -- 172.21.15.116:0/2421139783 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f6a20003f90 con 0x7f6a30100d80 2024-03-30T00:57:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.572+0000 7f6a2cc00700 1 --2- 172.21.15.116:0/2421139783 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6a140a1220 0x7f6a140a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.572+0000 7f6a2cc00700 1 -- 172.21.15.116:0/2421139783 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6a20014070 con 0x7f6a30100d80 2024-03-30T00:57:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.572+0000 7f6a2f400700 1 --2- 172.21.15.116:0/2421139783 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6a140a1220 0x7f6a140a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.573+0000 7f6a2f400700 1 --2- 172.21.15.116:0/2421139783 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6a140a1220 0x7f6a140a36e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c00b580 tx=0x7f6a1c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.576+0000 7f6a2cc00700 1 -- 172.21.15.116:0/2421139783 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6a20096320 con 0x7f6a30100d80 2024-03-30T00:57:47.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:47 smithi116 ceph-mon[38086]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:47.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:47 smithi116 ceph-mon[38086]: from='client.25460 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:47.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:47 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2129463955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:47.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:47 smithi181 ceph-mon[47622]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:47.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:47 smithi181 ceph-mon[47622]: from='client.25460 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:47.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:47 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2129463955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:47.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.781+0000 7f6a36a00700 1 -- 172.21.15.116:0/2421139783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6a301a17a0 con 0x7f6a30100d80 2024-03-30T00:57:47.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.782+0000 7f6a2cc00700 1 -- 172.21.15.116:0/2421139783 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6a20099940 con 0x7f6a30100d80 2024-03-30T00:57:47.789 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-30T00:57:47.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.784+0000 7f6a36a00700 1 -- 172.21.15.116:0/2421139783 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6a140a1220 msgr2=0x7f6a140a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:47.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.784+0000 7f6a36a00700 1 --2- 172.21.15.116:0/2421139783 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6a140a1220 0x7f6a140a36e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c00b580 tx=0x7f6a1c01a040 comp rx=0 tx=0).stop 2024-03-30T00:57:47.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.784+0000 7f6a36a00700 1 -- 172.21.15.116:0/2421139783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30100d80 msgr2=0x7f6a30198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:47.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.784+0000 7f6a36a00700 1 --2- 172.21.15.116:0/2421139783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30100d80 0x7f6a30198ef0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000e970 tx=0x7f6a2000ed30 comp rx=0 tx=0).stop 2024-03-30T00:57:47.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.784+0000 7f6a36a00700 1 -- 172.21.15.116:0/2421139783 shutdown_connections 2024-03-30T00:57:47.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.784+0000 7f6a36a00700 1 --2- 172.21.15.116:0/2421139783 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6a140a1220 0x7f6a140a36e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.784+0000 7f6a36a00700 1 --2- 172.21.15.116:0/2421139783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a30102910 0x7f6a301a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.784+0000 7f6a36a00700 1 --2- 172.21.15.116:0/2421139783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30100d80 0x7f6a30198ef0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.784+0000 7f6a36a00700 1 --2- 172.21.15.116:0/2421139783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a30101f70 0x7f6a30199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:47.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.784+0000 7f6a36a00700 1 -- 172.21.15.116:0/2421139783 >> 172.21.15.116:0/2421139783 conn(0x7f6a300fc2f0 msgr2=0x7f6a30072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:47.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.785+0000 7f6a36a00700 1 -- 172.21.15.116:0/2421139783 shutdown_connections 2024-03-30T00:57:47.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:47.785+0000 7f6a36a00700 1 -- 172.21.15.116:0/2421139783 wait complete. 2024-03-30T00:57:47.810 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T00:57:47.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:47 smithi130 ceph-mon[47626]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:47.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:47 smithi130 ceph-mon[47626]: from='client.25460 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:47.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:47 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2129463955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:48.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.000+0000 7f136c600700 1 -- 172.21.15.116:0/154288694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1364100d80 msgr2=0x7f13641011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:48.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.000+0000 7f136c600700 1 --2- 172.21.15.116:0/154288694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1364100d80 0x7f13641011a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f13580099c0 tx=0x7f1358009cd0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.000+0000 7f136c600700 1 -- 172.21.15.116:0/154288694 shutdown_connections 2024-03-30T00:57:48.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.000+0000 7f136c600700 1 --2- 172.21.15.116:0/154288694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1364102910 0x7f13641091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.000+0000 7f136c600700 1 --2- 172.21.15.116:0/154288694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1364100d80 0x7f13641011a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.000+0000 7f136c600700 1 --2- 172.21.15.116:0/154288694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1364101f70 0x7f13641023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.000+0000 7f136c600700 1 -- 172.21.15.116:0/154288694 >> 172.21.15.116:0/154288694 conn(0x7f13640fc2f0 msgr2=0x7f13640fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:48.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.000+0000 7f136c600700 1 -- 172.21.15.116:0/154288694 shutdown_connections 2024-03-30T00:57:48.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.000+0000 7f136c600700 1 -- 172.21.15.116:0/154288694 wait complete. 2024-03-30T00:57:48.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.001+0000 7f136c600700 1 Processor -- start 2024-03-30T00:57:48.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.001+0000 7f136c600700 1 -- start start 2024-03-30T00:57:48.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.001+0000 7f136c600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1364100d80 0x7f1364196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:48.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.001+0000 7f136c600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1364101f70 0x7f1364197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:48.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.001+0000 7f136c600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1364102910 0x7f136419e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.001+0000 7f136c600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136419e870 con 0x7f1364101f70 2024-03-30T00:57:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.001+0000 7f136c600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136419e9e0 con 0x7f1364102910 2024-03-30T00:57:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.001+0000 7f136c600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136419eba0 con 0x7f1364100d80 2024-03-30T00:57:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.002+0000 7f1369c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1364100d80 0x7f1364196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.002+0000 7f1369200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1364101f70 0x7f1364197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.002+0000 7f1369c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1364100d80 0x7f1364196dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:57572/0 (socket says 172.21.15.116:57572) 2024-03-30T00:57:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.002+0000 7f1369c00700 1 -- 172.21.15.116:0/1637341137 learned_addr learned my addr 172.21.15.116:0/1637341137 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.002+0000 7f136a600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1364102910 0x7f136419e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.002+0000 7f1369c00700 1 -- 172.21.15.116:0/1637341137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1364102910 msgr2=0x7f136419e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.002+0000 7f1369c00700 1 --2- 172.21.15.116:0/1637341137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1364102910 0x7f136419e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.002+0000 7f1369c00700 1 -- 172.21.15.116:0/1637341137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1364101f70 msgr2=0x7f1364197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.002+0000 7f1369c00700 1 --2- 172.21.15.116:0/1637341137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1364101f70 0x7f1364197310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.002+0000 7f1369c00700 1 -- 172.21.15.116:0/1637341137 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13580095f0 con 0x7f1364100d80 2024-03-30T00:57:48.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.002+0000 7f1369200700 1 --2- 172.21.15.116:0/1637341137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1364101f70 0x7f1364197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:48.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.002+0000 7f1369c00700 1 --2- 172.21.15.116:0/1637341137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1364100d80 0x7f1364196dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1358006010 tx=0x7f1358005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:48.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.002+0000 7f1362000700 1 -- 172.21.15.116:0/1637341137 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135801d070 con 0x7f1364100d80 2024-03-30T00:57:48.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.003+0000 7f136c600700 1 -- 172.21.15.116:0/1637341137 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1364072fe0 con 0x7f1364100d80 2024-03-30T00:57:48.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.003+0000 7f1362000700 1 -- 172.21.15.116:0/1637341137 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1358003f90 con 0x7f1364100d80 2024-03-30T00:57:48.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.003+0000 7f1362000700 1 -- 172.21.15.116:0/1637341137 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1358022cb0 con 0x7f1364100d80 2024-03-30T00:57:48.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.003+0000 7f136c600700 1 -- 172.21.15.116:0/1637341137 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1364073490 con 0x7f1364100d80 2024-03-30T00:57:48.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.004+0000 7f136c600700 1 -- 172.21.15.116:0/1637341137 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13641082a0 con 0x7f1364100d80 2024-03-30T00:57:48.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.005+0000 7f1362000700 1 -- 172.21.15.116:0/1637341137 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f1358022470 con 0x7f1364100d80 2024-03-30T00:57:48.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.007+0000 7f1362000700 1 --2- 172.21.15.116:0/1637341137 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f134c0a1120 0x7f134c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:48.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.007+0000 7f1362000700 1 -- 172.21.15.116:0/1637341137 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f13580ce760 con 0x7f1364100d80 2024-03-30T00:57:48.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.007+0000 7f1369200700 1 --2- 172.21.15.116:0/1637341137 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f134c0a1120 0x7f134c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:48.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.007+0000 7f1369200700 1 --2- 172.21.15.116:0/1637341137 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f134c0a1120 0x7f134c0a35e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f135000ffd0 tx=0x7f135000f820 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:48.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.009+0000 7f1362000700 1 -- 172.21.15.116:0/1637341137 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1358097c10 con 0x7f1364100d80 2024-03-30T00:57:48.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.216+0000 7f136c600700 1 -- 172.21.15.116:0/1637341137 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f136410c5e0 con 0x7f1364100d80 2024-03-30T00:57:48.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.216+0000 7f1362000700 1 -- 172.21.15.116:0/1637341137 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f135809b230 con 0x7f1364100d80 2024-03-30T00:57:48.224 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:57:48.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.219+0000 7f136c600700 1 -- 172.21.15.116:0/1637341137 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f134c0a1120 msgr2=0x7f134c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:48.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.219+0000 7f136c600700 1 --2- 172.21.15.116:0/1637341137 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f134c0a1120 0x7f134c0a35e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f135000ffd0 tx=0x7f135000f820 comp rx=0 tx=0).stop 2024-03-30T00:57:48.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.219+0000 7f136c600700 1 -- 172.21.15.116:0/1637341137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1364100d80 msgr2=0x7f1364196dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.219+0000 7f136c600700 1 --2- 172.21.15.116:0/1637341137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1364100d80 0x7f1364196dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1358006010 tx=0x7f1358005740 comp rx=0 tx=0).stop 2024-03-30T00:57:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.219+0000 7f136c600700 1 -- 172.21.15.116:0/1637341137 shutdown_connections 2024-03-30T00:57:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.219+0000 7f136c600700 1 --2- 172.21.15.116:0/1637341137 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f134c0a1120 0x7f134c0a35e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.219+0000 7f136c600700 1 --2- 172.21.15.116:0/1637341137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1364100d80 0x7f1364196dd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.219+0000 7f136c600700 1 --2- 172.21.15.116:0/1637341137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1364102910 0x7f136419e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.219+0000 7f136c600700 1 --2- 172.21.15.116:0/1637341137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1364101f70 0x7f1364197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.219+0000 7f136c600700 1 -- 172.21.15.116:0/1637341137 >> 172.21.15.116:0/1637341137 conn(0x7f13640fc2f0 msgr2=0x7f13641053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.220+0000 7f136c600700 1 -- 172.21.15.116:0/1637341137 shutdown_connections 2024-03-30T00:57:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.220+0000 7f136c600700 1 -- 172.21.15.116:0/1637341137 wait complete. 2024-03-30T00:57:48.244 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T00:57:48.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.433+0000 7f721b400700 1 -- 172.21.15.116:0/1734467644 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7214101f70 msgr2=0x7f72141023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:48.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.433+0000 7f721b400700 1 --2- 172.21.15.116:0/1734467644 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7214101f70 0x7f72141023d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f72000098f0 tx=0x7f7200009c00 comp rx=0 tx=0).stop 2024-03-30T00:57:48.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.433+0000 7f721b400700 1 -- 172.21.15.116:0/1734467644 shutdown_connections 2024-03-30T00:57:48.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.433+0000 7f721b400700 1 --2- 172.21.15.116:0/1734467644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7214102910 0x7f72141091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.433+0000 7f721b400700 1 --2- 172.21.15.116:0/1734467644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7214100d80 0x7f72141011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.433+0000 7f721b400700 1 --2- 172.21.15.116:0/1734467644 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7214101f70 0x7f72141023d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.433+0000 7f721b400700 1 -- 172.21.15.116:0/1734467644 >> 172.21.15.116:0/1734467644 conn(0x7f72140fc2f0 msgr2=0x7f72140fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:48.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.433+0000 7f721b400700 1 -- 172.21.15.116:0/1734467644 shutdown_connections 2024-03-30T00:57:48.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.433+0000 7f721b400700 1 -- 172.21.15.116:0/1734467644 wait complete. 2024-03-30T00:57:48.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.435+0000 7f721b400700 1 Processor -- start 2024-03-30T00:57:48.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.435+0000 7f721b400700 1 -- start start 2024-03-30T00:57:48.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.435+0000 7f721b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7214100d80 0x7f7214198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:48.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.435+0000 7f721b400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7214101f70 0x7f72141994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:48.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.435+0000 7f721b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7214102910 0x7f72141a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:48.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.435+0000 7f721b400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72141a0ac0 con 0x7f7214102910 2024-03-30T00:57:48.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.435+0000 7f721b400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72141a0c00 con 0x7f7214100d80 2024-03-30T00:57:48.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.435+0000 7f721b400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72141a0d90 con 0x7f7214101f70 2024-03-30T00:57:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.436+0000 7f7218a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7214100d80 0x7f7214198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.436+0000 7f7218a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7214100d80 0x7f7214198f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:60504/0 (socket says 172.21.15.116:60504) 2024-03-30T00:57:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.436+0000 7f7218a00700 1 -- 172.21.15.116:0/3157379813 learned_addr learned my addr 172.21.15.116:0/3157379813 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.436+0000 7f7219400700 1 --2- 172.21.15.116:0/3157379813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7214102910 0x7f72141a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.436+0000 7f7213e00700 1 --2- 172.21.15.116:0/3157379813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7214101f70 0x7f72141994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.436+0000 7f7218a00700 1 -- 172.21.15.116:0/3157379813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7214101f70 msgr2=0x7f72141994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.436+0000 7f7218a00700 1 --2- 172.21.15.116:0/3157379813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7214101f70 0x7f72141994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.436+0000 7f7218a00700 1 -- 172.21.15.116:0/3157379813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7214102910 msgr2=0x7f72141a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.436+0000 7f7218a00700 1 --2- 172.21.15.116:0/3157379813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7214102910 0x7f72141a04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.436+0000 7f7218a00700 1 -- 172.21.15.116:0/3157379813 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7200009520 con 0x7f7214100d80 2024-03-30T00:57:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.436+0000 7f7218a00700 1 --2- 172.21.15.116:0/3157379813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7214100d80 0x7f7214198f90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f720800e970 tx=0x7f720800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.436+0000 7f7211600700 1 -- 172.21.15.116:0/3157379813 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7208009790 con 0x7f7214100d80 2024-03-30T00:57:48.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.437+0000 7f721b400700 1 -- 172.21.15.116:0/3157379813 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72141a1070 con 0x7f7214100d80 2024-03-30T00:57:48.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.437+0000 7f7211600700 1 -- 172.21.15.116:0/3157379813 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f720800c7e0 con 0x7f7214100d80 2024-03-30T00:57:48.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.437+0000 7f7211600700 1 -- 172.21.15.116:0/3157379813 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7208003c70 con 0x7f7214100d80 2024-03-30T00:57:48.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.437+0000 7f721b400700 1 -- 172.21.15.116:0/3157379813 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72141a15d0 con 0x7f7214100d80 2024-03-30T00:57:48.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.437+0000 7f721b400700 1 -- 172.21.15.116:0/3157379813 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7214067af0 con 0x7f7214100d80 2024-03-30T00:57:48.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.440+0000 7f7211600700 1 -- 172.21.15.116:0/3157379813 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f720800cc50 con 0x7f7214100d80 2024-03-30T00:57:48.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.441+0000 7f7211600700 1 --2- 172.21.15.116:0/3157379813 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f71fc0a1120 0x7f71fc0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:48.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.441+0000 7f7211600700 1 -- 172.21.15.116:0/3157379813 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7208014070 con 0x7f7214100d80 2024-03-30T00:57:48.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.441+0000 7f7213e00700 1 --2- 172.21.15.116:0/3157379813 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f71fc0a1120 0x7f71fc0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:48.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.442+0000 7f7213e00700 1 --2- 172.21.15.116:0/3157379813 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f71fc0a1120 0x7f71fc0a35e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f720000fa70 tx=0x7f7200019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:48.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.443+0000 7f7211600700 1 -- 172.21.15.116:0/3157379813 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f72080963a0 con 0x7f7214100d80 2024-03-30T00:57:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.653+0000 7f721b400700 1 -- 172.21.15.116:0/3157379813 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f721410c790 con 0x7f7214100d80 2024-03-30T00:57:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.654+0000 7f7211600700 1 -- 172.21.15.116:0/3157379813 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7208003dd0 con 0x7f7214100d80 2024-03-30T00:57:48.661 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-30T00:57:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.657+0000 7f721b400700 1 -- 172.21.15.116:0/3157379813 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f71fc0a1120 msgr2=0x7f71fc0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.657+0000 7f721b400700 1 --2- 172.21.15.116:0/3157379813 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f71fc0a1120 0x7f71fc0a35e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f720000fa70 tx=0x7f7200019040 comp rx=0 tx=0).stop 2024-03-30T00:57:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.657+0000 7f721b400700 1 -- 172.21.15.116:0/3157379813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7214100d80 msgr2=0x7f7214198f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.657+0000 7f721b400700 1 --2- 172.21.15.116:0/3157379813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7214100d80 0x7f7214198f90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f720800e970 tx=0x7f720800ec80 comp rx=0 tx=0).stop 2024-03-30T00:57:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.657+0000 7f721b400700 1 -- 172.21.15.116:0/3157379813 shutdown_connections 2024-03-30T00:57:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.657+0000 7f721b400700 1 --2- 172.21.15.116:0/3157379813 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f71fc0a1120 0x7f71fc0a35e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.657+0000 7f721b400700 1 --2- 172.21.15.116:0/3157379813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7214102910 0x7f72141a04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.657+0000 7f721b400700 1 --2- 172.21.15.116:0/3157379813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7214100d80 0x7f7214198f90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.657+0000 7f721b400700 1 --2- 172.21.15.116:0/3157379813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7214101f70 0x7f72141994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.657+0000 7f721b400700 1 -- 172.21.15.116:0/3157379813 >> 172.21.15.116:0/3157379813 conn(0x7f72140fc2f0 msgr2=0x7f7214072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.657+0000 7f721b400700 1 -- 172.21.15.116:0/3157379813 shutdown_connections 2024-03-30T00:57:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.657+0000 7f721b400700 1 -- 172.21.15.116:0/3157379813 wait complete. 2024-03-30T00:57:48.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:48 smithi116 ceph-mon[38086]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:48.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:48 smithi116 ceph-mon[38086]: pgmap v289: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-03-30T00:57:48.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:48 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2421139783' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:48.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:48 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1637341137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:48.682 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T00:57:48.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:48 smithi181 ceph-mon[47622]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:48.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:48 smithi181 ceph-mon[47622]: pgmap v289: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-03-30T00:57:48.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:48 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2421139783' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:48.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:48 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1637341137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:48.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:48 smithi130 ceph-mon[47626]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T00:57:48.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:48 smithi130 ceph-mon[47626]: pgmap v289: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-03-30T00:57:48.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:48 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2421139783' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T00:57:48.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:48 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1637341137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T00:57:48.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.870+0000 7f9094e00700 1 -- 172.21.15.116:0/3073470037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9090102910 msgr2=0x7f90901091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:48.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.870+0000 7f9094e00700 1 --2- 172.21.15.116:0/3073470037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9090102910 0x7f90901091c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f908800b0d0 tx=0x7f908800b3e0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.870+0000 7f9094e00700 1 -- 172.21.15.116:0/3073470037 shutdown_connections 2024-03-30T00:57:48.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.870+0000 7f9094e00700 1 --2- 172.21.15.116:0/3073470037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9090100d80 0x7f90901011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.871+0000 7f9094e00700 1 --2- 172.21.15.116:0/3073470037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9090102910 0x7f90901091c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.871+0000 7f9094e00700 1 --2- 172.21.15.116:0/3073470037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9090101f70 0x7f90901023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.871+0000 7f9094e00700 1 -- 172.21.15.116:0/3073470037 >> 172.21.15.116:0/3073470037 conn(0x7f90900fc2f0 msgr2=0x7f90900fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:48.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.871+0000 7f9094e00700 1 -- 172.21.15.116:0/3073470037 shutdown_connections 2024-03-30T00:57:48.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.871+0000 7f9094e00700 1 -- 172.21.15.116:0/3073470037 wait complete. 2024-03-30T00:57:48.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.872+0000 7f9094e00700 1 Processor -- start 2024-03-30T00:57:48.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.872+0000 7f9094e00700 1 -- start start 2024-03-30T00:57:48.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f9094e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9090100d80 0x7f9090199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:48.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f9094e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9090101f70 0x7f9090199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:48.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f9094e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9090102910 0x7f90901a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:48.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f9094e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90901a0b60 con 0x7f9090102910 2024-03-30T00:57:48.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f9094e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90901a0ca0 con 0x7f9090101f70 2024-03-30T00:57:48.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f9094e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90901a0e30 con 0x7f9090100d80 2024-03-30T00:57:48.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f908e800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9090102910 0x7f90901a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:48.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f908d400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9090101f70 0x7f9090199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:48.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f908de00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9090100d80 0x7f9090199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:48.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f908e800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9090102910 0x7f90901a0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58636/0 (socket says 172.21.15.116:58636) 2024-03-30T00:57:48.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f908e800700 1 -- 172.21.15.116:0/494105626 learned_addr learned my addr 172.21.15.116:0/494105626 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:48.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f908de00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9090100d80 0x7f9090199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:57614/0 (socket says 172.21.15.116:57614) 2024-03-30T00:57:48.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f908d400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9090101f70 0x7f9090199570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:60526/0 (socket says 172.21.15.116:60526) 2024-03-30T00:57:48.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f908e800700 1 -- 172.21.15.116:0/494105626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9090100d80 msgr2=0x7f9090199030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:48.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f908e800700 1 --2- 172.21.15.116:0/494105626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9090100d80 0x7f9090199030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f908e800700 1 -- 172.21.15.116:0/494105626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9090101f70 msgr2=0x7f9090199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:48.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f908e800700 1 --2- 172.21.15.116:0/494105626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9090101f70 0x7f9090199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:48.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f908e800700 1 -- 172.21.15.116:0/494105626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9088009d00 con 0x7f9090102910 2024-03-30T00:57:48.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.873+0000 7f908d400700 1 --2- 172.21.15.116:0/494105626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9090101f70 0x7f9090199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:48.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.874+0000 7f908e800700 1 --2- 172.21.15.116:0/494105626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9090102910 0x7f90901a0590 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9088007b60 tx=0x7f9088008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:48.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.874+0000 7f9086a00700 1 -- 172.21.15.116:0/494105626 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908800e040 con 0x7f9090102910 2024-03-30T00:57:48.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.874+0000 7f9086a00700 1 -- 172.21.15.116:0/494105626 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f908801d930 con 0x7f9090102910 2024-03-30T00:57:48.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.874+0000 7f9094e00700 1 -- 172.21.15.116:0/494105626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90901a10b0 con 0x7f9090102910 2024-03-30T00:57:48.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.874+0000 7f9086a00700 1 -- 172.21.15.116:0/494105626 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908801b9f0 con 0x7f9090102910 2024-03-30T00:57:48.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.874+0000 7f9094e00700 1 -- 172.21.15.116:0/494105626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90901a1530 con 0x7f9090102910 2024-03-30T00:57:48.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.875+0000 7f9094e00700 1 -- 172.21.15.116:0/494105626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9090067af0 con 0x7f9090102910 2024-03-30T00:57:48.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.877+0000 7f9086a00700 1 -- 172.21.15.116:0/494105626 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f908801daa0 con 0x7f9090102910 2024-03-30T00:57:48.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.878+0000 7f9086a00700 1 --2- 172.21.15.116:0/494105626 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f90740a11a0 0x7f90740a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:48.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.878+0000 7f9086a00700 1 -- 172.21.15.116:0/494105626 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f90880ce6b0 con 0x7f9090102910 2024-03-30T00:57:48.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.879+0000 7f908de00700 1 --2- 172.21.15.116:0/494105626 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f90740a11a0 0x7f90740a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:48.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.879+0000 7f908de00700 1 --2- 172.21.15.116:0/494105626 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f90740a11a0 0x7f90740a3660 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9090101dd0 tx=0x7f9080009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:48.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:48.881+0000 7f9086a00700 1 -- 172.21.15.116:0/494105626 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9088097bc0 con 0x7f9090102910 2024-03-30T00:57:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.082+0000 7f9094e00700 1 -- 172.21.15.116:0/494105626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f909010c7d0 con 0x7f9090102910 2024-03-30T00:57:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.083+0000 7f9086a00700 1 -- 172.21.15.116:0/494105626 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v83) v1 ==== 103+0+11 (secure 0 0 0) 0x7f908809b1e0 con 0x7f9090102910 2024-03-30T00:57:49.089 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T00:57:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.085+0000 7f9094e00700 1 -- 172.21.15.116:0/494105626 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f90740a11a0 msgr2=0x7f90740a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.085+0000 7f9094e00700 1 --2- 172.21.15.116:0/494105626 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f90740a11a0 0x7f90740a3660 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9090101dd0 tx=0x7f9080009260 comp rx=0 tx=0).stop 2024-03-30T00:57:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.085+0000 7f9094e00700 1 -- 172.21.15.116:0/494105626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9090102910 msgr2=0x7f90901a0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.085+0000 7f9094e00700 1 --2- 172.21.15.116:0/494105626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9090102910 0x7f90901a0590 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9088007b60 tx=0x7f9088008e40 comp rx=0 tx=0).stop 2024-03-30T00:57:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.085+0000 7f9094e00700 1 -- 172.21.15.116:0/494105626 shutdown_connections 2024-03-30T00:57:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.085+0000 7f9094e00700 1 --2- 172.21.15.116:0/494105626 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f90740a11a0 0x7f90740a3660 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.085+0000 7f9094e00700 1 --2- 172.21.15.116:0/494105626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9090100d80 0x7f9090199030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.085+0000 7f9094e00700 1 --2- 172.21.15.116:0/494105626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9090102910 0x7f90901a0590 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.085+0000 7f9094e00700 1 --2- 172.21.15.116:0/494105626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9090101f70 0x7f9090199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.085+0000 7f9094e00700 1 -- 172.21.15.116:0/494105626 >> 172.21.15.116:0/494105626 conn(0x7f90900fc2f0 msgr2=0x7f9090072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.086+0000 7f9094e00700 1 -- 172.21.15.116:0/494105626 shutdown_connections 2024-03-30T00:57:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.086+0000 7f9094e00700 1 -- 172.21.15.116:0/494105626 wait complete. 2024-03-30T00:57:49.112 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:57:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.299+0000 7fb283c00700 1 -- 172.21.15.116:0/3349032692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c073810 msgr2=0x7fb27c073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.299+0000 7fb283c00700 1 --2- 172.21.15.116:0/3349032692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c073810 0x7fb27c073c90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb268009990 tx=0x7fb268009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.300+0000 7fb283c00700 1 -- 172.21.15.116:0/3349032692 shutdown_connections 2024-03-30T00:57:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.300+0000 7fb283c00700 1 --2- 172.21.15.116:0/3349032692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb27c074df0 0x7fb27c073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.300+0000 7fb283c00700 1 --2- 172.21.15.116:0/3349032692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb27c108b40 0x7fb27c10afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.300+0000 7fb283c00700 1 --2- 172.21.15.116:0/3349032692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c073810 0x7fb27c073c90 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.300+0000 7fb283c00700 1 -- 172.21.15.116:0/3349032692 >> 172.21.15.116:0/3349032692 conn(0x7fb27c0fc1f0 msgr2=0x7fb27c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.300+0000 7fb283c00700 1 -- 172.21.15.116:0/3349032692 shutdown_connections 2024-03-30T00:57:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.300+0000 7fb283c00700 1 -- 172.21.15.116:0/3349032692 wait complete. 2024-03-30T00:57:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.301+0000 7fb283c00700 1 Processor -- start 2024-03-30T00:57:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.301+0000 7fb283c00700 1 -- start start 2024-03-30T00:57:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.301+0000 7fb283c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb27c073810 0x7fb27c198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.301+0000 7fb283c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb27c074df0 0x7fb27c1993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.301+0000 7fb283c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c108b40 0x7fb27c1a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.301+0000 7fb283c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c1a0990 con 0x7fb27c108b40 2024-03-30T00:57:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.301+0000 7fb283c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c1a0ad0 con 0x7fb27c073810 2024-03-30T00:57:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.301+0000 7fb283c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c1a0c60 con 0x7fb27c074df0 2024-03-30T00:57:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.301+0000 7fb281c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c108b40 0x7fb27c1a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.301+0000 7fb281c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c108b40 0x7fb27c1a03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58654/0 (socket says 172.21.15.116:58654) 2024-03-30T00:57:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.301+0000 7fb281200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb27c073810 0x7fb27c198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.301+0000 7fb281c00700 1 -- 172.21.15.116:0/2176568122 learned_addr learned my addr 172.21.15.116:0/2176568122 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.301+0000 7fb27be00700 1 --2- 172.21.15.116:0/2176568122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb27c074df0 0x7fb27c1993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.302+0000 7fb281c00700 1 -- 172.21.15.116:0/2176568122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb27c074df0 msgr2=0x7fb27c1993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.302+0000 7fb281c00700 1 --2- 172.21.15.116:0/2176568122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb27c074df0 0x7fb27c1993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.302+0000 7fb281c00700 1 -- 172.21.15.116:0/2176568122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb27c073810 msgr2=0x7fb27c198e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.302+0000 7fb281c00700 1 --2- 172.21.15.116:0/2176568122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb27c073810 0x7fb27c198e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.302+0000 7fb281c00700 1 -- 172.21.15.116:0/2176568122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2680095f0 con 0x7fb27c108b40 2024-03-30T00:57:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.302+0000 7fb27be00700 1 --2- 172.21.15.116:0/2176568122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb27c074df0 0x7fb27c1993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.302+0000 7fb281c00700 1 --2- 172.21.15.116:0/2176568122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c108b40 0x7fb27c1a03c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb274007ee0 tx=0x7fb27400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.302+0000 7fb279600700 1 -- 172.21.15.116:0/2176568122 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb274017070 con 0x7fb27c108b40 2024-03-30T00:57:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.302+0000 7fb279600700 1 -- 172.21.15.116:0/2176568122 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb27400f040 con 0x7fb27c108b40 2024-03-30T00:57:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.302+0000 7fb283c00700 1 -- 172.21.15.116:0/2176568122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27c1a0f40 con 0x7fb27c108b40 2024-03-30T00:57:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.302+0000 7fb279600700 1 -- 172.21.15.116:0/2176568122 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb274005580 con 0x7fb27c108b40 2024-03-30T00:57:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.302+0000 7fb283c00700 1 -- 172.21.15.116:0/2176568122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27c1a1410 con 0x7fb27c108b40 2024-03-30T00:57:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.304+0000 7fb283c00700 1 -- 172.21.15.116:0/2176568122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27c107900 con 0x7fb27c108b40 2024-03-30T00:57:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.306+0000 7fb279600700 1 -- 172.21.15.116:0/2176568122 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fb27400db20 con 0x7fb27c108b40 2024-03-30T00:57:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.307+0000 7fb279600700 1 --2- 172.21.15.116:0/2176568122 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb2640a11a0 0x7fb2640a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.307+0000 7fb279600700 1 -- 172.21.15.116:0/2176568122 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb2740cc270 con 0x7fb27c108b40 2024-03-30T00:57:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.307+0000 7fb281200700 1 --2- 172.21.15.116:0/2176568122 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb2640a11a0 0x7fb2640a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.308+0000 7fb281200700 1 --2- 172.21.15.116:0/2176568122 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb2640a11a0 0x7fb2640a3660 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb270009c60 tx=0x7fb270009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.311+0000 7fb279600700 1 -- 172.21.15.116:0/2176568122 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb2740956f0 con 0x7fb27c108b40 2024-03-30T00:57:49.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.563+0000 7fb283c00700 1 -- 172.21.15.116:0/2176568122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb27c067a10 con 0x7fb27c108b40 2024-03-30T00:57:49.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.564+0000 7fb279600700 1 -- 172.21.15.116:0/2176568122 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10886 (secure 0 0 0) 0x7fb274098d10 con 0x7fb27c108b40 2024-03-30T00:57:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:49.571 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:57:40.442321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T00:57:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.566+0000 7fb283c00700 1 -- 172.21.15.116:0/2176568122 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb2640a11a0 msgr2=0x7fb2640a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.566+0000 7fb283c00700 1 --2- 172.21.15.116:0/2176568122 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb2640a11a0 0x7fb2640a3660 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb270009c60 tx=0x7fb270009260 comp rx=0 tx=0).stop 2024-03-30T00:57:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.566+0000 7fb283c00700 1 -- 172.21.15.116:0/2176568122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c108b40 msgr2=0x7fb27c1a03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.566+0000 7fb283c00700 1 --2- 172.21.15.116:0/2176568122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c108b40 0x7fb27c1a03c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb274007ee0 tx=0x7fb27400d3b0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.567+0000 7fb283c00700 1 -- 172.21.15.116:0/2176568122 shutdown_connections 2024-03-30T00:57:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.567+0000 7fb283c00700 1 --2- 172.21.15.116:0/2176568122 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb2640a11a0 0x7fb2640a3660 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.567+0000 7fb283c00700 1 --2- 172.21.15.116:0/2176568122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c108b40 0x7fb27c1a03c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.567+0000 7fb283c00700 1 --2- 172.21.15.116:0/2176568122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb27c073810 0x7fb27c198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.567+0000 7fb283c00700 1 --2- 172.21.15.116:0/2176568122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb27c074df0 0x7fb27c1993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.567+0000 7fb283c00700 1 -- 172.21.15.116:0/2176568122 >> 172.21.15.116:0/2176568122 conn(0x7fb27c0fc1f0 msgr2=0x7fb27c104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.567+0000 7fb283c00700 1 -- 172.21.15.116:0/2176568122 shutdown_connections 2024-03-30T00:57:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.567+0000 7fb283c00700 1 -- 172.21.15.116:0/2176568122 wait complete. 2024-03-30T00:57:49.575 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 23 2024-03-30T00:57:49.593 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T00:57:49.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:49 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3157379813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:49.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:49 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/494105626' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.778+0000 7f1bafa00700 1 -- 172.21.15.116:0/3842486252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8074d70 msgr2=0x7f1ba80731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.778+0000 7f1bafa00700 1 --2- 172.21.15.116:0/3842486252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8074d70 0x7f1ba80731c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c009990 tx=0x7f1b9c009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.778+0000 7f1bafa00700 1 -- 172.21.15.116:0/3842486252 shutdown_connections 2024-03-30T00:57:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.778+0000 7f1bafa00700 1 --2- 172.21.15.116:0/3842486252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba8108b70 0x7f1ba810b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.778+0000 7f1bafa00700 1 --2- 172.21.15.116:0/3842486252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8074d70 0x7f1ba80731c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.778+0000 7f1bafa00700 1 --2- 172.21.15.116:0/3842486252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ba8073790 0x7f1ba8073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.778+0000 7f1bafa00700 1 -- 172.21.15.116:0/3842486252 >> 172.21.15.116:0/3842486252 conn(0x7f1ba80fc1f0 msgr2=0x7f1ba80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.778+0000 7f1bafa00700 1 -- 172.21.15.116:0/3842486252 shutdown_connections 2024-03-30T00:57:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.778+0000 7f1bafa00700 1 -- 172.21.15.116:0/3842486252 wait complete. 2024-03-30T00:57:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.779+0000 7f1bafa00700 1 Processor -- start 2024-03-30T00:57:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.780+0000 7f1bafa00700 1 -- start start 2024-03-30T00:57:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.780+0000 7f1bafa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba8073790 0x7f1ba8103fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.780+0000 7f1bafa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ba8074d70 0x7f1ba8104510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.780+0000 7f1bafa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8108b70 0x7f1ba8100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.780+0000 7f1bafa00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba8101300 con 0x7f1ba8108b70 2024-03-30T00:57:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.780+0000 7f1bafa00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba8101470 con 0x7f1ba8074d70 2024-03-30T00:57:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.780+0000 7f1bafa00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba8101630 con 0x7f1ba8073790 2024-03-30T00:57:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.780+0000 7f1bada00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8108b70 0x7f1ba8100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.780+0000 7f1bada00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8108b70 0x7f1ba8100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58670/0 (socket says 172.21.15.116:58670) 2024-03-30T00:57:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.781+0000 7f1bada00700 1 -- 172.21.15.116:0/2492771317 learned_addr learned my addr 172.21.15.116:0/2492771317 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.781+0000 7f1bad000700 1 --2- 172.21.15.116:0/2492771317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba8073790 0x7f1ba8103fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.781+0000 7f1ba7e00700 1 --2- 172.21.15.116:0/2492771317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ba8074d70 0x7f1ba8104510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.781+0000 7f1bada00700 1 -- 172.21.15.116:0/2492771317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba8073790 msgr2=0x7f1ba8103fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.781+0000 7f1bada00700 1 --2- 172.21.15.116:0/2492771317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba8073790 0x7f1ba8103fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.781+0000 7f1bada00700 1 -- 172.21.15.116:0/2492771317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ba8074d70 msgr2=0x7f1ba8104510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.781+0000 7f1bada00700 1 --2- 172.21.15.116:0/2492771317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ba8074d70 0x7f1ba8104510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.781+0000 7f1bada00700 1 -- 172.21.15.116:0/2492771317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c0095f0 con 0x7f1ba8108b70 2024-03-30T00:57:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.781+0000 7f1bada00700 1 --2- 172.21.15.116:0/2492771317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8108b70 0x7f1ba8100b60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000ef30 tx=0x7f1ba000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.781+0000 7f1ba5600700 1 -- 172.21.15.116:0/2492771317 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0018070 con 0x7f1ba8108b70 2024-03-30T00:57:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.781+0000 7f1ba5600700 1 -- 172.21.15.116:0/2492771317 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ba000f040 con 0x7f1ba8108b70 2024-03-30T00:57:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.781+0000 7f1bafa00700 1 -- 172.21.15.116:0/2492771317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba81a52b0 con 0x7f1ba8108b70 2024-03-30T00:57:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.781+0000 7f1ba5600700 1 -- 172.21.15.116:0/2492771317 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba000cd00 con 0x7f1ba8108b70 2024-03-30T00:57:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.782+0000 7f1bafa00700 1 -- 172.21.15.116:0/2492771317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba81a5790 con 0x7f1ba8108b70 2024-03-30T00:57:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.783+0000 7f1bafa00700 1 -- 172.21.15.116:0/2492771317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba8107980 con 0x7f1ba8108b70 2024-03-30T00:57:49.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.784+0000 7f1ba5600700 1 -- 172.21.15.116:0/2492771317 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f1ba0003a00 con 0x7f1ba8108b70 2024-03-30T00:57:49.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.786+0000 7f1ba5600700 1 --2- 172.21.15.116:0/2492771317 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1b900a1120 0x7f1b900a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:49.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.786+0000 7f1ba5600700 1 -- 172.21.15.116:0/2492771317 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1ba00cc770 con 0x7f1ba8108b70 2024-03-30T00:57:49.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.786+0000 7f1bad000700 1 --2- 172.21.15.116:0/2492771317 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1b900a1120 0x7f1b900a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:49.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.787+0000 7f1bad000700 1 --2- 172.21.15.116:0/2492771317 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1b900a1120 0x7f1b900a35e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c00b2e0 tx=0x7f1b9c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:49.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:49.790+0000 7f1ba5600700 1 -- 172.21.15.116:0/2492771317 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1ba0095c20 con 0x7f1ba8108b70 2024-03-30T00:57:49.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:49 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3157379813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:49.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:49 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/494105626' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:49.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:49 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3157379813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T00:57:49.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:49 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/494105626' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T00:57:50.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.028+0000 7f1bafa00700 1 -- 172.21.15.116:0/2492771317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ba8067a10 con 0x7f1ba8108b70 2024-03-30T00:57:50.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.030+0000 7f1ba5600700 1 -- 172.21.15.116:0/2492771317 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+24556 (secure 0 0 0) 0x7f1ba0099240 con 0x7f1ba8108b70 2024-03-30T00:57:50.037 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:57:50.037 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:56:36.456069+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T00:57:50.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.032+0000 7f1bafa00700 1 -- 172.21.15.116:0/2492771317 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1b900a1120 msgr2=0x7f1b900a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.032+0000 7f1bafa00700 1 --2- 172.21.15.116:0/2492771317 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1b900a1120 0x7f1b900a35e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c00b2e0 tx=0x7f1b9c005fd0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.032+0000 7f1bafa00700 1 -- 172.21.15.116:0/2492771317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8108b70 msgr2=0x7f1ba8100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.032+0000 7f1bafa00700 1 --2- 172.21.15.116:0/2492771317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8108b70 0x7f1ba8100b60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000ef30 tx=0x7f1ba000c3a0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.032+0000 7f1bafa00700 1 -- 172.21.15.116:0/2492771317 shutdown_connections 2024-03-30T00:57:50.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.032+0000 7f1bafa00700 1 --2- 172.21.15.116:0/2492771317 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1b900a1120 0x7f1b900a35e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.032+0000 7f1bafa00700 1 --2- 172.21.15.116:0/2492771317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba8073790 0x7f1ba8103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.032+0000 7f1bafa00700 1 --2- 172.21.15.116:0/2492771317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ba8108b70 0x7f1ba8100b60 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.032+0000 7f1bafa00700 1 --2- 172.21.15.116:0/2492771317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ba8074d70 0x7f1ba8104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.032+0000 7f1bafa00700 1 -- 172.21.15.116:0/2492771317 >> 172.21.15.116:0/2492771317 conn(0x7f1ba80fc1f0 msgr2=0x7f1ba8104ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:50.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.033+0000 7f1bafa00700 1 -- 172.21.15.116:0/2492771317 shutdown_connections 2024-03-30T00:57:50.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.033+0000 7f1bafa00700 1 -- 172.21.15.116:0/2492771317 wait complete. 2024-03-30T00:57:50.058 DEBUG:tasks.fwd_scrub:Yielding 2024-03-30T00:57:50.058 INFO:teuthology.run_tasks:Running task workunit... 2024-03-30T00:57:50.071 INFO:tasks.workunit:Pulling workunits from ref 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T00:57:50.071 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-30T00:57:50.071 INFO:tasks.workunit:timeout=3h 2024-03-30T00:57:50.071 INFO:tasks.workunit:cleanup=True 2024-03-30T00:57:50.072 DEBUG:teuthology.orchestra.run.smithi116:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-30T00:57:50.072 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-30T00:57:50.072 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T00:57:50.072 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T00:57:50.117 INFO:teuthology.orchestra.run.smithi116.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-30T00:57:50.117 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-30T00:57:50.117 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 7ah/122d Inode: 1099511628283 Links: 2 2024-03-30T00:57:50.117 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T00:57:50.117 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:cephfs_t:s0 2024-03-30T00:57:50.117 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-30 00:56:24.118536905 +0000 2024-03-30T00:57:50.117 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-30 00:56:24.118536905 +0000 2024-03-30T00:57:50.118 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-30 00:57:38.927665880 +0000 2024-03-30T00:57:50.118 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-30 00:56:24.118536905 +0000 2024-03-30T00:57:50.118 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-30T00:57:50.118 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-30T00:57:50.236 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T00:57:50.285 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-30T00:57:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.280+0000 7f0f75600700 1 -- 172.21.15.116:0/2359956951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70104140 msgr2=0x7f0f70106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.280+0000 7f0f75600700 1 --2- 172.21.15.116:0/2359956951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70104140 0x7f0f70106530 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c009990 tx=0x7f0f5c009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.280+0000 7f0f75600700 1 -- 172.21.15.116:0/2359956951 shutdown_connections 2024-03-30T00:57:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.280+0000 7f0f75600700 1 --2- 172.21.15.116:0/2359956951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f70101810 0x7f0f70103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.280+0000 7f0f75600700 1 --2- 172.21.15.116:0/2359956951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f70106a70 0x7f0f70108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.280+0000 7f0f75600700 1 --2- 172.21.15.116:0/2359956951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70104140 0x7f0f70106530 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.280+0000 7f0f75600700 1 -- 172.21.15.116:0/2359956951 >> 172.21.15.116:0/2359956951 conn(0x7f0f700fb200 msgr2=0x7f0f700fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.281+0000 7f0f75600700 1 -- 172.21.15.116:0/2359956951 shutdown_connections 2024-03-30T00:57:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.281+0000 7f0f75600700 1 -- 172.21.15.116:0/2359956951 wait complete. 2024-03-30T00:57:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.282+0000 7f0f75600700 1 Processor -- start 2024-03-30T00:57:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.282+0000 7f0f75600700 1 -- start start 2024-03-30T00:57:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.282+0000 7f0f75600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f70101810 0x7f0f70198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.282+0000 7f0f75600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f70104140 0x7f0f701994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.282+0000 7f0f75600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70106a70 0x7f0f701a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.282+0000 7f0f75600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f701a0aa0 con 0x7f0f70106a70 2024-03-30T00:57:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.282+0000 7f0f75600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f701a0be0 con 0x7f0f70101810 2024-03-30T00:57:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.282+0000 7f0f75600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f701a0d70 con 0x7f0f70104140 2024-03-30T00:57:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.282+0000 7f0f6f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70106a70 0x7f0f701a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.282+0000 7f0f6e000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f70104140 0x7f0f701994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.282+0000 7f0f6f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70106a70 0x7f0f701a04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58692/0 (socket says 172.21.15.116:58692) 2024-03-30T00:57:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.282+0000 7f0f6f400700 1 -- 172.21.15.116:0/2131771398 learned_addr learned my addr 172.21.15.116:0/2131771398 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.282+0000 7f0f6ea00700 1 --2- 172.21.15.116:0/2131771398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f70101810 0x7f0f70198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.283+0000 7f0f6e000700 1 -- 172.21.15.116:0/2131771398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f70101810 msgr2=0x7f0f70198f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.283+0000 7f0f6e000700 1 --2- 172.21.15.116:0/2131771398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f70101810 0x7f0f70198f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.283+0000 7f0f6e000700 1 -- 172.21.15.116:0/2131771398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70106a70 msgr2=0x7f0f701a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.283+0000 7f0f6e000700 1 --2- 172.21.15.116:0/2131771398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70106a70 0x7f0f701a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.283+0000 7f0f6e000700 1 -- 172.21.15.116:0/2131771398 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f68009c40 con 0x7f0f70104140 2024-03-30T00:57:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.283+0000 7f0f6e000700 1 --2- 172.21.15.116:0/2131771398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f70104140 0x7f0f701994b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c000c00 tx=0x7f0f5c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.283+0000 7f0f67400700 1 -- 172.21.15.116:0/2131771398 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c01d070 con 0x7f0f70104140 2024-03-30T00:57:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.283+0000 7f0f75600700 1 -- 172.21.15.116:0/2131771398 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5c0095f0 con 0x7f0f70104140 2024-03-30T00:57:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.283+0000 7f0f67400700 1 -- 172.21.15.116:0/2131771398 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f5c021410 con 0x7f0f70104140 2024-03-30T00:57:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.283+0000 7f0f67400700 1 -- 172.21.15.116:0/2131771398 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c017650 con 0x7f0f70104140 2024-03-30T00:57:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.284+0000 7f0f75600700 1 -- 172.21.15.116:0/2131771398 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f701a1230 con 0x7f0f70104140 2024-03-30T00:57:50.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.287+0000 7f0f67400700 1 -- 172.21.15.116:0/2131771398 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f0f5c021580 con 0x7f0f70104140 2024-03-30T00:57:50.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.288+0000 7f0f8913cb80 1 Processor -- start 2024-03-30T00:57:50.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.288+0000 7f0f8913cb80 1 -- start start 2024-03-30T00:57:50.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.288+0000 7f0f67400700 1 --2- 172.21.15.116:0/2131771398 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0f540a1120 0x7f0f540a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.288+0000 7f0f67400700 1 -- 172.21.15.116:0/2131771398 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0f5c0cdf50 con 0x7f0f70104140 2024-03-30T00:57:50.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.288+0000 7f0f8913cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f6166850 0x55c2f6166c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.288+0000 7f0f8913cb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f61671b0 0x55c2f6167a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.288+0000 7f0f8913cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6168050 0x55c2f61684d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.288+0000 7f0f8913cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f60bea10 con 0x55c2f6168050 2024-03-30T00:57:50.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.288+0000 7f0f8913cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f60bc3d0 con 0x55c2f6166850 2024-03-30T00:57:50.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.288+0000 7f0f8913cb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f60b7cc0 con 0x55c2f61671b0 2024-03-30T00:57:50.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.289+0000 7f0f6f400700 1 --2- 172.21.15.116:0/2131771398 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0f540a1120 0x7f0f540a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.289+0000 7f0f6e000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f61671b0 0x55c2f6167a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.289+0000 7f0f6e000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f61671b0 0x55c2f6167a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:57678/0 (socket says 172.21.15.116:57678) 2024-03-30T00:57:50.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.289+0000 7f0f6e000700 1 -- 172.21.15.116:0/1837787088 learned_addr learned my addr 172.21.15.116:0/1837787088 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:50.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.289+0000 7f0f6f400700 1 --2- 172.21.15.116:0/1837787088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6168050 0x55c2f61684d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.289+0000 7f0f6ea00700 1 --2- 172.21.15.116:0/1837787088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f6166850 0x55c2f6166c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.289+0000 7f0f6e000700 1 -- 172.21.15.116:0/1837787088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f6166850 msgr2=0x55c2f6166c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.289+0000 7f0f6e000700 1 --2- 172.21.15.116:0/1837787088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f6166850 0x55c2f6166c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.289+0000 7f0f6e000700 1 -- 172.21.15.116:0/1837787088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6168050 msgr2=0x55c2f61684d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.289+0000 7f0f6e000700 1 --2- 172.21.15.116:0/1837787088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6168050 0x55c2f61684d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.289+0000 7f0f6e000700 1 -- 172.21.15.116:0/1837787088 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2f5f39430 con 0x55c2f61671b0 2024-03-30T00:57:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.289+0000 7f0f6f400700 1 --2- 172.21.15.116:0/2131771398 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0f540a1120 0x7f0f540a35e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0f68009690 tx=0x7f0f68009430 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.290+0000 7f0f6e000700 1 --2- 172.21.15.116:0/1837787088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f61671b0 0x55c2f6167a70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c09adc0 tx=0x7f0f5c009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76e6294525a840cc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.290+0000 7f0f5b400700 1 -- 172.21.15.116:0/1837787088 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c09b7c0 con 0x55c2f61671b0 2024-03-30T00:57:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.290+0000 7f0f5b400700 1 -- 172.21.15.116:0/1837787088 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f5c09b0c0 con 0x55c2f61671b0 2024-03-30T00:57:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.290+0000 7f0f5b400700 1 -- 172.21.15.116:0/1837787088 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c09b580 con 0x55c2f61671b0 2024-03-30T00:57:50.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.290+0000 7f0f8913cb80 1 -- 172.21.15.116:0/1837787088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f61671b0 msgr2=0x55c2f6167a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.290+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/1837787088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f61671b0 0x55c2f6167a70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c09adc0 tx=0x7f0f5c009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.291+0000 7f0f8913cb80 1 -- 172.21.15.116:0/1837787088 shutdown_connections 2024-03-30T00:57:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.291+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/1837787088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6168050 0x55c2f61684d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.291+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/1837787088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f6166850 0x55c2f6166c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.291+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/1837787088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f61671b0 0x55c2f6167a70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.291+0000 7f0f8913cb80 1 -- 172.21.15.116:0/1837787088 >> 172.21.15.116:0/1837787088 conn(0x55c2f60c5720 msgr2=0x55c2f6161a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.291+0000 7f0f8913cb80 1 -- 172.21.15.116:0/1837787088 shutdown_connections 2024-03-30T00:57:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.291+0000 7f0f8913cb80 1 -- 172.21.15.116:0/1837787088 wait complete. 2024-03-30T00:57:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.291+0000 7f0f8913cb80 1 -- start start 2024-03-30T00:57:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.291+0000 7f0f8913cb80 1 Processor -- start 2024-03-30T00:57:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.292+0000 7f0f8913cb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6166850 0x55c2f61702f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.292+0000 7f0f8913cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 0x55c2f6170830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.292+0000 7f0f8913cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6168050 0x55c2f6177870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.292+0000 7f0f8913cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f6053e00 con 0x55c2f6168050 2024-03-30T00:57:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.292+0000 7f0f8913cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f6053aa0 con 0x55c2f61671b0 2024-03-30T00:57:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.292+0000 7f0f8913cb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f5f3b720 con 0x55c2f6166850 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.292+0000 7f0f8913cb80 10 client.0 _ll_register_callbacks cb 0x55c2f605d4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.292+0000 7f0f6f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6168050 0x55c2f6177870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.292+0000 7f0f6ea00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6166850 0x55c2f61702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.292+0000 7f0f6ea00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6166850 0x55c2f61702f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:57688/0 (socket says 172.21.15.116:57688) 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.292+0000 7f0f6f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6168050 0x55c2f6177870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58714/0 (socket says 172.21.15.116:58714) 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.292+0000 7f0f6ea00700 1 -- 172.21.15.116:0/3498061251 learned_addr learned my addr 172.21.15.116:0/3498061251 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.292+0000 7f0f6e000700 1 --2- 172.21.15.116:0/3498061251 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 0x55c2f6170830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.293+0000 7f0f6ea00700 1 -- 172.21.15.116:0/3498061251 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 msgr2=0x55c2f6170830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.293+0000 7f0f6ea00700 1 --2- 172.21.15.116:0/3498061251 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 0x55c2f6170830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.293+0000 7f0f6ea00700 1 -- 172.21.15.116:0/3498061251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6168050 msgr2=0x55c2f6177870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.293+0000 7f0f6ea00700 1 --2- 172.21.15.116:0/3498061251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6168050 0x55c2f6177870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.293+0000 7f0f6ea00700 1 -- 172.21.15.116:0/3498061251 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f701a1230 con 0x55c2f6166850 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.293+0000 7f0f6e000700 1 --2- 172.21.15.116:0/3498061251 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 0x55c2f6170830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.293+0000 7f0f6f400700 1 --2- 172.21.15.116:0/3498061251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6168050 0x55c2f6177870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.293+0000 7f0f6ea00700 1 --2- 172.21.15.116:0/3498061251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6166850 0x55c2f61702f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0f60011830 tx=0x7f0f60011b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=775b1e58a0dfd1c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.293+0000 7f0f4fe00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T00:57:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.293+0000 7f0f4fe00700 1 -- 172.21.15.116:0/3498061251 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60016070 con 0x55c2f6166850 2024-03-30T00:57:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.293+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3498061251 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c2f6078400 con 0x55c2f6166850 2024-03-30T00:57:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.293+0000 7f0f4fe00700 1 -- 172.21.15.116:0/3498061251 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0f6000f890 con 0x55c2f6166850 2024-03-30T00:57:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.294+0000 7f0f4fe00700 1 -- 172.21.15.116:0/3498061251 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f6000fa50 con 0x55c2f6166850 2024-03-30T00:57:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.294+0000 7f0f8913cb80 10 client.25501 fetch_fsmap learned FSMap version 23 2024-03-30T00:57:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.294+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3498061251 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c2f5ee5be0 con 0x55c2f6166850 2024-03-30T00:57:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.294+0000 7f0f4fe00700 1 -- 172.21.15.116:0/3498061251 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c2f6078400 con 0x55c2f6166850 2024-03-30T00:57:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.294+0000 7f0f4fe00700 1 -- 172.21.15.116:0/3498061251 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0f6001acc0 con 0x55c2f6166850 2024-03-30T00:57:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.294+0000 7f0f8913cb80 10 client.25501 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:57:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.294+0000 7f0f8913cb80 10 client.25501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:57:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.294+0000 7f0f8913cb80 20 client.25501 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.294+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/3498061251 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c2f617c100 0x55c2f617e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.294+0000 7f0f8913cb80 4 client.25501 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:57:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.294+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3498061251 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2f60b0780 con 0x55c2f617c100 2024-03-30T00:57:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.295+0000 7f0f6e000700 1 --2- 172.21.15.116:0/3498061251 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c2f617c100 0x55c2f617e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.295+0000 7f0f4fe00700 10 client.25501 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.295+0000 7f0f6e000700 1 --2- 172.21.15.116:0/3498061251 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c2f617c100 0x55c2f617e520 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89249a97dfb2d028 server_cookie=67b74105c1404eb8 in_seq=0 out_seq=0 2024-03-30T00:57:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.296+0000 7f0f4fe00700 1 -- 172.21.15.116:0/3498061251 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c2f60b0780 con 0x55c2f617c100 2024-03-30T00:57:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.296+0000 7f0f4fe00700 10 client.25501 handle_command_reply: tid=0 2024-03-30T00:57:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.296+0000 7f0f8913cb80 1 client.25501 shutdown 2024-03-30T00:57:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.296+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3498061251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6166850 msgr2=0x55c2f61702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.296+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/3498061251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6166850 0x55c2f61702f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0f60011830 tx=0x7f0f60011b40 comp rx=0 tx=0).stop 2024-03-30T00:57:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.296+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3498061251 shutdown_connections 2024-03-30T00:57:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.296+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/3498061251 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c2f617c100 0x55c2f617e520 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.296+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/3498061251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6166850 0x55c2f61702f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.296+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/3498061251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6168050 0x55c2f6177870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.296+0000 7f0f4fe00700 0 client.25501 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.296+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/3498061251 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 0x55c2f6170830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.296+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3498061251 >> 172.21.15.116:0/3498061251 conn(0x55c2f60c5720 msgr2=0x55c2f61651f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.296+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3498061251 shutdown_connections 2024-03-30T00:57:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.296+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3498061251 wait complete. 2024-03-30T00:57:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.297+0000 7f0f8913cb80 20 client.25501 trim_cache size 0 max 16384 2024-03-30T00:57:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f8913cb80 1 Processor -- start 2024-03-30T00:57:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f8913cb80 1 -- start start 2024-03-30T00:57:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f8913cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6166850 0x55c2f61707b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f8913cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 0x55c2f6177790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f8913cb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6168050 0x55c2f616aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f8913cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f6167ed0 con 0x55c2f6166850 2024-03-30T00:57:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f8913cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f616f310 con 0x55c2f61671b0 2024-03-30T00:57:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f8913cb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f616fe80 con 0x55c2f6168050 2024-03-30T00:57:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f6ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6166850 0x55c2f61707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f6ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6166850 0x55c2f61707b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58720/0 (socket says 172.21.15.116:58720) 2024-03-30T00:57:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f6ea00700 1 -- 172.21.15.116:0/185078880 learned_addr learned my addr 172.21.15.116:0/185078880 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f6e000700 1 --2- 172.21.15.116:0/185078880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 0x55c2f6177790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f6f400700 1 --2- 172.21.15.116:0/185078880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6168050 0x55c2f616aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f6ea00700 1 -- 172.21.15.116:0/185078880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6168050 msgr2=0x55c2f616aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f6ea00700 1 --2- 172.21.15.116:0/185078880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6168050 0x55c2f616aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f6ea00700 1 -- 172.21.15.116:0/185078880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 msgr2=0x55c2f6177790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f6ea00700 1 --2- 172.21.15.116:0/185078880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 0x55c2f6177790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f6ea00700 1 -- 172.21.15.116:0/185078880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2f615b2c0 con 0x55c2f6166850 2024-03-30T00:57:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.315+0000 7f0f6ea00700 1 --2- 172.21.15.116:0/185078880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6166850 0x55c2f61707b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0f60011800 tx=0x7f0f60019600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1bd3453d27b31da server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.316+0000 7f0f5be00700 1 -- 172.21.15.116:0/185078880 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60016070 con 0x55c2f6166850 2024-03-30T00:57:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.316+0000 7f0f5be00700 1 -- 172.21.15.116:0/185078880 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f600196d0 con 0x55c2f6166850 2024-03-30T00:57:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.316+0000 7f0f5be00700 1 -- 172.21.15.116:0/185078880 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60005110 con 0x55c2f6166850 2024-03-30T00:57:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.316+0000 7f0f8913cb80 1 -- 172.21.15.116:0/185078880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6166850 msgr2=0x55c2f61707b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.316+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/185078880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6166850 0x55c2f61707b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0f60011800 tx=0x7f0f60019600 comp rx=0 tx=0).stop 2024-03-30T00:57:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.316+0000 7f0f8913cb80 1 -- 172.21.15.116:0/185078880 shutdown_connections 2024-03-30T00:57:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.316+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/185078880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6168050 0x55c2f616aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.316+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/185078880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6166850 0x55c2f61707b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.316+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/185078880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 0x55c2f6177790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.316+0000 7f0f8913cb80 1 -- 172.21.15.116:0/185078880 >> 172.21.15.116:0/185078880 conn(0x55c2f60c5720 msgr2=0x55c2f61700a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.317+0000 7f0f8913cb80 1 -- 172.21.15.116:0/185078880 shutdown_connections 2024-03-30T00:57:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.317+0000 7f0f8913cb80 1 -- 172.21.15.116:0/185078880 wait complete. 2024-03-30T00:57:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.317+0000 7f0f8913cb80 1 -- start start 2024-03-30T00:57:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.317+0000 7f0f8913cb80 1 Processor -- start 2024-03-30T00:57:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.317+0000 7f0f8913cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6166850 0x55c2f6176f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f8913cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 0x55c2f6177450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f8913cb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6168050 0x55c2f61d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f8913cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f61d0b20 con 0x55c2f6166850 2024-03-30T00:57:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f8913cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f61d0c60 con 0x55c2f61671b0 2024-03-30T00:57:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f8913cb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f61d0df0 con 0x55c2f6168050 2024-03-30T00:57:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f8913cb80 10 client.0 _ll_register_callbacks cb 0x55c2f605d4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f6ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6166850 0x55c2f6176f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f6ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6166850 0x55c2f6176f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58736/0 (socket says 172.21.15.116:58736) 2024-03-30T00:57:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f6ea00700 1 -- 172.21.15.116:0/3596958436 learned_addr learned my addr 172.21.15.116:0/3596958436 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f6f400700 1 --2- 172.21.15.116:0/3596958436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6168050 0x55c2f61d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f6e000700 1 --2- 172.21.15.116:0/3596958436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 0x55c2f6177450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f6ea00700 1 -- 172.21.15.116:0/3596958436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6168050 msgr2=0x55c2f61d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f6ea00700 1 --2- 172.21.15.116:0/3596958436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6168050 0x55c2f61d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f6ea00700 1 -- 172.21.15.116:0/3596958436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 msgr2=0x55c2f6177450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f6ea00700 1 --2- 172.21.15.116:0/3596958436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 0x55c2f6177450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f6ea00700 1 -- 172.21.15.116:0/3596958436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2f5ee5be0 con 0x55c2f6166850 2024-03-30T00:57:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f6ea00700 1 --2- 172.21.15.116:0/3596958436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6166850 0x55c2f6176f10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0f60011550 tx=0x7f0f6000fb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce30598657dfd0c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.318+0000 7f0f4fe00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:57:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.319+0000 7f0f4fe00700 1 -- 172.21.15.116:0/3596958436 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60016070 con 0x55c2f6166850 2024-03-30T00:57:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.319+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3596958436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f5c09b0c0 con 0x55c2f6166850 2024-03-30T00:57:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.319+0000 7f0f4fe00700 1 -- 172.21.15.116:0/3596958436 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0f60000e40 con 0x55c2f6166850 2024-03-30T00:57:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.319+0000 7f0f4fe00700 1 -- 172.21.15.116:0/3596958436 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f6000f9d0 con 0x55c2f6166850 2024-03-30T00:57:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.319+0000 7f0f8913cb80 10 client.15468 fetch_fsmap learned FSMap version 23 2024-03-30T00:57:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.319+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3596958436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c2f615a8f0 con 0x55c2f6166850 2024-03-30T00:57:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.319+0000 7f0f4fe00700 1 -- 172.21.15.116:0/3596958436 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f5c09b0c0 con 0x55c2f6166850 2024-03-30T00:57:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.319+0000 7f0f4fe00700 1 -- 172.21.15.116:0/3596958436 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0f60003710 con 0x55c2f6166850 2024-03-30T00:57:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.319+0000 7f0f8913cb80 10 client.15468 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:57:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.319+0000 7f0f8913cb80 10 client.15468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:57:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.319+0000 7f0f8913cb80 20 client.15468 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.319+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/3596958436 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c2f61d2540 0x55c2f61d4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.319+0000 7f0f8913cb80 4 client.15468 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T00:57:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.319+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3596958436 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c2f5f06d70 con 0x55c2f61d2540 2024-03-30T00:57:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.320+0000 7f0f6e000700 1 --2- 172.21.15.116:0/3596958436 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c2f61d2540 0x55c2f61d4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.320+0000 7f0f6e000700 1 --2- 172.21.15.116:0/3596958436 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c2f61d2540 0x55c2f61d4960 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=def1c36d28e6fb54 server_cookie=551dc498d6367d22 in_seq=0 out_seq=0 2024-03-30T00:57:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.320+0000 7f0f4fe00700 10 client.15468 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.321+0000 7f0f4fe00700 1 -- 172.21.15.116:0/3596958436 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c2f5f06d70 con 0x55c2f61d2540 2024-03-30T00:57:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.321+0000 7f0f4fe00700 10 client.15468 handle_command_reply: tid=0 2024-03-30T00:57:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.321+0000 7f0f8913cb80 1 client.15468 shutdown 2024-03-30T00:57:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.321+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3596958436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6166850 msgr2=0x55c2f6176f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.321+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/3596958436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6166850 0x55c2f6176f10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0f60011550 tx=0x7f0f6000fb80 comp rx=0 tx=0).stop 2024-03-30T00:57:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.322+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3596958436 shutdown_connections 2024-03-30T00:57:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.322+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/3596958436 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c2f61d2540 0x55c2f61d4960 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.322+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/3596958436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c2f6168050 0x55c2f61d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.322+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/3596958436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2f6166850 0x55c2f6176f10 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.322+0000 7f0f4fe00700 0 client.15468 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.322+0000 7f0f8913cb80 1 --2- 172.21.15.116:0/3596958436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2f61671b0 0x55c2f6177450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.322+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3596958436 >> 172.21.15.116:0/3596958436 conn(0x55c2f60c5720 msgr2=0x55c2f616aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.322+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3596958436 shutdown_connections 2024-03-30T00:57:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.322+0000 7f0f8913cb80 1 -- 172.21.15.116:0/3596958436 wait complete. 2024-03-30T00:57:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.322+0000 7f0f8913cb80 20 client.15468 trim_cache size 0 max 16384 2024-03-30T00:57:50.338 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:57:50.338 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-30T00:57:50.338 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "4690422e-0773-41b7-8ab9-35a0cbe0c0d4", 2024-03-30T00:57:50.338 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-30T00:57:50.339 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:57:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.322+0000 7f0f75600700 1 -- 172.21.15.116:0/2131771398 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0f540a1120 msgr2=0x7f0f540a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.322+0000 7f0f75600700 1 --2- 172.21.15.116:0/2131771398 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0f540a1120 0x7f0f540a35e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0f68009690 tx=0x7f0f68009430 comp rx=0 tx=0).stop 2024-03-30T00:57:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.322+0000 7f0f75600700 1 -- 172.21.15.116:0/2131771398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f70104140 msgr2=0x7f0f701994b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.322+0000 7f0f75600700 1 --2- 172.21.15.116:0/2131771398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f70104140 0x7f0f701994b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c000c00 tx=0x7f0f5c009580 comp rx=0 tx=0).stop 2024-03-30T00:57:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.323+0000 7f0f75600700 1 -- 172.21.15.116:0/2131771398 shutdown_connections 2024-03-30T00:57:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.323+0000 7f0f75600700 1 --2- 172.21.15.116:0/2131771398 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0f540a1120 0x7f0f540a35e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.323+0000 7f0f75600700 1 --2- 172.21.15.116:0/2131771398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70106a70 0x7f0f701a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.323+0000 7f0f75600700 1 --2- 172.21.15.116:0/2131771398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f70101810 0x7f0f70198f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.323+0000 7f0f75600700 1 --2- 172.21.15.116:0/2131771398 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f70104140 0x7f0f701994b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.323+0000 7f0f75600700 1 -- 172.21.15.116:0/2131771398 >> 172.21.15.116:0/2131771398 conn(0x7f0f700fb200 msgr2=0x7f0f70101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.323+0000 7f0f75600700 1 -- 172.21.15.116:0/2131771398 shutdown_connections 2024-03-30T00:57:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.323+0000 7f0f75600700 1 -- 172.21.15.116:0/2131771398 wait complete. 2024-03-30T00:57:50.349 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T00:57:50.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.550+0000 7fa5d7200700 1 -- 172.21.15.116:0/778251505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d0074df0 msgr2=0x7fa5d0073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.550+0000 7fa5d7200700 1 --2- 172.21.15.116:0/778251505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d0074df0 0x7fa5d0073240 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0009990 tx=0x7fa5c0009ca0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.550+0000 7fa5d7200700 1 -- 172.21.15.116:0/778251505 shutdown_connections 2024-03-30T00:57:50.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.550+0000 7fa5d7200700 1 --2- 172.21.15.116:0/778251505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5d0108c40 0x7fa5d010b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.550+0000 7fa5d7200700 1 --2- 172.21.15.116:0/778251505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d0074df0 0x7fa5d0073240 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.550+0000 7fa5d7200700 1 --2- 172.21.15.116:0/778251505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5d0073810 0x7fa5d0073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.550+0000 7fa5d7200700 1 -- 172.21.15.116:0/778251505 >> 172.21.15.116:0/778251505 conn(0x7fa5d00fc2f0 msgr2=0x7fa5d00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:50.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.550+0000 7fa5d7200700 1 -- 172.21.15.116:0/778251505 shutdown_connections 2024-03-30T00:57:50.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.550+0000 7fa5d7200700 1 -- 172.21.15.116:0/778251505 wait complete. 2024-03-30T00:57:50.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.551+0000 7fa5d7200700 1 Processor -- start 2024-03-30T00:57:50.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.551+0000 7fa5d7200700 1 -- start start 2024-03-30T00:57:50.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.551+0000 7fa5d7200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5d0073810 0x7fa5d0198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.551+0000 7fa5d7200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d0074df0 0x7fa5d0199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.551+0000 7fa5d7200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5d0108c40 0x7fa5d01a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.551+0000 7fa5d7200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d01a0a50 con 0x7fa5d0074df0 2024-03-30T00:57:50.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.551+0000 7fa5d7200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d01a0b90 con 0x7fa5d0073810 2024-03-30T00:57:50.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.551+0000 7fa5d7200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d01a0d20 con 0x7fa5d0108c40 2024-03-30T00:57:50.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.552+0000 7fa5cf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d0074df0 0x7fa5d0199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.552+0000 7fa5cf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d0074df0 0x7fa5d0199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58752/0 (socket says 172.21.15.116:58752) 2024-03-30T00:57:50.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.552+0000 7fa5cf400700 1 -- 172.21.15.116:0/824304339 learned_addr learned my addr 172.21.15.116:0/824304339 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:50.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.552+0000 7fa5cfe00700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5d0073810 0x7fa5d0198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.552+0000 7fa5d5200700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5d0108c40 0x7fa5d01a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.552+0000 7fa5cf400700 1 -- 172.21.15.116:0/824304339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5d0108c40 msgr2=0x7fa5d01a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.552+0000 7fa5cf400700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5d0108c40 0x7fa5d01a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.552+0000 7fa5cf400700 1 -- 172.21.15.116:0/824304339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5d0073810 msgr2=0x7fa5d0198f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.552+0000 7fa5cf400700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5d0073810 0x7fa5d0198f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.552+0000 7fa5cf400700 1 -- 172.21.15.116:0/824304339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c8009c40 con 0x7fa5d0074df0 2024-03-30T00:57:50.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.552+0000 7fa5cfe00700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5d0073810 0x7fa5d0198f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:57:50.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.552+0000 7fa5cf400700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d0074df0 0x7fa5d0199460 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc00d980 tx=0x7fa5bc00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:50.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.553+0000 7fa5ccc00700 1 -- 172.21.15.116:0/824304339 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc00bb60 con 0x7fa5d0074df0 2024-03-30T00:57:50.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.553+0000 7fa5ccc00700 1 -- 172.21.15.116:0/824304339 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5bc007620 con 0x7fa5d0074df0 2024-03-30T00:57:50.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.553+0000 7fa5ccc00700 1 -- 172.21.15.116:0/824304339 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc00f3c0 con 0x7fa5d0074df0 2024-03-30T00:57:50.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.553+0000 7fa5d7200700 1 -- 172.21.15.116:0/824304339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c00095f0 con 0x7fa5d0074df0 2024-03-30T00:57:50.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.553+0000 7fa5d7200700 1 -- 172.21.15.116:0/824304339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d01a1240 con 0x7fa5d0074df0 2024-03-30T00:57:50.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.556+0000 7fa5ccc00700 1 -- 172.21.15.116:0/824304339 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fa5bc010460 con 0x7fa5d0074df0 2024-03-30T00:57:50.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.557+0000 7fa5eaddeb80 1 Processor -- start 2024-03-30T00:57:50.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.557+0000 7fa5eaddeb80 1 -- start start 2024-03-30T00:57:50.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.557+0000 7fa5ccc00700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5b40a1120 0x7fa5b40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.558+0000 7fa5ccc00700 1 -- 172.21.15.116:0/824304339 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa5bc0ccde0 con 0x7fa5d0074df0 2024-03-30T00:57:50.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.558+0000 7fa5eaddeb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfdfe0 0x561200bfe400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.558+0000 7fa5eaddeb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200bff200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.558+0000 7fa5eaddeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bff7e0 0x561200bffc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.558+0000 7fa5eaddeb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561200b4d990 con 0x561200bfe940 2024-03-30T00:57:50.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.558+0000 7fa5d5200700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5b40a1120 0x7fa5b40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.558+0000 7fa5eaddeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561200b4b350 con 0x561200bff7e0 2024-03-30T00:57:50.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.558+0000 7fa5eaddeb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561200b46c40 con 0x561200bfdfe0 2024-03-30T00:57:50.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.558+0000 7fa5cfe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfdfe0 0x561200bfe400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.558+0000 7fa5cfe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfdfe0 0x561200bfe400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:57730/0 (socket says 172.21.15.116:57730) 2024-03-30T00:57:50.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.558+0000 7fa5cfe00700 1 -- 172.21.15.116:0/2754760871 learned_addr learned my addr 172.21.15.116:0/2754760871 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:50.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.558+0000 7fa5d5200700 1 --2- 172.21.15.116:0/2754760871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bff7e0 0x561200bffc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.558+0000 7fa5cf400700 1 --2- 172.21.15.116:0/2754760871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200bff200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.559+0000 7fa5cfe00700 1 -- 172.21.15.116:0/2754760871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bff7e0 msgr2=0x561200bffc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.559+0000 7fa5cfe00700 1 --2- 172.21.15.116:0/2754760871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bff7e0 0x561200bffc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.559+0000 7fa5cfe00700 1 -- 172.21.15.116:0/2754760871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 msgr2=0x561200bff200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.559+0000 7fa5cfe00700 1 --2- 172.21.15.116:0/2754760871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200bff200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.559+0000 7fa5cfe00700 1 -- 172.21.15.116:0/2754760871 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612009c83b0 con 0x561200bfdfe0 2024-03-30T00:57:50.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.559+0000 7fa5d5200700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5b40a1120 0x7fa5b40a35e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8009fd0 tx=0x7fa5c800fde0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:50.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.559+0000 7fa5cfe00700 1 --2- 172.21.15.116:0/2754760871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfdfe0 0x561200bfe400 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0015000 tx=0x7fa5c0005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=946487c9568dc081 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:50.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.559+0000 7fa5c5600700 1 -- 172.21.15.116:0/2754760871 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c000be60 con 0x561200bfdfe0 2024-03-30T00:57:50.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.559+0000 7fa5c5600700 1 -- 172.21.15.116:0/2754760871 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5c001e040 con 0x561200bfdfe0 2024-03-30T00:57:50.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.559+0000 7fa5c5600700 1 -- 172.21.15.116:0/2754760871 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0004ba0 con 0x561200bfdfe0 2024-03-30T00:57:50.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.560+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/2754760871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfdfe0 msgr2=0x561200bfe400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.560+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/2754760871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfdfe0 0x561200bfe400 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0015000 tx=0x7fa5c0005f00 comp rx=0 tx=0).stop 2024-03-30T00:57:50.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.560+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/2754760871 shutdown_connections 2024-03-30T00:57:50.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.560+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/2754760871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfdfe0 0x561200bfe400 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.560+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/2754760871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bff7e0 0x561200bffc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.560+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/2754760871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200bff200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.560+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/2754760871 >> 172.21.15.116:0/2754760871 conn(0x561200b546a0 msgr2=0x561200bf09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:50.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.560+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/2754760871 shutdown_connections 2024-03-30T00:57:50.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.560+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/2754760871 wait complete. 2024-03-30T00:57:50.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.561+0000 7fa5eaddeb80 1 -- start start 2024-03-30T00:57:50.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.561+0000 7fa5eaddeb80 1 Processor -- start 2024-03-30T00:57:50.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5eaddeb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfdfe0 0x561200c07eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5eaddeb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200c083f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5eaddeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bff7e0 0x561200c0f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5eaddeb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561200ae2d80 con 0x561200bfe940 2024-03-30T00:57:50.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5eaddeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561200ae2a20 con 0x561200bff7e0 2024-03-30T00:57:50.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5eaddeb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612009ca6a0 con 0x561200bfdfe0 2024-03-30T00:57:50.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5eaddeb80 10 client.0 _ll_register_callbacks cb 0x561200aec470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:50.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5cf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200c083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5cf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200c083f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58772/0 (socket says 172.21.15.116:58772) 2024-03-30T00:57:50.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5cf400700 1 -- 172.21.15.116:0/2508573439 learned_addr learned my addr 172.21.15.116:0/2508573439 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:50.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5cfe00700 1 --2- 172.21.15.116:0/2508573439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfdfe0 0x561200c07eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5d5200700 1 --2- 172.21.15.116:0/2508573439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bff7e0 0x561200c0f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5cf400700 1 -- 172.21.15.116:0/2508573439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfdfe0 msgr2=0x561200c07eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5cf400700 1 --2- 172.21.15.116:0/2508573439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfdfe0 0x561200c07eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5cf400700 1 -- 172.21.15.116:0/2508573439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bff7e0 msgr2=0x561200c0f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5cf400700 1 --2- 172.21.15.116:0/2508573439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bff7e0 0x561200c0f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.562+0000 7fa5cf400700 1 -- 172.21.15.116:0/2508573439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d010bcf0 con 0x561200bfe940 2024-03-30T00:57:50.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.563+0000 7fa5cf400700 1 --2- 172.21.15.116:0/2508573439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200c083f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc099eb0 tx=0x7fa5bc099fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51cbb354e1de90e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:50.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.563+0000 7fa5ba000700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:57:50.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.563+0000 7fa5ba000700 1 -- 172.21.15.116:0/2508573439 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc0982c0 con 0x561200bfe940 2024-03-30T00:57:50.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.563+0000 7fa5ba000700 1 -- 172.21.15.116:0/2508573439 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa5bc09a180 con 0x561200bfe940 2024-03-30T00:57:50.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.563+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/2508573439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561200b07380 con 0x561200bfe940 2024-03-30T00:57:50.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.563+0000 7fa5ba000700 1 -- 172.21.15.116:0/2508573439 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5bc09a340 con 0x561200bfe940 2024-03-30T00:57:50.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.563+0000 7fa5eaddeb80 10 client.15480 fetch_fsmap learned FSMap version 23 2024-03-30T00:57:50.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.563+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/2508573439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561200974b60 con 0x561200bfe940 2024-03-30T00:57:50.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.563+0000 7fa5ba000700 1 -- 172.21.15.116:0/2508573439 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5bc063080 con 0x561200bfe940 2024-03-30T00:57:50.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.564+0000 7fa5ba000700 1 -- 172.21.15.116:0/2508573439 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa5bc0a1f10 con 0x561200bfe940 2024-03-30T00:57:50.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.564+0000 7fa5eaddeb80 10 client.15480 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:57:50.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.564+0000 7fa5eaddeb80 10 client.15480 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:57:50.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.564+0000 7fa5eaddeb80 20 client.15480 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:50.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.564+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/2508573439 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561200c13b20 0x561200c15f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.564+0000 7fa5eaddeb80 4 client.15480 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:57:50.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.564+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/2508573439 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561200b3f700 con 0x561200c13b20 2024-03-30T00:57:50.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.564+0000 7fa5cfe00700 1 --2- 172.21.15.116:0/2508573439 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561200c13b20 0x561200c15f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.565+0000 7fa5cfe00700 1 --2- 172.21.15.116:0/2508573439 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561200c13b20 0x561200c15f40 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c39500664849a897 server_cookie=bbac838f873343ad in_seq=0 out_seq=0 2024-03-30T00:57:50.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.565+0000 7fa5ba000700 10 client.15480 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:50.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.565+0000 7fa5ba000700 1 -- 172.21.15.116:0/2508573439 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561200b3f700 con 0x561200c13b20 2024-03-30T00:57:50.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.565+0000 7fa5ba000700 10 client.15480 handle_command_reply: tid=0 2024-03-30T00:57:50.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.565+0000 7fa5eaddeb80 1 client.15480 shutdown 2024-03-30T00:57:50.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.565+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/2508573439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 msgr2=0x561200c083f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.565+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/2508573439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200c083f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc099eb0 tx=0x7fa5bc099fa0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.566+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/2508573439 shutdown_connections 2024-03-30T00:57:50.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.566+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/2508573439 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561200c13b20 0x561200c15f40 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.566+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/2508573439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfdfe0 0x561200c07eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.566+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/2508573439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bff7e0 0x561200c0f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.566+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/2508573439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200c083f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.566+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/2508573439 >> 172.21.15.116:0/2508573439 conn(0x561200b546a0 msgr2=0x561200bf4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:50.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.566+0000 7fa5ba000700 0 client.15480 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:50.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.566+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/2508573439 shutdown_connections 2024-03-30T00:57:50.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.566+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/2508573439 wait complete. 2024-03-30T00:57:50.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.566+0000 7fa5eaddeb80 20 client.15480 trim_cache size 0 max 16384 2024-03-30T00:57:50.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.586+0000 7fa5eaddeb80 1 Processor -- start 2024-03-30T00:57:50.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.586+0000 7fa5eaddeb80 1 -- start start 2024-03-30T00:57:50.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.586+0000 7fa5eaddeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bfdfe0 0x561200c07420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.586+0000 7fa5eaddeb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200c0f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.586+0000 7fa5eaddeb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bff7e0 0x561200bffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.586+0000 7fa5eaddeb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561200bf14b0 con 0x561200bfe940 2024-03-30T00:57:50.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.586+0000 7fa5eaddeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561200c11120 con 0x561200bfdfe0 2024-03-30T00:57:50.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.586+0000 7fa5eaddeb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561200c10560 con 0x561200bff7e0 2024-03-30T00:57:50.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.587+0000 7fa5cfe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bfdfe0 0x561200c07420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.587+0000 7fa5cf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200c0f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.587+0000 7fa5cfe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bfdfe0 0x561200c07420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:60686/0 (socket says 172.21.15.116:60686) 2024-03-30T00:57:50.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.587+0000 7fa5cfe00700 1 -- 172.21.15.116:0/1198797668 learned_addr learned my addr 172.21.15.116:0/1198797668 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:50.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.587+0000 7fa5d5200700 1 --2- 172.21.15.116:0/1198797668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bff7e0 0x561200bffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.587+0000 7fa5cfe00700 1 -- 172.21.15.116:0/1198797668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bff7e0 msgr2=0x561200bffbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.587+0000 7fa5cfe00700 1 --2- 172.21.15.116:0/1198797668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bff7e0 0x561200bffbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.587+0000 7fa5cfe00700 1 -- 172.21.15.116:0/1198797668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 msgr2=0x561200c0f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.587+0000 7fa5cfe00700 1 --2- 172.21.15.116:0/1198797668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200c0f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.587+0000 7fa5cfe00700 1 -- 172.21.15.116:0/1198797668 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561200b0cea0 con 0x561200bfdfe0 2024-03-30T00:57:50.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.587+0000 7fa5cf400700 1 --2- 172.21.15.116:0/1198797668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200c0f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:57:50.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.587+0000 7fa5cfe00700 1 --2- 172.21.15.116:0/1198797668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bfdfe0 0x561200c07420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0025040 tx=0x7fa5c0009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e02df007cf1ef0c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:50.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.588+0000 7fa5c6000700 1 -- 172.21.15.116:0/1198797668 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c000bb30 con 0x561200bfdfe0 2024-03-30T00:57:50.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.588+0000 7fa5c6000700 1 -- 172.21.15.116:0/1198797668 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5c0005500 con 0x561200bfdfe0 2024-03-30T00:57:50.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.588+0000 7fa5c6000700 1 -- 172.21.15.116:0/1198797668 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0004ea0 con 0x561200bfdfe0 2024-03-30T00:57:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.588+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/1198797668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bfdfe0 msgr2=0x561200c07420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.588+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/1198797668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bfdfe0 0x561200c07420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0025040 tx=0x7fa5c0009530 comp rx=0 tx=0).stop 2024-03-30T00:57:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.588+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/1198797668 shutdown_connections 2024-03-30T00:57:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.588+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/1198797668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bff7e0 0x561200bffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.588+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/1198797668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200bfdfe0 0x561200c07420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.588+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/1198797668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bfe940 0x561200c0f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.588+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/1198797668 >> 172.21.15.116:0/1198797668 conn(0x561200b546a0 msgr2=0x561200be95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:50.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.588+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/1198797668 shutdown_connections 2024-03-30T00:57:50.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.588+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/1198797668 wait complete. 2024-03-30T00:57:50.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.589+0000 7fa5eaddeb80 1 -- start start 2024-03-30T00:57:50.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.589+0000 7fa5eaddeb80 1 Processor -- start 2024-03-30T00:57:50.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.589+0000 7fa5eaddeb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfe940 0x561200c0d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.589+0000 7fa5eaddeb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bff7e0 0x561200c0d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.589+0000 7fa5eaddeb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200c0e580 0x561200c0ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5eaddeb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561200c0ef40 con 0x561200bff7e0 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5eaddeb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561200c0f0b0 con 0x561200c0e580 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5eaddeb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561200c0f270 con 0x561200bfe940 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5eaddeb80 10 client.0 _ll_register_callbacks cb 0x561200aec470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5cf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bff7e0 0x561200c0d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5cf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bff7e0 0x561200c0d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58782/0 (socket says 172.21.15.116:58782) 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5cf400700 1 -- 172.21.15.116:0/3049832990 learned_addr learned my addr 172.21.15.116:0/3049832990 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5cfe00700 1 --2- 172.21.15.116:0/3049832990 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfe940 0x561200c0d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5d5200700 1 --2- 172.21.15.116:0/3049832990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200c0e580 0x561200c0ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5cf400700 1 -- 172.21.15.116:0/3049832990 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfe940 msgr2=0x561200c0d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5cf400700 1 --2- 172.21.15.116:0/3049832990 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfe940 0x561200c0d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5cf400700 1 -- 172.21.15.116:0/3049832990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200c0e580 msgr2=0x561200c0ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5cf400700 1 --2- 172.21.15.116:0/3049832990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200c0e580 0x561200c0ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5cf400700 1 -- 172.21.15.116:0/3049832990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561200974b60 con 0x561200bff7e0 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5cf400700 1 --2- 172.21.15.116:0/3049832990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bff7e0 0x561200c0d5c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc096420 tx=0x7fa5bc0a2480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd306ecff8b84df8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:57:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.590+0000 7fa5ba000700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:57:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.591+0000 7fa5ba000700 1 -- 172.21.15.116:0/3049832990 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc014040 con 0x561200bff7e0 2024-03-30T00:57:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.591+0000 7fa5ba000700 1 -- 172.21.15.116:0/3049832990 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa5bc083770 con 0x561200bff7e0 2024-03-30T00:57:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.591+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/3049832990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5c001e040 con 0x561200bff7e0 2024-03-30T00:57:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.591+0000 7fa5ba000700 1 -- 172.21.15.116:0/3049832990 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5bc099a60 con 0x561200bff7e0 2024-03-30T00:57:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.591+0000 7fa5eaddeb80 10 client.15486 fetch_fsmap learned FSMap version 23 2024-03-30T00:57:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.591+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/3049832990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5612009c83b0 con 0x561200bff7e0 2024-03-30T00:57:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.591+0000 7fa5ba000700 1 -- 172.21.15.116:0/3049832990 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5c001e040 con 0x561200bff7e0 2024-03-30T00:57:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.591+0000 7fa5ba000700 1 -- 172.21.15.116:0/3049832990 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa5bc095d90 con 0x561200bff7e0 2024-03-30T00:57:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.591+0000 7fa5eaddeb80 10 client.15486 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:57:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.591+0000 7fa5eaddeb80 10 client.15486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:57:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.591+0000 7fa5eaddeb80 20 client.15486 populate_metadata read hostname 'smithi116' 2024-03-30T00:57:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.591+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/3049832990 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561200c68700 0x561200c6ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:57:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.591+0000 7fa5eaddeb80 4 client.15486 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T00:57:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.591+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/3049832990 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561200995cf0 con 0x561200c68700 2024-03-30T00:57:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.592+0000 7fa5cfe00700 1 --2- 172.21.15.116:0/3049832990 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561200c68700 0x561200c6ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:57:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.592+0000 7fa5cfe00700 1 --2- 172.21.15.116:0/3049832990 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561200c68700 0x561200c6ab20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fb2a9fd56a2a06a server_cookie=69e1d2c62a23f416 in_seq=0 out_seq=0 2024-03-30T00:57:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.592+0000 7fa5ba000700 10 client.15486 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.592+0000 7fa5ba000700 1 -- 172.21.15.116:0/3049832990 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x561200995cf0 con 0x561200c68700 2024-03-30T00:57:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.592+0000 7fa5ba000700 10 client.15486 handle_command_reply: tid=0 2024-03-30T00:57:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.592+0000 7fa5eaddeb80 1 client.15486 shutdown 2024-03-30T00:57:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.593+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/3049832990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bff7e0 msgr2=0x561200c0d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.593+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/3049832990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bff7e0 0x561200c0d5c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc096420 tx=0x7fa5bc0a2480 comp rx=0 tx=0).stop 2024-03-30T00:57:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.593+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/3049832990 shutdown_connections 2024-03-30T00:57:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.593+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/3049832990 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561200c68700 0x561200c6ab20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.593+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/3049832990 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561200bfe940 0x561200c0d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.593+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/3049832990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561200c0e580 0x561200c0ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.593+0000 7fa5eaddeb80 1 --2- 172.21.15.116:0/3049832990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561200bff7e0 0x561200c0d5c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.593+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/3049832990 >> 172.21.15.116:0/3049832990 conn(0x561200b546a0 msgr2=0x561200c05dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:50.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.593+0000 7fa5ba000700 0 client.15486 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:57:50.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.593+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/3049832990 shutdown_connections 2024-03-30T00:57:50.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.593+0000 7fa5eaddeb80 1 -- 172.21.15.116:0/3049832990 wait complete. 2024-03-30T00:57:50.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.593+0000 7fa5eaddeb80 20 client.15486 trim_cache size 0 max 16384 2024-03-30T00:57:50.607 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:57:50.607 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T00:57:50.607 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-30T00:57:50.607 INFO:teuthology.orchestra.run.smithi116.stdout: "4690422e-0773-41b7-8ab9-35a0cbe0c0d4": { 2024-03-30T00:57:50.607 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-30T00:57:50.607 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "4690422e-0773-41b7-8ab9-35a0cbe0c0d4", 2024-03-30T00:57:50.608 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-30T00:57:50.608 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T00:57:50.608 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T00:57:50.608 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:57:50.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.594+0000 7fa5d7200700 1 -- 172.21.15.116:0/824304339 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5b40a1120 msgr2=0x7fa5b40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.594+0000 7fa5d7200700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5b40a1120 0x7fa5b40a35e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8009fd0 tx=0x7fa5c800fde0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.594+0000 7fa5d7200700 1 -- 172.21.15.116:0/824304339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d0074df0 msgr2=0x7fa5d0199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:57:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.594+0000 7fa5d7200700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d0074df0 0x7fa5d0199460 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc00d980 tx=0x7fa5bc00dd40 comp rx=0 tx=0).stop 2024-03-30T00:57:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.594+0000 7fa5d7200700 1 -- 172.21.15.116:0/824304339 shutdown_connections 2024-03-30T00:57:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.594+0000 7fa5d7200700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5b40a1120 0x7fa5b40a35e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.594+0000 7fa5d7200700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5d0108c40 0x7fa5d01a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.594+0000 7fa5d7200700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5d0073810 0x7fa5d0198f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.594+0000 7fa5d7200700 1 --2- 172.21.15.116:0/824304339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d0074df0 0x7fa5d0199460 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:57:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.594+0000 7fa5d7200700 1 -- 172.21.15.116:0/824304339 >> 172.21.15.116:0/824304339 conn(0x7fa5d00fc2f0 msgr2=0x7fa5d0104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:57:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.594+0000 7fa5d7200700 1 -- 172.21.15.116:0/824304339 shutdown_connections 2024-03-30T00:57:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:57:50.594+0000 7fa5d7200700 1 -- 172.21.15.116:0/824304339 wait complete. 2024-03-30T00:57:50.616 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:50 smithi116 ceph-mon[38086]: pgmap v290: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-30T00:57:50.616 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:50 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2176568122' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:50.616 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:50 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2492771317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:50.618 INFO:tasks.cephfs.filesystem:scrub status for tag:4690422e-0773-41b7-8ab9-35a0cbe0c0d4 - {'path': '/', 'tag': '4690422e-0773-41b7-8ab9-35a0cbe0c0d4', 'options': 'recursive,force'} 2024-03-30T00:57:50.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:50 smithi181 ceph-mon[47622]: pgmap v290: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-30T00:57:50.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:50 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2176568122' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:50.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:50 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2492771317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:50.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:50 smithi130 ceph-mon[47626]: pgmap v290: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-30T00:57:50.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:50 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2176568122' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:57:50.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:50 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2492771317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T00:57:51.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:51 smithi181 ceph-mon[47622]: scrub queued for path: / 2024-03-30T00:57:51.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:51 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T00:57:51.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:51 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T00:57:51.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:51 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T00:57:51.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:51 smithi130 ceph-mon[47626]: scrub queued for path: / 2024-03-30T00:57:51.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:51 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T00:57:51.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:51 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T00:57:51.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:51 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T00:57:51.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:51 smithi116 ceph-mon[38086]: scrub queued for path: / 2024-03-30T00:57:51.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:51 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T00:57:51.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:51 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T00:57:51.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:51 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T00:57:52.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:52 smithi130 ceph-mon[47626]: pgmap v291: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-30T00:57:52.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:52 smithi116 ceph-mon[38086]: pgmap v291: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-30T00:57:53.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:52 smithi181 ceph-mon[47622]: pgmap v291: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-30T00:57:54.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:54 smithi130 ceph-mon[47626]: pgmap v292: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-30T00:57:54.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:54 smithi116 ceph-mon[38086]: pgmap v292: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-30T00:57:55.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:54 smithi181 ceph-mon[47622]: pgmap v292: 129 pgs: 129 active+clean; 647 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-30T00:57:56.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:56 smithi130 ceph-mon[47626]: pgmap v293: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-30T00:57:56.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:56 smithi116 ceph-mon[38086]: pgmap v293: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-30T00:57:57.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:56 smithi181 ceph-mon[47622]: pgmap v293: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-30T00:57:58.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:57:58 smithi130 ceph-mon[47626]: pgmap v294: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-30T00:57:58.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:57:58 smithi116 ceph-mon[38086]: pgmap v294: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-30T00:57:59.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:57:58 smithi181 ceph-mon[47622]: pgmap v294: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-30T00:58:00.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:00 smithi130 ceph-mon[47626]: pgmap v295: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-30T00:58:00.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:00 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:58:00.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:00 smithi116 ceph-mon[38086]: pgmap v295: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-30T00:58:00.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:00 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:58:01.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:00 smithi181 ceph-mon[47622]: pgmap v295: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-30T00:58:01.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:00 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:58:02.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:02 smithi181 ceph-mon[47622]: pgmap v296: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-30T00:58:02.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:02 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T00:58:02.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:02 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:02 smithi130 ceph-mon[47626]: pgmap v296: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-30T00:58:02.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:02 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T00:58:02.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:02 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:02 smithi116 ceph-mon[38086]: pgmap v296: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-30T00:58:02.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:02 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T00:58:02.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:02.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:02 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:03.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:58:03.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:58:03.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:03.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:58:03.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:58:03.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:03.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:58:03.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:58:03.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:58:04.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:04 smithi181 ceph-mon[47622]: pgmap v297: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-30T00:58:04.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:04 smithi130 ceph-mon[47626]: pgmap v297: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-30T00:58:04.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:04 smithi116 ceph-mon[38086]: pgmap v297: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-30T00:58:06.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:06 smithi181 ceph-mon[47622]: pgmap v298: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-30T00:58:06.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:06 smithi130 ceph-mon[47626]: pgmap v298: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-30T00:58:06.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:06 smithi116 ceph-mon[38086]: pgmap v298: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-30T00:58:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:08 smithi181 ceph-mon[47622]: pgmap v299: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-30T00:58:08.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:08 smithi130 ceph-mon[47626]: pgmap v299: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-30T00:58:08.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:08 smithi116 ceph-mon[38086]: pgmap v299: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-30T00:58:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:10 smithi181 ceph-mon[47622]: pgmap v300: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:58:10.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:10 smithi130 ceph-mon[47626]: pgmap v300: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:58:10.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:10 smithi116 ceph-mon[38086]: pgmap v300: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:58:12.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:12 smithi181 ceph-mon[47622]: pgmap v301: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:58:12.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:12 smithi130 ceph-mon[47626]: pgmap v301: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:58:12.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:12 smithi116 ceph-mon[38086]: pgmap v301: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:58:14.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:14 smithi181 ceph-mon[47622]: pgmap v302: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:14.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:14 smithi130 ceph-mon[47626]: pgmap v302: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:14.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:14 smithi116 ceph-mon[38086]: pgmap v302: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:16.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:16 smithi181 ceph-mon[47622]: pgmap v303: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:58:16.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:16 smithi130 ceph-mon[47626]: pgmap v303: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:58:16.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:16 smithi116 ceph-mon[38086]: pgmap v303: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:58:18.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:18 smithi181 ceph-mon[47622]: pgmap v304: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:18.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:18 smithi130 ceph-mon[47626]: pgmap v304: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:18.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:18 smithi116 ceph-mon[38086]: pgmap v304: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:20.620 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T00:58:20.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:20 smithi181 ceph-mon[47622]: pgmap v305: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:58:20.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.822+0000 7f54aec00700 1 -- 172.21.15.116:0/3743723600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a8101f70 msgr2=0x7f54a81023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:20.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.822+0000 7f54aec00700 1 --2- 172.21.15.116:0/3743723600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a8101f70 0x7f54a81023d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5494009920 tx=0x7f5494009c30 comp rx=0 tx=0).stop 2024-03-30T00:58:20.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.822+0000 7f54aec00700 1 -- 172.21.15.116:0/3743723600 shutdown_connections 2024-03-30T00:58:20.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.822+0000 7f54aec00700 1 --2- 172.21.15.116:0/3743723600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54a8102910 0x7f54a81091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.822+0000 7f54aec00700 1 --2- 172.21.15.116:0/3743723600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54a8100d80 0x7f54a81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.822+0000 7f54aec00700 1 --2- 172.21.15.116:0/3743723600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a8101f70 0x7f54a81023d0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.822+0000 7f54aec00700 1 -- 172.21.15.116:0/3743723600 >> 172.21.15.116:0/3743723600 conn(0x7f54a80fc2f0 msgr2=0x7f54a80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:20.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.822+0000 7f54aec00700 1 -- 172.21.15.116:0/3743723600 shutdown_connections 2024-03-30T00:58:20.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.822+0000 7f54aec00700 1 -- 172.21.15.116:0/3743723600 wait complete. 2024-03-30T00:58:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.823+0000 7f54aec00700 1 Processor -- start 2024-03-30T00:58:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.823+0000 7f54aec00700 1 -- start start 2024-03-30T00:58:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.824+0000 7f54aec00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54a8100d80 0x7f54a8074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.824+0000 7f54aec00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54a8101f70 0x7f54a80731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.824+0000 7f54aec00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a8102910 0x7f54a80736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.824+0000 7f54aec00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a8073e80 con 0x7f54a8102910 2024-03-30T00:58:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.824+0000 7f54aec00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a8073ff0 con 0x7f54a8101f70 2024-03-30T00:58:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.824+0000 7f54aec00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a81a0e40 con 0x7f54a8100d80 2024-03-30T00:58:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.824+0000 7f54acc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a8102910 0x7f54a80736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.824+0000 7f54acc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a8102910 0x7f54a80736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40460/0 (socket says 172.21.15.116:40460) 2024-03-30T00:58:20.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.824+0000 7f54acc00700 1 -- 172.21.15.116:0/4219222140 learned_addr learned my addr 172.21.15.116:0/4219222140 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:20.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.824+0000 7f54a7e00700 1 --2- 172.21.15.116:0/4219222140 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54a8100d80 0x7f54a8074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.824+0000 7f54a7400700 1 --2- 172.21.15.116:0/4219222140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54a8101f70 0x7f54a80731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.825+0000 7f54acc00700 1 -- 172.21.15.116:0/4219222140 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54a8100d80 msgr2=0x7f54a8074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:20.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.825+0000 7f54acc00700 1 --2- 172.21.15.116:0/4219222140 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54a8100d80 0x7f54a8074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.825+0000 7f54acc00700 1 -- 172.21.15.116:0/4219222140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54a8101f70 msgr2=0x7f54a80731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:20.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.825+0000 7f54acc00700 1 --2- 172.21.15.116:0/4219222140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54a8101f70 0x7f54a80731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.825+0000 7f54acc00700 1 -- 172.21.15.116:0/4219222140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a0009c40 con 0x7f54a8102910 2024-03-30T00:58:20.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.825+0000 7f54acc00700 1 --2- 172.21.15.116:0/4219222140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a8102910 0x7f54a80736e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f54a000c370 tx=0x7f54a000c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:20.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.825+0000 7f54a4c00700 1 -- 172.21.15.116:0/4219222140 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0017070 con 0x7f54a8102910 2024-03-30T00:58:20.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.825+0000 7f54a4c00700 1 -- 172.21.15.116:0/4219222140 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54a000f040 con 0x7f54a8102910 2024-03-30T00:58:20.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.825+0000 7f54aec00700 1 -- 172.21.15.116:0/4219222140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5494009520 con 0x7f54a8102910 2024-03-30T00:58:20.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.825+0000 7f54a4c00700 1 -- 172.21.15.116:0/4219222140 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a000e050 con 0x7f54a8102910 2024-03-30T00:58:20.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.825+0000 7f54aec00700 1 -- 172.21.15.116:0/4219222140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a81a13f0 con 0x7f54a8102910 2024-03-30T00:58:20.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.828+0000 7f54a4c00700 1 -- 172.21.15.116:0/4219222140 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f54a0003a00 con 0x7f54a8102910 2024-03-30T00:58:20.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.829+0000 7f54a4c00700 1 --2- 172.21.15.116:0/4219222140 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f548c0a1120 0x7f548c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.829+0000 7f54c275ab80 1 Processor -- start 2024-03-30T00:58:20.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54a4c00700 1 -- 172.21.15.116:0/4219222140 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f54a00cc4b0 con 0x7f54a8102910 2024-03-30T00:58:20.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54a7e00700 1 --2- 172.21.15.116:0/4219222140 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f548c0a1120 0x7f548c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54c275ab80 1 -- start start 2024-03-30T00:58:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54c275ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e60fe0 0x55ba03e61400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54c275ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e61940 0x55ba03e62200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54c275ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e627e0 0x55ba03e62c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54c275ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba03db0990 con 0x55ba03e627e0 2024-03-30T00:58:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54c275ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba03dae350 con 0x55ba03e61940 2024-03-30T00:58:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54c275ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba03da9c40 con 0x55ba03e60fe0 2024-03-30T00:58:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54a7e00700 1 --2- 172.21.15.116:0/4219222140 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f548c0a1120 0x7f548c0a35e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f54a8101dd0 tx=0x7f5498009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54a7400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e61940 0x55ba03e62200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54a7e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e60fe0 0x55ba03e61400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54acc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e627e0 0x55ba03e62c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54a7400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e61940 0x55ba03e62200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:42402/0 (socket says 172.21.15.116:42402) 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.830+0000 7f54a7400700 1 -- 172.21.15.116:0/601573556 learned_addr learned my addr 172.21.15.116:0/601573556 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.831+0000 7f54a7400700 1 -- 172.21.15.116:0/601573556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e60fe0 msgr2=0x55ba03e61400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.831+0000 7f54a7400700 1 --2- 172.21.15.116:0/601573556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e60fe0 0x55ba03e61400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.831+0000 7f54a7400700 1 -- 172.21.15.116:0/601573556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e627e0 msgr2=0x55ba03e62c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.831+0000 7f54a7400700 1 --2- 172.21.15.116:0/601573556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e627e0 0x55ba03e62c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.831+0000 7f54a7400700 1 -- 172.21.15.116:0/601573556 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba03c2b3b0 con 0x55ba03e61940 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.831+0000 7f54acc00700 1 --2- 172.21.15.116:0/601573556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e627e0 0x55ba03e62c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.831+0000 7f54a7e00700 1 --2- 172.21.15.116:0/601573556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e60fe0 0x55ba03e61400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.831+0000 7f54a7400700 1 --2- 172.21.15.116:0/601573556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e61940 0x55ba03e62200 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5494000c00 tx=0x7f5494009f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=758e30923df6c2ad server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.831+0000 7f549d200700 1 -- 172.21.15.116:0/601573556 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549400f660 con 0x55ba03e61940 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.831+0000 7f549d200700 1 -- 172.21.15.116:0/601573556 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f549401d040 con 0x55ba03e61940 2024-03-30T00:58:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.831+0000 7f549d200700 1 -- 172.21.15.116:0/601573556 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5494005430 con 0x55ba03e61940 2024-03-30T00:58:20.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.832+0000 7f54c275ab80 1 -- 172.21.15.116:0/601573556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e61940 msgr2=0x55ba03e62200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:20.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.832+0000 7f54c275ab80 1 --2- 172.21.15.116:0/601573556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e61940 0x55ba03e62200 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5494000c00 tx=0x7f5494009f80 comp rx=0 tx=0).stop 2024-03-30T00:58:20.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.832+0000 7f54c275ab80 1 -- 172.21.15.116:0/601573556 shutdown_connections 2024-03-30T00:58:20.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.832+0000 7f54c275ab80 1 --2- 172.21.15.116:0/601573556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e60fe0 0x55ba03e61400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.832+0000 7f54c275ab80 1 --2- 172.21.15.116:0/601573556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e627e0 0x55ba03e62c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.832+0000 7f54c275ab80 1 --2- 172.21.15.116:0/601573556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e61940 0x55ba03e62200 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.832+0000 7f54c275ab80 1 -- 172.21.15.116:0/601573556 >> 172.21.15.116:0/601573556 conn(0x55ba03db76a0 msgr2=0x55ba03e539a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:20.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.832+0000 7f54c275ab80 1 -- 172.21.15.116:0/601573556 shutdown_connections 2024-03-30T00:58:20.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.832+0000 7f54c275ab80 1 -- 172.21.15.116:0/601573556 wait complete. 2024-03-30T00:58:20.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.833+0000 7f54c275ab80 1 -- start start 2024-03-30T00:58:20.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.833+0000 7f54c275ab80 1 Processor -- start 2024-03-30T00:58:20.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.834+0000 7f54c275ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e60fe0 0x55ba03e6adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.834+0000 7f54c275ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e61940 0x55ba03e6b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.834+0000 7f54c275ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e627e0 0x55ba03e72330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.834+0000 7f54c275ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba03d45d80 con 0x55ba03e60fe0 2024-03-30T00:58:20.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.834+0000 7f54c275ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba03d45a20 con 0x55ba03e61940 2024-03-30T00:58:20.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.834+0000 7f54c275ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba03c2d6a0 con 0x55ba03e627e0 2024-03-30T00:58:20.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.834+0000 7f54c275ab80 10 client.0 _ll_register_callbacks cb 0x55ba03d4f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:20.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.834+0000 7f54a7e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e60fe0 0x55ba03e6adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.834+0000 7f54a7e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e60fe0 0x55ba03e6adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40480/0 (socket says 172.21.15.116:40480) 2024-03-30T00:58:20.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.834+0000 7f54a7e00700 1 -- 172.21.15.116:0/2349736287 learned_addr learned my addr 172.21.15.116:0/2349736287 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:20.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.834+0000 7f54acc00700 1 --2- 172.21.15.116:0/2349736287 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e627e0 0x55ba03e72330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.834+0000 7f54a7400700 1 --2- 172.21.15.116:0/2349736287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e61940 0x55ba03e6b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.834+0000 7f54a7e00700 1 -- 172.21.15.116:0/2349736287 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e627e0 msgr2=0x55ba03e72330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:20.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.834+0000 7f54a7e00700 1 --2- 172.21.15.116:0/2349736287 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e627e0 0x55ba03e72330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.835+0000 7f54a7e00700 1 -- 172.21.15.116:0/2349736287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e61940 msgr2=0x55ba03e6b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:20.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.835+0000 7f54a7e00700 1 --2- 172.21.15.116:0/2349736287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e61940 0x55ba03e6b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.835+0000 7f54a7e00700 1 -- 172.21.15.116:0/2349736287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a81a13f0 con 0x55ba03e60fe0 2024-03-30T00:58:20.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.835+0000 7f54acc00700 1 --2- 172.21.15.116:0/2349736287 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e627e0 0x55ba03e72330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:58:20.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.835+0000 7f54a7e00700 1 --2- 172.21.15.116:0/2349736287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e60fe0 0x55ba03e6adb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f549801dc40 tx=0x7f54980047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9e9750b94741e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:20.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.835+0000 7f5491600700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:58:20.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.835+0000 7f5491600700 1 -- 172.21.15.116:0/2349736287 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498022070 con 0x55ba03e60fe0 2024-03-30T00:58:20.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.835+0000 7f5491600700 1 -- 172.21.15.116:0/2349736287 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f549801ca20 con 0x55ba03e60fe0 2024-03-30T00:58:20.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.835+0000 7f54c275ab80 1 -- 172.21.15.116:0/2349736287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ba03d6a380 con 0x55ba03e60fe0 2024-03-30T00:58:20.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.835+0000 7f5491600700 1 -- 172.21.15.116:0/2349736287 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f549801cbe0 con 0x55ba03e60fe0 2024-03-30T00:58:20.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.835+0000 7f54c275ab80 10 client.15498 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:20.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.835+0000 7f54c275ab80 1 -- 172.21.15.116:0/2349736287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f54a8109b10 con 0x55ba03e60fe0 2024-03-30T00:58:20.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.836+0000 7f5491600700 1 -- 172.21.15.116:0/2349736287 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ba03d6a380 con 0x55ba03e60fe0 2024-03-30T00:58:20.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.836+0000 7f5491600700 1 -- 172.21.15.116:0/2349736287 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f549800e3b0 con 0x55ba03e60fe0 2024-03-30T00:58:20.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.836+0000 7f54c275ab80 10 client.15498 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:20.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.836+0000 7f54c275ab80 10 client.15498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:20.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.836+0000 7f54c275ab80 20 client.15498 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:20.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.836+0000 7f54c275ab80 1 --2- 172.21.15.116:0/2349736287 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ba03e76bf0 0x55ba03e79010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.836+0000 7f54c275ab80 4 client.15498 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:58:20.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.836+0000 7f54c275ab80 1 -- 172.21.15.116:0/2349736287 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ba03da2700 con 0x55ba03e76bf0 2024-03-30T00:58:20.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.836+0000 7f54a7400700 1 --2- 172.21.15.116:0/2349736287 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ba03e76bf0 0x55ba03e79010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.837+0000 7f54a7400700 1 --2- 172.21.15.116:0/2349736287 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ba03e76bf0 0x55ba03e79010 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35e053095d5697cf server_cookie=a75f2fdc9704641c in_seq=0 out_seq=0 2024-03-30T00:58:20.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.837+0000 7f5491600700 10 client.15498 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:20.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.837+0000 7f5491600700 1 -- 172.21.15.116:0/2349736287 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ba03da2700 con 0x55ba03e76bf0 2024-03-30T00:58:20.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.837+0000 7f5491600700 10 client.15498 handle_command_reply: tid=0 2024-03-30T00:58:20.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.838+0000 7f54c275ab80 1 client.15498 shutdown 2024-03-30T00:58:20.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.838+0000 7f54c275ab80 1 -- 172.21.15.116:0/2349736287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e60fe0 msgr2=0x55ba03e6adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:20.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.838+0000 7f54c275ab80 1 --2- 172.21.15.116:0/2349736287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e60fe0 0x55ba03e6adb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f549801dc40 tx=0x7f54980047c0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.838+0000 7f54c275ab80 1 -- 172.21.15.116:0/2349736287 shutdown_connections 2024-03-30T00:58:20.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.838+0000 7f54c275ab80 1 --2- 172.21.15.116:0/2349736287 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ba03e76bf0 0x55ba03e79010 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.838+0000 7f54c275ab80 1 --2- 172.21.15.116:0/2349736287 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e627e0 0x55ba03e72330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.838+0000 7f54c275ab80 1 --2- 172.21.15.116:0/2349736287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e60fe0 0x55ba03e6adb0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.838+0000 7f54c275ab80 1 --2- 172.21.15.116:0/2349736287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e61940 0x55ba03e6b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.838+0000 7f5491600700 0 client.15498 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:20.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.838+0000 7f54c275ab80 1 -- 172.21.15.116:0/2349736287 >> 172.21.15.116:0/2349736287 conn(0x55ba03db76a0 msgr2=0x55ba03e57170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:20.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.838+0000 7f54c275ab80 1 -- 172.21.15.116:0/2349736287 shutdown_connections 2024-03-30T00:58:20.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.838+0000 7f54c275ab80 1 -- 172.21.15.116:0/2349736287 wait complete. 2024-03-30T00:58:20.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.838+0000 7f54c275ab80 20 client.15498 trim_cache size 0 max 16384 2024-03-30T00:58:20.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:20 smithi130 ceph-mon[47626]: pgmap v305: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:58:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.855+0000 7f54c275ab80 1 Processor -- start 2024-03-30T00:58:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.855+0000 7f54c275ab80 1 -- start start 2024-03-30T00:58:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.855+0000 7f54c275ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e60fe0 0x55ba03e62bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.855+0000 7f54c275ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e61940 0x55ba03e722f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.855+0000 7f54c275ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e627e0 0x55ba03e52f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.855+0000 7f54c275ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba03e626a0 con 0x55ba03e61940 2024-03-30T00:58:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.855+0000 7f54c275ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba03e734b0 con 0x55ba03e627e0 2024-03-30T00:58:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.855+0000 7f54c275ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba03e61780 con 0x55ba03e60fe0 2024-03-30T00:58:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.856+0000 7f54a7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e61940 0x55ba03e722f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.856+0000 7f54a7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e61940 0x55ba03e722f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40484/0 (socket says 172.21.15.116:40484) 2024-03-30T00:58:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.856+0000 7f54a7400700 1 -- 172.21.15.116:0/4080943758 learned_addr learned my addr 172.21.15.116:0/4080943758 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.856+0000 7f54a7e00700 1 --2- 172.21.15.116:0/4080943758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e60fe0 0x55ba03e62bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.856+0000 7f54acc00700 1 --2- 172.21.15.116:0/4080943758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e627e0 0x55ba03e52f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.856+0000 7f54a7400700 1 -- 172.21.15.116:0/4080943758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e60fe0 msgr2=0x55ba03e62bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.856+0000 7f54a7400700 1 --2- 172.21.15.116:0/4080943758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e60fe0 0x55ba03e62bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.856+0000 7f54a7400700 1 -- 172.21.15.116:0/4080943758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e627e0 msgr2=0x55ba03e52f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.856+0000 7f54a7400700 1 --2- 172.21.15.116:0/4080943758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e627e0 0x55ba03e52f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.856+0000 7f54a7400700 1 -- 172.21.15.116:0/4080943758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba03bd7b60 con 0x55ba03e61940 2024-03-30T00:58:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.856+0000 7f54a7400700 1 --2- 172.21.15.116:0/4080943758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e61940 0x55ba03e722f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5494000c00 tx=0x7f5494009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202c5f8001b6567 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.857+0000 7f549dc00700 1 -- 172.21.15.116:0/4080943758 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54940321e0 con 0x55ba03e61940 2024-03-30T00:58:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.857+0000 7f549dc00700 1 -- 172.21.15.116:0/4080943758 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f549400ba20 con 0x55ba03e61940 2024-03-30T00:58:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.857+0000 7f549dc00700 1 -- 172.21.15.116:0/4080943758 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54940052f0 con 0x55ba03e61940 2024-03-30T00:58:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.857+0000 7f54c275ab80 1 -- 172.21.15.116:0/4080943758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e61940 msgr2=0x55ba03e722f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.857+0000 7f54c275ab80 1 --2- 172.21.15.116:0/4080943758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e61940 0x55ba03e722f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5494000c00 tx=0x7f5494009f80 comp rx=0 tx=0).stop 2024-03-30T00:58:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.857+0000 7f54c275ab80 1 -- 172.21.15.116:0/4080943758 shutdown_connections 2024-03-30T00:58:20.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.857+0000 7f54c275ab80 1 --2- 172.21.15.116:0/4080943758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e60fe0 0x55ba03e62bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.857+0000 7f54c275ab80 1 --2- 172.21.15.116:0/4080943758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e627e0 0x55ba03e52f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.857+0000 7f54c275ab80 1 --2- 172.21.15.116:0/4080943758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e61940 0x55ba03e722f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.857+0000 7f54c275ab80 1 -- 172.21.15.116:0/4080943758 >> 172.21.15.116:0/4080943758 conn(0x55ba03db76a0 msgr2=0x55ba03e7d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:20.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.857+0000 7f54c275ab80 1 -- 172.21.15.116:0/4080943758 shutdown_connections 2024-03-30T00:58:20.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.857+0000 7f54c275ab80 1 -- 172.21.15.116:0/4080943758 wait complete. 2024-03-30T00:58:20.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.858+0000 7f54c275ab80 1 -- start start 2024-03-30T00:58:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.858+0000 7f54c275ab80 1 Processor -- start 2024-03-30T00:58:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54c275ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e60fe0 0x55ba03e70330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54c275ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e61940 0x55ba03e70870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54c275ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e627e0 0x55ba03e717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54c275ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba03e71d20 con 0x55ba03e627e0 2024-03-30T00:58:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54c275ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba03e71e90 con 0x55ba03e60fe0 2024-03-30T00:58:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54c275ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba03e72050 con 0x55ba03e61940 2024-03-30T00:58:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54c275ab80 10 client.0 _ll_register_callbacks cb 0x55ba03d4f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54acc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e627e0 0x55ba03e717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54acc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e627e0 0x55ba03e717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40490/0 (socket says 172.21.15.116:40490) 2024-03-30T00:58:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54acc00700 1 -- 172.21.15.116:0/168700160 learned_addr learned my addr 172.21.15.116:0/168700160 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54a7e00700 1 --2- 172.21.15.116:0/168700160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e60fe0 0x55ba03e70330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54a7400700 1 --2- 172.21.15.116:0/168700160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e61940 0x55ba03e70870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54acc00700 1 -- 172.21.15.116:0/168700160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e61940 msgr2=0x55ba03e70870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54acc00700 1 --2- 172.21.15.116:0/168700160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e61940 0x55ba03e70870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54acc00700 1 -- 172.21.15.116:0/168700160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e60fe0 msgr2=0x55ba03e70330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54acc00700 1 --2- 172.21.15.116:0/168700160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e60fe0 0x55ba03e70330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54acc00700 1 -- 172.21.15.116:0/168700160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba03c2b3b0 con 0x55ba03e627e0 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.859+0000 7f54a7400700 1 --2- 172.21.15.116:0/168700160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e61940 0x55ba03e70870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f54acc00700 1 --2- 172.21.15.116:0/168700160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e627e0 0x55ba03e717e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f54a00c9c40 tx=0x7f54a000ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b4e217edfc5939 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f5491600700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f5491600700 1 -- 172.21.15.116:0/168700160 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0099c90 con 0x55ba03e627e0 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f5491600700 1 -- 172.21.15.116:0/168700160 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f54a0099590 con 0x55ba03e627e0 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f54c275ab80 1 -- 172.21.15.116:0/168700160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ba03e69de0 con 0x55ba03e627e0 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f5491600700 1 -- 172.21.15.116:0/168700160 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54a009d3c0 con 0x55ba03e627e0 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f54c275ab80 10 client.15504 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f54c275ab80 1 -- 172.21.15.116:0/168700160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ba03d6fea0 con 0x55ba03e627e0 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f5491600700 1 -- 172.21.15.116:0/168700160 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54a00b3070 con 0x55ba03e627e0 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f5491600700 1 -- 172.21.15.116:0/168700160 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f54a00997f0 con 0x55ba03e627e0 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f54c275ab80 10 client.15504 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f54c275ab80 10 client.15504 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f54c275ab80 20 client.15504 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f54c275ab80 1 --2- 172.21.15.116:0/168700160 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ba03ecbcc0 0x55ba03ece0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f54c275ab80 4 client.15504 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.860+0000 7f54c275ab80 1 -- 172.21.15.116:0/168700160 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f549401d040 con 0x55ba03ecbcc0 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.861+0000 7f54a7e00700 1 --2- 172.21.15.116:0/168700160 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ba03ecbcc0 0x55ba03ece0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.861+0000 7f54a7e00700 1 --2- 172.21.15.116:0/168700160 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ba03ecbcc0 0x55ba03ece0e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b249925eaf2d6ee2 server_cookie=3bcd7b4e6f7f90d5 in_seq=0 out_seq=0 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.861+0000 7f5491600700 10 client.15504 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.861+0000 7f5491600700 1 -- 172.21.15.116:0/168700160 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7f549401d040 con 0x55ba03ecbcc0 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.861+0000 7f5491600700 10 client.15504 handle_command_reply: tid=0 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.861+0000 7f54c275ab80 1 client.15504 shutdown 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.861+0000 7f54c275ab80 1 -- 172.21.15.116:0/168700160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e627e0 msgr2=0x55ba03e717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.861+0000 7f54c275ab80 1 --2- 172.21.15.116:0/168700160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e627e0 0x55ba03e717e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f54a00c9c40 tx=0x7f54a000ce00 comp rx=0 tx=0).stop 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.861+0000 7f54c275ab80 1 -- 172.21.15.116:0/168700160 shutdown_connections 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.861+0000 7f54c275ab80 1 --2- 172.21.15.116:0/168700160 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ba03ecbcc0 0x55ba03ece0e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54c275ab80 1 --2- 172.21.15.116:0/168700160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba03e627e0 0x55ba03e717e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54c275ab80 1 --2- 172.21.15.116:0/168700160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba03e60fe0 0x55ba03e70330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f5491600700 0 client.15504 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54c275ab80 1 --2- 172.21.15.116:0/168700160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ba03e61940 0x55ba03e70870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54c275ab80 1 -- 172.21.15.116:0/168700160 >> 172.21.15.116:0/168700160 conn(0x55ba03db76a0 msgr2=0x55ba03e4c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54c275ab80 1 -- 172.21.15.116:0/168700160 shutdown_connections 2024-03-30T00:58:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54c275ab80 1 -- 172.21.15.116:0/168700160 wait complete. 2024-03-30T00:58:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54c275ab80 20 client.15504 trim_cache size 0 max 16384 2024-03-30T00:58:20.873 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:58:20.873 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-30T00:58:20.873 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-30T00:58:20.873 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:58:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54aec00700 1 -- 172.21.15.116:0/4219222140 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f548c0a1120 msgr2=0x7f548c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54aec00700 1 --2- 172.21.15.116:0/4219222140 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f548c0a1120 0x7f548c0a35e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f54a8101dd0 tx=0x7f5498009310 comp rx=0 tx=0).stop 2024-03-30T00:58:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54aec00700 1 -- 172.21.15.116:0/4219222140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a8102910 msgr2=0x7f54a80736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54aec00700 1 --2- 172.21.15.116:0/4219222140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a8102910 0x7f54a80736e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f54a000c370 tx=0x7f54a000c680 comp rx=0 tx=0).stop 2024-03-30T00:58:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54aec00700 1 -- 172.21.15.116:0/4219222140 shutdown_connections 2024-03-30T00:58:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54aec00700 1 --2- 172.21.15.116:0/4219222140 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f548c0a1120 0x7f548c0a35e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54aec00700 1 --2- 172.21.15.116:0/4219222140 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54a8100d80 0x7f54a8074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54aec00700 1 --2- 172.21.15.116:0/4219222140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54a8102910 0x7f54a80736e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54aec00700 1 --2- 172.21.15.116:0/4219222140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54a8101f70 0x7f54a80731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.862+0000 7f54aec00700 1 -- 172.21.15.116:0/4219222140 >> 172.21.15.116:0/4219222140 conn(0x7f54a80fc2f0 msgr2=0x7f54a81053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.863+0000 7f54aec00700 1 -- 172.21.15.116:0/4219222140 shutdown_connections 2024-03-30T00:58:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:20.863+0000 7f54aec00700 1 -- 172.21.15.116:0/4219222140 wait complete. 2024-03-30T00:58:20.885 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:20 smithi116 ceph-mon[38086]: pgmap v305: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:58:20.887 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T00:58:20.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:58:21.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.071+0000 7f15be600700 1 -- 172.21.15.116:0/3544148566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8100c80 msgr2=0x7f15b81010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.071+0000 7f15be600700 1 --2- 172.21.15.116:0/3544148566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8100c80 0x7f15b81010a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f15a80099c0 tx=0x7f15a8009cd0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.071+0000 7f15be600700 1 -- 172.21.15.116:0/3544148566 shutdown_connections 2024-03-30T00:58:21.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.071+0000 7f15be600700 1 --2- 172.21.15.116:0/3544148566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b8102810 0x7f15b81090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.071+0000 7f15be600700 1 --2- 172.21.15.116:0/3544148566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8100c80 0x7f15b81010a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.071+0000 7f15be600700 1 --2- 172.21.15.116:0/3544148566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15b8101e70 0x7f15b81022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.071+0000 7f15be600700 1 -- 172.21.15.116:0/3544148566 >> 172.21.15.116:0/3544148566 conn(0x7f15b80fc1f0 msgr2=0x7f15b80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:21.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.071+0000 7f15be600700 1 -- 172.21.15.116:0/3544148566 shutdown_connections 2024-03-30T00:58:21.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.071+0000 7f15be600700 1 -- 172.21.15.116:0/3544148566 wait complete. 2024-03-30T00:58:21.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.072+0000 7f15be600700 1 Processor -- start 2024-03-30T00:58:21.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.072+0000 7f15be600700 1 -- start start 2024-03-30T00:58:21.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15be600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b8100c80 0x7f15b8074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15be600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8101e70 0x7f15b80731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15be600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15b8102810 0x7f15b80736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15be600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b8073e80 con 0x7f15b8101e70 2024-03-30T00:58:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15be600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b8073ff0 con 0x7f15b8102810 2024-03-30T00:58:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15be600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b81a0c80 con 0x7f15b8100c80 2024-03-30T00:58:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15b6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8101e70 0x7f15b80731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15b6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8101e70 0x7f15b80731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40508/0 (socket says 172.21.15.116:40508) 2024-03-30T00:58:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15b6a00700 1 -- 172.21.15.116:0/1161832630 learned_addr learned my addr 172.21.15.116:0/1161832630 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15b7400700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b8100c80 0x7f15b8074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15b7e00700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15b8102810 0x7f15b80736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15b6a00700 1 -- 172.21.15.116:0/1161832630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b8100c80 msgr2=0x7f15b8074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15b6a00700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b8100c80 0x7f15b8074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15b6a00700 1 -- 172.21.15.116:0/1161832630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15b8102810 msgr2=0x7f15b80736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15b6a00700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15b8102810 0x7f15b80736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15b6a00700 1 -- 172.21.15.116:0/1161832630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a80095f0 con 0x7f15b8101e70 2024-03-30T00:58:21.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.073+0000 7f15b7e00700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15b8102810 0x7f15b80736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:58:21.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.074+0000 7f15b6a00700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8101e70 0x7f15b80731a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f15a400d980 tx=0x7f15a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:21.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.074+0000 7f15afe00700 1 -- 172.21.15.116:0/1161832630 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a4004220 con 0x7f15b8101e70 2024-03-30T00:58:21.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.074+0000 7f15be600700 1 -- 172.21.15.116:0/1161832630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b81a0f10 con 0x7f15b8101e70 2024-03-30T00:58:21.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.074+0000 7f15afe00700 1 -- 172.21.15.116:0/1161832630 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15a4007640 con 0x7f15b8101e70 2024-03-30T00:58:21.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.074+0000 7f15afe00700 1 -- 172.21.15.116:0/1161832630 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a4005610 con 0x7f15b8101e70 2024-03-30T00:58:21.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.074+0000 7f15be600700 1 -- 172.21.15.116:0/1161832630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b81a1390 con 0x7f15b8101e70 2024-03-30T00:58:21.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.075+0000 7f15be600700 1 -- 172.21.15.116:0/1161832630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b81080d0 con 0x7f15b8101e70 2024-03-30T00:58:21.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.077+0000 7f15afe00700 1 -- 172.21.15.116:0/1161832630 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f15a4004750 con 0x7f15b8101e70 2024-03-30T00:58:21.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.078+0000 7f15afe00700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f159c0a1120 0x7f159c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.079+0000 7f15afe00700 1 -- 172.21.15.116:0/1161832630 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f15a40ccf70 con 0x7f15b8101e70 2024-03-30T00:58:21.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.079+0000 7f15b7400700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f159c0a1120 0x7f159c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.079+0000 7f15b7400700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f159c0a1120 0x7f159c0a35e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f15a8003730 tx=0x7f15a8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:21.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.081+0000 7f15afe00700 1 -- 172.21.15.116:0/1161832630 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f15a4096390 con 0x7f15b8101e70 2024-03-30T00:58:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.334+0000 7f15be600700 1 -- 172.21.15.116:0/1161832630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15b8067a10 con 0x7f15b8101e70 2024-03-30T00:58:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.335+0000 7f15afe00700 1 -- 172.21.15.116:0/1161832630 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10886 (secure 0 0 0) 0x7f15a4016020 con 0x7f15b8101e70 2024-03-30T00:58:21.342 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:58:21.343 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:57:40.442321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T00:58:21.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.337+0000 7f15be600700 1 -- 172.21.15.116:0/1161832630 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f159c0a1120 msgr2=0x7f159c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.337+0000 7f15be600700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f159c0a1120 0x7f159c0a35e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f15a8003730 tx=0x7f15a8000bc0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.338+0000 7f15be600700 1 -- 172.21.15.116:0/1161832630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8101e70 msgr2=0x7f15b80731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.338+0000 7f15be600700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8101e70 0x7f15b80731a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f15a400d980 tx=0x7f15a400dc90 comp rx=0 tx=0).stop 2024-03-30T00:58:21.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.338+0000 7f15be600700 1 -- 172.21.15.116:0/1161832630 shutdown_connections 2024-03-30T00:58:21.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.338+0000 7f15be600700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f159c0a1120 0x7f159c0a35e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.338+0000 7f15be600700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b8100c80 0x7f15b8074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.338+0000 7f15be600700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15b8102810 0x7f15b80736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.338+0000 7f15be600700 1 --2- 172.21.15.116:0/1161832630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8101e70 0x7f15b80731a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.338+0000 7f15be600700 1 -- 172.21.15.116:0/1161832630 >> 172.21.15.116:0/1161832630 conn(0x7f15b80fc1f0 msgr2=0x7f15b8105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.338+0000 7f15be600700 1 -- 172.21.15.116:0/1161832630 shutdown_connections 2024-03-30T00:58:21.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.338+0000 7f15be600700 1 -- 172.21.15.116:0/1161832630 wait complete. 2024-03-30T00:58:21.346 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 23 2024-03-30T00:58:21.363 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T00:58:21.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.554+0000 7f05be800700 1 -- 172.21.15.116:0/3362864318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b8102910 msgr2=0x7f05b81091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.554+0000 7f05be800700 1 --2- 172.21.15.116:0/3362864318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b8102910 0x7f05b81091c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f05b000b0d0 tx=0x7f05b000b3e0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.554+0000 7f05be800700 1 -- 172.21.15.116:0/3362864318 shutdown_connections 2024-03-30T00:58:21.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.554+0000 7f05be800700 1 --2- 172.21.15.116:0/3362864318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05b8100d80 0x7f05b81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.554+0000 7f05be800700 1 --2- 172.21.15.116:0/3362864318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b8102910 0x7f05b81091c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.554+0000 7f05be800700 1 --2- 172.21.15.116:0/3362864318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05b8101f70 0x7f05b81023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.554+0000 7f05be800700 1 -- 172.21.15.116:0/3362864318 >> 172.21.15.116:0/3362864318 conn(0x7f05b80fc2f0 msgr2=0x7f05b80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:21.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.554+0000 7f05be800700 1 -- 172.21.15.116:0/3362864318 shutdown_connections 2024-03-30T00:58:21.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.554+0000 7f05be800700 1 -- 172.21.15.116:0/3362864318 wait complete. 2024-03-30T00:58:21.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.555+0000 7f05be800700 1 Processor -- start 2024-03-30T00:58:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.555+0000 7f05be800700 1 -- start start 2024-03-30T00:58:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.555+0000 7f05be800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b8100d80 0x7f05b8074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05be800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05b8101f70 0x7f05b80731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05be800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05b8102910 0x7f05b80736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05be800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b8073e80 con 0x7f05b8100d80 2024-03-30T00:58:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05be800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b8073ff0 con 0x7f05b8101f70 2024-03-30T00:58:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05be800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b81a0eb0 con 0x7f05b8102910 2024-03-30T00:58:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05b7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b8100d80 0x7f05b8074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05b7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b8100d80 0x7f05b8074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40530/0 (socket says 172.21.15.116:40530) 2024-03-30T00:58:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05b7400700 1 -- 172.21.15.116:0/2601318724 learned_addr learned my addr 172.21.15.116:0/2601318724 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05b6a00700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05b8101f70 0x7f05b80731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05b7e00700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05b8102910 0x7f05b80736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05b7400700 1 -- 172.21.15.116:0/2601318724 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05b8102910 msgr2=0x7f05b80736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05b7400700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05b8102910 0x7f05b80736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05b7400700 1 -- 172.21.15.116:0/2601318724 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05b8101f70 msgr2=0x7f05b80731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05b7400700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05b8101f70 0x7f05b80731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05b7400700 1 -- 172.21.15.116:0/2601318724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b0009d00 con 0x7f05b8100d80 2024-03-30T00:58:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05b7e00700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05b8102910 0x7f05b80736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:58:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.556+0000 7f05b7400700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b8100d80 0x7f05b8074b10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f05a800e970 tx=0x7f05a800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.557+0000 7f05afe00700 1 -- 172.21.15.116:0/2601318724 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a8009790 con 0x7f05b8100d80 2024-03-30T00:58:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.557+0000 7f05afe00700 1 -- 172.21.15.116:0/2601318724 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05a800eec0 con 0x7f05b8100d80 2024-03-30T00:58:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.557+0000 7f05be800700 1 -- 172.21.15.116:0/2601318724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b81a1140 con 0x7f05b8100d80 2024-03-30T00:58:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.557+0000 7f05afe00700 1 -- 172.21.15.116:0/2601318724 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a8003c90 con 0x7f05b8100d80 2024-03-30T00:58:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.557+0000 7f05be800700 1 -- 172.21.15.116:0/2601318724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b81a16a0 con 0x7f05b8100d80 2024-03-30T00:58:21.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.560+0000 7f05afe00700 1 -- 172.21.15.116:0/2601318724 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f05a800cc70 con 0x7f05b8100d80 2024-03-30T00:58:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.561+0000 7f05d22d4b80 1 Processor -- start 2024-03-30T00:58:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.561+0000 7f05afe00700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f059c0a11a0 0x7f059c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.561+0000 7f05d22d4b80 1 -- start start 2024-03-30T00:58:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.561+0000 7f05afe00700 1 -- 172.21.15.116:0/2601318724 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f05a8014070 con 0x7f05b8100d80 2024-03-30T00:58:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.561+0000 7f05b7e00700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f059c0a11a0 0x7f059c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.561+0000 7f05d22d4b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee37d0 0x5579b5ee3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.561+0000 7f05d22d4b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee4130 0x5579b5ee49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.561+0000 7f05d22d4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5ee5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.561+0000 7f05d22d4b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579b5e3b990 con 0x5579b5ee37d0 2024-03-30T00:58:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.562+0000 7f05d22d4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579b5e39350 con 0x5579b5ee4fd0 2024-03-30T00:58:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.562+0000 7f05d22d4b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579b5e34c40 con 0x5579b5ee4130 2024-03-30T00:58:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.562+0000 7f05b7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee37d0 0x5579b5ee3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.562+0000 7f05b7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee37d0 0x5579b5ee3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40534/0 (socket says 172.21.15.116:40534) 2024-03-30T00:58:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.562+0000 7f05b7400700 1 -- 172.21.15.116:0/1280686703 learned_addr learned my addr 172.21.15.116:0/1280686703 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.562+0000 7f05b6a00700 1 --2- 172.21.15.116:0/1280686703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee4130 0x5579b5ee49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.562+0000 7f05b7e00700 1 --2- 172.21.15.116:0/1280686703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5ee5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.562+0000 7f05b7400700 1 -- 172.21.15.116:0/1280686703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee4130 msgr2=0x5579b5ee49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.562+0000 7f05b7400700 1 --2- 172.21.15.116:0/1280686703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee4130 0x5579b5ee49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.562+0000 7f05b7400700 1 -- 172.21.15.116:0/1280686703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 msgr2=0x5579b5ee5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.562+0000 7f05b7400700 1 --2- 172.21.15.116:0/1280686703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5ee5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.562+0000 7f05b7400700 1 -- 172.21.15.116:0/1280686703 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579b5cb63b0 con 0x5579b5ee37d0 2024-03-30T00:58:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.562+0000 7f05b7e00700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f059c0a11a0 0x7f059c0a3660 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009fd0 tx=0x7f05b0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.562+0000 7f05b7400700 1 --2- 172.21.15.116:0/1280686703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee37d0 0x5579b5ee3bf0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f05a8096920 tx=0x7f05a8099810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39df2cf9ee87aced server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.563+0000 7f05acc00700 1 -- 172.21.15.116:0/1280686703 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a80b1070 con 0x5579b5ee37d0 2024-03-30T00:58:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.563+0000 7f05acc00700 1 -- 172.21.15.116:0/1280686703 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05a809dce0 con 0x5579b5ee37d0 2024-03-30T00:58:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.563+0000 7f05acc00700 1 -- 172.21.15.116:0/1280686703 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a809a110 con 0x5579b5ee37d0 2024-03-30T00:58:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.563+0000 7f05d22d4b80 1 -- 172.21.15.116:0/1280686703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee37d0 msgr2=0x5579b5ee3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.563+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/1280686703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee37d0 0x5579b5ee3bf0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f05a8096920 tx=0x7f05a8099810 comp rx=0 tx=0).stop 2024-03-30T00:58:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.563+0000 7f05d22d4b80 1 -- 172.21.15.116:0/1280686703 shutdown_connections 2024-03-30T00:58:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.563+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/1280686703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5ee5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.563+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/1280686703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee37d0 0x5579b5ee3bf0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.563+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/1280686703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee4130 0x5579b5ee49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.563+0000 7f05d22d4b80 1 -- 172.21.15.116:0/1280686703 >> 172.21.15.116:0/1280686703 conn(0x5579b5e426a0 msgr2=0x5579b5ede9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.563+0000 7f05d22d4b80 1 -- 172.21.15.116:0/1280686703 shutdown_connections 2024-03-30T00:58:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.563+0000 7f05d22d4b80 1 -- 172.21.15.116:0/1280686703 wait complete. 2024-03-30T00:58:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.564+0000 7f05d22d4b80 1 -- start start 2024-03-30T00:58:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.564+0000 7f05d22d4b80 1 Processor -- start 2024-03-30T00:58:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05d22d4b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee37d0 0x5579b5eed600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05d22d4b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee4130 0x5579b5eedb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05d22d4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5ef4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05d22d4b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579b5dd0d80 con 0x5579b5ee37d0 2024-03-30T00:58:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05d22d4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579b5dd0a20 con 0x5579b5ee4fd0 2024-03-30T00:58:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05d22d4b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579b5cb86a0 con 0x5579b5ee4130 2024-03-30T00:58:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05d22d4b80 10 client.0 _ll_register_callbacks cb 0x5579b5dda470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05b7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee37d0 0x5579b5eed600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05b6a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee4130 0x5579b5eedb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05b7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee37d0 0x5579b5eed600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40542/0 (socket says 172.21.15.116:40542) 2024-03-30T00:58:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05b7400700 1 -- 172.21.15.116:0/633843992 learned_addr learned my addr 172.21.15.116:0/633843992 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05b7e00700 1 --2- 172.21.15.116:0/633843992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5ef4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05b7400700 1 -- 172.21.15.116:0/633843992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee4130 msgr2=0x5579b5eedb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05b7400700 1 --2- 172.21.15.116:0/633843992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee4130 0x5579b5eedb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05b7400700 1 -- 172.21.15.116:0/633843992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 msgr2=0x5579b5ef4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05b7400700 1 --2- 172.21.15.116:0/633843992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5ef4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05b7400700 1 -- 172.21.15.116:0/633843992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b81a16a0 con 0x5579b5ee37d0 2024-03-30T00:58:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05b7e00700 1 --2- 172.21.15.116:0/633843992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5ef4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:58:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05b7400700 1 --2- 172.21.15.116:0/633843992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee37d0 0x5579b5eed600 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f05a8099e40 tx=0x7f05a8095b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5136cd4d04e71929 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.565+0000 7f05a1600700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:58:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.566+0000 7f05a1600700 1 -- 172.21.15.116:0/633843992 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a80b6070 con 0x5579b5ee37d0 2024-03-30T00:58:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.566+0000 7f05a1600700 1 -- 172.21.15.116:0/633843992 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f05a809a320 con 0x5579b5ee37d0 2024-03-30T00:58:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.566+0000 7f05d22d4b80 1 -- 172.21.15.116:0/633843992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5579b5df5380 con 0x5579b5ee37d0 2024-03-30T00:58:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.566+0000 7f05a1600700 1 -- 172.21.15.116:0/633843992 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05a80b1070 con 0x5579b5ee37d0 2024-03-30T00:58:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.566+0000 7f05d22d4b80 10 client.15522 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.566+0000 7f05d22d4b80 1 -- 172.21.15.116:0/633843992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5579b5c62b60 con 0x5579b5ee37d0 2024-03-30T00:58:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.566+0000 7f05a1600700 1 -- 172.21.15.116:0/633843992 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5579b5df5380 con 0x5579b5ee37d0 2024-03-30T00:58:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.566+0000 7f05a1600700 1 -- 172.21.15.116:0/633843992 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f05a8099070 con 0x5579b5ee37d0 2024-03-30T00:58:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.566+0000 7f05d22d4b80 10 client.15522 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.566+0000 7f05d22d4b80 10 client.15522 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T00:58:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.566+0000 7f05d22d4b80 20 client.15522 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.566+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/633843992 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5579b5ef8e70 0x5579b5efb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.566+0000 7f05d22d4b80 4 client.15522 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:58:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.566+0000 7f05d22d4b80 1 -- 172.21.15.116:0/633843992 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5579b5e2d700 con 0x5579b5ef8e70 2024-03-30T00:58:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.567+0000 7f05b6a00700 1 --2- 172.21.15.116:0/633843992 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5579b5ef8e70 0x5579b5efb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.567+0000 7f05a1600700 10 client.15522 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T00:58:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.567+0000 7f05b6a00700 1 --2- 172.21.15.116:0/633843992 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5579b5ef8e70 0x5579b5efb260 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d6aefce81a033fc3 server_cookie=778b14b339cd9941 in_seq=0 out_seq=0 2024-03-30T00:58:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.568+0000 7f05a1600700 1 -- 172.21.15.116:0/633843992 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5579b5e2d700 con 0x5579b5ef8e70 2024-03-30T00:58:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.568+0000 7f05a1600700 10 client.15522 handle_command_reply: tid=0 2024-03-30T00:58:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.568+0000 7f05d22d4b80 1 client.15522 shutdown 2024-03-30T00:58:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.568+0000 7f05d22d4b80 1 -- 172.21.15.116:0/633843992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee37d0 msgr2=0x5579b5eed600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.568+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/633843992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee37d0 0x5579b5eed600 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f05a8099e40 tx=0x7f05a8095b80 comp rx=0 tx=0).stop 2024-03-30T00:58:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.568+0000 7f05d22d4b80 1 -- 172.21.15.116:0/633843992 shutdown_connections 2024-03-30T00:58:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.568+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/633843992 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5579b5ef8e70 0x5579b5efb260 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.568+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/633843992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5ef4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.568+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/633843992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee37d0 0x5579b5eed600 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.568+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/633843992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee4130 0x5579b5eedb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.568+0000 7f05a1600700 0 client.15522 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T00:58:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.568+0000 7f05d22d4b80 1 -- 172.21.15.116:0/633843992 >> 172.21.15.116:0/633843992 conn(0x5579b5e426a0 msgr2=0x5579b5ee2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.569+0000 7f05d22d4b80 1 -- 172.21.15.116:0/633843992 shutdown_connections 2024-03-30T00:58:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.569+0000 7f05d22d4b80 1 -- 172.21.15.116:0/633843992 wait complete. 2024-03-30T00:58:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.569+0000 7f05d22d4b80 20 client.15522 trim_cache size 0 max 16384 2024-03-30T00:58:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.582+0000 7f05d22d4b80 1 Processor -- start 2024-03-30T00:58:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.582+0000 7f05d22d4b80 1 -- start start 2024-03-30T00:58:21.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.582+0000 7f05d22d4b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee37d0 0x5579b5ef5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.582+0000 7f05d22d4b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee4130 0x5579b5ef4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.582+0000 7f05d22d4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5eddf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.582+0000 7f05d22d4b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579b5ee4e50 con 0x5579b5ee4130 2024-03-30T00:58:21.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.582+0000 7f05d22d4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579b5ef5d00 con 0x5579b5ee4fd0 2024-03-30T00:58:21.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.582+0000 7f05d22d4b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579b5eee690 con 0x5579b5ee37d0 2024-03-30T00:58:21.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.583+0000 7f05b6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee4130 0x5579b5ef4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.583+0000 7f05b6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee4130 0x5579b5ef4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40548/0 (socket says 172.21.15.116:40548) 2024-03-30T00:58:21.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.583+0000 7f05b6a00700 1 -- 172.21.15.116:0/2663938097 learned_addr learned my addr 172.21.15.116:0/2663938097 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:21.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.583+0000 7f05b7e00700 1 --2- 172.21.15.116:0/2663938097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5eddf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.583+0000 7f05b7400700 1 --2- 172.21.15.116:0/2663938097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee37d0 0x5579b5ef5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.583+0000 7f05b6a00700 1 -- 172.21.15.116:0/2663938097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee37d0 msgr2=0x5579b5ef5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.583+0000 7f05b6a00700 1 --2- 172.21.15.116:0/2663938097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee37d0 0x5579b5ef5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.583+0000 7f05b6a00700 1 -- 172.21.15.116:0/2663938097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 msgr2=0x5579b5eddf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.583+0000 7f05b6a00700 1 --2- 172.21.15.116:0/2663938097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5eddf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.583+0000 7f05b6a00700 1 -- 172.21.15.116:0/2663938097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579b5dfaea0 con 0x5579b5ee4130 2024-03-30T00:58:21.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.583+0000 7f05b6a00700 1 --2- 172.21.15.116:0/2663938097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee4130 0x5579b5ef4b40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f05a40097c0 tx=0x7f05a4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a7319b66ba0202 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:21.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.583+0000 7f05ad600700 1 -- 172.21.15.116:0/2663938097 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a4017070 con 0x5579b5ee4130 2024-03-30T00:58:21.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.584+0000 7f05ad600700 1 -- 172.21.15.116:0/2663938097 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05a4018040 con 0x5579b5ee4130 2024-03-30T00:58:21.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.584+0000 7f05ad600700 1 -- 172.21.15.116:0/2663938097 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a4010690 con 0x5579b5ee4130 2024-03-30T00:58:21.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.584+0000 7f05d22d4b80 1 -- 172.21.15.116:0/2663938097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee4130 msgr2=0x5579b5ef4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.584+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/2663938097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee4130 0x5579b5ef4b40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f05a40097c0 tx=0x7f05a4009260 comp rx=0 tx=0).stop 2024-03-30T00:58:21.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.584+0000 7f05d22d4b80 1 -- 172.21.15.116:0/2663938097 shutdown_connections 2024-03-30T00:58:21.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.584+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/2663938097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee37d0 0x5579b5ef5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.584+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/2663938097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5eddf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.584+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/2663938097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee4130 0x5579b5ef4b40 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.584+0000 7f05d22d4b80 1 -- 172.21.15.116:0/2663938097 >> 172.21.15.116:0/2663938097 conn(0x5579b5e426a0 msgr2=0x5579b5ed75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:21.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.584+0000 7f05d22d4b80 1 -- 172.21.15.116:0/2663938097 shutdown_connections 2024-03-30T00:58:21.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.584+0000 7f05d22d4b80 1 -- 172.21.15.116:0/2663938097 wait complete. 2024-03-30T00:58:21.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.585+0000 7f05d22d4b80 1 -- start start 2024-03-30T00:58:21.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.585+0000 7f05d22d4b80 1 Processor -- start 2024-03-30T00:58:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.585+0000 7f05d22d4b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee37d0 0x5579b5ef2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.585+0000 7f05d22d4b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee4130 0x5579b5ef2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.585+0000 7f05d22d4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5ef3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.585+0000 7f05d22d4b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579b5ef44e0 con 0x5579b5ee4130 2024-03-30T00:58:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.585+0000 7f05d22d4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579b5ef4650 con 0x5579b5ee4fd0 2024-03-30T00:58:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.585+0000 7f05d22d4b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579b5ef4810 con 0x5579b5ee37d0 2024-03-30T00:58:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.585+0000 7f05d22d4b80 10 client.0 _ll_register_callbacks cb 0x5579b5dda470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.585+0000 7f05b6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee4130 0x5579b5ef2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.586+0000 7f05b6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee4130 0x5579b5ef2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40564/0 (socket says 172.21.15.116:40564) 2024-03-30T00:58:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.586+0000 7f05b7400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee37d0 0x5579b5ef2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.586+0000 7f05b6a00700 1 -- 172.21.15.116:0/2961633048 learned_addr learned my addr 172.21.15.116:0/2961633048 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.586+0000 7f05b7e00700 1 --2- 172.21.15.116:0/2961633048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5ef3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.586+0000 7f05b7400700 1 -- 172.21.15.116:0/2961633048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 msgr2=0x5579b5ef3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.586+0000 7f05b7400700 1 --2- 172.21.15.116:0/2961633048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5ef3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.586+0000 7f05b7400700 1 -- 172.21.15.116:0/2961633048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee4130 msgr2=0x5579b5ef2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.586+0000 7f05b7400700 1 --2- 172.21.15.116:0/2961633048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee4130 0x5579b5ef2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.586+0000 7f05b7400700 1 -- 172.21.15.116:0/2961633048 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a4009b30 con 0x5579b5ee37d0 2024-03-30T00:58:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.586+0000 7f05b7400700 1 --2- 172.21.15.116:0/2961633048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee37d0 0x5579b5ef2aa0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f05a809a760 tx=0x7f05a801fde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a649fb4bed03675 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.586+0000 7f05a1600700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T00:58:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.586+0000 7f05a1600700 1 -- 172.21.15.116:0/2961633048 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a80b1070 con 0x5579b5ee37d0 2024-03-30T00:58:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.586+0000 7f05d22d4b80 1 -- 172.21.15.116:0/2961633048 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05a809dce0 con 0x5579b5ee37d0 2024-03-30T00:58:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.586+0000 7f05a1600700 1 -- 172.21.15.116:0/2961633048 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f05a8098300 con 0x5579b5ee37d0 2024-03-30T00:58:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.587+0000 7f05a1600700 1 -- 172.21.15.116:0/2961633048 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05a80b82b0 con 0x5579b5ee37d0 2024-03-30T00:58:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.587+0000 7f05d22d4b80 10 client.25567 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:21.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.587+0000 7f05d22d4b80 1 -- 172.21.15.116:0/2961633048 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5579b5cbacd0 con 0x5579b5ee37d0 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.587+0000 7f05a1600700 1 -- 172.21.15.116:0/2961633048 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05a809dce0 con 0x5579b5ee37d0 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.587+0000 7f05a1600700 1 -- 172.21.15.116:0/2961633048 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f05a80a1ea0 con 0x5579b5ee37d0 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.587+0000 7f05d22d4b80 10 client.25567 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.587+0000 7f05d22d4b80 10 client.25567 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.587+0000 7f05d22d4b80 20 client.25567 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.587+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/2961633048 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5579b5f4d820 0x5579b5f4fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.587+0000 7f05d22d4b80 4 client.25567 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.587+0000 7f05d22d4b80 1 -- 172.21.15.116:0/2961633048 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5579b5c83cf0 con 0x5579b5f4d820 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.588+0000 7f05b6a00700 1 --2- 172.21.15.116:0/2961633048 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5579b5f4d820 0x5579b5f4fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.588+0000 7f05a1600700 10 client.25567 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.588+0000 7f05b6a00700 1 --2- 172.21.15.116:0/2961633048 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5579b5f4d820 0x5579b5f4fc10 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=17b01fd0f9317f5d server_cookie=2f3cb01335a5523e in_seq=0 out_seq=0 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.588+0000 7f05a1600700 1 -- 172.21.15.116:0/2961633048 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5579b5c83cf0 con 0x5579b5f4d820 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.588+0000 7f05a1600700 10 client.25567 handle_command_reply: tid=0 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.588+0000 7f05d22d4b80 1 client.25567 shutdown 2024-03-30T00:58:21.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.589+0000 7f05d22d4b80 1 -- 172.21.15.116:0/2961633048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee37d0 msgr2=0x5579b5ef2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.589+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/2961633048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee37d0 0x5579b5ef2aa0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f05a809a760 tx=0x7f05a801fde0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.589+0000 7f05d22d4b80 1 -- 172.21.15.116:0/2961633048 shutdown_connections 2024-03-30T00:58:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.589+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/2961633048 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5579b5f4d820 0x5579b5f4fc10 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.589+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/2961633048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5579b5ee37d0 0x5579b5ef2aa0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.589+0000 7f05a1600700 0 client.25567 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T00:58:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.589+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/2961633048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579b5ee4fd0 0x5579b5ef3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.589+0000 7f05d22d4b80 1 --2- 172.21.15.116:0/2961633048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579b5ee4130 0x5579b5ef2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.589+0000 7f05d22d4b80 1 -- 172.21.15.116:0/2961633048 >> 172.21.15.116:0/2961633048 conn(0x5579b5e426a0 msgr2=0x5579b5ef5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.589+0000 7f05d22d4b80 1 -- 172.21.15.116:0/2961633048 shutdown_connections 2024-03-30T00:58:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.589+0000 7f05d22d4b80 1 -- 172.21.15.116:0/2961633048 wait complete. 2024-03-30T00:58:21.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.589+0000 7f05d22d4b80 20 client.25567 trim_cache size 0 max 16384 2024-03-30T00:58:21.600 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T00:58:21.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.589+0000 7f05be800700 1 -- 172.21.15.116:0/2601318724 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f059c0a11a0 msgr2=0x7f059c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.589+0000 7f05be800700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f059c0a11a0 0x7f059c0a3660 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009fd0 tx=0x7f05b0009500 comp rx=0 tx=0).stop 2024-03-30T00:58:21.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.590+0000 7f05be800700 1 -- 172.21.15.116:0/2601318724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b8100d80 msgr2=0x7f05b8074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.590+0000 7f05be800700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b8100d80 0x7f05b8074b10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f05a800e970 tx=0x7f05a800ec80 comp rx=0 tx=0).stop 2024-03-30T00:58:21.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.590+0000 7f05be800700 1 -- 172.21.15.116:0/2601318724 shutdown_connections 2024-03-30T00:58:21.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.590+0000 7f05be800700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f059c0a11a0 0x7f059c0a3660 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.590+0000 7f05be800700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05b8102910 0x7f05b80736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.590+0000 7f05be800700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b8100d80 0x7f05b8074b10 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.590+0000 7f05be800700 1 --2- 172.21.15.116:0/2601318724 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05b8101f70 0x7f05b80731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.590+0000 7f05be800700 1 -- 172.21.15.116:0/2601318724 >> 172.21.15.116:0/2601318724 conn(0x7f05b80fc2f0 msgr2=0x7f05b81052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:21.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.590+0000 7f05be800700 1 -- 172.21.15.116:0/2601318724 shutdown_connections 2024-03-30T00:58:21.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.590+0000 7f05be800700 1 -- 172.21.15.116:0/2601318724 wait complete. 2024-03-30T00:58:21.612 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:21 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1161832630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:58:21.614 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T00:58:21.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:21 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1161832630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:58:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.801+0000 7f4640600700 1 -- 172.21.15.116:0/2281454497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4638100d80 msgr2=0x7f46381011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.801+0000 7f4640600700 1 --2- 172.21.15.116:0/2281454497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4638100d80 0x7f46381011a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f462c0099c0 tx=0x7f462c009cd0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.801+0000 7f4640600700 1 -- 172.21.15.116:0/2281454497 shutdown_connections 2024-03-30T00:58:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.801+0000 7f4640600700 1 --2- 172.21.15.116:0/2281454497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4638102910 0x7f46381091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.801+0000 7f4640600700 1 --2- 172.21.15.116:0/2281454497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4638100d80 0x7f46381011a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.801+0000 7f4640600700 1 --2- 172.21.15.116:0/2281454497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4638101f70 0x7f46381023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.801+0000 7f4640600700 1 -- 172.21.15.116:0/2281454497 >> 172.21.15.116:0/2281454497 conn(0x7f46380fc2f0 msgr2=0x7f46380fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:21.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.801+0000 7f4640600700 1 -- 172.21.15.116:0/2281454497 shutdown_connections 2024-03-30T00:58:21.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.801+0000 7f4640600700 1 -- 172.21.15.116:0/2281454497 wait complete. 2024-03-30T00:58:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.802+0000 7f4640600700 1 Processor -- start 2024-03-30T00:58:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.802+0000 7f4640600700 1 -- start start 2024-03-30T00:58:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.802+0000 7f4640600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4638100d80 0x7f4638074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f4640600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4638101f70 0x7f46380731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f4640600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4638102910 0x7f46380736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f4640600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4638073e80 con 0x7f4638102910 2024-03-30T00:58:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f4640600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4638073ff0 con 0x7f4638101f70 2024-03-30T00:58:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f4640600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46381a0dc0 con 0x7f4638100d80 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f463e600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4638102910 0x7f46380736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f463e600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4638102910 0x7f46380736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40584/0 (socket says 172.21.15.116:40584) 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f463e600700 1 -- 172.21.15.116:0/3748133209 learned_addr learned my addr 172.21.15.116:0/3748133209 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f463d200700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4638101f70 0x7f46380731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f463dc00700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4638100d80 0x7f4638074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f463e600700 1 -- 172.21.15.116:0/3748133209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4638100d80 msgr2=0x7f4638074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f463e600700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4638100d80 0x7f4638074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f463e600700 1 -- 172.21.15.116:0/3748133209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4638101f70 msgr2=0x7f46380731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f463e600700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4638101f70 0x7f46380731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f463e600700 1 -- 172.21.15.116:0/3748133209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4624009520 con 0x7f4638102910 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f463d200700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4638101f70 0x7f46380731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.803+0000 7f463e600700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4638102910 0x7f46380736e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f463000c370 tx=0x7f463000c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.804+0000 7f4636000700 1 -- 172.21.15.116:0/3748133209 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4630017070 con 0x7f4638102910 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.804+0000 7f4636000700 1 -- 172.21.15.116:0/3748133209 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f463000f040 con 0x7f4638102910 2024-03-30T00:58:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.804+0000 7f4636000700 1 -- 172.21.15.116:0/3748133209 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463000e050 con 0x7f4638102910 2024-03-30T00:58:21.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.804+0000 7f4640600700 1 -- 172.21.15.116:0/3748133209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4630009c40 con 0x7f4638102910 2024-03-30T00:58:21.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.804+0000 7f4640600700 1 -- 172.21.15.116:0/3748133209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462c0095f0 con 0x7f4638102910 2024-03-30T00:58:21.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.807+0000 7f4636000700 1 -- 172.21.15.116:0/3748133209 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f463000cb00 con 0x7f4638102910 2024-03-30T00:58:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.808+0000 7f4654065b80 1 Processor -- start 2024-03-30T00:58:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.808+0000 7f4636000700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46200a1120 0x7f46200a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.808+0000 7f4654065b80 1 -- start start 2024-03-30T00:58:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.808+0000 7f4636000700 1 -- 172.21.15.116:0/3748133209 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f46300cc560 con 0x7f4638102910 2024-03-30T00:58:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f463d200700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46200a1120 0x7f46200a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f4654065b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb347d0 0x55bc6fb34bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f4654065b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb35130 0x55bc6fb359f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f4654065b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb36450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f4654065b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6fa8c990 con 0x55bc6fb35fd0 2024-03-30T00:58:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f4654065b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6fa8a350 con 0x55bc6fb347d0 2024-03-30T00:58:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f4654065b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6fa85c40 con 0x55bc6fb35130 2024-03-30T00:58:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f463e600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb36450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f463e600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb36450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40598/0 (socket says 172.21.15.116:40598) 2024-03-30T00:58:21.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f463e600700 1 -- 172.21.15.116:0/4184688879 learned_addr learned my addr 172.21.15.116:0/4184688879 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:21.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f463dc00700 1 --2- 172.21.15.116:0/4184688879 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb347d0 0x55bc6fb34bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f463d200700 1 --2- 172.21.15.116:0/4184688879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb35130 0x55bc6fb359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f463d200700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46200a1120 0x7f46200a35e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4624011630 tx=0x7f46240119f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:21.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f463e600700 1 -- 172.21.15.116:0/4184688879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb35130 msgr2=0x55bc6fb359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f463e600700 1 --2- 172.21.15.116:0/4184688879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb35130 0x55bc6fb359f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f463e600700 1 -- 172.21.15.116:0/4184688879 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb347d0 msgr2=0x55bc6fb34bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f463e600700 1 --2- 172.21.15.116:0/4184688879 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb347d0 0x55bc6fb34bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.809+0000 7f463e600700 1 -- 172.21.15.116:0/4184688879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc6f9073b0 con 0x55bc6fb35fd0 2024-03-30T00:58:21.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.810+0000 7f463e600700 1 --2- 172.21.15.116:0/4184688879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb36450 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f46300090a0 tx=0x7f4630099180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a94d28b9701907 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:21.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.810+0000 7f462aa00700 1 -- 172.21.15.116:0/4184688879 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46300092e0 con 0x55bc6fb35fd0 2024-03-30T00:58:21.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.810+0000 7f462aa00700 1 -- 172.21.15.116:0/4184688879 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4630082e90 con 0x55bc6fb35fd0 2024-03-30T00:58:21.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.810+0000 7f462aa00700 1 -- 172.21.15.116:0/4184688879 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46300979e0 con 0x55bc6fb35fd0 2024-03-30T00:58:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.811+0000 7f4654065b80 1 -- 172.21.15.116:0/4184688879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35fd0 msgr2=0x55bc6fb36450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.811+0000 7f4654065b80 1 --2- 172.21.15.116:0/4184688879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb36450 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f46300090a0 tx=0x7f4630099180 comp rx=0 tx=0).stop 2024-03-30T00:58:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.811+0000 7f4654065b80 1 -- 172.21.15.116:0/4184688879 shutdown_connections 2024-03-30T00:58:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.811+0000 7f4654065b80 1 --2- 172.21.15.116:0/4184688879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb36450 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.811+0000 7f4654065b80 1 --2- 172.21.15.116:0/4184688879 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb347d0 0x55bc6fb34bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.811+0000 7f4654065b80 1 --2- 172.21.15.116:0/4184688879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb35130 0x55bc6fb359f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.811+0000 7f4654065b80 1 -- 172.21.15.116:0/4184688879 >> 172.21.15.116:0/4184688879 conn(0x55bc6fa936a0 msgr2=0x55bc6fb2f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.811+0000 7f4654065b80 1 -- 172.21.15.116:0/4184688879 shutdown_connections 2024-03-30T00:58:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.811+0000 7f4654065b80 1 -- 172.21.15.116:0/4184688879 wait complete. 2024-03-30T00:58:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.811+0000 7f4654065b80 1 -- start start 2024-03-30T00:58:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.812+0000 7f4654065b80 1 Processor -- start 2024-03-30T00:58:21.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.812+0000 7f4654065b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb347d0 0x55bc6fb3e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.812+0000 7f4654065b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35130 0x55bc6fb3eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.812+0000 7f4654065b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb45b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.812+0000 7f4654065b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6fa21d80 con 0x55bc6fb35130 2024-03-30T00:58:21.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.812+0000 7f4654065b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6fa21a20 con 0x55bc6fb35fd0 2024-03-30T00:58:21.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.812+0000 7f4654065b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6f9096a0 con 0x55bc6fb347d0 2024-03-30T00:58:21.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.812+0000 7f4654065b80 10 client.0 _ll_register_callbacks cb 0x55bc6fa2b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:21.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.812+0000 7f463d200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35130 0x55bc6fb3eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.812+0000 7f463d200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35130 0x55bc6fb3eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40614/0 (socket says 172.21.15.116:40614) 2024-03-30T00:58:21.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.812+0000 7f463d200700 1 -- 172.21.15.116:0/1364733483 learned_addr learned my addr 172.21.15.116:0/1364733483 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:21.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.812+0000 7f463e600700 1 --2- 172.21.15.116:0/1364733483 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb45b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.812+0000 7f463dc00700 1 --2- 172.21.15.116:0/1364733483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb347d0 0x55bc6fb3e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.813+0000 7f463d200700 1 -- 172.21.15.116:0/1364733483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb347d0 msgr2=0x55bc6fb3e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.813+0000 7f463d200700 1 --2- 172.21.15.116:0/1364733483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb347d0 0x55bc6fb3e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.813+0000 7f463d200700 1 -- 172.21.15.116:0/1364733483 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35fd0 msgr2=0x55bc6fb45b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.813+0000 7f463d200700 1 --2- 172.21.15.116:0/1364733483 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb45b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.813+0000 7f463d200700 1 -- 172.21.15.116:0/1364733483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462c0095f0 con 0x55bc6fb35130 2024-03-30T00:58:21.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.813+0000 7f463e600700 1 --2- 172.21.15.116:0/1364733483 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb45b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:58:21.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.813+0000 7f463d200700 1 --2- 172.21.15.116:0/1364733483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35130 0x55bc6fb3eb90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f462401ec50 tx=0x7f46240047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e6c187b1bcf5f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.813+0000 7f461f400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:58:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.813+0000 7f461f400700 1 -- 172.21.15.116:0/1364733483 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462401e540 con 0x55bc6fb35130 2024-03-30T00:58:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.813+0000 7f461f400700 1 -- 172.21.15.116:0/1364733483 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f462400f960 con 0x55bc6fb35130 2024-03-30T00:58:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.813+0000 7f4654065b80 1 -- 172.21.15.116:0/1364733483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc6fa46380 con 0x55bc6fb35130 2024-03-30T00:58:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.813+0000 7f461f400700 1 -- 172.21.15.116:0/1364733483 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f462400fb20 con 0x55bc6fb35130 2024-03-30T00:58:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.813+0000 7f4654065b80 10 client.15540 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.813+0000 7f4654065b80 1 -- 172.21.15.116:0/1364733483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bc6f8b3b60 con 0x55bc6fb35130 2024-03-30T00:58:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.814+0000 7f461f400700 1 -- 172.21.15.116:0/1364733483 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bc6fa46380 con 0x55bc6fb35130 2024-03-30T00:58:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.814+0000 7f461f400700 1 -- 172.21.15.116:0/1364733483 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f46240274d0 con 0x55bc6fb35130 2024-03-30T00:58:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.814+0000 7f4654065b80 10 client.15540 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.814+0000 7f4654065b80 10 client.15540 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T00:58:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.814+0000 7f4654065b80 20 client.15540 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.814+0000 7f4654065b80 1 --2- 172.21.15.116:0/1364733483 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55bc6fb4a510 0x55bc6fb4c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.814+0000 7f4654065b80 4 client.15540 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:58:21.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.814+0000 7f4654065b80 1 -- 172.21.15.116:0/1364733483 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bc6fa7e700 con 0x55bc6fb4a510 2024-03-30T00:58:21.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.814+0000 7f463dc00700 1 --2- 172.21.15.116:0/1364733483 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55bc6fb4a510 0x55bc6fb4c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.815+0000 7f461f400700 10 client.15540 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T00:58:21.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.815+0000 7f463dc00700 1 --2- 172.21.15.116:0/1364733483 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55bc6fb4a510 0x55bc6fb4c930 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9a6e13a9f7e41717 server_cookie=c785f0e5b7006ffd in_seq=0 out_seq=0 2024-03-30T00:58:21.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.816+0000 7f461f400700 1 -- 172.21.15.116:0/1364733483 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55bc6fa7e700 con 0x55bc6fb4a510 2024-03-30T00:58:21.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.816+0000 7f461f400700 10 client.15540 handle_command_reply: tid=0 2024-03-30T00:58:21.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.816+0000 7f4654065b80 1 client.15540 shutdown 2024-03-30T00:58:21.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.816+0000 7f4654065b80 1 -- 172.21.15.116:0/1364733483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35130 msgr2=0x55bc6fb3eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.816+0000 7f4654065b80 1 --2- 172.21.15.116:0/1364733483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35130 0x55bc6fb3eb90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f462401ec50 tx=0x7f46240047c0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.816+0000 7f4654065b80 1 -- 172.21.15.116:0/1364733483 shutdown_connections 2024-03-30T00:58:21.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.816+0000 7f4654065b80 1 --2- 172.21.15.116:0/1364733483 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55bc6fb4a510 0x55bc6fb4c930 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.816+0000 7f4654065b80 1 --2- 172.21.15.116:0/1364733483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb347d0 0x55bc6fb3e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.816+0000 7f4654065b80 1 --2- 172.21.15.116:0/1364733483 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb45b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.816+0000 7f4654065b80 1 --2- 172.21.15.116:0/1364733483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35130 0x55bc6fb3eb90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.816+0000 7f461f400700 0 client.15540 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T00:58:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.816+0000 7f4654065b80 1 -- 172.21.15.116:0/1364733483 >> 172.21.15.116:0/1364733483 conn(0x55bc6fa936a0 msgr2=0x55bc6fb33170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.816+0000 7f4654065b80 1 -- 172.21.15.116:0/1364733483 shutdown_connections 2024-03-30T00:58:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.816+0000 7f4654065b80 1 -- 172.21.15.116:0/1364733483 wait complete. 2024-03-30T00:58:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.817+0000 7f4654065b80 20 client.15540 trim_cache size 0 max 16384 2024-03-30T00:58:21.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.830+0000 7f4654065b80 1 Processor -- start 2024-03-30T00:58:21.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.830+0000 7f4654065b80 1 -- start start 2024-03-30T00:58:21.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.830+0000 7f4654065b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb347d0 0x55bc6fb34bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.830+0000 7f4654065b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35130 0x55bc6fb45b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.830+0000 7f4654065b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb3db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.830+0000 7f4654065b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6fb35e90 con 0x55bc6fb35fd0 2024-03-30T00:58:21.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.830+0000 7f4654065b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6fb304e0 con 0x55bc6fb35130 2024-03-30T00:58:21.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.830+0000 7f4654065b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6fb3f6b0 con 0x55bc6fb347d0 2024-03-30T00:58:21.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.830+0000 7f463e600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb3db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.830+0000 7f463e600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb3db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40624/0 (socket says 172.21.15.116:40624) 2024-03-30T00:58:21.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.830+0000 7f463e600700 1 -- 172.21.15.116:0/761365195 learned_addr learned my addr 172.21.15.116:0/761365195 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:21.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.830+0000 7f463dc00700 1 --2- 172.21.15.116:0/761365195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb347d0 0x55bc6fb34bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.830+0000 7f463d200700 1 --2- 172.21.15.116:0/761365195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35130 0x55bc6fb45b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.831+0000 7f463e600700 1 -- 172.21.15.116:0/761365195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb347d0 msgr2=0x55bc6fb34bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.831+0000 7f463e600700 1 --2- 172.21.15.116:0/761365195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb347d0 0x55bc6fb34bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.831+0000 7f463e600700 1 -- 172.21.15.116:0/761365195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35130 msgr2=0x55bc6fb45b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.831+0000 7f463e600700 1 --2- 172.21.15.116:0/761365195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35130 0x55bc6fb45b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.831+0000 7f463e600700 1 -- 172.21.15.116:0/761365195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc6fa4bea0 con 0x55bc6fb35fd0 2024-03-30T00:58:21.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.831+0000 7f463e600700 1 --2- 172.21.15.116:0/761365195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb3db60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f46300992f0 tx=0x7f46300b9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d2f77b2ba28800 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:21.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.831+0000 7f462b400700 1 -- 172.21.15.116:0/761365195 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46300092e0 con 0x55bc6fb35fd0 2024-03-30T00:58:21.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.831+0000 7f462b400700 1 -- 172.21.15.116:0/761365195 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4630099d40 con 0x55bc6fb35fd0 2024-03-30T00:58:21.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.831+0000 7f462b400700 1 -- 172.21.15.116:0/761365195 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46300996b0 con 0x55bc6fb35fd0 2024-03-30T00:58:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.832+0000 7f4654065b80 1 -- 172.21.15.116:0/761365195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35fd0 msgr2=0x55bc6fb3db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.832+0000 7f4654065b80 1 --2- 172.21.15.116:0/761365195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb3db60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f46300992f0 tx=0x7f46300b9040 comp rx=0 tx=0).stop 2024-03-30T00:58:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.832+0000 7f4654065b80 1 -- 172.21.15.116:0/761365195 shutdown_connections 2024-03-30T00:58:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.832+0000 7f4654065b80 1 --2- 172.21.15.116:0/761365195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb347d0 0x55bc6fb34bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.832+0000 7f4654065b80 1 --2- 172.21.15.116:0/761365195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb3db60 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.832+0000 7f4654065b80 1 --2- 172.21.15.116:0/761365195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35130 0x55bc6fb45b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.832+0000 7f4654065b80 1 -- 172.21.15.116:0/761365195 >> 172.21.15.116:0/761365195 conn(0x55bc6fa936a0 msgr2=0x55bc6fb3c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.832+0000 7f4654065b80 1 -- 172.21.15.116:0/761365195 shutdown_connections 2024-03-30T00:58:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.832+0000 7f4654065b80 1 -- 172.21.15.116:0/761365195 wait complete. 2024-03-30T00:58:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.832+0000 7f4654065b80 1 -- start start 2024-03-30T00:58:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.832+0000 7f4654065b80 1 Processor -- start 2024-03-30T00:58:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f4654065b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb347d0 0x55bc6fb43650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f4654065b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb35130 0x55bc6fb43b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f4654065b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb44b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f4654065b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6fb45040 con 0x55bc6fb347d0 2024-03-30T00:58:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f4654065b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6fb451b0 con 0x55bc6fb35fd0 2024-03-30T00:58:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f4654065b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6fb45370 con 0x55bc6fb35130 2024-03-30T00:58:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f4654065b80 10 client.0 _ll_register_callbacks cb 0x55bc6fa2b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f463dc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb347d0 0x55bc6fb43650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f463dc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb347d0 0x55bc6fb43650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40628/0 (socket says 172.21.15.116:40628) 2024-03-30T00:58:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f463dc00700 1 -- 172.21.15.116:0/1927548666 learned_addr learned my addr 172.21.15.116:0/1927548666 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f463d200700 1 --2- 172.21.15.116:0/1927548666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb35130 0x55bc6fb43b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f463e600700 1 --2- 172.21.15.116:0/1927548666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb44b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f463dc00700 1 -- 172.21.15.116:0/1927548666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb35130 msgr2=0x55bc6fb43b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f463dc00700 1 --2- 172.21.15.116:0/1927548666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb35130 0x55bc6fb43b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f463dc00700 1 -- 172.21.15.116:0/1927548666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35fd0 msgr2=0x55bc6fb44b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f463dc00700 1 --2- 172.21.15.116:0/1927548666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb44b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.833+0000 7f463dc00700 1 -- 172.21.15.116:0/1927548666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc6f9073b0 con 0x55bc6fb347d0 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.834+0000 7f463dc00700 1 --2- 172.21.15.116:0/1927548666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb347d0 0x55bc6fb43650 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f462c0095c0 tx=0x7f462c005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8003dc1c7ac242f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.834+0000 7f461f400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.834+0000 7f461f400700 1 -- 172.21.15.116:0/1927548666 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c028140 con 0x55bc6fb347d0 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.834+0000 7f461f400700 1 -- 172.21.15.116:0/1927548666 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f462c0282a0 con 0x55bc6fb347d0 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.834+0000 7f4654065b80 1 -- 172.21.15.116:0/1927548666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4630082e90 con 0x55bc6fb347d0 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.834+0000 7f461f400700 1 -- 172.21.15.116:0/1927548666 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f462c028460 con 0x55bc6fb347d0 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.834+0000 7f4654065b80 10 client.15546 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.834+0000 7f4654065b80 1 -- 172.21.15.116:0/1927548666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bc6f90bcd0 con 0x55bc6fb347d0 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.834+0000 7f461f400700 1 -- 172.21.15.116:0/1927548666 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4630082e90 con 0x55bc6fb347d0 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.834+0000 7f461f400700 1 -- 172.21.15.116:0/1927548666 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f462c003ad0 con 0x55bc6fb347d0 2024-03-30T00:58:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.835+0000 7f4654065b80 10 client.15546 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.835+0000 7f4654065b80 10 client.15546 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T00:58:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.835+0000 7f4654065b80 20 client.15546 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.835+0000 7f4654065b80 1 --2- 172.21.15.116:0/1927548666 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55bc6fb9efa0 0x55bc6fba1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.835+0000 7f4654065b80 4 client.15546 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T00:58:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.835+0000 7f4654065b80 1 -- 172.21.15.116:0/1927548666 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bc6f8d4cf0 con 0x55bc6fb9efa0 2024-03-30T00:58:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.835+0000 7f463d200700 1 --2- 172.21.15.116:0/1927548666 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55bc6fb9efa0 0x55bc6fba1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f461f400700 10 client.15546 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T00:58:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f463d200700 1 --2- 172.21.15.116:0/1927548666 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55bc6fb9efa0 0x55bc6fba1390 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bd752863b78ba86f server_cookie=5ccb086137fbb677 in_seq=0 out_seq=0 2024-03-30T00:58:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f461f400700 1 -- 172.21.15.116:0/1927548666 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bc6f8d4cf0 con 0x55bc6fb9efa0 2024-03-30T00:58:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f461f400700 10 client.15546 handle_command_reply: tid=0 2024-03-30T00:58:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f4654065b80 1 client.15546 shutdown 2024-03-30T00:58:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f4654065b80 1 -- 172.21.15.116:0/1927548666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb347d0 msgr2=0x55bc6fb43650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f4654065b80 1 --2- 172.21.15.116:0/1927548666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb347d0 0x55bc6fb43650 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f462c0095c0 tx=0x7f462c005720 comp rx=0 tx=0).stop 2024-03-30T00:58:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f4654065b80 1 -- 172.21.15.116:0/1927548666 shutdown_connections 2024-03-30T00:58:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f4654065b80 1 --2- 172.21.15.116:0/1927548666 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55bc6fb9efa0 0x55bc6fba1390 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f4654065b80 1 --2- 172.21.15.116:0/1927548666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bc6fb35fd0 0x55bc6fb44b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f4654065b80 1 --2- 172.21.15.116:0/1927548666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc6fb347d0 0x55bc6fb43650 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f461f400700 0 client.15546 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T00:58:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f4654065b80 1 --2- 172.21.15.116:0/1927548666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bc6fb35130 0x55bc6fb43b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f4654065b80 1 -- 172.21.15.116:0/1927548666 >> 172.21.15.116:0/1927548666 conn(0x55bc6fa936a0 msgr2=0x55bc6fb3c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f4654065b80 1 -- 172.21.15.116:0/1927548666 shutdown_connections 2024-03-30T00:58:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f4654065b80 1 -- 172.21.15.116:0/1927548666 wait complete. 2024-03-30T00:58:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.836+0000 7f4654065b80 20 client.15546 trim_cache size 0 max 16384 2024-03-30T00:58:21.847 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T00:58:21.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.837+0000 7f4640600700 1 -- 172.21.15.116:0/3748133209 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46200a1120 msgr2=0x7f46200a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.837+0000 7f4640600700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46200a1120 0x7f46200a35e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4624011630 tx=0x7f46240119f0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.837+0000 7f4640600700 1 -- 172.21.15.116:0/3748133209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4638102910 msgr2=0x7f46380736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:21.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.837+0000 7f4640600700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4638102910 0x7f46380736e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f463000c370 tx=0x7f463000c680 comp rx=0 tx=0).stop 2024-03-30T00:58:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.837+0000 7f4640600700 1 -- 172.21.15.116:0/3748133209 shutdown_connections 2024-03-30T00:58:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.837+0000 7f4640600700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46200a1120 0x7f46200a35e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.837+0000 7f4640600700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4638100d80 0x7f4638074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.837+0000 7f4640600700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4638102910 0x7f46380736e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.837+0000 7f4640600700 1 --2- 172.21.15.116:0/3748133209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4638101f70 0x7f46380731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.837+0000 7f4640600700 1 -- 172.21.15.116:0/3748133209 >> 172.21.15.116:0/3748133209 conn(0x7f46380fc2f0 msgr2=0x7f4638105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.837+0000 7f4640600700 1 -- 172.21.15.116:0/3748133209 shutdown_connections 2024-03-30T00:58:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:21.837+0000 7f4640600700 1 -- 172.21.15.116:0/3748133209 wait complete. 2024-03-30T00:58:21.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:21 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1161832630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:58:21.862 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T00:58:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.050+0000 7f66d4200700 1 -- 172.21.15.116:0/3122655177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66cc102910 msgr2=0x7f66cc1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.050+0000 7f66d4200700 1 --2- 172.21.15.116:0/3122655177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66cc102910 0x7f66cc1091c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f66c4009fc0 tx=0x7f66c400b330 comp rx=0 tx=0).stop 2024-03-30T00:58:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.050+0000 7f66d4200700 1 -- 172.21.15.116:0/3122655177 shutdown_connections 2024-03-30T00:58:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.050+0000 7f66d4200700 1 --2- 172.21.15.116:0/3122655177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66cc102910 0x7f66cc1091c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.050+0000 7f66d4200700 1 --2- 172.21.15.116:0/3122655177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66cc100d80 0x7f66cc1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.050+0000 7f66d4200700 1 --2- 172.21.15.116:0/3122655177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66cc101f70 0x7f66cc1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.050+0000 7f66d4200700 1 -- 172.21.15.116:0/3122655177 >> 172.21.15.116:0/3122655177 conn(0x7f66cc0fc2f0 msgr2=0x7f66cc0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.050+0000 7f66d4200700 1 -- 172.21.15.116:0/3122655177 shutdown_connections 2024-03-30T00:58:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.050+0000 7f66d4200700 1 -- 172.21.15.116:0/3122655177 wait complete. 2024-03-30T00:58:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.051+0000 7f66d4200700 1 Processor -- start 2024-03-30T00:58:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.051+0000 7f66d4200700 1 -- start start 2024-03-30T00:58:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.051+0000 7f66d4200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66cc100d80 0x7f66cc196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.051+0000 7f66d4200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66cc101f70 0x7f66cc197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.051+0000 7f66d4200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66cc102910 0x7f66cc19e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.051+0000 7f66d4200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc19e950 con 0x7f66cc101f70 2024-03-30T00:58:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.051+0000 7f66d4200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc19ea90 con 0x7f66cc102910 2024-03-30T00:58:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.051+0000 7f66d4200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc19ec20 con 0x7f66cc100d80 2024-03-30T00:58:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.052+0000 7f66d0e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66cc101f70 0x7f66cc197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.052+0000 7f66d1800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66cc100d80 0x7f66cc196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.052+0000 7f66d0e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66cc101f70 0x7f66cc197360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40642/0 (socket says 172.21.15.116:40642) 2024-03-30T00:58:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.052+0000 7f66d0e00700 1 -- 172.21.15.116:0/3981987760 learned_addr learned my addr 172.21.15.116:0/3981987760 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.052+0000 7f66d2200700 1 --2- 172.21.15.116:0/3981987760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66cc102910 0x7f66cc19e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.052+0000 7f66d0e00700 1 -- 172.21.15.116:0/3981987760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66cc100d80 msgr2=0x7f66cc196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.052+0000 7f66d0e00700 1 --2- 172.21.15.116:0/3981987760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66cc100d80 0x7f66cc196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.052+0000 7f66d0e00700 1 -- 172.21.15.116:0/3981987760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66cc102910 msgr2=0x7f66cc19e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.052+0000 7f66d0e00700 1 --2- 172.21.15.116:0/3981987760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66cc102910 0x7f66cc19e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.052+0000 7f66d0e00700 1 -- 172.21.15.116:0/3981987760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c4009c40 con 0x7f66cc101f70 2024-03-30T00:58:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.052+0000 7f66d1800700 1 --2- 172.21.15.116:0/3981987760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66cc100d80 0x7f66cc196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:58:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.052+0000 7f66d0e00700 1 --2- 172.21.15.116:0/3981987760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66cc101f70 0x7f66cc197360 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f66b800e9a0 tx=0x7f66b800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.053+0000 7f66ca000700 1 -- 172.21.15.116:0/3981987760 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b8009790 con 0x7f66cc101f70 2024-03-30T00:58:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.053+0000 7f66ca000700 1 -- 172.21.15.116:0/3981987760 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f66b800ce80 con 0x7f66cc101f70 2024-03-30T00:58:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.053+0000 7f66ca000700 1 -- 172.21.15.116:0/3981987760 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b8018680 con 0x7f66cc101f70 2024-03-30T00:58:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.053+0000 7f66d4200700 1 -- 172.21.15.116:0/3981987760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66cc19ef00 con 0x7f66cc101f70 2024-03-30T00:58:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.053+0000 7f66d4200700 1 -- 172.21.15.116:0/3981987760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66cc19f380 con 0x7f66cc101f70 2024-03-30T00:58:22.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.056+0000 7f66ca000700 1 -- 172.21.15.116:0/3981987760 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f66b8010430 con 0x7f66cc101f70 2024-03-30T00:58:22.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.057+0000 7f66e7dafb80 1 Processor -- start 2024-03-30T00:58:22.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.057+0000 7f66ca000700 1 --2- 172.21.15.116:0/3981987760 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f66b40a11a0 0x7f66b40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.057+0000 7f66e7dafb80 1 -- start start 2024-03-30T00:58:22.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.057+0000 7f66ca000700 1 -- 172.21.15.116:0/3981987760 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f66b8014070 con 0x7f66cc101f70 2024-03-30T00:58:22.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.057+0000 7f66d2200700 1 --2- 172.21.15.116:0/3981987760 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f66b40a11a0 0x7f66b40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.057+0000 7f66e7dafb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac467d0 0x55724ac46bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66e7dafb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47130 0x55724ac479f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66e7dafb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47fd0 0x55724ac48450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66e7dafb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ab9e990 con 0x55724ac47130 2024-03-30T00:58:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66e7dafb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ab9c350 con 0x55724ac467d0 2024-03-30T00:58:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66e7dafb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ab97c40 con 0x55724ac47fd0 2024-03-30T00:58:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66d0e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47130 0x55724ac479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66d0e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47130 0x55724ac479f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40658/0 (socket says 172.21.15.116:40658) 2024-03-30T00:58:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66d0e00700 1 -- 172.21.15.116:0/452547515 learned_addr learned my addr 172.21.15.116:0/452547515 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66d2200700 1 --2- 172.21.15.116:0/452547515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47fd0 0x55724ac48450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66d1800700 1 --2- 172.21.15.116:0/452547515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac467d0 0x55724ac46bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66d2200700 1 --2- 172.21.15.116:0/3981987760 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f66b40a11a0 0x7f66b40a3660 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f66c4007a30 tx=0x7f66c4009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66d0e00700 1 -- 172.21.15.116:0/452547515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47fd0 msgr2=0x55724ac48450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66d0e00700 1 --2- 172.21.15.116:0/452547515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47fd0 0x55724ac48450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66d0e00700 1 -- 172.21.15.116:0/452547515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac467d0 msgr2=0x55724ac46bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66d0e00700 1 --2- 172.21.15.116:0/452547515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac467d0 0x55724ac46bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.058+0000 7f66d0e00700 1 -- 172.21.15.116:0/452547515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55724aa193b0 con 0x55724ac47130 2024-03-30T00:58:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.059+0000 7f66d0e00700 1 --2- 172.21.15.116:0/452547515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47130 0x55724ac479f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f66b8096360 tx=0x7f66b8099ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7973524d1eac9bd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.059+0000 7f66bea00700 1 -- 172.21.15.116:0/452547515 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b8099e60 con 0x55724ac47130 2024-03-30T00:58:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.059+0000 7f66bea00700 1 -- 172.21.15.116:0/452547515 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f66b809dc90 con 0x55724ac47130 2024-03-30T00:58:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.059+0000 7f66bea00700 1 -- 172.21.15.116:0/452547515 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b80a1e50 con 0x55724ac47130 2024-03-30T00:58:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.059+0000 7f66e7dafb80 1 -- 172.21.15.116:0/452547515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47130 msgr2=0x55724ac479f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.059+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/452547515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47130 0x55724ac479f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f66b8096360 tx=0x7f66b8099ad0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.060+0000 7f66e7dafb80 1 -- 172.21.15.116:0/452547515 shutdown_connections 2024-03-30T00:58:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.060+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/452547515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47fd0 0x55724ac48450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.060+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/452547515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac467d0 0x55724ac46bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.060+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/452547515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47130 0x55724ac479f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.060+0000 7f66e7dafb80 1 -- 172.21.15.116:0/452547515 >> 172.21.15.116:0/452547515 conn(0x55724aba56a0 msgr2=0x55724ac419a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.060+0000 7f66e7dafb80 1 -- 172.21.15.116:0/452547515 shutdown_connections 2024-03-30T00:58:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.060+0000 7f66e7dafb80 1 -- 172.21.15.116:0/452547515 wait complete. 2024-03-30T00:58:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.060+0000 7f66e7dafb80 1 -- start start 2024-03-30T00:58:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.061+0000 7f66e7dafb80 1 Processor -- start 2024-03-30T00:58:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.061+0000 7f66e7dafb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac467d0 0x55724ac506b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.061+0000 7f66e7dafb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac47130 0x55724ac50bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.061+0000 7f66e7dafb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47fd0 0x55724ac57c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.061+0000 7f66e7dafb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ab33d80 con 0x55724ac467d0 2024-03-30T00:58:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.061+0000 7f66e7dafb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ab33a20 con 0x55724ac47130 2024-03-30T00:58:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.061+0000 7f66e7dafb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724aa1b6a0 con 0x55724ac47fd0 2024-03-30T00:58:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.061+0000 7f66e7dafb80 10 client.0 _ll_register_callbacks cb 0x55724ab3d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.061+0000 7f66d1800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac467d0 0x55724ac506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.061+0000 7f66d2200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47fd0 0x55724ac57c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.062+0000 7f66d2200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47fd0 0x55724ac57c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:36732/0 (socket says 172.21.15.116:36732) 2024-03-30T00:58:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.062+0000 7f66d2200700 1 -- 172.21.15.116:0/2414912613 learned_addr learned my addr 172.21.15.116:0/2414912613 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.062+0000 7f66d0e00700 1 --2- 172.21.15.116:0/2414912613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac47130 0x55724ac50bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.062+0000 7f66d2200700 1 -- 172.21.15.116:0/2414912613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac47130 msgr2=0x55724ac50bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.062+0000 7f66d2200700 1 --2- 172.21.15.116:0/2414912613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac47130 0x55724ac50bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.062+0000 7f66d2200700 1 -- 172.21.15.116:0/2414912613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac467d0 msgr2=0x55724ac506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.062+0000 7f66d2200700 1 --2- 172.21.15.116:0/2414912613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac467d0 0x55724ac506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.062+0000 7f66d2200700 1 -- 172.21.15.116:0/2414912613 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66cc19f380 con 0x55724ac47fd0 2024-03-30T00:58:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.062+0000 7f66d2200700 1 --2- 172.21.15.116:0/2414912613 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47fd0 0x55724ac57c30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f66c4003c80 tx=0x7f66c400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44ef95c706e76dd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.062+0000 7f66b3400700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T00:58:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.062+0000 7f66b3400700 1 -- 172.21.15.116:0/2414912613 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c400e040 con 0x55724ac47fd0 2024-03-30T00:58:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.063+0000 7f66e7dafb80 1 -- 172.21.15.116:0/2414912613 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55724ab58380 con 0x55724ac47fd0 2024-03-30T00:58:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.063+0000 7f66b3400700 1 -- 172.21.15.116:0/2414912613 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f66c401c650 con 0x55724ac47fd0 2024-03-30T00:58:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.063+0000 7f66b3400700 1 -- 172.21.15.116:0/2414912613 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66c401c810 con 0x55724ac47fd0 2024-03-30T00:58:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.063+0000 7f66e7dafb80 10 client.25597 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.063+0000 7f66e7dafb80 1 -- 172.21.15.116:0/2414912613 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55724a9c5b60 con 0x55724ac47fd0 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.063+0000 7f66b3400700 1 -- 172.21.15.116:0/2414912613 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55724ab58380 con 0x55724ac47fd0 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.063+0000 7f66b3400700 1 -- 172.21.15.116:0/2414912613 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f66c402f070 con 0x55724ac47fd0 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.063+0000 7f66e7dafb80 10 client.25597 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.063+0000 7f66e7dafb80 10 client.25597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.063+0000 7f66e7dafb80 20 client.25597 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.063+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/2414912613 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55724ac5c460 0x55724ac5e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.063+0000 7f66e7dafb80 4 client.25597 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.063+0000 7f66e7dafb80 1 -- 172.21.15.116:0/2414912613 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55724ab90700 con 0x55724ac5c460 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.064+0000 7f66d1800700 1 --2- 172.21.15.116:0/2414912613 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55724ac5c460 0x55724ac5e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.064+0000 7f66d1800700 1 --2- 172.21.15.116:0/2414912613 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55724ac5c460 0x55724ac5e880 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94375f6fadda084e server_cookie=e30b5d607214d8c6 in_seq=0 out_seq=0 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.064+0000 7f66b3400700 10 client.25597 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66b3400700 1 -- 172.21.15.116:0/2414912613 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55724ab90700 con 0x55724ac5c460 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66b3400700 10 client.25597 handle_command_reply: tid=0 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66e7dafb80 1 client.25597 shutdown 2024-03-30T00:58:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66e7dafb80 1 -- 172.21.15.116:0/2414912613 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47fd0 msgr2=0x55724ac57c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/2414912613 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47fd0 0x55724ac57c30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f66c4003c80 tx=0x7f66c400b330 comp rx=0 tx=0).stop 2024-03-30T00:58:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66e7dafb80 1 -- 172.21.15.116:0/2414912613 shutdown_connections 2024-03-30T00:58:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/2414912613 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55724ac5c460 0x55724ac5e880 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/2414912613 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47fd0 0x55724ac57c30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/2414912613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac467d0 0x55724ac506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/2414912613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac47130 0x55724ac50bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66b3400700 0 client.25597 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66e7dafb80 1 -- 172.21.15.116:0/2414912613 >> 172.21.15.116:0/2414912613 conn(0x55724aba56a0 msgr2=0x55724ac45170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66e7dafb80 1 -- 172.21.15.116:0/2414912613 shutdown_connections 2024-03-30T00:58:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66e7dafb80 1 -- 172.21.15.116:0/2414912613 wait complete. 2024-03-30T00:58:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.065+0000 7f66e7dafb80 20 client.25597 trim_cache size 0 max 16384 2024-03-30T00:58:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.078+0000 7f66e7dafb80 1 Processor -- start 2024-03-30T00:58:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66e7dafb80 1 -- start start 2024-03-30T00:58:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66e7dafb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac467d0 0x55724ac4b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66e7dafb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47130 0x55724ac582e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66e7dafb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47fd0 0x55724ac57bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66e7dafb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ac50220 con 0x55724ac47fd0 2024-03-30T00:58:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66e7dafb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ac3d480 con 0x55724ac467d0 2024-03-30T00:58:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66e7dafb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ac518a0 con 0x55724ac47130 2024-03-30T00:58:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66d2200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47fd0 0x55724ac57bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66d2200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47fd0 0x55724ac57bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40688/0 (socket says 172.21.15.116:40688) 2024-03-30T00:58:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66d2200700 1 -- 172.21.15.116:0/945408374 learned_addr learned my addr 172.21.15.116:0/945408374 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66d1800700 1 --2- 172.21.15.116:0/945408374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac467d0 0x55724ac4b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66d0e00700 1 --2- 172.21.15.116:0/945408374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47130 0x55724ac582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66d2200700 1 -- 172.21.15.116:0/945408374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47130 msgr2=0x55724ac582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66d2200700 1 --2- 172.21.15.116:0/945408374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47130 0x55724ac582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66d2200700 1 -- 172.21.15.116:0/945408374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac467d0 msgr2=0x55724ac4b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:22.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.079+0000 7f66d2200700 1 --2- 172.21.15.116:0/945408374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac467d0 0x55724ac4b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.080+0000 7f66d2200700 1 -- 172.21.15.116:0/945408374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55724ab5dea0 con 0x55724ac47fd0 2024-03-30T00:58:22.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.080+0000 7f66d1800700 1 --2- 172.21.15.116:0/945408374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac467d0 0x55724ac4b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:58:22.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.080+0000 7f66d2200700 1 --2- 172.21.15.116:0/945408374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47fd0 0x55724ac57bf0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f66c40061a0 tx=0x7f66c40094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9a43f849634428 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:22.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.080+0000 7f66bf400700 1 -- 172.21.15.116:0/945408374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c400e030 con 0x55724ac47fd0 2024-03-30T00:58:22.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.080+0000 7f66bf400700 1 -- 172.21.15.116:0/945408374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f66c40052e0 con 0x55724ac47fd0 2024-03-30T00:58:22.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.080+0000 7f66bf400700 1 -- 172.21.15.116:0/945408374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4031430 con 0x55724ac47fd0 2024-03-30T00:58:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.081+0000 7f66e7dafb80 1 -- 172.21.15.116:0/945408374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47fd0 msgr2=0x55724ac57bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.081+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/945408374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47fd0 0x55724ac57bf0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f66c40061a0 tx=0x7f66c40094a0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.081+0000 7f66e7dafb80 1 -- 172.21.15.116:0/945408374 shutdown_connections 2024-03-30T00:58:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.081+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/945408374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47fd0 0x55724ac57bf0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.081+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/945408374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac467d0 0x55724ac4b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.081+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/945408374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac47130 0x55724ac582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.081+0000 7f66e7dafb80 1 -- 172.21.15.116:0/945408374 >> 172.21.15.116:0/945408374 conn(0x55724aba56a0 msgr2=0x55724ac3a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.081+0000 7f66e7dafb80 1 -- 172.21.15.116:0/945408374 shutdown_connections 2024-03-30T00:58:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.081+0000 7f66e7dafb80 1 -- 172.21.15.116:0/945408374 wait complete. 2024-03-30T00:58:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.081+0000 7f66e7dafb80 1 -- start start 2024-03-30T00:58:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.081+0000 7f66e7dafb80 1 Processor -- start 2024-03-30T00:58:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66e7dafb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac467d0 0x55724ac55ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66e7dafb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47130 0x55724ac56000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66e7dafb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac47fd0 0x55724ac56f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66e7dafb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ac574b0 con 0x55724ac47130 2024-03-30T00:58:22.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66e7dafb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ac57620 con 0x55724ac47fd0 2024-03-30T00:58:22.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66e7dafb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ac577e0 con 0x55724ac467d0 2024-03-30T00:58:22.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66e7dafb80 10 client.0 _ll_register_callbacks cb 0x55724ab3d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:22.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66d1800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac467d0 0x55724ac55ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66d2200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac47fd0 0x55724ac56f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66d1800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac467d0 0x55724ac55ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:36748/0 (socket says 172.21.15.116:36748) 2024-03-30T00:58:22.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66d2200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac47fd0 0x55724ac56f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:42620/0 (socket says 172.21.15.116:42620) 2024-03-30T00:58:22.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66d1800700 1 -- 172.21.15.116:0/1568723775 learned_addr learned my addr 172.21.15.116:0/1568723775 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:22.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66d0e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47130 0x55724ac56000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66d1800700 1 -- 172.21.15.116:0/1568723775 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac47fd0 msgr2=0x55724ac56f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:22.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66d1800700 1 --2- 172.21.15.116:0/1568723775 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac47fd0 0x55724ac56f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66d1800700 1 -- 172.21.15.116:0/1568723775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47130 msgr2=0x55724ac56000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:22.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66d1800700 1 --2- 172.21.15.116:0/1568723775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47130 0x55724ac56000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.082+0000 7f66d1800700 1 -- 172.21.15.116:0/1568723775 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c0012170 con 0x55724ac467d0 2024-03-30T00:58:22.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.083+0000 7f66d1800700 1 --2- 172.21.15.116:0/1568723775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac467d0 0x55724ac55ac0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x55724ac5f160 tx=0x7f66c001e3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51465105f9cad7b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:22.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.083+0000 7f66b3400700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T00:58:22.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.083+0000 7f66b3400700 1 -- 172.21.15.116:0/1568723775 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c0017040 con 0x55724ac467d0 2024-03-30T00:58:22.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.083+0000 7f66e7dafb80 1 -- 172.21.15.116:0/1568723775 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f66b809dc90 con 0x55724ac467d0 2024-03-30T00:58:22.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.083+0000 7f66b3400700 1 -- 172.21.15.116:0/1568723775 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f66c0004510 con 0x55724ac467d0 2024-03-30T00:58:22.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.083+0000 7f66b3400700 1 -- 172.21.15.116:0/1568723775 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66c0004830 con 0x55724ac467d0 2024-03-30T00:58:22.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.083+0000 7f66e7dafb80 10 client.25603 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:22.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.083+0000 7f66e7dafb80 1 -- 172.21.15.116:0/1568723775 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55724a9c5b60 con 0x55724ac467d0 2024-03-30T00:58:22.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.084+0000 7f66b3400700 1 -- 172.21.15.116:0/1568723775 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f66b809dc90 con 0x55724ac467d0 2024-03-30T00:58:22.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.084+0000 7f66b3400700 1 -- 172.21.15.116:0/1568723775 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f66c00126f0 con 0x55724ac467d0 2024-03-30T00:58:22.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.084+0000 7f66e7dafb80 10 client.25603 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:22.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.084+0000 7f66e7dafb80 10 client.25603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.084+0000 7f66e7dafb80 20 client.25603 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.084+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/1568723775 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55724acb12a0 0x55724acb36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.084+0000 7f66e7dafb80 4 client.25603 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.084+0000 7f66e7dafb80 1 -- 172.21.15.116:0/1568723775 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55724a9e6cf0 con 0x55724acb12a0 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.084+0000 7f66d0e00700 1 --2- 172.21.15.116:0/1568723775 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55724acb12a0 0x55724acb36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.084+0000 7f66d0e00700 1 --2- 172.21.15.116:0/1568723775 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55724acb12a0 0x55724acb36c0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42aa36a75f2c8c40 server_cookie=567f2fec164afd8a in_seq=0 out_seq=0 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.084+0000 7f66b3400700 10 client.25603 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66b3400700 1 -- 172.21.15.116:0/1568723775 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55724a9e6cf0 con 0x55724acb12a0 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66b3400700 10 client.25603 handle_command_reply: tid=0 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66e7dafb80 1 client.25603 shutdown 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66e7dafb80 1 -- 172.21.15.116:0/1568723775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac467d0 msgr2=0x55724ac55ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/1568723775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac467d0 0x55724ac55ac0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x55724ac5f160 tx=0x7f66c001e3a0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66e7dafb80 1 -- 172.21.15.116:0/1568723775 shutdown_connections 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/1568723775 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55724acb12a0 0x55724acb36c0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/1568723775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55724ac467d0 0x55724ac55ac0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/1568723775 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55724ac47fd0 0x55724ac56f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66e7dafb80 1 --2- 172.21.15.116:0/1568723775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724ac47130 0x55724ac56000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66b3400700 0 client.25603 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:22.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66e7dafb80 1 -- 172.21.15.116:0/1568723775 >> 172.21.15.116:0/1568723775 conn(0x55724aba56a0 msgr2=0x55724ac4e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:22.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66e7dafb80 1 -- 172.21.15.116:0/1568723775 shutdown_connections 2024-03-30T00:58:22.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66e7dafb80 1 -- 172.21.15.116:0/1568723775 wait complete. 2024-03-30T00:58:22.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.085+0000 7f66e7dafb80 20 client.25603 trim_cache size 0 max 16384 2024-03-30T00:58:22.097 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T00:58:22.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.086+0000 7f66d4200700 1 -- 172.21.15.116:0/3981987760 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f66b40a11a0 msgr2=0x7f66b40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:22.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.086+0000 7f66d4200700 1 --2- 172.21.15.116:0/3981987760 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f66b40a11a0 0x7f66b40a3660 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f66c4007a30 tx=0x7f66c4009f10 comp rx=0 tx=0).stop 2024-03-30T00:58:22.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.086+0000 7f66d4200700 1 -- 172.21.15.116:0/3981987760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66cc101f70 msgr2=0x7f66cc197360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:22.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.086+0000 7f66d4200700 1 --2- 172.21.15.116:0/3981987760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66cc101f70 0x7f66cc197360 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f66b800e9a0 tx=0x7f66b800ecb0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.086+0000 7f66d4200700 1 -- 172.21.15.116:0/3981987760 shutdown_connections 2024-03-30T00:58:22.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.086+0000 7f66d4200700 1 --2- 172.21.15.116:0/3981987760 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f66b40a11a0 0x7f66b40a3660 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.086+0000 7f66d4200700 1 --2- 172.21.15.116:0/3981987760 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66cc100d80 0x7f66cc196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.086+0000 7f66d4200700 1 --2- 172.21.15.116:0/3981987760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66cc102910 0x7f66cc19e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.086+0000 7f66d4200700 1 --2- 172.21.15.116:0/3981987760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66cc101f70 0x7f66cc197360 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:22.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.086+0000 7f66d4200700 1 -- 172.21.15.116:0/3981987760 >> 172.21.15.116:0/3981987760 conn(0x7f66cc0fc2f0 msgr2=0x7f66cc105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:22.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.086+0000 7f66d4200700 1 -- 172.21.15.116:0/3981987760 shutdown_connections 2024-03-30T00:58:22.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:22.086+0000 7f66d4200700 1 -- 172.21.15.116:0/3981987760 wait complete. 2024-03-30T00:58:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:22 smithi181 ceph-mon[47622]: pgmap v306: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:58:22.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:22 smithi130 ceph-mon[47626]: pgmap v306: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:58:22.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:22 smithi116 ceph-mon[38086]: pgmap v306: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:58:23.111 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T00:58:23.112 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T00:58:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.315+0000 7f117aa00700 1 -- 172.21.15.116:0/1188362628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1174102910 msgr2=0x7f11741091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.315+0000 7f117aa00700 1 --2- 172.21.15.116:0/1188362628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1174102910 0x7f11741091c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f116c00b0d0 tx=0x7f116c00b3e0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.315+0000 7f117aa00700 1 -- 172.21.15.116:0/1188362628 shutdown_connections 2024-03-30T00:58:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.315+0000 7f117aa00700 1 --2- 172.21.15.116:0/1188362628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1174100d80 0x7f11741011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.315+0000 7f117aa00700 1 --2- 172.21.15.116:0/1188362628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1174102910 0x7f11741091c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.315+0000 7f117aa00700 1 --2- 172.21.15.116:0/1188362628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1174101f70 0x7f11741023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.315+0000 7f117aa00700 1 -- 172.21.15.116:0/1188362628 >> 172.21.15.116:0/1188362628 conn(0x7f11740fc2f0 msgr2=0x7f11740fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.315+0000 7f117aa00700 1 -- 172.21.15.116:0/1188362628 shutdown_connections 2024-03-30T00:58:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.315+0000 7f117aa00700 1 -- 172.21.15.116:0/1188362628 wait complete. 2024-03-30T00:58:23.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.316+0000 7f117aa00700 1 Processor -- start 2024-03-30T00:58:23.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.317+0000 7f117aa00700 1 -- start start 2024-03-30T00:58:23.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.317+0000 7f117aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1174100d80 0x7f1174074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.317+0000 7f117aa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1174101f70 0x7f11740731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.317+0000 7f117aa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1174102910 0x7f11740736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.317+0000 7f117aa00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1174073e80 con 0x7f1174100d80 2024-03-30T00:58:23.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.317+0000 7f117aa00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1174073ff0 con 0x7f1174102910 2024-03-30T00:58:23.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.317+0000 7f117aa00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11741a0e50 con 0x7f1174101f70 2024-03-30T00:58:23.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.317+0000 7f1173e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1174100d80 0x7f1174074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.317+0000 7f1173e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1174100d80 0x7f1174074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40730/0 (socket says 172.21.15.116:40730) 2024-03-30T00:58:23.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.317+0000 7f1173400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1174101f70 0x7f11740731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.317+0000 7f1173e00700 1 -- 172.21.15.116:0/2416432699 learned_addr learned my addr 172.21.15.116:0/2416432699 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:23.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.317+0000 7f1178a00700 1 --2- 172.21.15.116:0/2416432699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1174102910 0x7f11740736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.318+0000 7f1173e00700 1 -- 172.21.15.116:0/2416432699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1174101f70 msgr2=0x7f11740731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.318+0000 7f1173e00700 1 --2- 172.21.15.116:0/2416432699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1174101f70 0x7f11740731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.318+0000 7f1173e00700 1 -- 172.21.15.116:0/2416432699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1174102910 msgr2=0x7f11740736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.318+0000 7f1173e00700 1 --2- 172.21.15.116:0/2416432699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1174102910 0x7f11740736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.318+0000 7f1173e00700 1 -- 172.21.15.116:0/2416432699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1164009520 con 0x7f1174100d80 2024-03-30T00:58:23.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.318+0000 7f1173e00700 1 --2- 172.21.15.116:0/2416432699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1174100d80 0x7f1174074b10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f116400ea50 tx=0x7f116400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:23.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.318+0000 7f1170c00700 1 -- 172.21.15.116:0/2416432699 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116400cb70 con 0x7f1174100d80 2024-03-30T00:58:23.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.318+0000 7f1170c00700 1 -- 172.21.15.116:0/2416432699 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f116400ccd0 con 0x7f1174100d80 2024-03-30T00:58:23.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.318+0000 7f1170c00700 1 -- 172.21.15.116:0/2416432699 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1164018680 con 0x7f1174100d80 2024-03-30T00:58:23.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.318+0000 7f117aa00700 1 -- 172.21.15.116:0/2416432699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116c009d00 con 0x7f1174100d80 2024-03-30T00:58:23.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.318+0000 7f117aa00700 1 -- 172.21.15.116:0/2416432699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11741a1410 con 0x7f1174100d80 2024-03-30T00:58:23.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.321+0000 7f1170c00700 1 -- 172.21.15.116:0/2416432699 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f1164010430 con 0x7f1174100d80 2024-03-30T00:58:23.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.323+0000 7f1170c00700 1 --2- 172.21.15.116:0/2416432699 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f11580a11a0 0x7f11580a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.323+0000 7f1170c00700 1 -- 172.21.15.116:0/2416432699 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1164014070 con 0x7f1174100d80 2024-03-30T00:58:23.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.323+0000 7f118e4e5b80 1 Processor -- start 2024-03-30T00:58:23.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.323+0000 7f1173400700 1 --2- 172.21.15.116:0/2416432699 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f11580a11a0 0x7f11580a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.323+0000 7f118e4e5b80 1 -- start start 2024-03-30T00:58:23.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.323+0000 7f118e4e5b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a239126c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.323+0000 7f118e4e5b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a2391271b0 0x55a239127a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.323+0000 7f118e4e5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a239128050 0x55a2391284d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.323+0000 7f118e4e5b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a23907ea10 con 0x55a239126850 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.323+0000 7f118e4e5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a23907c3d0 con 0x55a239128050 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.323+0000 7f118e4e5b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a239077cc0 con 0x55a2391271b0 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.324+0000 7f1173400700 1 --2- 172.21.15.116:0/2416432699 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f11580a11a0 0x7f11580a3660 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1160009c60 tx=0x7f1160009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.324+0000 7f1173e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a239126c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.324+0000 7f1173e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a239126c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40740/0 (socket says 172.21.15.116:40740) 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.324+0000 7f1173e00700 1 -- 172.21.15.116:0/342555249 learned_addr learned my addr 172.21.15.116:0/342555249 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.324+0000 7f1173400700 1 --2- 172.21.15.116:0/342555249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a2391271b0 0x55a239127a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.324+0000 7f1178a00700 1 --2- 172.21.15.116:0/342555249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a239128050 0x55a2391284d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.324+0000 7f1173e00700 1 -- 172.21.15.116:0/342555249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a2391271b0 msgr2=0x55a239127a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.324+0000 7f1173e00700 1 --2- 172.21.15.116:0/342555249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a2391271b0 0x55a239127a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.324+0000 7f1173e00700 1 -- 172.21.15.116:0/342555249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a239128050 msgr2=0x55a2391284d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.324+0000 7f1173e00700 1 --2- 172.21.15.116:0/342555249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a239128050 0x55a2391284d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.324+0000 7f1173e00700 1 -- 172.21.15.116:0/342555249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a238ef9430 con 0x55a239126850 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.324+0000 7f1173e00700 1 --2- 172.21.15.116:0/342555249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a239126c70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f11640a20f0 tx=0x7f1164099de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3259a0fa6b496edb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.325+0000 7f1169200700 1 -- 172.21.15.116:0/342555249 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116409a8a0 con 0x55a239126850 2024-03-30T00:58:23.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.325+0000 7f1169200700 1 -- 172.21.15.116:0/342555249 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f116409a1a0 con 0x55a239126850 2024-03-30T00:58:23.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.325+0000 7f1169200700 1 -- 172.21.15.116:0/342555249 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1164099690 con 0x55a239126850 2024-03-30T00:58:23.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.325+0000 7f118e4e5b80 1 -- 172.21.15.116:0/342555249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 msgr2=0x55a239126c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.325+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/342555249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a239126c70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f11640a20f0 tx=0x7f1164099de0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.325+0000 7f118e4e5b80 1 -- 172.21.15.116:0/342555249 shutdown_connections 2024-03-30T00:58:23.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.325+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/342555249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a239128050 0x55a2391284d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.325+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/342555249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a239126c70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.325+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/342555249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a2391271b0 0x55a239127a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.325+0000 7f118e4e5b80 1 -- 172.21.15.116:0/342555249 >> 172.21.15.116:0/342555249 conn(0x55a239085720 msgr2=0x55a239121a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:23.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.326+0000 7f118e4e5b80 1 -- 172.21.15.116:0/342555249 shutdown_connections 2024-03-30T00:58:23.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.326+0000 7f118e4e5b80 1 -- 172.21.15.116:0/342555249 wait complete. 2024-03-30T00:58:23.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.326+0000 7f118e4e5b80 1 -- start start 2024-03-30T00:58:23.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.326+0000 7f118e4e5b80 1 Processor -- start 2024-03-30T00:58:23.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.327+0000 7f118e4e5b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a239130740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.327+0000 7f118e4e5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a2391271b0 0x55a239130c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.327+0000 7f118e4e5b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239128050 0x55a239137c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.327+0000 7f118e4e5b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a239013e00 con 0x55a239126850 2024-03-30T00:58:23.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.327+0000 7f118e4e5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a239013aa0 con 0x55a2391271b0 2024-03-30T00:58:23.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.327+0000 7f118e4e5b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a238efb720 con 0x55a239128050 2024-03-30T00:58:23.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.327+0000 7f118e4e5b80 10 client.0 _ll_register_callbacks cb 0x55a23901d4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:23.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.327+0000 7f1173e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a239130740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.327+0000 7f1173400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a2391271b0 0x55a239130c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.327+0000 7f1173400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a2391271b0 0x55a239130c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:42650/0 (socket says 172.21.15.116:42650) 2024-03-30T00:58:23.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.327+0000 7f1178a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239128050 0x55a239137c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.327+0000 7f1173400700 1 -- 172.21.15.116:0/1051830911 learned_addr learned my addr 172.21.15.116:0/1051830911 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:23.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.328+0000 7f1173400700 1 -- 172.21.15.116:0/1051830911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239128050 msgr2=0x55a239137c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.328+0000 7f1173400700 1 --2- 172.21.15.116:0/1051830911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239128050 0x55a239137c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.328+0000 7f1173400700 1 -- 172.21.15.116:0/1051830911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 msgr2=0x55a239130740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.328+0000 7f1173400700 1 --2- 172.21.15.116:0/1051830911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a239130740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.328+0000 7f1173400700 1 -- 172.21.15.116:0/1051830911 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11741a1410 con 0x55a2391271b0 2024-03-30T00:58:23.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.328+0000 7f1173400700 1 --2- 172.21.15.116:0/1051830911 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a2391271b0 0x55a239130c80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1160005910 tx=0x7f11600047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10a92f90643f0403 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:23.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.328+0000 7f115d600700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T00:58:23.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.328+0000 7f115d600700 1 -- 172.21.15.116:0/1051830911 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1160022070 con 0x55a2391271b0 2024-03-30T00:58:23.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.328+0000 7f118e4e5b80 1 -- 172.21.15.116:0/1051830911 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a239038400 con 0x55a2391271b0 2024-03-30T00:58:23.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.328+0000 7f115d600700 1 -- 172.21.15.116:0/1051830911 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f116001d620 con 0x55a2391271b0 2024-03-30T00:58:23.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.329+0000 7f115d600700 1 -- 172.21.15.116:0/1051830911 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1160004580 con 0x55a2391271b0 2024-03-30T00:58:23.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.329+0000 7f118e4e5b80 10 client.25634 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:23.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.329+0000 7f118e4e5b80 1 -- 172.21.15.116:0/1051830911 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a238ea5be0 con 0x55a2391271b0 2024-03-30T00:58:23.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.329+0000 7f115d600700 1 -- 172.21.15.116:0/1051830911 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a239038400 con 0x55a2391271b0 2024-03-30T00:58:23.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.329+0000 7f115d600700 1 -- 172.21.15.116:0/1051830911 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1160027970 con 0x55a2391271b0 2024-03-30T00:58:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.329+0000 7f118e4e5b80 10 client.25634 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.329+0000 7f118e4e5b80 10 client.25634 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.329+0000 7f118e4e5b80 20 client.25634 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.329+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/1051830911 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a23913c500 0x55a23913e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.329+0000 7f118e4e5b80 4 client.25634 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:58:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.329+0000 7f118e4e5b80 1 -- 172.21.15.116:0/1051830911 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a239070780 con 0x55a23913c500 2024-03-30T00:58:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.329+0000 7f1173e00700 1 --2- 172.21.15.116:0/1051830911 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a23913c500 0x55a23913e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.330+0000 7f1173e00700 1 --2- 172.21.15.116:0/1051830911 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a23913c500 0x55a23913e920 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc073cf08e497d14 server_cookie=aa7ca2ff8f72986c in_seq=0 out_seq=0 2024-03-30T00:58:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.330+0000 7f115d600700 10 client.25634 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f115d600700 1 -- 172.21.15.116:0/1051830911 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a239070780 con 0x55a23913c500 2024-03-30T00:58:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f115d600700 10 client.25634 handle_command_reply: tid=0 2024-03-30T00:58:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f118e4e5b80 1 client.25634 shutdown 2024-03-30T00:58:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f118e4e5b80 1 -- 172.21.15.116:0/1051830911 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a2391271b0 msgr2=0x55a239130c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/1051830911 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a2391271b0 0x55a239130c80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1160005910 tx=0x7f11600047c0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f118e4e5b80 1 -- 172.21.15.116:0/1051830911 shutdown_connections 2024-03-30T00:58:23.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/1051830911 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a23913c500 0x55a23913e920 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/1051830911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239128050 0x55a239137c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/1051830911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a239130740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f115d600700 0 client.25634 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:23.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/1051830911 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a2391271b0 0x55a239130c80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f118e4e5b80 1 -- 172.21.15.116:0/1051830911 >> 172.21.15.116:0/1051830911 conn(0x55a239085720 msgr2=0x55a2391251f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:23.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f118e4e5b80 1 -- 172.21.15.116:0/1051830911 shutdown_connections 2024-03-30T00:58:23.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f118e4e5b80 1 -- 172.21.15.116:0/1051830911 wait complete. 2024-03-30T00:58:23.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.331+0000 7f118e4e5b80 20 client.25634 trim_cache size 0 max 16384 2024-03-30T00:58:23.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.348+0000 7f118e4e5b80 1 Processor -- start 2024-03-30T00:58:23.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.348+0000 7f118e4e5b80 1 -- start start 2024-03-30T00:58:23.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f118e4e5b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a23912b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f118e4e5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a2391271b0 0x55a239137c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f118e4e5b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239128050 0x55a239121610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f118e4e5b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a239038400 con 0x55a239126850 2024-03-30T00:58:23.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f118e4e5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a239122530 con 0x55a2391271b0 2024-03-30T00:58:23.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f118e4e5b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a239127ed0 con 0x55a239128050 2024-03-30T00:58:23.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f1173e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a23912b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f1173e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a23912b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40762/0 (socket says 172.21.15.116:40762) 2024-03-30T00:58:23.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f1173e00700 1 -- 172.21.15.116:0/3484987386 learned_addr learned my addr 172.21.15.116:0/3484987386 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:23.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f1173400700 1 --2- 172.21.15.116:0/3484987386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a2391271b0 0x55a239137c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f1178a00700 1 --2- 172.21.15.116:0/3484987386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239128050 0x55a239121610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f1173e00700 1 -- 172.21.15.116:0/3484987386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239128050 msgr2=0x55a239121610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f1173e00700 1 --2- 172.21.15.116:0/3484987386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239128050 0x55a239121610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f1173e00700 1 -- 172.21.15.116:0/3484987386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a2391271b0 msgr2=0x55a239137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f1173e00700 1 --2- 172.21.15.116:0/3484987386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a2391271b0 0x55a239137c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.349+0000 7f1173e00700 1 -- 172.21.15.116:0/3484987386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a239143550 con 0x55a239126850 2024-03-30T00:58:23.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.350+0000 7f1173e00700 1 --2- 172.21.15.116:0/3484987386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a23912b260 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f116409a130 tx=0x7f1164096610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9aee3a72b0f5093 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:23.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.350+0000 7f1169c00700 1 -- 172.21.15.116:0/3484987386 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11640b8040 con 0x55a239126850 2024-03-30T00:58:23.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.350+0000 7f1169c00700 1 -- 172.21.15.116:0/3484987386 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f116409e050 con 0x55a239126850 2024-03-30T00:58:23.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.350+0000 7f1169c00700 1 -- 172.21.15.116:0/3484987386 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11640998f0 con 0x55a239126850 2024-03-30T00:58:23.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.350+0000 7f118e4e5b80 1 -- 172.21.15.116:0/3484987386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 msgr2=0x55a23912b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.350+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/3484987386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a23912b260 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f116409a130 tx=0x7f1164096610 comp rx=0 tx=0).stop 2024-03-30T00:58:23.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.350+0000 7f118e4e5b80 1 -- 172.21.15.116:0/3484987386 shutdown_connections 2024-03-30T00:58:23.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.350+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/3484987386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239128050 0x55a239121610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.350+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/3484987386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a239126850 0x55a23912b260 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.350+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/3484987386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a2391271b0 0x55a239137c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.350+0000 7f118e4e5b80 1 -- 172.21.15.116:0/3484987386 >> 172.21.15.116:0/3484987386 conn(0x55a239085720 msgr2=0x55a23912e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:23.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.350+0000 7f118e4e5b80 1 -- 172.21.15.116:0/3484987386 shutdown_connections 2024-03-30T00:58:23.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.350+0000 7f118e4e5b80 1 -- 172.21.15.116:0/3484987386 wait complete. 2024-03-30T00:58:23.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.351+0000 7f118e4e5b80 1 -- start start 2024-03-30T00:58:23.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.351+0000 7f118e4e5b80 1 Processor -- start 2024-03-30T00:58:23.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.351+0000 7f118e4e5b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239126850 0x55a239136c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.351+0000 7f118e4e5b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2391271b0 0x55a239137170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f118e4e5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a239128050 0x55a239190320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f118e4e5b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a239190860 con 0x55a2391271b0 2024-03-30T00:58:23.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f118e4e5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2391909a0 con 0x55a239128050 2024-03-30T00:58:23.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f118e4e5b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a239190b30 con 0x55a239126850 2024-03-30T00:58:23.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f118e4e5b80 10 client.0 _ll_register_callbacks cb 0x55a23901d4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:23.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f1173400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2391271b0 0x55a239137170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f1173e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239126850 0x55a239136c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f1173400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2391271b0 0x55a239137170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40774/0 (socket says 172.21.15.116:40774) 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f1173400700 1 -- 172.21.15.116:0/2861517861 learned_addr learned my addr 172.21.15.116:0/2861517861 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f1178a00700 1 --2- 172.21.15.116:0/2861517861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a239128050 0x55a239190320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f1173400700 1 -- 172.21.15.116:0/2861517861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239126850 msgr2=0x55a239136c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f1173400700 1 --2- 172.21.15.116:0/2861517861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239126850 0x55a239136c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f1173400700 1 -- 172.21.15.116:0/2861517861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a239128050 msgr2=0x55a239190320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f1173400700 1 --2- 172.21.15.116:0/2861517861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a239128050 0x55a239190320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f1173400700 1 -- 172.21.15.116:0/2861517861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a238ef9430 con 0x55a2391271b0 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f1173e00700 1 --2- 172.21.15.116:0/2861517861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239126850 0x55a239136c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f1173400700 1 --2- 172.21.15.116:0/2861517861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2391271b0 0x55a239137170 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1160005910 tx=0x7f1160009330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f7653ede52d758 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.352+0000 7f115d600700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.353+0000 7f115d600700 1 -- 172.21.15.116:0/2861517861 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1160022040 con 0x55a2391271b0 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.353+0000 7f115d600700 1 -- 172.21.15.116:0/2861517861 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f116001d820 con 0x55a2391271b0 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.353+0000 7f118e4e5b80 1 -- 172.21.15.116:0/2861517861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f116409a1a0 con 0x55a2391271b0 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.353+0000 7f115d600700 1 -- 172.21.15.116:0/2861517861 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f116001da20 con 0x55a2391271b0 2024-03-30T00:58:23.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.353+0000 7f118e4e5b80 10 client.15582 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:23.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.353+0000 7f118e4e5b80 1 -- 172.21.15.116:0/2861517861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a23903df20 con 0x55a2391271b0 2024-03-30T00:58:23.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.353+0000 7f115d600700 1 -- 172.21.15.116:0/2861517861 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f116409a1a0 con 0x55a2391271b0 2024-03-30T00:58:23.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.353+0000 7f115d600700 1 -- 172.21.15.116:0/2861517861 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1160035b00 con 0x55a2391271b0 2024-03-30T00:58:23.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.353+0000 7f118e4e5b80 10 client.15582 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:23.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.353+0000 7f118e4e5b80 10 client.15582 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:23.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.353+0000 7f118e4e5b80 20 client.15582 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:23.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.353+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/2861517861 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a2391924e0 0x55a239194900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.354+0000 7f118e4e5b80 4 client.15582 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T00:58:23.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.354+0000 7f118e4e5b80 1 -- 172.21.15.116:0/2861517861 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a238ec6d70 con 0x55a2391924e0 2024-03-30T00:58:23.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.354+0000 7f1173e00700 1 --2- 172.21.15.116:0/2861517861 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a2391924e0 0x55a239194900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.354+0000 7f115d600700 10 client.15582 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:23.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.354+0000 7f1173e00700 1 --2- 172.21.15.116:0/2861517861 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a2391924e0 0x55a239194900 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f2f1c58df21876e server_cookie=597f26a1ef87733a in_seq=0 out_seq=0 2024-03-30T00:58:23.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.355+0000 7f115d600700 1 -- 172.21.15.116:0/2861517861 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a238ec6d70 con 0x55a2391924e0 2024-03-30T00:58:23.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.355+0000 7f115d600700 10 client.15582 handle_command_reply: tid=0 2024-03-30T00:58:23.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.355+0000 7f118e4e5b80 1 client.15582 shutdown 2024-03-30T00:58:23.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.355+0000 7f118e4e5b80 1 -- 172.21.15.116:0/2861517861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2391271b0 msgr2=0x55a239137170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.355+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/2861517861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2391271b0 0x55a239137170 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1160005910 tx=0x7f1160009330 comp rx=0 tx=0).stop 2024-03-30T00:58:23.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.355+0000 7f118e4e5b80 1 -- 172.21.15.116:0/2861517861 shutdown_connections 2024-03-30T00:58:23.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.355+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/2861517861 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a2391924e0 0x55a239194900 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.355+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/2861517861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a239126850 0x55a239136c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.355+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/2861517861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a239128050 0x55a239190320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.355+0000 7f115d600700 0 client.15582 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:23.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.355+0000 7f118e4e5b80 1 --2- 172.21.15.116:0/2861517861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2391271b0 0x55a239137170 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.355+0000 7f118e4e5b80 1 -- 172.21.15.116:0/2861517861 >> 172.21.15.116:0/2861517861 conn(0x55a239085720 msgr2=0x55a23912e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:23.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.356+0000 7f118e4e5b80 1 -- 172.21.15.116:0/2861517861 shutdown_connections 2024-03-30T00:58:23.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.356+0000 7f118e4e5b80 1 -- 172.21.15.116:0/2861517861 wait complete. 2024-03-30T00:58:23.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.356+0000 7f118e4e5b80 20 client.15582 trim_cache size 0 max 16384 2024-03-30T00:58:23.367 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:58:23.367 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-30T00:58:23.367 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "54cb865d-91e6-4433-80ed-87c187414eae", 2024-03-30T00:58:23.367 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-30T00:58:23.367 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:58:23.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.356+0000 7f117aa00700 1 -- 172.21.15.116:0/2416432699 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f11580a11a0 msgr2=0x7f11580a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.356+0000 7f117aa00700 1 --2- 172.21.15.116:0/2416432699 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f11580a11a0 0x7f11580a3660 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1160009c60 tx=0x7f1160009260 comp rx=0 tx=0).stop 2024-03-30T00:58:23.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.356+0000 7f117aa00700 1 -- 172.21.15.116:0/2416432699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1174100d80 msgr2=0x7f1174074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.356+0000 7f117aa00700 1 --2- 172.21.15.116:0/2416432699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1174100d80 0x7f1174074b10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f116400ea50 tx=0x7f116400ed60 comp rx=0 tx=0).stop 2024-03-30T00:58:23.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.356+0000 7f117aa00700 1 -- 172.21.15.116:0/2416432699 shutdown_connections 2024-03-30T00:58:23.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.356+0000 7f117aa00700 1 --2- 172.21.15.116:0/2416432699 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f11580a11a0 0x7f11580a3660 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.356+0000 7f117aa00700 1 --2- 172.21.15.116:0/2416432699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1174102910 0x7f11740736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.356+0000 7f117aa00700 1 --2- 172.21.15.116:0/2416432699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1174100d80 0x7f1174074b10 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.356+0000 7f117aa00700 1 --2- 172.21.15.116:0/2416432699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1174101f70 0x7f11740731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.356+0000 7f117aa00700 1 -- 172.21.15.116:0/2416432699 >> 172.21.15.116:0/2416432699 conn(0x7f11740fc2f0 msgr2=0x7f1174105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:23.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.357+0000 7f117aa00700 1 -- 172.21.15.116:0/2416432699 shutdown_connections 2024-03-30T00:58:23.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.357+0000 7f117aa00700 1 -- 172.21.15.116:0/2416432699 wait complete. 2024-03-30T00:58:23.381 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T00:58:23.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.569+0000 7efdf1e00700 1 -- 172.21.15.116:0/1409037830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdec100d80 msgr2=0x7efdec1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.569+0000 7efdf1e00700 1 --2- 172.21.15.116:0/1409037830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdec100d80 0x7efdec1011a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7efddc0099c0 tx=0x7efddc009cd0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.570+0000 7efdf1e00700 1 -- 172.21.15.116:0/1409037830 shutdown_connections 2024-03-30T00:58:23.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.570+0000 7efdf1e00700 1 --2- 172.21.15.116:0/1409037830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdec102910 0x7efdec1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.570+0000 7efdf1e00700 1 --2- 172.21.15.116:0/1409037830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdec100d80 0x7efdec1011a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.570+0000 7efdf1e00700 1 --2- 172.21.15.116:0/1409037830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efdec101f70 0x7efdec1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.570+0000 7efdf1e00700 1 -- 172.21.15.116:0/1409037830 >> 172.21.15.116:0/1409037830 conn(0x7efdec0fc2f0 msgr2=0x7efdec0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:23.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.570+0000 7efdf1e00700 1 -- 172.21.15.116:0/1409037830 shutdown_connections 2024-03-30T00:58:23.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.570+0000 7efdf1e00700 1 -- 172.21.15.116:0/1409037830 wait complete. 2024-03-30T00:58:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.571+0000 7efdf1e00700 1 Processor -- start 2024-03-30T00:58:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.571+0000 7efdf1e00700 1 -- start start 2024-03-30T00:58:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.571+0000 7efdf1e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdec100d80 0x7efdec196d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.571+0000 7efdf1e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdec101f70 0x7efdec197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.571+0000 7efdf1e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efdec102910 0x7efdec19e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.571+0000 7efdf1e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec19e840 con 0x7efdec100d80 2024-03-30T00:58:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.571+0000 7efdf1e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec19e9b0 con 0x7efdec102910 2024-03-30T00:58:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.571+0000 7efdf1e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec19eb70 con 0x7efdec101f70 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.571+0000 7efdeaa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdec100d80 0x7efdec196d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.571+0000 7efdeaa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdec100d80 0x7efdec196d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40794/0 (socket says 172.21.15.116:40794) 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.571+0000 7efdeaa00700 1 -- 172.21.15.116:0/1825260416 learned_addr learned my addr 172.21.15.116:0/1825260416 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.572+0000 7efdeb400700 1 --2- 172.21.15.116:0/1825260416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efdec102910 0x7efdec19e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.572+0000 7efdea000700 1 --2- 172.21.15.116:0/1825260416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdec101f70 0x7efdec197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.572+0000 7efdeaa00700 1 -- 172.21.15.116:0/1825260416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdec101f70 msgr2=0x7efdec197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.572+0000 7efdeaa00700 1 --2- 172.21.15.116:0/1825260416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdec101f70 0x7efdec197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.572+0000 7efdeaa00700 1 -- 172.21.15.116:0/1825260416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efdec102910 msgr2=0x7efdec19e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.572+0000 7efdeaa00700 1 --2- 172.21.15.116:0/1825260416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efdec102910 0x7efdec19e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.572+0000 7efdeaa00700 1 -- 172.21.15.116:0/1825260416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efddc0095f0 con 0x7efdec100d80 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.572+0000 7efdeaa00700 1 --2- 172.21.15.116:0/1825260416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdec100d80 0x7efdec196d50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7efddc003730 tx=0x7efddc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.572+0000 7efdf0a00700 1 -- 172.21.15.116:0/1825260416 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc01d070 con 0x7efdec100d80 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.572+0000 7efdf1e00700 1 -- 172.21.15.116:0/1825260416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdec19edf0 con 0x7efdec100d80 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.572+0000 7efdf0a00700 1 -- 172.21.15.116:0/1825260416 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efddc005050 con 0x7efdec100d80 2024-03-30T00:58:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.573+0000 7efdf1e00700 1 -- 172.21.15.116:0/1825260416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdec19f1c0 con 0x7efdec100d80 2024-03-30T00:58:23.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.573+0000 7efdf0a00700 1 -- 172.21.15.116:0/1825260416 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc004a30 con 0x7efdec100d80 2024-03-30T00:58:23.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.575+0000 7efdf0a00700 1 -- 172.21.15.116:0/1825260416 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7efddc0051c0 con 0x7efdec100d80 2024-03-30T00:58:23.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.577+0000 7efdf0a00700 1 --2- 172.21.15.116:0/1825260416 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efdd00a1120 0x7efdd00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.577+0000 7efe05934b80 1 Processor -- start 2024-03-30T00:58:23.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.577+0000 7efdf0a00700 1 -- 172.21.15.116:0/1825260416 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efddc0cd7f0 con 0x7efdec100d80 2024-03-30T00:58:23.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.577+0000 7efe05934b80 1 -- start start 2024-03-30T00:58:23.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.577+0000 7efdea000700 1 --2- 172.21.15.116:0/1825260416 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efdd00a1120 0x7efdd00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.577+0000 7efe05934b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790974be0 0x563790975000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.577+0000 7efe05934b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790975540 0x563790975e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.577+0000 7efe05934b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 0x563790976860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.577+0000 7efe05934b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637908c8990 con 0x563790975540 2024-03-30T00:58:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.577+0000 7efe05934b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637908c6350 con 0x563790974be0 2024-03-30T00:58:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.577+0000 7efe05934b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637908c1c40 con 0x5637909763e0 2024-03-30T00:58:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.578+0000 7efdeaa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790974be0 0x563790975000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.578+0000 7efdeaa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790974be0 0x563790975000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:42674/0 (socket says 172.21.15.116:42674) 2024-03-30T00:58:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.578+0000 7efdeaa00700 1 -- 172.21.15.116:0/2654373793 learned_addr learned my addr 172.21.15.116:0/2654373793 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.578+0000 7efdea000700 1 --2- 172.21.15.116:0/2654373793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790975540 0x563790975e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.578+0000 7efdeb400700 1 --2- 172.21.15.116:0/2654373793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 0x563790976860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.578+0000 7efdeaa00700 1 -- 172.21.15.116:0/2654373793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 msgr2=0x563790976860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.578+0000 7efdeaa00700 1 --2- 172.21.15.116:0/2654373793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 0x563790976860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.578+0000 7efdeaa00700 1 -- 172.21.15.116:0/2654373793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790975540 msgr2=0x563790975e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.578+0000 7efdeaa00700 1 --2- 172.21.15.116:0/2654373793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790975540 0x563790975e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.578+0000 7efdea000700 1 --2- 172.21.15.116:0/1825260416 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efdd00a1120 0x7efdd00a35e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efdd8009cb0 tx=0x7efdd8011720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.578+0000 7efdeaa00700 1 -- 172.21.15.116:0/2654373793 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637907433b0 con 0x563790974be0 2024-03-30T00:58:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.578+0000 7efdeaa00700 1 --2- 172.21.15.116:0/2654373793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790974be0 0x563790975000 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efddc096e60 tx=0x7efddc09a2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d50a95bdffbba60 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.578+0000 7efdd7e00700 1 -- 172.21.15.116:0/2654373793 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc09a960 con 0x563790974be0 2024-03-30T00:58:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.579+0000 7efdd7e00700 1 -- 172.21.15.116:0/2654373793 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efddc09e790 con 0x563790974be0 2024-03-30T00:58:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.579+0000 7efdd7e00700 1 -- 172.21.15.116:0/2654373793 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc09a0b0 con 0x563790974be0 2024-03-30T00:58:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.579+0000 7efe05934b80 1 -- 172.21.15.116:0/2654373793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790974be0 msgr2=0x563790975000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.579+0000 7efe05934b80 1 --2- 172.21.15.116:0/2654373793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790974be0 0x563790975000 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efddc096e60 tx=0x7efddc09a2c0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.579+0000 7efe05934b80 1 -- 172.21.15.116:0/2654373793 shutdown_connections 2024-03-30T00:58:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.579+0000 7efe05934b80 1 --2- 172.21.15.116:0/2654373793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 0x563790976860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.579+0000 7efe05934b80 1 --2- 172.21.15.116:0/2654373793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790974be0 0x563790975000 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.579+0000 7efe05934b80 1 --2- 172.21.15.116:0/2654373793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790975540 0x563790975e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.579+0000 7efe05934b80 1 -- 172.21.15.116:0/2654373793 >> 172.21.15.116:0/2654373793 conn(0x5637908cf6a0 msgr2=0x56379096b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.579+0000 7efe05934b80 1 -- 172.21.15.116:0/2654373793 shutdown_connections 2024-03-30T00:58:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.579+0000 7efe05934b80 1 -- 172.21.15.116:0/2654373793 wait complete. 2024-03-30T00:58:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.580+0000 7efe05934b80 1 -- start start 2024-03-30T00:58:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.580+0000 7efe05934b80 1 Processor -- start 2024-03-30T00:58:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.580+0000 7efe05934b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x56379097e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.580+0000 7efe05934b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563790975540 0x56379097ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.580+0000 7efe05934b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637909763e0 0x563790985c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.580+0000 7efe05934b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56379085dd80 con 0x563790974be0 2024-03-30T00:58:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.580+0000 7efe05934b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56379085da20 con 0x5637909763e0 2024-03-30T00:58:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.580+0000 7efe05934b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637907456a0 con 0x563790975540 2024-03-30T00:58:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efe05934b80 10 client.0 _ll_register_callbacks cb 0x563790867470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efdeaa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x56379097e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efdeaa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x56379097e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40824/0 (socket says 172.21.15.116:40824) 2024-03-30T00:58:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efdeaa00700 1 -- 172.21.15.116:0/516128355 learned_addr learned my addr 172.21.15.116:0/516128355 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efdeb400700 1 --2- 172.21.15.116:0/516128355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637909763e0 0x563790985c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efdea000700 1 --2- 172.21.15.116:0/516128355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563790975540 0x56379097ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efdeaa00700 1 -- 172.21.15.116:0/516128355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563790975540 msgr2=0x56379097ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efdeaa00700 1 --2- 172.21.15.116:0/516128355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563790975540 0x56379097ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efdeaa00700 1 -- 172.21.15.116:0/516128355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637909763e0 msgr2=0x563790985c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efdeaa00700 1 --2- 172.21.15.116:0/516128355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637909763e0 0x563790985c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efdeaa00700 1 -- 172.21.15.116:0/516128355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdec19f1c0 con 0x563790974be0 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efdea000700 1 --2- 172.21.15.116:0/516128355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563790975540 0x56379097ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efdeaa00700 1 --2- 172.21.15.116:0/516128355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x56379097e720 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efddc009990 tx=0x7efddc09af30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=855794ec5c882489 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efdd4c00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.581+0000 7efdd4c00700 1 -- 172.21.15.116:0/516128355 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc09a960 con 0x563790974be0 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.582+0000 7efdd4c00700 1 -- 172.21.15.116:0/516128355 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efddc0a28c0 con 0x563790974be0 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.582+0000 7efe05934b80 1 -- 172.21.15.116:0/516128355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563790882380 con 0x563790974be0 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.582+0000 7efdd4c00700 1 -- 172.21.15.116:0/516128355 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7efddc0a2a80 con 0x563790974be0 2024-03-30T00:58:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.582+0000 7efe05934b80 10 client.15594 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.582+0000 7efe05934b80 1 -- 172.21.15.116:0/516128355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5637906efb60 con 0x563790974be0 2024-03-30T00:58:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.582+0000 7efdd4c00700 1 -- 172.21.15.116:0/516128355 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efddc07f070 con 0x563790974be0 2024-03-30T00:58:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.582+0000 7efdd4c00700 1 -- 172.21.15.116:0/516128355 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7efddc0ba100 con 0x563790974be0 2024-03-30T00:58:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.582+0000 7efe05934b80 10 client.15594 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.582+0000 7efe05934b80 10 client.15594 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.582+0000 7efe05934b80 20 client.15594 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.582+0000 7efe05934b80 1 --2- 172.21.15.116:0/516128355 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56379098a070 0x56379098c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.582+0000 7efe05934b80 4 client.15594 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:58:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.582+0000 7efe05934b80 1 -- 172.21.15.116:0/516128355 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5637908ba700 con 0x56379098a070 2024-03-30T00:58:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.583+0000 7efdea000700 1 --2- 172.21.15.116:0/516128355 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56379098a070 0x56379098c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.583+0000 7efdea000700 1 --2- 172.21.15.116:0/516128355 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56379098a070 0x56379098c490 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b11fee2e8a32a38 server_cookie=ed99ba9bd8f18af3 in_seq=0 out_seq=0 2024-03-30T00:58:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.583+0000 7efdd4c00700 10 client.15594 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efdd4c00700 1 -- 172.21.15.116:0/516128355 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5637908ba700 con 0x56379098a070 2024-03-30T00:58:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efdd4c00700 10 client.15594 handle_command_reply: tid=0 2024-03-30T00:58:23.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efe05934b80 1 client.15594 shutdown 2024-03-30T00:58:23.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efe05934b80 1 -- 172.21.15.116:0/516128355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 msgr2=0x56379097e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efe05934b80 1 --2- 172.21.15.116:0/516128355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x56379097e720 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efddc009990 tx=0x7efddc09af30 comp rx=0 tx=0).stop 2024-03-30T00:58:23.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efe05934b80 1 -- 172.21.15.116:0/516128355 shutdown_connections 2024-03-30T00:58:23.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efe05934b80 1 --2- 172.21.15.116:0/516128355 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56379098a070 0x56379098c490 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efe05934b80 1 --2- 172.21.15.116:0/516128355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5637909763e0 0x563790985c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efe05934b80 1 --2- 172.21.15.116:0/516128355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x56379097e720 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efe05934b80 1 --2- 172.21.15.116:0/516128355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563790975540 0x56379097ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efe05934b80 1 -- 172.21.15.116:0/516128355 >> 172.21.15.116:0/516128355 conn(0x5637908cf6a0 msgr2=0x56379096f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:23.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efdd4c00700 0 client.15594 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:23.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efe05934b80 1 -- 172.21.15.116:0/516128355 shutdown_connections 2024-03-30T00:58:23.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efe05934b80 1 -- 172.21.15.116:0/516128355 wait complete. 2024-03-30T00:58:23.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.584+0000 7efe05934b80 20 client.15594 trim_cache size 0 max 16384 2024-03-30T00:58:23.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.601+0000 7efe05934b80 1 Processor -- start 2024-03-30T00:58:23.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.602+0000 7efe05934b80 1 -- start start 2024-03-30T00:58:23.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.602+0000 7efe05934b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x56379098c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.602+0000 7efe05934b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790975540 0x563790985c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.602+0000 7efe05934b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 0x56379096b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.602+0000 7efe05934b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56379097f3a0 con 0x563790974be0 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.602+0000 7efe05934b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637909747e0 con 0x563790975540 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.602+0000 7efe05934b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56379097c660 con 0x5637909763e0 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.602+0000 7efdeaa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x56379098c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.602+0000 7efdeaa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x56379098c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40834/0 (socket says 172.21.15.116:40834) 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.602+0000 7efdeaa00700 1 -- 172.21.15.116:0/2392440023 learned_addr learned my addr 172.21.15.116:0/2392440023 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.602+0000 7efdeb400700 1 --2- 172.21.15.116:0/2392440023 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 0x56379096b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.602+0000 7efdea000700 1 --2- 172.21.15.116:0/2392440023 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790975540 0x563790985c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.603+0000 7efdeaa00700 1 -- 172.21.15.116:0/2392440023 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 msgr2=0x56379096b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.603+0000 7efdeaa00700 1 --2- 172.21.15.116:0/2392440023 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 0x56379096b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.603+0000 7efdeaa00700 1 -- 172.21.15.116:0/2392440023 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790975540 msgr2=0x563790985c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.603+0000 7efdeaa00700 1 --2- 172.21.15.116:0/2392440023 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790975540 0x563790985c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.603+0000 7efdeaa00700 1 -- 172.21.15.116:0/2392440023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563790887ea0 con 0x563790974be0 2024-03-30T00:58:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.603+0000 7efdeaa00700 1 --2- 172.21.15.116:0/2392440023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x56379098c080 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efddc096e60 tx=0x7efddc005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae52246a6c8b85d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.603+0000 7efde1200700 1 -- 172.21.15.116:0/2392440023 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc09a2c0 con 0x563790974be0 2024-03-30T00:58:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.603+0000 7efde1200700 1 -- 172.21.15.116:0/2392440023 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efddc07f070 con 0x563790974be0 2024-03-30T00:58:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.603+0000 7efde1200700 1 -- 172.21.15.116:0/2392440023 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc0ba0b0 con 0x563790974be0 2024-03-30T00:58:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.603+0000 7efe05934b80 1 -- 172.21.15.116:0/2392440023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 msgr2=0x56379098c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.603+0000 7efe05934b80 1 --2- 172.21.15.116:0/2392440023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x56379098c080 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efddc096e60 tx=0x7efddc005ef0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.604+0000 7efe05934b80 1 -- 172.21.15.116:0/2392440023 shutdown_connections 2024-03-30T00:58:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.604+0000 7efe05934b80 1 --2- 172.21.15.116:0/2392440023 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 0x56379096b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.604+0000 7efe05934b80 1 --2- 172.21.15.116:0/2392440023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x56379098c080 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.604+0000 7efe05934b80 1 --2- 172.21.15.116:0/2392440023 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790975540 0x563790985c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.604+0000 7efe05934b80 1 -- 172.21.15.116:0/2392440023 >> 172.21.15.116:0/2392440023 conn(0x5637908cf6a0 msgr2=0x56379097c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.604+0000 7efe05934b80 1 -- 172.21.15.116:0/2392440023 shutdown_connections 2024-03-30T00:58:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.604+0000 7efe05934b80 1 -- 172.21.15.116:0/2392440023 wait complete. 2024-03-30T00:58:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.604+0000 7efe05934b80 1 -- start start 2024-03-30T00:58:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.604+0000 7efe05934b80 1 Processor -- start 2024-03-30T00:58:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efe05934b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x563790985630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efe05934b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790975540 0x563790985b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efe05934b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 0x563790983d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efe05934b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563790984420 con 0x563790974be0 2024-03-30T00:58:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efe05934b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563790984590 con 0x563790975540 2024-03-30T00:58:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efe05934b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x563790984750 con 0x5637909763e0 2024-03-30T00:58:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efe05934b80 10 client.0 _ll_register_callbacks cb 0x563790867470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efdeaa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x563790985630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efdeaa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x563790985630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40850/0 (socket says 172.21.15.116:40850) 2024-03-30T00:58:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efdeaa00700 1 -- 172.21.15.116:0/1788280931 learned_addr learned my addr 172.21.15.116:0/1788280931 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efdea000700 1 --2- 172.21.15.116:0/1788280931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790975540 0x563790985b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efdeb400700 1 --2- 172.21.15.116:0/1788280931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 0x563790983d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efdeaa00700 1 -- 172.21.15.116:0/1788280931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 msgr2=0x563790983d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efdeaa00700 1 --2- 172.21.15.116:0/1788280931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 0x563790983d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efdeaa00700 1 -- 172.21.15.116:0/1788280931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790975540 msgr2=0x563790985b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:23.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efdeaa00700 1 --2- 172.21.15.116:0/1788280931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790975540 0x563790985b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efdeaa00700 1 -- 172.21.15.116:0/1788280931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5637906efb60 con 0x563790974be0 2024-03-30T00:58:23.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efdeb400700 1 --2- 172.21.15.116:0/1788280931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 0x563790983d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:58:23.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efdeaa00700 1 --2- 172.21.15.116:0/1788280931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x563790985630 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efddc009990 tx=0x7efddc0ba3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f481377ed02030 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:23.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.605+0000 7efdd4c00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:58:23.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.606+0000 7efdd4c00700 1 -- 172.21.15.116:0/1788280931 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc09a2c0 con 0x563790974be0 2024-03-30T00:58:23.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.606+0000 7efdd4c00700 1 -- 172.21.15.116:0/1788280931 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efddc09a960 con 0x563790974be0 2024-03-30T00:58:23.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.606+0000 7efe05934b80 1 -- 172.21.15.116:0/1788280931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efddc09e790 con 0x563790974be0 2024-03-30T00:58:23.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.606+0000 7efdd4c00700 1 -- 172.21.15.116:0/1788280931 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7efddc0a2d50 con 0x563790974be0 2024-03-30T00:58:23.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.606+0000 7efe05934b80 10 client.15600 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.606+0000 7efe05934b80 1 -- 172.21.15.116:0/1788280931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563790747cd0 con 0x563790974be0 2024-03-30T00:58:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.606+0000 7efdd4c00700 1 -- 172.21.15.116:0/1788280931 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efddc09e790 con 0x563790974be0 2024-03-30T00:58:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.606+0000 7efdd4c00700 1 -- 172.21.15.116:0/1788280931 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7efddc0a9020 con 0x563790974be0 2024-03-30T00:58:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.606+0000 7efe05934b80 10 client.15600 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.606+0000 7efe05934b80 10 client.15600 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.606+0000 7efe05934b80 20 client.15600 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.606+0000 7efe05934b80 1 --2- 172.21.15.116:0/1788280931 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5637909dec80 0x5637909e10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.606+0000 7efe05934b80 4 client.15600 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T00:58:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.606+0000 7efe05934b80 1 -- 172.21.15.116:0/1788280931 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563790710cf0 con 0x5637909dec80 2024-03-30T00:58:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.607+0000 7efdea000700 1 --2- 172.21.15.116:0/1788280931 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5637909dec80 0x5637909e10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.607+0000 7efdea000700 1 --2- 172.21.15.116:0/1788280931 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5637909dec80 0x5637909e10a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7df7812a10a7693 server_cookie=e9f9ad1ee5c6aa19 in_seq=0 out_seq=0 2024-03-30T00:58:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.607+0000 7efdd4c00700 10 client.15600 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.607+0000 7efdd4c00700 1 -- 172.21.15.116:0/1788280931 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563790710cf0 con 0x5637909dec80 2024-03-30T00:58:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.607+0000 7efdd4c00700 10 client.15600 handle_command_reply: tid=0 2024-03-30T00:58:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.607+0000 7efe05934b80 1 client.15600 shutdown 2024-03-30T00:58:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efe05934b80 1 -- 172.21.15.116:0/1788280931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 msgr2=0x563790985630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efe05934b80 1 --2- 172.21.15.116:0/1788280931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x563790985630 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efddc009990 tx=0x7efddc0ba3e0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efe05934b80 1 -- 172.21.15.116:0/1788280931 shutdown_connections 2024-03-30T00:58:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efe05934b80 1 --2- 172.21.15.116:0/1788280931 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5637909dec80 0x5637909e10a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efe05934b80 1 --2- 172.21.15.116:0/1788280931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5637909763e0 0x563790983d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efdd4c00700 0 client.15600 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efe05934b80 1 --2- 172.21.15.116:0/1788280931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563790974be0 0x563790985630 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efe05934b80 1 --2- 172.21.15.116:0/1788280931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563790975540 0x563790985b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efe05934b80 1 -- 172.21.15.116:0/1788280931 >> 172.21.15.116:0/1788280931 conn(0x5637908cf6a0 msgr2=0x56379098c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efe05934b80 1 -- 172.21.15.116:0/1788280931 shutdown_connections 2024-03-30T00:58:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efe05934b80 1 -- 172.21.15.116:0/1788280931 wait complete. 2024-03-30T00:58:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efe05934b80 20 client.15600 trim_cache size 0 max 16384 2024-03-30T00:58:23.621 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:58:23.621 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T00:58:23.621 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-30T00:58:23.621 INFO:teuthology.orchestra.run.smithi116.stdout: "54cb865d-91e6-4433-80ed-87c187414eae": { 2024-03-30T00:58:23.621 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-30T00:58:23.621 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "54cb865d-91e6-4433-80ed-87c187414eae", 2024-03-30T00:58:23.621 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-30T00:58:23.621 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T00:58:23.621 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T00:58:23.621 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:58:23.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efdf1e00700 1 -- 172.21.15.116:0/1825260416 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efdd00a1120 msgr2=0x7efdd00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efdf1e00700 1 --2- 172.21.15.116:0/1825260416 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efdd00a1120 0x7efdd00a35e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efdd8009cb0 tx=0x7efdd8011720 comp rx=0 tx=0).stop 2024-03-30T00:58:23.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efdf1e00700 1 -- 172.21.15.116:0/1825260416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdec100d80 msgr2=0x7efdec196d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:23.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.608+0000 7efdf1e00700 1 --2- 172.21.15.116:0/1825260416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdec100d80 0x7efdec196d50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7efddc003730 tx=0x7efddc00f690 comp rx=0 tx=0).stop 2024-03-30T00:58:23.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.609+0000 7efdf1e00700 1 -- 172.21.15.116:0/1825260416 shutdown_connections 2024-03-30T00:58:23.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.609+0000 7efdf1e00700 1 --2- 172.21.15.116:0/1825260416 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efdd00a1120 0x7efdd00a35e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.609+0000 7efdf1e00700 1 --2- 172.21.15.116:0/1825260416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efdec102910 0x7efdec19e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.609+0000 7efdf1e00700 1 --2- 172.21.15.116:0/1825260416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdec100d80 0x7efdec196d50 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.609+0000 7efdf1e00700 1 --2- 172.21.15.116:0/1825260416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdec101f70 0x7efdec197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:23.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.609+0000 7efdf1e00700 1 -- 172.21.15.116:0/1825260416 >> 172.21.15.116:0/1825260416 conn(0x7efdec0fc2f0 msgr2=0x7efdec1053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:23.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.609+0000 7efdf1e00700 1 -- 172.21.15.116:0/1825260416 shutdown_connections 2024-03-30T00:58:23.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:23.609+0000 7efdf1e00700 1 -- 172.21.15.116:0/1825260416 wait complete. 2024-03-30T00:58:23.633 INFO:tasks.cephfs.filesystem:scrub status for tag:54cb865d-91e6-4433-80ed-87c187414eae - {'path': '/', 'tag': '54cb865d-91e6-4433-80ed-87c187414eae', 'options': 'recursive,force'} 2024-03-30T00:58:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:24 smithi181 ceph-mon[47622]: pgmap v307: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:24 smithi181 ceph-mon[47622]: scrub queued for path: / 2024-03-30T00:58:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:24 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T00:58:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:24 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T00:58:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:24 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T00:58:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:24 smithi130 ceph-mon[47626]: pgmap v307: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:24 smithi130 ceph-mon[47626]: scrub queued for path: / 2024-03-30T00:58:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:24 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T00:58:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:24 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T00:58:24.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:24 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T00:58:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:24 smithi116 ceph-mon[38086]: pgmap v307: 129 pgs: 129 active+clean; 652 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:24 smithi116 ceph-mon[38086]: scrub queued for path: / 2024-03-30T00:58:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:24 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T00:58:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:24 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T00:58:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:24 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T00:58:26.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:26 smithi181 ceph-mon[47622]: pgmap v308: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-30T00:58:26.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:26 smithi130 ceph-mon[47626]: pgmap v308: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-30T00:58:26.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:26 smithi116 ceph-mon[38086]: pgmap v308: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-30T00:58:27.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:27 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T00:58:27.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:27 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T00:58:27.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:27 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T00:58:28.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:28 smithi181 ceph-mon[47622]: pgmap v309: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-30T00:58:28.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:28 smithi130 ceph-mon[47626]: pgmap v309: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-30T00:58:28.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:28 smithi116 ceph-mon[38086]: pgmap v309: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-30T00:58:30.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:30 smithi181 ceph-mon[47622]: pgmap v310: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-30T00:58:30.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:30 smithi130 ceph-mon[47626]: pgmap v310: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-30T00:58:30.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:30 smithi116 ceph-mon[38086]: pgmap v310: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-30T00:58:32.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:32 smithi181 ceph-mon[47622]: pgmap v311: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-30T00:58:32.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:32 smithi130 ceph-mon[47626]: pgmap v311: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-30T00:58:32.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:32 smithi116 ceph-mon[38086]: pgmap v311: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-30T00:58:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:34 smithi181 ceph-mon[47622]: pgmap v312: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-30T00:58:34.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:34 smithi130 ceph-mon[47626]: pgmap v312: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-30T00:58:34.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:34 smithi116 ceph-mon[38086]: pgmap v312: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-30T00:58:36.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:36 smithi181 ceph-mon[47622]: pgmap v313: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-30T00:58:36.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:36 smithi130 ceph-mon[47626]: pgmap v313: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-30T00:58:36.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:36 smithi116 ceph-mon[38086]: pgmap v313: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-30T00:58:38.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:38 smithi181 ceph-mon[47622]: pgmap v314: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-30T00:58:38.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:38 smithi130 ceph-mon[47626]: pgmap v314: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-30T00:58:38.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:38 smithi116 ceph-mon[38086]: pgmap v314: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-30T00:58:40.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:40 smithi130 ceph-mon[47626]: pgmap v315: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-30T00:58:40.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:40 smithi116 ceph-mon[38086]: pgmap v315: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-30T00:58:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:40 smithi181 ceph-mon[47622]: pgmap v315: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-30T00:58:42.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:42 smithi130 ceph-mon[47626]: pgmap v316: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-30T00:58:42.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:42 smithi116 ceph-mon[38086]: pgmap v316: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-30T00:58:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:42 smithi181 ceph-mon[47622]: pgmap v316: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-30T00:58:44.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:44 smithi130 ceph-mon[47626]: pgmap v317: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-30T00:58:44.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:44 smithi116 ceph-mon[38086]: pgmap v317: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-30T00:58:45.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:44 smithi181 ceph-mon[47622]: pgmap v317: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-30T00:58:46.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:46 smithi130 ceph-mon[47626]: pgmap v318: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-30T00:58:46.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:46 smithi116 ceph-mon[38086]: pgmap v318: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-30T00:58:47.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:46 smithi181 ceph-mon[47622]: pgmap v318: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-30T00:58:48.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:48 smithi130 ceph-mon[47626]: pgmap v319: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:48.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:48 smithi116 ceph-mon[38086]: pgmap v319: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:49.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:48 smithi181 ceph-mon[47622]: pgmap v319: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:50.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:50 smithi130 ceph-mon[47626]: pgmap v320: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:58:50.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:50 smithi116 ceph-mon[38086]: pgmap v320: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:58:51.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:50 smithi181 ceph-mon[47622]: pgmap v320: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:58:52.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:52 smithi130 ceph-mon[47626]: pgmap v321: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:58:52.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:52 smithi116 ceph-mon[38086]: pgmap v321: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:58:53.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:52 smithi181 ceph-mon[47622]: pgmap v321: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T00:58:53.634 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T00:58:53.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.842+0000 7f642c800700 1 -- 172.21.15.116:0/3099983341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6424102f90 msgr2=0x7f64241033b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:53.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.842+0000 7f642c800700 1 --2- 172.21.15.116:0/3099983341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6424102f90 0x7f64241033b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6418009920 tx=0x7f6418009c30 comp rx=0 tx=0).stop 2024-03-30T00:58:53.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.842+0000 7f642c800700 1 -- 172.21.15.116:0/3099983341 shutdown_connections 2024-03-30T00:58:53.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.842+0000 7f642c800700 1 --2- 172.21.15.116:0/3099983341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6424104b20 0x7f642410b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.843+0000 7f642c800700 1 --2- 172.21.15.116:0/3099983341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6424102f90 0x7f64241033b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.843+0000 7f642c800700 1 --2- 172.21.15.116:0/3099983341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6424104180 0x7f64241045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.843+0000 7f642c800700 1 -- 172.21.15.116:0/3099983341 >> 172.21.15.116:0/3099983341 conn(0x7f64240fe500 msgr2=0x7f6424100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:53.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.843+0000 7f642c800700 1 -- 172.21.15.116:0/3099983341 shutdown_connections 2024-03-30T00:58:53.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.843+0000 7f642c800700 1 -- 172.21.15.116:0/3099983341 wait complete. 2024-03-30T00:58:53.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.844+0000 7f642c800700 1 Processor -- start 2024-03-30T00:58:53.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.844+0000 7f642c800700 1 -- start start 2024-03-30T00:58:53.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.844+0000 7f642c800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6424102f90 0x7f6424199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.844+0000 7f642c800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6424104180 0x7f6424199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.844+0000 7f642c800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6424104b20 0x7f64241a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.844+0000 7f642c800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64241a0b40 con 0x7f6424102f90 2024-03-30T00:58:53.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.844+0000 7f642c800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64241a0c80 con 0x7f6424104180 2024-03-30T00:58:53.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.844+0000 7f642c800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64241a0e10 con 0x7f6424104b20 2024-03-30T00:58:53.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.845+0000 7f642a800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6424104b20 0x7f64241a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.845+0000 7f642a800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6424104b20 0x7f64241a0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:55022/0 (socket says 172.21.15.116:55022) 2024-03-30T00:58:53.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.845+0000 7f642a800700 1 -- 172.21.15.116:0/4098954861 learned_addr learned my addr 172.21.15.116:0/4098954861 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:53.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.845+0000 7f6429400700 1 --2- 172.21.15.116:0/4098954861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6424104180 0x7f6424199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.845+0000 7f6429e00700 1 --2- 172.21.15.116:0/4098954861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6424102f90 0x7f6424199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.845+0000 7f6429400700 1 -- 172.21.15.116:0/4098954861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6424104b20 msgr2=0x7f64241a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:53.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.845+0000 7f6429400700 1 --2- 172.21.15.116:0/4098954861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6424104b20 0x7f64241a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.845+0000 7f6429400700 1 -- 172.21.15.116:0/4098954861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6424102f90 msgr2=0x7f6424199010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:53.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.845+0000 7f6429400700 1 --2- 172.21.15.116:0/4098954861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6424102f90 0x7f6424199010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.845+0000 7f6429400700 1 -- 172.21.15.116:0/4098954861 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6420009c40 con 0x7f6424104180 2024-03-30T00:58:53.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.846+0000 7f6429400700 1 --2- 172.21.15.116:0/4098954861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6424104180 0x7f6424199550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f641400e9c0 tx=0x7f641400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:53.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.846+0000 7f641ea00700 1 -- 172.21.15.116:0/4098954861 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641400c9c0 con 0x7f6424104180 2024-03-30T00:58:53.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.846+0000 7f641ea00700 1 -- 172.21.15.116:0/4098954861 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6414010430 con 0x7f6424104180 2024-03-30T00:58:53.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.846+0000 7f641ea00700 1 -- 172.21.15.116:0/4098954861 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414018680 con 0x7f6424104180 2024-03-30T00:58:53.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.846+0000 7f642c800700 1 -- 172.21.15.116:0/4098954861 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6418009520 con 0x7f6424104180 2024-03-30T00:58:53.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.847+0000 7f642c800700 1 -- 172.21.15.116:0/4098954861 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64241a1470 con 0x7f6424104180 2024-03-30T00:58:53.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.851+0000 7f644024eb80 1 Processor -- start 2024-03-30T00:58:53.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.851+0000 7f644024eb80 1 -- start start 2024-03-30T00:58:53.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f644024eb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca47d0 0x55d675ca4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f644024eb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5130 0x55d675ca59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f644024eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca5fd0 0x55d675ca6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f644024eb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d675bfc990 con 0x55d675ca5130 2024-03-30T00:58:53.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f644024eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d675bfa350 con 0x55d675ca5fd0 2024-03-30T00:58:53.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f644024eb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d675bf5c40 con 0x55d675ca47d0 2024-03-30T00:58:53.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f6429e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5130 0x55d675ca59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f642a800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca47d0 0x55d675ca4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f6429e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5130 0x55d675ca59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56852/0 (socket says 172.21.15.116:56852) 2024-03-30T00:58:53.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f6429e00700 1 -- 172.21.15.116:0/1744749217 learned_addr learned my addr 172.21.15.116:0/1744749217 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:53.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f6429e00700 1 -- 172.21.15.116:0/1744749217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca47d0 msgr2=0x55d675ca4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:53.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f6429e00700 1 --2- 172.21.15.116:0/1744749217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca47d0 0x55d675ca4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f6429e00700 1 -- 172.21.15.116:0/1744749217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca5fd0 msgr2=0x55d675ca6450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T00:58:53.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f6429e00700 1 --2- 172.21.15.116:0/1744749217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca5fd0 0x55d675ca6450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.852+0000 7f6429e00700 1 -- 172.21.15.116:0/1744749217 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d675a773b0 con 0x55d675ca5130 2024-03-30T00:58:53.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.853+0000 7f6429e00700 1 --2- 172.21.15.116:0/1744749217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5130 0x55d675ca59f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f641800fa40 tx=0x7f6418009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6906c4e0d42748 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:53.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.853+0000 7f6413000700 1 -- 172.21.15.116:0/1744749217 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641800f6e0 con 0x55d675ca5130 2024-03-30T00:58:53.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.853+0000 7f641ea00700 1 -- 172.21.15.116:0/4098954861 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f6414010860 con 0x7f6424104180 2024-03-30T00:58:53.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.854+0000 7f6413000700 1 -- 172.21.15.116:0/1744749217 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f641801d040 con 0x55d675ca5130 2024-03-30T00:58:53.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.854+0000 7f6413000700 1 -- 172.21.15.116:0/1744749217 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6418005710 con 0x55d675ca5130 2024-03-30T00:58:53.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.855+0000 7f644024eb80 1 -- 172.21.15.116:0/1744749217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5130 msgr2=0x55d675ca59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:53.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.855+0000 7f644024eb80 1 --2- 172.21.15.116:0/1744749217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5130 0x55d675ca59f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f641800fa40 tx=0x7f6418009f80 comp rx=0 tx=0).stop 2024-03-30T00:58:53.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.855+0000 7f644024eb80 1 -- 172.21.15.116:0/1744749217 shutdown_connections 2024-03-30T00:58:53.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.855+0000 7f644024eb80 1 --2- 172.21.15.116:0/1744749217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca47d0 0x55d675ca4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.855+0000 7f644024eb80 1 --2- 172.21.15.116:0/1744749217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca5fd0 0x55d675ca6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.855+0000 7f644024eb80 1 --2- 172.21.15.116:0/1744749217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5130 0x55d675ca59f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.855+0000 7f644024eb80 1 -- 172.21.15.116:0/1744749217 >> 172.21.15.116:0/1744749217 conn(0x55d675c036a0 msgr2=0x55d675c9f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:53.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.855+0000 7f644024eb80 1 -- 172.21.15.116:0/1744749217 shutdown_connections 2024-03-30T00:58:53.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.855+0000 7f641ea00700 1 --2- 172.21.15.116:0/4098954861 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f640c0a10c0 0x7f640c0a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.855+0000 7f644024eb80 1 -- 172.21.15.116:0/1744749217 wait complete. 2024-03-30T00:58:53.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.856+0000 7f641ea00700 1 -- 172.21.15.116:0/4098954861 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6414014070 con 0x7f6424104180 2024-03-30T00:58:53.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.856+0000 7f6429e00700 1 --2- 172.21.15.116:0/4098954861 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f640c0a10c0 0x7f640c0a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.857+0000 7f644024eb80 1 -- start start 2024-03-30T00:58:53.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.857+0000 7f6429e00700 1 --2- 172.21.15.116:0/4098954861 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f640c0a10c0 0x7f640c0a3580 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6418005a80 tx=0x7f6418009c30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:53.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.858+0000 7f644024eb80 1 Processor -- start 2024-03-30T00:58:53.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.858+0000 7f644024eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca47d0 0x55d675cb6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.858+0000 7f644024eb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca5130 0x55d675cb7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.858+0000 7f644024eb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5fd0 0x55d675cbe370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.858+0000 7f644024eb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d675b91d80 con 0x55d675ca5fd0 2024-03-30T00:58:53.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.858+0000 7f644024eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d675b91a20 con 0x55d675ca47d0 2024-03-30T00:58:53.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.858+0000 7f644024eb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d675a796a0 con 0x55d675ca5130 2024-03-30T00:58:53.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.858+0000 7f644024eb80 10 client.0 _ll_register_callbacks cb 0x55d675b9b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:53.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.859+0000 7f642a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5fd0 0x55d675cbe370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.859+0000 7f642a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5fd0 0x55d675cbe370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56856/0 (socket says 172.21.15.116:56856) 2024-03-30T00:58:53.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.859+0000 7f642a800700 1 -- 172.21.15.116:0/542456270 learned_addr learned my addr 172.21.15.116:0/542456270 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:53.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.859+0000 7f6429e00700 1 --2- 172.21.15.116:0/542456270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca47d0 0x55d675cb6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.859+0000 7f642a800700 1 -- 172.21.15.116:0/542456270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca5130 msgr2=0x55d675cb7330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T00:58:53.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.859+0000 7f642a800700 1 --2- 172.21.15.116:0/542456270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca5130 0x55d675cb7330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.859+0000 7f642a800700 1 -- 172.21.15.116:0/542456270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca47d0 msgr2=0x55d675cb6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:53.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.859+0000 7f642a800700 1 --2- 172.21.15.116:0/542456270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca47d0 0x55d675cb6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.859+0000 7f642a800700 1 -- 172.21.15.116:0/542456270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642410bd20 con 0x55d675ca5fd0 2024-03-30T00:58:53.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.859+0000 7f642a800700 1 --2- 172.21.15.116:0/542456270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5fd0 0x55d675cbe370 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6420007fb0 tx=0x7f6420009350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47712b390fd5f95b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:53.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.859+0000 7f6407400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:58:53.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.860+0000 7f6407400700 1 -- 172.21.15.116:0/542456270 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420003c00 con 0x55d675ca5fd0 2024-03-30T00:58:53.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.860+0000 7f6407400700 1 -- 172.21.15.116:0/542456270 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6420003d60 con 0x55d675ca5fd0 2024-03-30T00:58:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.860+0000 7f644024eb80 1 -- 172.21.15.116:0/542456270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d675bb6380 con 0x55d675ca5fd0 2024-03-30T00:58:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.860+0000 7f6407400700 1 -- 172.21.15.116:0/542456270 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f642000e030 con 0x55d675ca5fd0 2024-03-30T00:58:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.860+0000 7f644024eb80 10 client.15612 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.860+0000 7f644024eb80 1 -- 172.21.15.116:0/542456270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d675a23b60 con 0x55d675ca5fd0 2024-03-30T00:58:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.861+0000 7f6407400700 1 -- 172.21.15.116:0/542456270 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6420026020 con 0x55d675ca5fd0 2024-03-30T00:58:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.861+0000 7f6407400700 1 -- 172.21.15.116:0/542456270 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6420004250 con 0x55d675ca5fd0 2024-03-30T00:58:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.861+0000 7f644024eb80 10 client.15612 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.861+0000 7f644024eb80 10 client.15612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.861+0000 7f644024eb80 20 client.15612 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.861+0000 7f644024eb80 1 --2- 172.21.15.116:0/542456270 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d675cc2b80 0x55d675cc4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.861+0000 7f644024eb80 4 client.15612 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:58:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.861+0000 7f644024eb80 1 -- 172.21.15.116:0/542456270 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d675bee700 con 0x55d675cc2b80 2024-03-30T00:58:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.862+0000 7f6429e00700 1 --2- 172.21.15.116:0/542456270 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d675cc2b80 0x55d675cc4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.863+0000 7f6407400700 10 client.15612 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.863+0000 7f6429e00700 1 --2- 172.21.15.116:0/542456270 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d675cc2b80 0x55d675cc4fa0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7025f3ff56d601a3 server_cookie=de3309035860d9ec in_seq=0 out_seq=0 2024-03-30T00:58:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.864+0000 7f6407400700 1 -- 172.21.15.116:0/542456270 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d675bee700 con 0x55d675cc2b80 2024-03-30T00:58:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.864+0000 7f6407400700 10 client.15612 handle_command_reply: tid=0 2024-03-30T00:58:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.864+0000 7f644024eb80 1 client.15612 shutdown 2024-03-30T00:58:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.864+0000 7f644024eb80 1 -- 172.21.15.116:0/542456270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5fd0 msgr2=0x55d675cbe370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.864+0000 7f644024eb80 1 --2- 172.21.15.116:0/542456270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5fd0 0x55d675cbe370 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6420007fb0 tx=0x7f6420009350 comp rx=0 tx=0).stop 2024-03-30T00:58:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.864+0000 7f644024eb80 1 -- 172.21.15.116:0/542456270 shutdown_connections 2024-03-30T00:58:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.864+0000 7f644024eb80 1 --2- 172.21.15.116:0/542456270 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d675cc2b80 0x55d675cc4fa0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.864+0000 7f644024eb80 1 --2- 172.21.15.116:0/542456270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5fd0 0x55d675cbe370 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.864+0000 7f644024eb80 1 --2- 172.21.15.116:0/542456270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca47d0 0x55d675cb6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.864+0000 7f6407400700 0 client.15612 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.864+0000 7f644024eb80 1 --2- 172.21.15.116:0/542456270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca5130 0x55d675cb7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.864+0000 7f644024eb80 1 -- 172.21.15.116:0/542456270 >> 172.21.15.116:0/542456270 conn(0x55d675c036a0 msgr2=0x55d675ca3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.864+0000 7f644024eb80 1 -- 172.21.15.116:0/542456270 shutdown_connections 2024-03-30T00:58:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.864+0000 7f644024eb80 1 -- 172.21.15.116:0/542456270 wait complete. 2024-03-30T00:58:53.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.865+0000 7f644024eb80 20 client.15612 trim_cache size 0 max 16384 2024-03-30T00:58:53.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.882+0000 7f644024eb80 1 Processor -- start 2024-03-30T00:58:53.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.882+0000 7f644024eb80 1 -- start start 2024-03-30T00:58:53.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.883+0000 7f644024eb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca47d0 0x55d675ca6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.883+0000 7f644024eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca5130 0x55d675cbe330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.883+0000 7f644024eb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5fd0 0x55d675cb6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.883+0000 7f644024eb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d675cc0100 con 0x55d675ca5fd0 2024-03-30T00:58:53.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.883+0000 7f644024eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d675cb6960 con 0x55d675ca5130 2024-03-30T00:58:53.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.883+0000 7f644024eb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d675ca57e0 con 0x55d675ca47d0 2024-03-30T00:58:53.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.883+0000 7f642a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5fd0 0x55d675cb6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.883+0000 7f642a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5fd0 0x55d675cb6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56870/0 (socket says 172.21.15.116:56870) 2024-03-30T00:58:53.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.883+0000 7f642a800700 1 -- 172.21.15.116:0/1819341170 learned_addr learned my addr 172.21.15.116:0/1819341170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:53.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.884+0000 7f6429e00700 1 --2- 172.21.15.116:0/1819341170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca47d0 0x55d675ca6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.884+0000 7f6429400700 1 --2- 172.21.15.116:0/1819341170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca5130 0x55d675cbe330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.884+0000 7f642a800700 1 -- 172.21.15.116:0/1819341170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca47d0 msgr2=0x55d675ca6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:53.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.884+0000 7f642a800700 1 --2- 172.21.15.116:0/1819341170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca47d0 0x55d675ca6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.884+0000 7f642a800700 1 -- 172.21.15.116:0/1819341170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca5130 msgr2=0x55d675cbe330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:53.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.884+0000 7f642a800700 1 --2- 172.21.15.116:0/1819341170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca5130 0x55d675cbe330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.884+0000 7f642a800700 1 -- 172.21.15.116:0/1819341170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d675bbbea0 con 0x55d675ca5fd0 2024-03-30T00:58:53.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.885+0000 7f642a800700 1 --2- 172.21.15.116:0/1819341170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5fd0 0x55d675cb6300 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6420009010 tx=0x7f6420016430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364098f5bcf2d40f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:53.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.885+0000 7f6413a00700 1 -- 172.21.15.116:0/1819341170 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420013bf0 con 0x55d675ca5fd0 2024-03-30T00:58:53.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.885+0000 7f6413a00700 1 -- 172.21.15.116:0/1819341170 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6420026030 con 0x55d675ca5fd0 2024-03-30T00:58:53.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.885+0000 7f6413a00700 1 -- 172.21.15.116:0/1819341170 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420013bf0 con 0x55d675ca5fd0 2024-03-30T00:58:53.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.886+0000 7f644024eb80 1 -- 172.21.15.116:0/1819341170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5fd0 msgr2=0x55d675cb6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:53.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.886+0000 7f644024eb80 1 --2- 172.21.15.116:0/1819341170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5fd0 0x55d675cb6300 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6420009010 tx=0x7f6420016430 comp rx=0 tx=0).stop 2024-03-30T00:58:53.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.887+0000 7f644024eb80 1 -- 172.21.15.116:0/1819341170 shutdown_connections 2024-03-30T00:58:53.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.887+0000 7f644024eb80 1 --2- 172.21.15.116:0/1819341170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca47d0 0x55d675ca6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.887+0000 7f644024eb80 1 --2- 172.21.15.116:0/1819341170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5fd0 0x55d675cb6300 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.887+0000 7f644024eb80 1 --2- 172.21.15.116:0/1819341170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca5130 0x55d675cbe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.887+0000 7f644024eb80 1 -- 172.21.15.116:0/1819341170 >> 172.21.15.116:0/1819341170 conn(0x55d675c036a0 msgr2=0x55d675c985a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:53.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.887+0000 7f644024eb80 1 -- 172.21.15.116:0/1819341170 shutdown_connections 2024-03-30T00:58:53.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.887+0000 7f644024eb80 1 -- 172.21.15.116:0/1819341170 wait complete. 2024-03-30T00:58:53.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.887+0000 7f644024eb80 1 -- start start 2024-03-30T00:58:53.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.888+0000 7f644024eb80 1 Processor -- start 2024-03-30T00:58:53.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.888+0000 7f644024eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca47d0 0x55d675cbdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.888+0000 7f644024eb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5130 0x55d675cbc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.888+0000 7f644024eb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca5fd0 0x55d675cbcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.888+0000 7f644024eb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d675cbe6d0 con 0x55d675ca5130 2024-03-30T00:58:53.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.888+0000 7f644024eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d675cbd030 con 0x55d675ca47d0 2024-03-30T00:58:53.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.888+0000 7f644024eb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d675cbd1c0 con 0x55d675ca5fd0 2024-03-30T00:58:53.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.889+0000 7f644024eb80 10 client.0 _ll_register_callbacks cb 0x55d675b9b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:53.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.889+0000 7f6429400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5130 0x55d675cbc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.889+0000 7f6429400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5130 0x55d675cbc5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56886/0 (socket says 172.21.15.116:56886) 2024-03-30T00:58:53.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.889+0000 7f642a800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca5fd0 0x55d675cbcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.889+0000 7f6429400700 1 -- 172.21.15.116:0/1398659254 learned_addr learned my addr 172.21.15.116:0/1398659254 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:53.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.890+0000 7f6429e00700 1 --2- 172.21.15.116:0/1398659254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca47d0 0x55d675cbdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.890+0000 7f6429400700 1 -- 172.21.15.116:0/1398659254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca5fd0 msgr2=0x55d675cbcaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:53.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.890+0000 7f6429400700 1 --2- 172.21.15.116:0/1398659254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca5fd0 0x55d675cbcaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.890+0000 7f6429400700 1 -- 172.21.15.116:0/1398659254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca47d0 msgr2=0x55d675cbdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:53.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.890+0000 7f6429400700 1 --2- 172.21.15.116:0/1398659254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca47d0 0x55d675cbdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.890+0000 7f6429400700 1 -- 172.21.15.116:0/1398659254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d675a23b60 con 0x55d675ca5130 2024-03-30T00:58:53.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.891+0000 7f6429400700 1 --2- 172.21.15.116:0/1398659254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5130 0x55d675cbc5b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f641400c5b0 tx=0x7f641409a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37440866c9f7dec3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:53.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.891+0000 7f6407400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:58:53.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.891+0000 7f6407400700 1 -- 172.21.15.116:0/1398659254 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414099d80 con 0x55d675ca5130 2024-03-30T00:58:53.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.891+0000 7f6407400700 1 -- 172.21.15.116:0/1398659254 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f641409dbb0 con 0x55d675ca5130 2024-03-30T00:58:53.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.891+0000 7f644024eb80 1 -- 172.21.15.116:0/1398659254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f641801d040 con 0x55d675ca5130 2024-03-30T00:58:53.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.891+0000 7f6407400700 1 -- 172.21.15.116:0/1398659254 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64140b8170 con 0x55d675ca5130 2024-03-30T00:58:53.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.891+0000 7f644024eb80 10 client.15618 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:53.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.891+0000 7f644024eb80 1 -- 172.21.15.116:0/1398659254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d675a7bcd0 con 0x55d675ca5130 2024-03-30T00:58:53.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.892+0000 7f6407400700 1 -- 172.21.15.116:0/1398659254 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f641407c080 con 0x55d675ca5130 2024-03-30T00:58:53.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.892+0000 7f6407400700 1 -- 172.21.15.116:0/1398659254 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6414095b70 con 0x55d675ca5130 2024-03-30T00:58:53.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.892+0000 7f644024eb80 10 client.15618 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:53.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.892+0000 7f644024eb80 10 client.15618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:53.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.892+0000 7f644024eb80 20 client.15618 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:53.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.892+0000 7f644024eb80 1 --2- 172.21.15.116:0/1398659254 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d675d18000 0x55d675d1a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:53.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.892+0000 7f644024eb80 4 client.15618 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T00:58:53.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.892+0000 7f644024eb80 1 -- 172.21.15.116:0/1398659254 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d675a44cf0 con 0x55d675d18000 2024-03-30T00:58:53.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.892+0000 7f6429e00700 1 --2- 172.21.15.116:0/1398659254 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d675d18000 0x55d675d1a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:53.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.893+0000 7f6429e00700 1 --2- 172.21.15.116:0/1398659254 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d675d18000 0x55d675d1a420 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=780aa4642534b5ee server_cookie=f04cfdf8dc3b9d9a in_seq=0 out_seq=0 2024-03-30T00:58:53.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.893+0000 7f6407400700 10 client.15618 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:53.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.893+0000 7f6407400700 1 -- 172.21.15.116:0/1398659254 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d675a44cf0 con 0x55d675d18000 2024-03-30T00:58:53.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.893+0000 7f6407400700 10 client.15618 handle_command_reply: tid=0 2024-03-30T00:58:53.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.893+0000 7f644024eb80 1 client.15618 shutdown 2024-03-30T00:58:53.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.894+0000 7f644024eb80 1 -- 172.21.15.116:0/1398659254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5130 msgr2=0x55d675cbc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:53.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.894+0000 7f644024eb80 1 --2- 172.21.15.116:0/1398659254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5130 0x55d675cbc5b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f641400c5b0 tx=0x7f641409a2c0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.894+0000 7f644024eb80 1 -- 172.21.15.116:0/1398659254 shutdown_connections 2024-03-30T00:58:53.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.894+0000 7f644024eb80 1 --2- 172.21.15.116:0/1398659254 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d675d18000 0x55d675d1a420 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.894+0000 7f644024eb80 1 --2- 172.21.15.116:0/1398659254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d675ca5fd0 0x55d675cbcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.894+0000 7f644024eb80 1 --2- 172.21.15.116:0/1398659254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d675ca47d0 0x55d675cbdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.894+0000 7f644024eb80 1 --2- 172.21.15.116:0/1398659254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d675ca5130 0x55d675cbc5b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.894+0000 7f6407400700 0 client.15618 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:53.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.894+0000 7f644024eb80 1 -- 172.21.15.116:0/1398659254 >> 172.21.15.116:0/1398659254 conn(0x55d675c036a0 msgr2=0x55d675cac4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:53.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.894+0000 7f644024eb80 1 -- 172.21.15.116:0/1398659254 shutdown_connections 2024-03-30T00:58:53.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.894+0000 7f644024eb80 1 -- 172.21.15.116:0/1398659254 wait complete. 2024-03-30T00:58:53.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.894+0000 7f644024eb80 20 client.15618 trim_cache size 0 max 16384 2024-03-30T00:58:53.914 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:58:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-30T00:58:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-30T00:58:53.914 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:58:53.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.895+0000 7f642c800700 1 -- 172.21.15.116:0/4098954861 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f640c0a10c0 msgr2=0x7f640c0a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:53.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.895+0000 7f642c800700 1 --2- 172.21.15.116:0/4098954861 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f640c0a10c0 0x7f640c0a3580 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6418005a80 tx=0x7f6418009c30 comp rx=0 tx=0).stop 2024-03-30T00:58:53.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.895+0000 7f642c800700 1 -- 172.21.15.116:0/4098954861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6424104180 msgr2=0x7f6424199550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:53.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.895+0000 7f642c800700 1 --2- 172.21.15.116:0/4098954861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6424104180 0x7f6424199550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f641400e9c0 tx=0x7f641400ecd0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.895+0000 7f642c800700 1 -- 172.21.15.116:0/4098954861 shutdown_connections 2024-03-30T00:58:53.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.895+0000 7f642c800700 1 --2- 172.21.15.116:0/4098954861 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f640c0a10c0 0x7f640c0a3580 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.895+0000 7f642c800700 1 --2- 172.21.15.116:0/4098954861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6424104b20 0x7f64241a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.895+0000 7f642c800700 1 --2- 172.21.15.116:0/4098954861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6424102f90 0x7f6424199010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.895+0000 7f642c800700 1 --2- 172.21.15.116:0/4098954861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6424104180 0x7f6424199550 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:53.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.895+0000 7f642c800700 1 -- 172.21.15.116:0/4098954861 >> 172.21.15.116:0/4098954861 conn(0x7f64240fe500 msgr2=0x7f6424107690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:53.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.896+0000 7f642c800700 1 -- 172.21.15.116:0/4098954861 shutdown_connections 2024-03-30T00:58:53.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:53.896+0000 7f642c800700 1 -- 172.21.15.116:0/4098954861 wait complete. 2024-03-30T00:58:53.930 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T00:58:53.931 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:58:54.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.141+0000 7f5ef3800700 1 -- 172.21.15.116:0/1141326158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5eec100c80 msgr2=0x7f5eec1010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.141+0000 7f5ef3800700 1 --2- 172.21.15.116:0/1141326158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5eec100c80 0x7f5eec1010a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0009990 tx=0x7f5ee0009ca0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.141+0000 7f5ef3800700 1 -- 172.21.15.116:0/1141326158 shutdown_connections 2024-03-30T00:58:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.141+0000 7f5ef3800700 1 --2- 172.21.15.116:0/1141326158 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5eec100c80 0x7f5eec1010a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.141+0000 7f5ef3800700 1 --2- 172.21.15.116:0/1141326158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eec102810 0x7f5eec1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.141+0000 7f5ef3800700 1 --2- 172.21.15.116:0/1141326158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5eec101e70 0x7f5eec1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.141+0000 7f5ef3800700 1 -- 172.21.15.116:0/1141326158 >> 172.21.15.116:0/1141326158 conn(0x7f5eec0fc210 msgr2=0x7f5eec0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.142+0000 7f5ef3800700 1 -- 172.21.15.116:0/1141326158 shutdown_connections 2024-03-30T00:58:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.142+0000 7f5ef3800700 1 -- 172.21.15.116:0/1141326158 wait complete. 2024-03-30T00:58:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.143+0000 7f5ef3800700 1 Processor -- start 2024-03-30T00:58:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.143+0000 7f5ef3800700 1 -- start start 2024-03-30T00:58:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.143+0000 7f5ef3800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5eec100c80 0x7f5eec198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.143+0000 7f5ef3800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eec101e70 0x7f5eec199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.143+0000 7f5ef3800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5eec102810 0x7f5eec1a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.143+0000 7f5ef3800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec1a08f0 con 0x7f5eec101e70 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.143+0000 7f5ef3800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec1a0a30 con 0x7f5eec100c80 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.143+0000 7f5ef3800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec1a0bc0 con 0x7f5eec102810 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.144+0000 7f5ef0e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5eec100c80 0x7f5eec198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.144+0000 7f5ef0e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5eec100c80 0x7f5eec198dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:46040/0 (socket says 172.21.15.116:46040) 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.144+0000 7f5ef0e00700 1 -- 172.21.15.116:0/1790998691 learned_addr learned my addr 172.21.15.116:0/1790998691 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.144+0000 7f5ef1800700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5eec102810 0x7f5eec1a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.144+0000 7f5ef0e00700 1 -- 172.21.15.116:0/1790998691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5eec102810 msgr2=0x7f5eec1a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.144+0000 7f5ef0e00700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5eec102810 0x7f5eec1a0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.144+0000 7f5ef0e00700 1 -- 172.21.15.116:0/1790998691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eec101e70 msgr2=0x7f5eec199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.144+0000 7f5eebe00700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eec101e70 0x7f5eec199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.144+0000 7f5ef0e00700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eec101e70 0x7f5eec199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.144+0000 7f5ef0e00700 1 -- 172.21.15.116:0/1790998691 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee00095f0 con 0x7f5eec100c80 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.144+0000 7f5eebe00700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eec101e70 0x7f5eec199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.145+0000 7f5ef0e00700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5eec100c80 0x7f5eec198dc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00095c0 tx=0x7f5ee0009f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.145+0000 7f5ee9600700 1 -- 172.21.15.116:0/1790998691 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0020670 con 0x7f5eec100c80 2024-03-30T00:58:54.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.145+0000 7f5ee9600700 1 -- 172.21.15.116:0/1790998691 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ee0005580 con 0x7f5eec100c80 2024-03-30T00:58:54.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.145+0000 7f5ef3800700 1 -- 172.21.15.116:0/1790998691 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eec103090 con 0x7f5eec100c80 2024-03-30T00:58:54.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.145+0000 7f5ee9600700 1 -- 172.21.15.116:0/1790998691 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee000fa90 con 0x7f5eec100c80 2024-03-30T00:58:54.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.145+0000 7f5ef3800700 1 -- 172.21.15.116:0/1790998691 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eec103590 con 0x7f5eec100c80 2024-03-30T00:58:54.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.146+0000 7f5ef3800700 1 -- 172.21.15.116:0/1790998691 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eec067180 con 0x7f5eec100c80 2024-03-30T00:58:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.150+0000 7f5ee9600700 1 -- 172.21.15.116:0/1790998691 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f5ee001d070 con 0x7f5eec100c80 2024-03-30T00:58:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.152+0000 7f5ee9600700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ed40a11a0 0x7f5ed40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.152+0000 7f5ee9600700 1 -- 172.21.15.116:0/1790998691 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5ee00df8a0 con 0x7f5eec100c80 2024-03-30T00:58:54.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.153+0000 7f5eebe00700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ed40a11a0 0x7f5ed40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.153+0000 7f5ee9600700 1 -- 172.21.15.116:0/1790998691 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5ee00ac3d0 con 0x7f5eec100c80 2024-03-30T00:58:54.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.153+0000 7f5eebe00700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ed40a11a0 0x7f5ed40a3660 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8005fd0 tx=0x7f5ed8005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:54.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.418+0000 7f5ef3800700 1 -- 172.21.15.116:0/1790998691 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5eec10c4e0 con 0x7f5eec100c80 2024-03-30T00:58:54.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.419+0000 7f5ee9600700 1 -- 172.21.15.116:0/1790998691 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10886 (secure 0 0 0) 0x7f5ee00ac5b0 con 0x7f5eec100c80 2024-03-30T00:58:54.425 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:58:54.425 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:57:40.442321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T00:58:54.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.421+0000 7f5ef3800700 1 -- 172.21.15.116:0/1790998691 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ed40a11a0 msgr2=0x7f5ed40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.421+0000 7f5ef3800700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ed40a11a0 0x7f5ed40a3660 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8005fd0 tx=0x7f5ed8005e20 comp rx=0 tx=0).stop 2024-03-30T00:58:54.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.421+0000 7f5ef3800700 1 -- 172.21.15.116:0/1790998691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5eec100c80 msgr2=0x7f5eec198dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.421+0000 7f5ef3800700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5eec100c80 0x7f5eec198dc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00095c0 tx=0x7f5ee0009f50 comp rx=0 tx=0).stop 2024-03-30T00:58:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.422+0000 7f5ef3800700 1 -- 172.21.15.116:0/1790998691 shutdown_connections 2024-03-30T00:58:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.422+0000 7f5ef3800700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5ed40a11a0 0x7f5ed40a3660 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.422+0000 7f5ef3800700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5eec102810 0x7f5eec1a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.422+0000 7f5ef3800700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5eec100c80 0x7f5eec198dc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.422+0000 7f5ef3800700 1 --2- 172.21.15.116:0/1790998691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eec101e70 0x7f5eec199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.422+0000 7f5ef3800700 1 -- 172.21.15.116:0/1790998691 >> 172.21.15.116:0/1790998691 conn(0x7f5eec0fc210 msgr2=0x7f5eec072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.422+0000 7f5ef3800700 1 -- 172.21.15.116:0/1790998691 shutdown_connections 2024-03-30T00:58:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.422+0000 7f5ef3800700 1 -- 172.21.15.116:0/1790998691 wait complete. 2024-03-30T00:58:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 23 2024-03-30T00:58:54.447 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T00:58:54.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.645+0000 7f30cc200700 1 -- 172.21.15.116:0/562570201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30c4074df0 msgr2=0x7f30c4073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.645+0000 7f30cc200700 1 --2- 172.21.15.116:0/562570201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30c4074df0 0x7f30c4073240 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f30b80098f0 tx=0x7f30b8009c00 comp rx=0 tx=0).stop 2024-03-30T00:58:54.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.645+0000 7f30cc200700 1 -- 172.21.15.116:0/562570201 shutdown_connections 2024-03-30T00:58:54.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.645+0000 7f30cc200700 1 --2- 172.21.15.116:0/562570201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30c4108c40 0x7f30c410b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.645+0000 7f30cc200700 1 --2- 172.21.15.116:0/562570201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30c4074df0 0x7f30c4073240 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.645+0000 7f30cc200700 1 --2- 172.21.15.116:0/562570201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30c4073810 0x7f30c4073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.645+0000 7f30cc200700 1 -- 172.21.15.116:0/562570201 >> 172.21.15.116:0/562570201 conn(0x7f30c40fc2f0 msgr2=0x7f30c40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:54.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.645+0000 7f30cc200700 1 -- 172.21.15.116:0/562570201 shutdown_connections 2024-03-30T00:58:54.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.645+0000 7f30cc200700 1 -- 172.21.15.116:0/562570201 wait complete. 2024-03-30T00:58:54.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.647+0000 7f30cc200700 1 Processor -- start 2024-03-30T00:58:54.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.647+0000 7f30cc200700 1 -- start start 2024-03-30T00:58:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.647+0000 7f30cc200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30c4073810 0x7f30c4198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.647+0000 7f30cc200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30c4074df0 0x7f30c41994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.647+0000 7f30cc200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30c4108c40 0x7f30c41a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.647+0000 7f30cc200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c41a0aa0 con 0x7f30c4074df0 2024-03-30T00:58:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.647+0000 7f30cc200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c41a0be0 con 0x7f30c4073810 2024-03-30T00:58:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.647+0000 7f30cc200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c41a0d70 con 0x7f30c4108c40 2024-03-30T00:58:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.648+0000 7f30c9800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30c4073810 0x7f30c4198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.648+0000 7f30c9800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30c4073810 0x7f30c4198f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:46050/0 (socket says 172.21.15.116:46050) 2024-03-30T00:58:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.648+0000 7f30c9800700 1 -- 172.21.15.116:0/1015156236 learned_addr learned my addr 172.21.15.116:0/1015156236 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.648+0000 7f30ca200700 1 --2- 172.21.15.116:0/1015156236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30c4108c40 0x7f30c41a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.648+0000 7f30c8e00700 1 --2- 172.21.15.116:0/1015156236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30c4074df0 0x7f30c41994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.648+0000 7f30c9800700 1 -- 172.21.15.116:0/1015156236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30c4108c40 msgr2=0x7f30c41a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.648+0000 7f30c9800700 1 --2- 172.21.15.116:0/1015156236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30c4108c40 0x7f30c41a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.648+0000 7f30c9800700 1 -- 172.21.15.116:0/1015156236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30c4074df0 msgr2=0x7f30c41994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.648+0000 7f30c9800700 1 --2- 172.21.15.116:0/1015156236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30c4074df0 0x7f30c41994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.648+0000 7f30c9800700 1 -- 172.21.15.116:0/1015156236 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b0009520 con 0x7f30c4073810 2024-03-30T00:58:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.649+0000 7f30c9800700 1 --2- 172.21.15.116:0/1015156236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30c4073810 0x7f30c4198f70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009810 tx=0x7f30b800b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.649+0000 7f30c2000700 1 -- 172.21.15.116:0/1015156236 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b801c070 con 0x7f30c4073810 2024-03-30T00:58:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.649+0000 7f30c2000700 1 -- 172.21.15.116:0/1015156236 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30b800f7d0 con 0x7f30c4073810 2024-03-30T00:58:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.649+0000 7f30c2000700 1 -- 172.21.15.116:0/1015156236 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8003a60 con 0x7f30c4073810 2024-03-30T00:58:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.649+0000 7f30cc200700 1 -- 172.21.15.116:0/1015156236 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b8009520 con 0x7f30c4073810 2024-03-30T00:58:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.649+0000 7f30cc200700 1 -- 172.21.15.116:0/1015156236 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c41a11d0 con 0x7f30c4073810 2024-03-30T00:58:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.654+0000 7f30dfcf0b80 1 Processor -- start 2024-03-30T00:58:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.654+0000 7f30dfcf0b80 1 -- start start 2024-03-30T00:58:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.654+0000 7f30c2000700 1 -- 172.21.15.116:0/1015156236 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f30b8020030 con 0x7f30c4073810 2024-03-30T00:58:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.654+0000 7f30dfcf0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecc7d0 0x55d721eccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.654+0000 7f30dfcf0b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecd130 0x55d721ecd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.654+0000 7f30dfcf0b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecdfd0 0x55d721ece450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.654+0000 7f30dfcf0b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721e24990 con 0x55d721ecdfd0 2024-03-30T00:58:54.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.654+0000 7f30dfcf0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721e22350 con 0x55d721ecc7d0 2024-03-30T00:58:54.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.654+0000 7f30dfcf0b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721e1dc40 con 0x55d721ecd130 2024-03-30T00:58:54.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.655+0000 7f30ca200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecc7d0 0x55d721eccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.655+0000 7f30ca200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecc7d0 0x55d721eccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:46064/0 (socket says 172.21.15.116:46064) 2024-03-30T00:58:54.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.655+0000 7f30ca200700 1 -- 172.21.15.116:0/434454222 learned_addr learned my addr 172.21.15.116:0/434454222 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:54.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.655+0000 7f30ca200700 1 -- 172.21.15.116:0/434454222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecd130 msgr2=0x55d721ecd9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T00:58:54.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.655+0000 7f30ca200700 1 --2- 172.21.15.116:0/434454222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecd130 0x55d721ecd9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.655+0000 7f30ca200700 1 -- 172.21.15.116:0/434454222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecdfd0 msgr2=0x55d721ece450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T00:58:54.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.655+0000 7f30ca200700 1 --2- 172.21.15.116:0/434454222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecdfd0 0x55d721ece450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.655+0000 7f30ca200700 1 -- 172.21.15.116:0/434454222 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d721c9f3b0 con 0x55d721ecc7d0 2024-03-30T00:58:54.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.655+0000 7f30ca200700 1 --2- 172.21.15.116:0/434454222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecc7d0 0x55d721eccbf0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009410 tx=0x7f30bc0099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d284d582d3ab294c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:54.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.655+0000 7f30b6a00700 1 -- 172.21.15.116:0/434454222 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc00e070 con 0x55d721ecc7d0 2024-03-30T00:58:54.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.655+0000 7f30b6a00700 1 -- 172.21.15.116:0/434454222 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30bc00f040 con 0x55d721ecc7d0 2024-03-30T00:58:54.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.655+0000 7f30b6a00700 1 -- 172.21.15.116:0/434454222 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc015b00 con 0x55d721ecc7d0 2024-03-30T00:58:54.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.656+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/434454222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecc7d0 msgr2=0x55d721eccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.656+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/434454222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecc7d0 0x55d721eccbf0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009410 tx=0x7f30bc0099e0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.656+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/434454222 shutdown_connections 2024-03-30T00:58:54.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.656+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/434454222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecdfd0 0x55d721ece450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.656+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/434454222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecc7d0 0x55d721eccbf0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.656+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/434454222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecd130 0x55d721ecd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.656+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/434454222 >> 172.21.15.116:0/434454222 conn(0x55d721e2b6a0 msgr2=0x55d721ec79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:54.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.657+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/434454222 shutdown_connections 2024-03-30T00:58:54.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.657+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/434454222 wait complete. 2024-03-30T00:58:54.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.657+0000 7f30c2000700 1 --2- 172.21.15.116:0/1015156236 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f30ac0a10b0 0x7f30ac0a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.657+0000 7f30c2000700 1 -- 172.21.15.116:0/1015156236 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f30b80cdb80 con 0x7f30c4073810 2024-03-30T00:58:54.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.657+0000 7f30dfcf0b80 1 -- start start 2024-03-30T00:58:54.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.657+0000 7f30dfcf0b80 1 Processor -- start 2024-03-30T00:58:54.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.658+0000 7f30dfcf0b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecc7d0 0x55d721ee76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.658+0000 7f30dfcf0b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecd130 0x55d721ee7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.658+0000 7f30c8e00700 1 --2- 172.21.15.116:0/1015156236 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f30ac0a10b0 0x7f30ac0a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.658+0000 7f30dfcf0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecdfd0 0x55d721eeec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.658+0000 7f30dfcf0b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721db9d80 con 0x55d721ecd130 2024-03-30T00:58:54.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.658+0000 7f30dfcf0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721db9a20 con 0x55d721ecdfd0 2024-03-30T00:58:54.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.658+0000 7f30dfcf0b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721ca16a0 con 0x55d721ecc7d0 2024-03-30T00:58:54.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.658+0000 7f30dfcf0b80 10 client.0 _ll_register_callbacks cb 0x55d721dc3470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:54.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.658+0000 7f30c9800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecc7d0 0x55d721ee76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.658+0000 7f30c9800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecc7d0 0x55d721ee76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:55074/0 (socket says 172.21.15.116:55074) 2024-03-30T00:58:54.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.658+0000 7f30c9800700 1 -- 172.21.15.116:0/3374682079 learned_addr learned my addr 172.21.15.116:0/3374682079 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:54.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.658+0000 7f30ca200700 1 --2- 172.21.15.116:0/3374682079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecdfd0 0x55d721eeec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.659+0000 7f30c9800700 1 -- 172.21.15.116:0/3374682079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecdfd0 msgr2=0x55d721eeec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:54.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.659+0000 7f30c9800700 1 --2- 172.21.15.116:0/3374682079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecdfd0 0x55d721eeec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.659+0000 7f30c9800700 1 -- 172.21.15.116:0/3374682079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecd130 msgr2=0x55d721ee7c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T00:58:54.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.659+0000 7f30c9800700 1 --2- 172.21.15.116:0/3374682079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecd130 0x55d721ee7c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.659+0000 7f30c9800700 1 -- 172.21.15.116:0/3374682079 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30bc0090f0 con 0x55d721ecc7d0 2024-03-30T00:58:54.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.659+0000 7f30c8e00700 1 --2- 172.21.15.116:0/1015156236 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f30ac0a10b0 0x7f30ac0a3570 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x55d721ecd7b0 tx=0x7f30b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:54.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.659+0000 7f30c9800700 1 --2- 172.21.15.116:0/3374682079 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecc7d0 0x55d721ee76e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f30b809aa50 tx=0x7f30b809a910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b65142f048c13339 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:54.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.659+0000 7f30ab400700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T00:58:54.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.660+0000 7f30ab400700 1 -- 172.21.15.116:0/3374682079 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b809ad50 con 0x55d721ecc7d0 2024-03-30T00:58:54.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.660+0000 7f30ab400700 1 -- 172.21.15.116:0/3374682079 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f30b809eb80 con 0x55d721ecc7d0 2024-03-30T00:58:54.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.660+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/3374682079 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d721dde380 con 0x55d721ecc7d0 2024-03-30T00:58:54.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.660+0000 7f30ab400700 1 -- 172.21.15.116:0/3374682079 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30b80a3140 con 0x55d721ecc7d0 2024-03-30T00:58:54.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.660+0000 7f30dfcf0b80 10 client.25669 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:54.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.660+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/3374682079 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d721c4bb60 con 0x55d721ecc7d0 2024-03-30T00:58:54.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.660+0000 7f30ab400700 1 -- 172.21.15.116:0/3374682079 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30b80b7080 con 0x55d721ecc7d0 2024-03-30T00:58:54.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.661+0000 7f30ab400700 1 -- 172.21.15.116:0/3374682079 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f30b8015020 con 0x55d721ecc7d0 2024-03-30T00:58:54.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.661+0000 7f30dfcf0b80 10 client.25669 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:54.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.661+0000 7f30dfcf0b80 10 client.25669 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T00:58:54.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.661+0000 7f30dfcf0b80 20 client.25669 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:54.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.661+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/3374682079 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55d721ef1840 0x55d721ef3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.661+0000 7f30dfcf0b80 4 client.25669 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:58:54.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.661+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/3374682079 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d721e16700 con 0x55d721ef1840 2024-03-30T00:58:54.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.661+0000 7f30c8e00700 1 --2- 172.21.15.116:0/3374682079 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55d721ef1840 0x55d721ef3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.662+0000 7f30c8e00700 1 --2- 172.21.15.116:0/3374682079 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55d721ef1840 0x55d721ef3c60 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dad31f474a0c121e server_cookie=d2bbfd8a1d18508a in_seq=0 out_seq=0 2024-03-30T00:58:54.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.662+0000 7f30ab400700 10 client.25669 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T00:58:54.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30ab400700 1 -- 172.21.15.116:0/3374682079 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d721e16700 con 0x55d721ef1840 2024-03-30T00:58:54.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30ab400700 10 client.25669 handle_command_reply: tid=0 2024-03-30T00:58:54.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30dfcf0b80 1 client.25669 shutdown 2024-03-30T00:58:54.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/3374682079 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecc7d0 msgr2=0x55d721ee76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/3374682079 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecc7d0 0x55d721ee76e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f30b809aa50 tx=0x7f30b809a910 comp rx=0 tx=0).stop 2024-03-30T00:58:54.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/3374682079 shutdown_connections 2024-03-30T00:58:54.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/3374682079 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55d721ef1840 0x55d721ef3c60 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/3374682079 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecc7d0 0x55d721ee76e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/3374682079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecdfd0 0x55d721eeec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30ab400700 0 client.25669 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T00:58:54.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/3374682079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecd130 0x55d721ee7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/3374682079 >> 172.21.15.116:0/3374682079 conn(0x55d721e2b6a0 msgr2=0x55d721ec9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:54.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/3374682079 shutdown_connections 2024-03-30T00:58:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/3374682079 wait complete. 2024-03-30T00:58:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.663+0000 7f30dfcf0b80 20 client.25669 trim_cache size 0 max 16384 2024-03-30T00:58:54.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.677+0000 7f30dfcf0b80 1 Processor -- start 2024-03-30T00:58:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.677+0000 7f30dfcf0b80 1 -- start start 2024-03-30T00:58:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.677+0000 7f30dfcf0b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecc7d0 0x55d721ef7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.677+0000 7f30dfcf0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecd130 0x55d721ece3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.677+0000 7f30dfcf0b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecdfd0 0x55d721ec0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.677+0000 7f30dfcf0b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721ef85a0 con 0x55d721ecc7d0 2024-03-30T00:58:54.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.677+0000 7f30dfcf0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721eccf70 con 0x55d721ecd130 2024-03-30T00:58:54.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.677+0000 7f30dfcf0b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721ec88b0 con 0x55d721ecdfd0 2024-03-30T00:58:54.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.678+0000 7f30ca200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecdfd0 0x55d721ec0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.678+0000 7f30ca200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecdfd0 0x55d721ec0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:55080/0 (socket says 172.21.15.116:55080) 2024-03-30T00:58:54.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.678+0000 7f30ca200700 1 -- 172.21.15.116:0/2260326086 learned_addr learned my addr 172.21.15.116:0/2260326086 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:54.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.678+0000 7f30c9800700 1 --2- 172.21.15.116:0/2260326086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecc7d0 0x55d721ef7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.678+0000 7f30c8e00700 1 --2- 172.21.15.116:0/2260326086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecd130 0x55d721ece3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.678+0000 7f30ca200700 1 -- 172.21.15.116:0/2260326086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecd130 msgr2=0x55d721ece3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.678+0000 7f30ca200700 1 --2- 172.21.15.116:0/2260326086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecd130 0x55d721ece3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.678+0000 7f30ca200700 1 -- 172.21.15.116:0/2260326086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecc7d0 msgr2=0x55d721ef7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.678+0000 7f30ca200700 1 --2- 172.21.15.116:0/2260326086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecc7d0 0x55d721ef7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.678+0000 7f30ca200700 1 -- 172.21.15.116:0/2260326086 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d721de3ea0 con 0x55d721ecdfd0 2024-03-30T00:58:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.678+0000 7f30c8e00700 1 --2- 172.21.15.116:0/2260326086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecd130 0x55d721ece3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:58:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.679+0000 7f30ca200700 1 --2- 172.21.15.116:0/2260326086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecdfd0 0x55d721ec0f70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f30bc018000 tx=0x7f30bc004a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b110ff5ae1af388 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.679+0000 7f30b7400700 1 -- 172.21.15.116:0/2260326086 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc00e040 con 0x55d721ecdfd0 2024-03-30T00:58:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.679+0000 7f30b7400700 1 -- 172.21.15.116:0/2260326086 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30bc004e80 con 0x55d721ecdfd0 2024-03-30T00:58:54.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.679+0000 7f30b7400700 1 -- 172.21.15.116:0/2260326086 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc013ab0 con 0x55d721ecdfd0 2024-03-30T00:58:54.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.680+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/2260326086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecdfd0 msgr2=0x55d721ec0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.680+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/2260326086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecdfd0 0x55d721ec0f70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f30bc018000 tx=0x7f30bc004a90 comp rx=0 tx=0).stop 2024-03-30T00:58:54.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.680+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/2260326086 shutdown_connections 2024-03-30T00:58:54.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.680+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/2260326086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecdfd0 0x55d721ec0f70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.680+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/2260326086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecc7d0 0x55d721ef7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.680+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/2260326086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecd130 0x55d721ece3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.680+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/2260326086 >> 172.21.15.116:0/2260326086 conn(0x55d721e2b6a0 msgr2=0x55d721ed2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:54.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.680+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/2260326086 shutdown_connections 2024-03-30T00:58:54.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.680+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/2260326086 wait complete. 2024-03-30T00:58:54.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.681+0000 7f30dfcf0b80 1 -- start start 2024-03-30T00:58:54.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.681+0000 7f30dfcf0b80 1 Processor -- start 2024-03-30T00:58:54.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.682+0000 7f30dfcf0b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecc7d0 0x55d721efc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.682+0000 7f30dfcf0b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecd130 0x55d721efcb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.682+0000 7f30dfcf0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecdfd0 0x55d721f463a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.682+0000 7f30dfcf0b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721f468e0 con 0x55d721ecc7d0 2024-03-30T00:58:54.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.682+0000 7f30dfcf0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721f46a50 con 0x55d721ecdfd0 2024-03-30T00:58:54.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.682+0000 7f30dfcf0b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721f46c10 con 0x55d721ecd130 2024-03-30T00:58:54.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.682+0000 7f30dfcf0b80 10 client.0 _ll_register_callbacks cb 0x55d721dc3470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:54.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.682+0000 7f30c9800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecc7d0 0x55d721efc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.682+0000 7f30c9800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecc7d0 0x55d721efc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56952/0 (socket says 172.21.15.116:56952) 2024-03-30T00:58:54.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.682+0000 7f30c9800700 1 -- 172.21.15.116:0/4175215985 learned_addr learned my addr 172.21.15.116:0/4175215985 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:54.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.682+0000 7f30c8e00700 1 --2- 172.21.15.116:0/4175215985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecd130 0x55d721efcb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.682+0000 7f30ca200700 1 --2- 172.21.15.116:0/4175215985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecdfd0 0x55d721f463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.683+0000 7f30c9800700 1 -- 172.21.15.116:0/4175215985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecd130 msgr2=0x55d721efcb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:54.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.683+0000 7f30c9800700 1 --2- 172.21.15.116:0/4175215985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecd130 0x55d721efcb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.683+0000 7f30c9800700 1 -- 172.21.15.116:0/4175215985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecdfd0 msgr2=0x55d721f463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:54.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.683+0000 7f30c9800700 1 --2- 172.21.15.116:0/4175215985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecdfd0 0x55d721f463a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.683+0000 7f30c9800700 1 -- 172.21.15.116:0/4175215985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d721c9f3b0 con 0x55d721ecc7d0 2024-03-30T00:58:54.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.683+0000 7f30c9800700 1 --2- 172.21.15.116:0/4175215985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecc7d0 0x55d721efc5e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f30b8097250 tx=0x7f30b8003bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7de4106618afd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:54.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.683+0000 7f30ab400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:58:54.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.683+0000 7f30ab400700 1 -- 172.21.15.116:0/4175215985 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b809eb80 con 0x55d721ecc7d0 2024-03-30T00:58:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.683+0000 7f30ab400700 1 -- 172.21.15.116:0/4175215985 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f30b8021460 con 0x55d721ecc7d0 2024-03-30T00:58:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.684+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/4175215985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d721c6ccf0 con 0x55d721ecc7d0 2024-03-30T00:58:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.684+0000 7f30ab400700 1 -- 172.21.15.116:0/4175215985 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30b809ad50 con 0x55d721ecc7d0 2024-03-30T00:58:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.684+0000 7f30dfcf0b80 10 client.15633 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.684+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/4175215985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d721ca3cd0 con 0x55d721ecc7d0 2024-03-30T00:58:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.684+0000 7f30ab400700 1 -- 172.21.15.116:0/4175215985 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d721c6ccf0 con 0x55d721ecc7d0 2024-03-30T00:58:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.684+0000 7f30ab400700 1 -- 172.21.15.116:0/4175215985 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f30b8015070 con 0x55d721ecc7d0 2024-03-30T00:58:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.685+0000 7f30dfcf0b80 10 client.15633 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.685+0000 7f30dfcf0b80 10 client.15633 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T00:58:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.685+0000 7f30dfcf0b80 20 client.15633 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.685+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/4175215985 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55d721f47ec0 0x55d721f4a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.685+0000 7f30dfcf0b80 4 client.15633 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T00:58:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.685+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/4175215985 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d721ca1ce0 con 0x55d721f47ec0 2024-03-30T00:58:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.685+0000 7f30c8e00700 1 --2- 172.21.15.116:0/4175215985 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55d721f47ec0 0x55d721f4a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.686+0000 7f30ab400700 10 client.15633 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T00:58:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.686+0000 7f30c8e00700 1 --2- 172.21.15.116:0/4175215985 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55d721f47ec0 0x55d721f4a2b0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=474c7f85c15f7cd2 server_cookie=da5cb4620f1dbacd in_seq=0 out_seq=0 2024-03-30T00:58:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.686+0000 7f30ab400700 1 -- 172.21.15.116:0/4175215985 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d721ca1ce0 con 0x55d721f47ec0 2024-03-30T00:58:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.686+0000 7f30ab400700 10 client.15633 handle_command_reply: tid=0 2024-03-30T00:58:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.686+0000 7f30dfcf0b80 1 client.15633 shutdown 2024-03-30T00:58:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.687+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/4175215985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecc7d0 msgr2=0x55d721efc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.687+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/4175215985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecc7d0 0x55d721efc5e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f30b8097250 tx=0x7f30b8003bc0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.687+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/4175215985 shutdown_connections 2024-03-30T00:58:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.687+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/4175215985 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55d721f47ec0 0x55d721f4a2b0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.687+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/4175215985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721ecdfd0 0x55d721f463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.687+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/4175215985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721ecc7d0 0x55d721efc5e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.687+0000 7f30dfcf0b80 1 --2- 172.21.15.116:0/4175215985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d721ecd130 0x55d721efcb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.687+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/4175215985 >> 172.21.15.116:0/4175215985 conn(0x55d721e2b6a0 msgr2=0x55d721ed23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.687+0000 7f30ab400700 0 client.15633 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T00:58:54.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.687+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/4175215985 shutdown_connections 2024-03-30T00:58:54.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.687+0000 7f30dfcf0b80 1 -- 172.21.15.116:0/4175215985 wait complete. 2024-03-30T00:58:54.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.687+0000 7f30dfcf0b80 20 client.15633 trim_cache size 0 max 16384 2024-03-30T00:58:54.695 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T00:58:54.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.688+0000 7f30cc200700 1 -- 172.21.15.116:0/1015156236 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f30ac0a10b0 msgr2=0x7f30ac0a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.688+0000 7f30cc200700 1 --2- 172.21.15.116:0/1015156236 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f30ac0a10b0 0x7f30ac0a3570 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x55d721ecd7b0 tx=0x7f30b0009260 comp rx=0 tx=0).stop 2024-03-30T00:58:54.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.688+0000 7f30cc200700 1 -- 172.21.15.116:0/1015156236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30c4073810 msgr2=0x7f30c4198f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.688+0000 7f30cc200700 1 --2- 172.21.15.116:0/1015156236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30c4073810 0x7f30c4198f70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009810 tx=0x7f30b800b790 comp rx=0 tx=0).stop 2024-03-30T00:58:54.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.688+0000 7f30cc200700 1 -- 172.21.15.116:0/1015156236 shutdown_connections 2024-03-30T00:58:54.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.688+0000 7f30cc200700 1 --2- 172.21.15.116:0/1015156236 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f30ac0a10b0 0x7f30ac0a3570 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.688+0000 7f30cc200700 1 --2- 172.21.15.116:0/1015156236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30c4108c40 0x7f30c41a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.688+0000 7f30cc200700 1 --2- 172.21.15.116:0/1015156236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30c4073810 0x7f30c4198f70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.688+0000 7f30cc200700 1 --2- 172.21.15.116:0/1015156236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30c4074df0 0x7f30c41994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.688+0000 7f30cc200700 1 -- 172.21.15.116:0/1015156236 >> 172.21.15.116:0/1015156236 conn(0x7f30c40fc2f0 msgr2=0x7f30c4104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:54.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.688+0000 7f30cc200700 1 -- 172.21.15.116:0/1015156236 shutdown_connections 2024-03-30T00:58:54.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.688+0000 7f30cc200700 1 -- 172.21.15.116:0/1015156236 wait complete. 2024-03-30T00:58:54.710 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:54 smithi116 ceph-mon[38086]: pgmap v322: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:54.710 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:54 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1790998691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:58:54.714 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T00:58:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:54 smithi130 ceph-mon[47626]: pgmap v322: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:54 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1790998691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:58:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.937+0000 7f40f7e00700 1 -- 172.21.15.116:0/2693495638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40f8103f60 msgr2=0x7f40f8106350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.937+0000 7f40f7e00700 1 --2- 172.21.15.116:0/2693495638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40f8103f60 0x7f40f8106350 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f40e4009990 tx=0x7f40e4009ca0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.937+0000 7f40f7e00700 1 -- 172.21.15.116:0/2693495638 shutdown_connections 2024-03-30T00:58:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.937+0000 7f40f7e00700 1 --2- 172.21.15.116:0/2693495638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40f8106890 0x7f40f8108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.937+0000 7f40f7e00700 1 --2- 172.21.15.116:0/2693495638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f8101630 0x7f40f8103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.937+0000 7f40f7e00700 1 --2- 172.21.15.116:0/2693495638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40f8103f60 0x7f40f8106350 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.937+0000 7f40f7e00700 1 -- 172.21.15.116:0/2693495638 >> 172.21.15.116:0/2693495638 conn(0x7f40f80fb000 msgr2=0x7f40f80fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.938+0000 7f40f7e00700 1 -- 172.21.15.116:0/2693495638 shutdown_connections 2024-03-30T00:58:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.938+0000 7f40f7e00700 1 -- 172.21.15.116:0/2693495638 wait complete. 2024-03-30T00:58:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.940+0000 7f40f7e00700 1 Processor -- start 2024-03-30T00:58:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.940+0000 7f40f7e00700 1 -- start start 2024-03-30T00:58:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.940+0000 7f40f7e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f8101630 0x7f40f8100f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.940+0000 7f40f7e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40f8103f60 0x7f40f80ff5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.940+0000 7f40f7e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40f8106890 0x7f40f80ffb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.940+0000 7f40f7e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f81002c0 con 0x7f40f8101630 2024-03-30T00:58:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.940+0000 7f40f7e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8100430 con 0x7f40f8103f60 2024-03-30T00:58:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.940+0000 7f40f7e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f81a0ba0 con 0x7f40f8106890 2024-03-30T00:58:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.941+0000 7f40f6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f8101630 0x7f40f8100f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.941+0000 7f40f7400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40f8106890 0x7f40f80ffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.941+0000 7f40f6000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40f8103f60 0x7f40f80ff5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.941+0000 7f40f6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f8101630 0x7f40f8100f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37918/0 (socket says 172.21.15.116:37918) 2024-03-30T00:58:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.941+0000 7f40f6a00700 1 -- 172.21.15.116:0/102707177 learned_addr learned my addr 172.21.15.116:0/102707177 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:54.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.941+0000 7f40f6a00700 1 -- 172.21.15.116:0/102707177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40f8106890 msgr2=0x7f40f80ffb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.941+0000 7f40f6a00700 1 --2- 172.21.15.116:0/102707177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40f8106890 0x7f40f80ffb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.941+0000 7f40f6a00700 1 -- 172.21.15.116:0/102707177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40f8103f60 msgr2=0x7f40f80ff5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.941+0000 7f40f6a00700 1 --2- 172.21.15.116:0/102707177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40f8103f60 0x7f40f80ff5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.941+0000 7f40f6a00700 1 -- 172.21.15.116:0/102707177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e40095f0 con 0x7f40f8101630 2024-03-30T00:58:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.941+0000 7f40f7400700 1 --2- 172.21.15.116:0/102707177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40f8106890 0x7f40f80ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:58:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.942+0000 7f40f6a00700 1 --2- 172.21.15.116:0/102707177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f8101630 0x7f40f8100f50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f40e800e9a0 tx=0x7f40e800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.942+0000 7f40ef400700 1 -- 172.21.15.116:0/102707177 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8009790 con 0x7f40f8101630 2024-03-30T00:58:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.942+0000 7f40ef400700 1 -- 172.21.15.116:0/102707177 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40e800ce70 con 0x7f40f8101630 2024-03-30T00:58:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.942+0000 7f40f7e00700 1 -- 172.21.15.116:0/102707177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f81a0e30 con 0x7f40f8101630 2024-03-30T00:58:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.942+0000 7f40ef400700 1 -- 172.21.15.116:0/102707177 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8018420 con 0x7f40f8101630 2024-03-30T00:58:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.942+0000 7f40f7e00700 1 -- 172.21.15.116:0/102707177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f81a1260 con 0x7f40f8101630 2024-03-30T00:58:54.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.947+0000 7f40ef400700 1 -- 172.21.15.116:0/102707177 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f40e80106f0 con 0x7f40f8101630 2024-03-30T00:58:54.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.949+0000 7f411153cb80 1 Processor -- start 2024-03-30T00:58:54.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.949+0000 7f411153cb80 1 -- start start 2024-03-30T00:58:54.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.949+0000 7f411153cb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a036e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.949+0000 7f411153cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a037340 0x56504a037bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.949+0000 7f411153cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a038190 0x56504a038610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.949+0000 7f411153cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x565049f8e990 con 0x56504a037340 2024-03-30T00:58:54.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.949+0000 7f411153cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x565049f8c350 con 0x56504a038190 2024-03-30T00:58:54.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.949+0000 7f411153cb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x565049f87c40 con 0x56504a0369e0 2024-03-30T00:58:54.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.950+0000 7f40f7400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.950+0000 7f40f7400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a036e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:35486/0 (socket says 172.21.15.116:35486) 2024-03-30T00:58:54.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.950+0000 7f40f7400700 1 -- 172.21.15.116:0/4256406965 learned_addr learned my addr 172.21.15.116:0/4256406965 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:54.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.950+0000 7f40f6000700 1 --2- 172.21.15.116:0/4256406965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a038190 0x56504a038610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.950+0000 7f40ef400700 1 --2- 172.21.15.116:0/102707177 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f40dc0a1220 0x7f40dc0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.950+0000 7f40f7400700 1 -- 172.21.15.116:0/4256406965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a038190 msgr2=0x56504a038610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:54.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.950+0000 7f40f7400700 1 --2- 172.21.15.116:0/4256406965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a038190 0x56504a038610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.950+0000 7f40f7400700 1 -- 172.21.15.116:0/4256406965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a037340 msgr2=0x56504a037bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:54.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.950+0000 7f40f7400700 1 --2- 172.21.15.116:0/4256406965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a037340 0x56504a037bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.950+0000 7f40f7400700 1 -- 172.21.15.116:0/4256406965 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565049e093b0 con 0x56504a0369e0 2024-03-30T00:58:54.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.950+0000 7f40ef400700 1 -- 172.21.15.116:0/102707177 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f40e8014070 con 0x7f40f8101630 2024-03-30T00:58:54.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.951+0000 7f40f7400700 1 --2- 172.21.15.116:0/102707177 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f40dc0a1220 0x7f40dc0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.951+0000 7f40f7400700 1 --2- 172.21.15.116:0/4256406965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a036e00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f40f0003eb0 tx=0x7f40f0009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eff14a55d4f0690 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:54.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.952+0000 7f40f7400700 1 --2- 172.21.15.116:0/102707177 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f40dc0a1220 0x7f40dc0a36e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f40f0020000 tx=0x7f40f0023000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:54.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.952+0000 7f40e3e00700 1 -- 172.21.15.116:0/4256406965 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0007d20 con 0x56504a0369e0 2024-03-30T00:58:54.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.952+0000 7f40e3e00700 1 -- 172.21.15.116:0/4256406965 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40f000f040 con 0x56504a0369e0 2024-03-30T00:58:54.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.952+0000 7f40e3e00700 1 -- 172.21.15.116:0/4256406965 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0015850 con 0x56504a0369e0 2024-03-30T00:58:54.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.953+0000 7f411153cb80 1 -- 172.21.15.116:0/4256406965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 msgr2=0x56504a036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.953+0000 7f411153cb80 1 --2- 172.21.15.116:0/4256406965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a036e00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f40f0003eb0 tx=0x7f40f0009920 comp rx=0 tx=0).stop 2024-03-30T00:58:54.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.953+0000 7f411153cb80 1 -- 172.21.15.116:0/4256406965 shutdown_connections 2024-03-30T00:58:54.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.953+0000 7f411153cb80 1 --2- 172.21.15.116:0/4256406965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a036e00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.953+0000 7f411153cb80 1 --2- 172.21.15.116:0/4256406965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a038190 0x56504a038610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.953+0000 7f411153cb80 1 --2- 172.21.15.116:0/4256406965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a037340 0x56504a037bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.953+0000 7f411153cb80 1 -- 172.21.15.116:0/4256406965 >> 172.21.15.116:0/4256406965 conn(0x565049f956a0 msgr2=0x56504a031ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:54.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.953+0000 7f411153cb80 1 -- 172.21.15.116:0/4256406965 shutdown_connections 2024-03-30T00:58:54.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.953+0000 7f411153cb80 1 -- 172.21.15.116:0/4256406965 wait complete. 2024-03-30T00:58:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.955+0000 7f411153cb80 1 -- start start 2024-03-30T00:58:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.956+0000 7f411153cb80 1 Processor -- start 2024-03-30T00:58:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.957+0000 7f411153cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a0369e0 0x56504a05a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.957+0000 7f411153cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a037340 0x56504a05a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.957+0000 7f411153cb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a038190 0x56504a061690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.957+0000 7f411153cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x565049f23d80 con 0x56504a037340 2024-03-30T00:58:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.957+0000 7f411153cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x565049f23a20 con 0x56504a0369e0 2024-03-30T00:58:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.957+0000 7f411153cb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x565049e0b6a0 con 0x56504a038190 2024-03-30T00:58:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.957+0000 7f411153cb80 10 client.0 _ll_register_callbacks cb 0x565049f2d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.958+0000 7f40f6000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a037340 0x56504a05a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.958+0000 7f40f6a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a0369e0 0x56504a05a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.958+0000 7f40f7400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a038190 0x56504a061690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.958+0000 7f40f6a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a0369e0 0x56504a05a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43796/0 (socket says 172.21.15.116:43796) 2024-03-30T00:58:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.958+0000 7f40f6a00700 1 -- 172.21.15.116:0/1118172944 learned_addr learned my addr 172.21.15.116:0/1118172944 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.958+0000 7f40f7400700 1 -- 172.21.15.116:0/1118172944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a0369e0 msgr2=0x56504a05a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.958+0000 7f40f7400700 1 --2- 172.21.15.116:0/1118172944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a0369e0 0x56504a05a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.958+0000 7f40f7400700 1 -- 172.21.15.116:0/1118172944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a037340 msgr2=0x56504a05a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.958+0000 7f40f7400700 1 --2- 172.21.15.116:0/1118172944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a037340 0x56504a05a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.958+0000 7f40f7400700 1 -- 172.21.15.116:0/1118172944 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f0003b90 con 0x56504a038190 2024-03-30T00:58:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.958+0000 7f40f6000700 1 --2- 172.21.15.116:0/1118172944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a037340 0x56504a05a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:58:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.958+0000 7f40f7400700 1 --2- 172.21.15.116:0/1118172944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a038190 0x56504a061690 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009fd0 tx=0x7f40f0009430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=862582e778d6a290 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.958+0000 7f40e0c00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T00:58:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.959+0000 7f40e0c00700 1 -- 172.21.15.116:0/1118172944 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0004e10 con 0x56504a038190 2024-03-30T00:58:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.959+0000 7f40e0c00700 1 -- 172.21.15.116:0/1118172944 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f40f0007e00 con 0x56504a038190 2024-03-30T00:58:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.959+0000 7f411153cb80 1 -- 172.21.15.116:0/1118172944 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565049f48380 con 0x56504a038190 2024-03-30T00:58:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.959+0000 7f40e0c00700 1 -- 172.21.15.116:0/1118172944 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40f000e4f0 con 0x56504a038190 2024-03-30T00:58:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.959+0000 7f411153cb80 10 client.25687 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.959+0000 7f411153cb80 1 -- 172.21.15.116:0/1118172944 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x565049db5b60 con 0x56504a038190 2024-03-30T00:58:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.959+0000 7f40e0c00700 1 -- 172.21.15.116:0/1118172944 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x565049f48380 con 0x56504a038190 2024-03-30T00:58:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.960+0000 7f40e0c00700 1 -- 172.21.15.116:0/1118172944 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f40f0015b80 con 0x56504a038190 2024-03-30T00:58:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.960+0000 7f411153cb80 10 client.25687 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.960+0000 7f411153cb80 10 client.25687 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T00:58:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.960+0000 7f411153cb80 20 client.25687 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.960+0000 7f411153cb80 1 --2- 172.21.15.116:0/1118172944 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x56504a065e40 0x56504a068260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.960+0000 7f411153cb80 4 client.25687 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:58:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.960+0000 7f411153cb80 1 -- 172.21.15.116:0/1118172944 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565049f80700 con 0x56504a065e40 2024-03-30T00:58:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.960+0000 7f40f6a00700 1 --2- 172.21.15.116:0/1118172944 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x56504a065e40 0x56504a068260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.961+0000 7f40e0c00700 10 client.25687 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T00:58:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.961+0000 7f40f6a00700 1 --2- 172.21.15.116:0/1118172944 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x56504a065e40 0x56504a068260 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f95f32bd4c7f7535 server_cookie=a6fae431a74e0347 in_seq=0 out_seq=0 2024-03-30T00:58:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.962+0000 7f40e0c00700 1 -- 172.21.15.116:0/1118172944 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x565049f80700 con 0x56504a065e40 2024-03-30T00:58:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.962+0000 7f40e0c00700 10 client.25687 handle_command_reply: tid=0 2024-03-30T00:58:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.962+0000 7f411153cb80 1 client.25687 shutdown 2024-03-30T00:58:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.962+0000 7f411153cb80 1 -- 172.21.15.116:0/1118172944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a038190 msgr2=0x56504a061690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.962+0000 7f411153cb80 1 --2- 172.21.15.116:0/1118172944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a038190 0x56504a061690 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009fd0 tx=0x7f40f0009430 comp rx=0 tx=0).stop 2024-03-30T00:58:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.962+0000 7f411153cb80 1 -- 172.21.15.116:0/1118172944 shutdown_connections 2024-03-30T00:58:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.962+0000 7f411153cb80 1 --2- 172.21.15.116:0/1118172944 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x56504a065e40 0x56504a068260 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.962+0000 7f411153cb80 1 --2- 172.21.15.116:0/1118172944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a038190 0x56504a061690 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.962+0000 7f411153cb80 1 --2- 172.21.15.116:0/1118172944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a0369e0 0x56504a05a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.962+0000 7f411153cb80 1 --2- 172.21.15.116:0/1118172944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a037340 0x56504a05a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.962+0000 7f411153cb80 1 -- 172.21.15.116:0/1118172944 >> 172.21.15.116:0/1118172944 conn(0x565049f956a0 msgr2=0x56504a0353c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.962+0000 7f40e0c00700 0 client.25687 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T00:58:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.962+0000 7f411153cb80 1 -- 172.21.15.116:0/1118172944 shutdown_connections 2024-03-30T00:58:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.962+0000 7f411153cb80 1 -- 172.21.15.116:0/1118172944 wait complete. 2024-03-30T00:58:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.963+0000 7f411153cb80 20 client.25687 trim_cache size 0 max 16384 2024-03-30T00:58:54.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.976+0000 7f411153cb80 1 Processor -- start 2024-03-30T00:58:54.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.976+0000 7f411153cb80 1 -- start start 2024-03-30T00:58:54.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.976+0000 7f411153cb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a061d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.976+0000 7f411153cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a037340 0x56504a061650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.976+0000 7f411153cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a038190 0x56504a059620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.976+0000 7f411153cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x565049f80700 con 0x56504a038190 2024-03-30T00:58:54.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.976+0000 7f411153cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x565049f48380 con 0x56504a037340 2024-03-30T00:58:54.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.976+0000 7f411153cb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56504a065c80 con 0x56504a0369e0 2024-03-30T00:58:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.977+0000 7f40f7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a038190 0x56504a059620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.977+0000 7f40f6000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a037340 0x56504a061650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.977+0000 7f40f6000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a037340 0x56504a061650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43810/0 (socket says 172.21.15.116:43810) 2024-03-30T00:58:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.977+0000 7f40f6000700 1 -- 172.21.15.116:0/664067030 learned_addr learned my addr 172.21.15.116:0/664067030 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.977+0000 7f40f6a00700 1 --2- 172.21.15.116:0/664067030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a061d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.978+0000 7f40f7400700 1 -- 172.21.15.116:0/664067030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 msgr2=0x56504a061d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:54.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.978+0000 7f40f7400700 1 --2- 172.21.15.116:0/664067030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a061d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.978+0000 7f40f7400700 1 -- 172.21.15.116:0/664067030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a037340 msgr2=0x56504a061650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:54.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.978+0000 7f40f7400700 1 --2- 172.21.15.116:0/664067030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a037340 0x56504a061650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.978+0000 7f40f7400700 1 -- 172.21.15.116:0/664067030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565049f4dea0 con 0x56504a038190 2024-03-30T00:58:54.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.978+0000 7f40f7400700 1 --2- 172.21.15.116:0/664067030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a038190 0x56504a059620 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f40f0004600 tx=0x7f40f0007420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=541a65b2d1ad5b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.978+0000 7f40ecc00700 1 -- 172.21.15.116:0/664067030 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f000e4f0 con 0x56504a038190 2024-03-30T00:58:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.978+0000 7f40ecc00700 1 -- 172.21.15.116:0/664067030 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40f000eb90 con 0x56504a038190 2024-03-30T00:58:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.978+0000 7f40ecc00700 1 -- 172.21.15.116:0/664067030 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0015b50 con 0x56504a038190 2024-03-30T00:58:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.979+0000 7f411153cb80 1 -- 172.21.15.116:0/664067030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a038190 msgr2=0x56504a059620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.979+0000 7f411153cb80 1 --2- 172.21.15.116:0/664067030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a038190 0x56504a059620 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f40f0004600 tx=0x7f40f0007420 comp rx=0 tx=0).stop 2024-03-30T00:58:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.980+0000 7f411153cb80 1 -- 172.21.15.116:0/664067030 shutdown_connections 2024-03-30T00:58:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.980+0000 7f411153cb80 1 --2- 172.21.15.116:0/664067030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a061d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.980+0000 7f411153cb80 1 --2- 172.21.15.116:0/664067030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a038190 0x56504a059620 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.980+0000 7f411153cb80 1 --2- 172.21.15.116:0/664067030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a037340 0x56504a061650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.980+0000 7f411153cb80 1 -- 172.21.15.116:0/664067030 >> 172.21.15.116:0/664067030 conn(0x565049f956a0 msgr2=0x56504a02a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.980+0000 7f411153cb80 1 -- 172.21.15.116:0/664067030 shutdown_connections 2024-03-30T00:58:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.980+0000 7f411153cb80 1 -- 172.21.15.116:0/664067030 wait complete. 2024-03-30T00:58:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.980+0000 7f411153cb80 1 -- start start 2024-03-30T00:58:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.980+0000 7f411153cb80 1 Processor -- start 2024-03-30T00:58:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.981+0000 7f411153cb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a05f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.981+0000 7f411153cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a037340 0x56504a05fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.981+0000 7f411153cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a038190 0x56504a0609d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.981+0000 7f411153cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56504a060f10 con 0x56504a038190 2024-03-30T00:58:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.981+0000 7f411153cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56504a061080 con 0x56504a037340 2024-03-30T00:58:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.981+0000 7f411153cb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56504a061240 con 0x56504a0369e0 2024-03-30T00:58:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.981+0000 7f40f6a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a05f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.981+0000 7f40f6a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a05f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:35502/0 (socket says 172.21.15.116:35502) 2024-03-30T00:58:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.981+0000 7f40f6a00700 1 -- 172.21.15.116:0/1225918482 learned_addr learned my addr 172.21.15.116:0/1225918482 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.982+0000 7f411153cb80 10 client.0 _ll_register_callbacks cb 0x565049f2d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.982+0000 7f40f6000700 1 --2- 172.21.15.116:0/1225918482 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a037340 0x56504a05fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.982+0000 7f40f7400700 1 --2- 172.21.15.116:0/1225918482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a038190 0x56504a0609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.982+0000 7f40f6a00700 1 -- 172.21.15.116:0/1225918482 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a037340 msgr2=0x56504a05fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.982+0000 7f40f6a00700 1 --2- 172.21.15.116:0/1225918482 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a037340 0x56504a05fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.982+0000 7f40f6a00700 1 -- 172.21.15.116:0/1225918482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a038190 msgr2=0x56504a0609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.982+0000 7f40f6a00700 1 --2- 172.21.15.116:0/1225918482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a038190 0x56504a0609d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.982+0000 7f40f6a00700 1 -- 172.21.15.116:0/1225918482 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565049db5b60 con 0x56504a0369e0 2024-03-30T00:58:54.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.983+0000 7f40f6a00700 1 --2- 172.21.15.116:0/1225918482 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a05f5c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f40e8096820 tx=0x7f40e8096220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58f6665bcd9c0c72 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:54.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.983+0000 7f40e0c00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T00:58:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.983+0000 7f40e0c00700 1 -- 172.21.15.116:0/1225918482 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e809dbe0 con 0x56504a0369e0 2024-03-30T00:58:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.983+0000 7f411153cb80 1 -- 172.21.15.116:0/1225918482 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40f000f040 con 0x56504a0369e0 2024-03-30T00:58:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.983+0000 7f40e0c00700 1 -- 172.21.15.116:0/1225918482 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f40e80a2230 con 0x56504a0369e0 2024-03-30T00:58:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.983+0000 7f40e0c00700 1 -- 172.21.15.116:0/1225918482 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40e8010430 con 0x56504a0369e0 2024-03-30T00:58:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.983+0000 7f411153cb80 10 client.25690 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.983+0000 7f411153cb80 1 -- 172.21.15.116:0/1225918482 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x565049e0dcd0 con 0x56504a0369e0 2024-03-30T00:58:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.983+0000 7f40e0c00700 1 -- 172.21.15.116:0/1225918482 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40f000f040 con 0x56504a0369e0 2024-03-30T00:58:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.983+0000 7f40e0c00700 1 -- 172.21.15.116:0/1225918482 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f40e80a1ea0 con 0x56504a0369e0 2024-03-30T00:58:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.984+0000 7f411153cb80 10 client.25690 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.984+0000 7f411153cb80 10 client.25690 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T00:58:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.984+0000 7f411153cb80 20 client.25690 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.984+0000 7f411153cb80 1 --2- 172.21.15.116:0/1225918482 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x56504a0baca0 0x56504a0bd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.984+0000 7f411153cb80 4 client.25690 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T00:58:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.984+0000 7f411153cb80 1 -- 172.21.15.116:0/1225918482 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x565049dd6cf0 con 0x56504a0baca0 2024-03-30T00:58:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.984+0000 7f40f6000700 1 --2- 172.21.15.116:0/1225918482 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x56504a0baca0 0x56504a0bd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f40f6000700 1 --2- 172.21.15.116:0/1225918482 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x56504a0baca0 0x56504a0bd0c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=30f2a804d21fd5f4 server_cookie=e1bdc650dd5a3e75 in_seq=0 out_seq=0 2024-03-30T00:58:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f40e0c00700 10 client.25690 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T00:58:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f40e0c00700 1 -- 172.21.15.116:0/1225918482 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x565049dd6cf0 con 0x56504a0baca0 2024-03-30T00:58:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f40e0c00700 10 client.25690 handle_command_reply: tid=0 2024-03-30T00:58:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f411153cb80 1 client.25690 shutdown 2024-03-30T00:58:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f411153cb80 1 -- 172.21.15.116:0/1225918482 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 msgr2=0x56504a05f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f411153cb80 1 --2- 172.21.15.116:0/1225918482 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a05f5c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f40e8096820 tx=0x7f40e8096220 comp rx=0 tx=0).stop 2024-03-30T00:58:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f411153cb80 1 -- 172.21.15.116:0/1225918482 shutdown_connections 2024-03-30T00:58:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f411153cb80 1 --2- 172.21.15.116:0/1225918482 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x56504a0baca0 0x56504a0bd0c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f411153cb80 1 --2- 172.21.15.116:0/1225918482 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56504a0369e0 0x56504a05f5c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f411153cb80 1 --2- 172.21.15.116:0/1225918482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56504a038190 0x56504a0609d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f411153cb80 1 --2- 172.21.15.116:0/1225918482 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56504a037340 0x56504a05fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f411153cb80 1 -- 172.21.15.116:0/1225918482 >> 172.21.15.116:0/1225918482 conn(0x565049f956a0 msgr2=0x56504a03e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f40e0c00700 0 client.25690 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T00:58:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f411153cb80 1 -- 172.21.15.116:0/1225918482 shutdown_connections 2024-03-30T00:58:54.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.985+0000 7f411153cb80 1 -- 172.21.15.116:0/1225918482 wait complete. 2024-03-30T00:58:54.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.986+0000 7f411153cb80 20 client.25690 trim_cache size 0 max 16384 2024-03-30T00:58:54.996 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T00:58:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.986+0000 7f40f7e00700 1 -- 172.21.15.116:0/102707177 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f40dc0a1220 msgr2=0x7f40dc0a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.986+0000 7f40f7e00700 1 --2- 172.21.15.116:0/102707177 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f40dc0a1220 0x7f40dc0a36e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f40f0020000 tx=0x7f40f0023000 comp rx=0 tx=0).stop 2024-03-30T00:58:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.986+0000 7f40f7e00700 1 -- 172.21.15.116:0/102707177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f8101630 msgr2=0x7f40f8100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.986+0000 7f40f7e00700 1 --2- 172.21.15.116:0/102707177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f8101630 0x7f40f8100f50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f40e800e9a0 tx=0x7f40e800ecb0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.987+0000 7f40f7e00700 1 -- 172.21.15.116:0/102707177 shutdown_connections 2024-03-30T00:58:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.987+0000 7f40f7e00700 1 --2- 172.21.15.116:0/102707177 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f40dc0a1220 0x7f40dc0a36e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.987+0000 7f40f7e00700 1 --2- 172.21.15.116:0/102707177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40f8106890 0x7f40f80ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.987+0000 7f40f7e00700 1 --2- 172.21.15.116:0/102707177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f8101630 0x7f40f8100f50 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.987+0000 7f40f7e00700 1 --2- 172.21.15.116:0/102707177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40f8103f60 0x7f40f80ff5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.987+0000 7f40f7e00700 1 -- 172.21.15.116:0/102707177 >> 172.21.15.116:0/102707177 conn(0x7f40f80fb000 msgr2=0x7f40f8103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.987+0000 7f40f7e00700 1 -- 172.21.15.116:0/102707177 shutdown_connections 2024-03-30T00:58:54.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:54.987+0000 7f40f7e00700 1 -- 172.21.15.116:0/102707177 wait complete. 2024-03-30T00:58:55.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:54 smithi181 ceph-mon[47622]: pgmap v322: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:58:55.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:54 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1790998691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:58:55.012 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T00:58:55.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.210+0000 7f6ebfe00700 1 -- 172.21.15.116:0/1220467395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ec01007a0 msgr2=0x7f6ec0100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:55.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.210+0000 7f6ebfe00700 1 --2- 172.21.15.116:0/1220467395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ec01007a0 0x7f6ec0100c20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8009f90 tx=0x7f6eb800b330 comp rx=0 tx=0).stop 2024-03-30T00:58:55.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.211+0000 7f6ebfe00700 1 -- 172.21.15.116:0/1220467395 shutdown_connections 2024-03-30T00:58:55.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.211+0000 7f6ebfe00700 1 --2- 172.21.15.116:0/1220467395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ec01007a0 0x7f6ec0100c20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.211+0000 7f6ebfe00700 1 --2- 172.21.15.116:0/1220467395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6ec00ff480 0x7f6ec00ff8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.211+0000 7f6ebfe00700 1 --2- 172.21.15.116:0/1220467395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec00ffde0 0x7f6ec0100260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.211+0000 7f6ebfe00700 1 -- 172.21.15.116:0/1220467395 >> 172.21.15.116:0/1220467395 conn(0x7f6ec00fb000 msgr2=0x7f6ec00fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:55.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.211+0000 7f6ebfe00700 1 -- 172.21.15.116:0/1220467395 shutdown_connections 2024-03-30T00:58:55.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.211+0000 7f6ebfe00700 1 -- 172.21.15.116:0/1220467395 wait complete. 2024-03-30T00:58:55.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.212+0000 7f6ebfe00700 1 Processor -- start 2024-03-30T00:58:55.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.212+0000 7f6ebfe00700 1 -- start start 2024-03-30T00:58:55.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.212+0000 7f6ebfe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6ec00ff480 0x7f6ec0198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.212+0000 7f6ebfe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec00ffde0 0x7f6ec01992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.213+0000 7f6ebfe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ec01007a0 0x7f6ec01a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.213+0000 7f6ebfe00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec01a08c0 con 0x7f6ec00ffde0 2024-03-30T00:58:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.213+0000 7f6ebfe00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec01a0a00 con 0x7f6ec00ff480 2024-03-30T00:58:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.213+0000 7f6ebfe00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec01a0b90 con 0x7f6ec01007a0 2024-03-30T00:58:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.213+0000 7f6ebea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6ec00ff480 0x7f6ec0198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.213+0000 7f6ebf400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ec01007a0 0x7f6ec01a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.213+0000 7f6ebf400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ec01007a0 0x7f6ec01a02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:35518/0 (socket says 172.21.15.116:35518) 2024-03-30T00:58:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.213+0000 7f6ebf400700 1 -- 172.21.15.116:0/4066105790 learned_addr learned my addr 172.21.15.116:0/4066105790 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.213+0000 7f6ebe000700 1 --2- 172.21.15.116:0/4066105790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec00ffde0 0x7f6ec01992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.213+0000 7f6ebf400700 1 -- 172.21.15.116:0/4066105790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6ec00ff480 msgr2=0x7f6ec0198d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.213+0000 7f6ebf400700 1 --2- 172.21.15.116:0/4066105790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6ec00ff480 0x7f6ec0198d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.213+0000 7f6ebf400700 1 -- 172.21.15.116:0/4066105790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec00ffde0 msgr2=0x7f6ec01992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.214+0000 7f6ebf400700 1 --2- 172.21.15.116:0/4066105790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec00ffde0 0x7f6ec01992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.214+0000 7f6ebf400700 1 -- 172.21.15.116:0/4066105790 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb8009c40 con 0x7f6ec01007a0 2024-03-30T00:58:55.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.214+0000 7f6ebe000700 1 --2- 172.21.15.116:0/4066105790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec00ffde0 0x7f6ec01992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T00:58:55.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.214+0000 7f6ebf400700 1 --2- 172.21.15.116:0/4066105790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ec01007a0 0x7f6ec01a02f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8009f60 tx=0x7f6eb800bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:55.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.214+0000 7f6eb7400700 1 -- 172.21.15.116:0/4066105790 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb800e040 con 0x7f6ec01007a0 2024-03-30T00:58:55.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.214+0000 7f6eb7400700 1 -- 172.21.15.116:0/4066105790 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6eb8004e60 con 0x7f6ec01007a0 2024-03-30T00:58:55.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.214+0000 7f6eb7400700 1 -- 172.21.15.116:0/4066105790 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb80054e0 con 0x7f6ec01007a0 2024-03-30T00:58:55.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.215+0000 7f6ebfe00700 1 -- 172.21.15.116:0/4066105790 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec01a0e10 con 0x7f6ec01007a0 2024-03-30T00:58:55.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.215+0000 7f6ebfe00700 1 -- 172.21.15.116:0/4066105790 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec01a11e0 con 0x7f6ec01007a0 2024-03-30T00:58:55.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.219+0000 7f6ed951cb80 1 Processor -- start 2024-03-30T00:58:55.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.220+0000 7f6eb7400700 1 -- 172.21.15.116:0/4066105790 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f6eb8005640 con 0x7f6ec01007a0 2024-03-30T00:58:55.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.220+0000 7f6ed951cb80 1 -- start start 2024-03-30T00:58:55.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.220+0000 7f6ed951cb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916ba1f0 0x560c916ba610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.220+0000 7f6ed951cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916bab50 0x560c916bb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.220+0000 7f6ed951cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916bbe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.220+0000 7f6ed951cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c91609990 con 0x560c916bb9a0 2024-03-30T00:58:55.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.220+0000 7f6ed951cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c91607350 con 0x560c916bab50 2024-03-30T00:58:55.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.220+0000 7f6ed951cb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c91602c40 con 0x560c916ba1f0 2024-03-30T00:58:55.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.221+0000 7f6ebe000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916ba1f0 0x560c916ba610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.221+0000 7f6ebe000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916ba1f0 0x560c916ba610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:35532/0 (socket says 172.21.15.116:35532) 2024-03-30T00:58:55.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.221+0000 7f6ebe000700 1 -- 172.21.15.116:0/3394947925 learned_addr learned my addr 172.21.15.116:0/3394947925 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:55.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.221+0000 7f6ebea00700 1 --2- 172.21.15.116:0/3394947925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916bab50 0x560c916bb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.221+0000 7f6ebe000700 1 --2- 172.21.15.116:0/3394947925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916bbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.221+0000 7f6ebe000700 1 -- 172.21.15.116:0/3394947925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916bab50 msgr2=0x560c916bb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:55.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.221+0000 7f6ebe000700 1 --2- 172.21.15.116:0/3394947925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916bab50 0x560c916bb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.221+0000 7f6ebe000700 1 -- 172.21.15.116:0/3394947925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 msgr2=0x560c916bbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:55.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.221+0000 7f6ebe000700 1 --2- 172.21.15.116:0/3394947925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916bbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.222+0000 7f6ebe000700 1 -- 172.21.15.116:0/3394947925 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c914843b0 con 0x560c916ba1f0 2024-03-30T00:58:55.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.222+0000 7f6ebe000700 1 --2- 172.21.15.116:0/3394947925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916ba1f0 0x560c916ba610 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0128a0 tx=0x7f6eac00a470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=321741d261687507 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:55.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.222+0000 7f6eabe00700 1 -- 172.21.15.116:0/3394947925 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac004a30 con 0x560c916ba1f0 2024-03-30T00:58:55.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.222+0000 7f6eabe00700 1 -- 172.21.15.116:0/3394947925 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6eac018030 con 0x560c916ba1f0 2024-03-30T00:58:55.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.222+0000 7f6eabe00700 1 -- 172.21.15.116:0/3394947925 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac010840 con 0x560c916ba1f0 2024-03-30T00:58:55.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.222+0000 7f6eb7400700 1 --2- 172.21.15.116:0/4066105790 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6ea40a11a0 0x7f6ea40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.223+0000 7f6eb7400700 1 -- 172.21.15.116:0/4066105790 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6eb80cdc20 con 0x7f6ec01007a0 2024-03-30T00:58:55.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.223+0000 7f6ebea00700 1 --2- 172.21.15.116:0/4066105790 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6ea40a11a0 0x7f6ea40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.223+0000 7f6ed951cb80 1 -- 172.21.15.116:0/3394947925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916ba1f0 msgr2=0x560c916ba610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:55.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.223+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/3394947925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916ba1f0 0x560c916ba610 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0128a0 tx=0x7f6eac00a470 comp rx=0 tx=0).stop 2024-03-30T00:58:55.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.223+0000 7f6ed951cb80 1 -- 172.21.15.116:0/3394947925 shutdown_connections 2024-03-30T00:58:55.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.223+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/3394947925 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916ba1f0 0x560c916ba610 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.223+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/3394947925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916bbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.223+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/3394947925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916bab50 0x560c916bb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.223+0000 7f6ed951cb80 1 -- 172.21.15.116:0/3394947925 >> 172.21.15.116:0/3394947925 conn(0x560c916106a0 msgr2=0x560c916acad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:55.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.224+0000 7f6ebea00700 1 --2- 172.21.15.116:0/4066105790 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6ea40a11a0 0x7f6ea40a3660 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0005950 tx=0x7f6eb00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:55.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.224+0000 7f6ed951cb80 1 -- 172.21.15.116:0/3394947925 shutdown_connections 2024-03-30T00:58:55.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.224+0000 7f6ed951cb80 1 -- 172.21.15.116:0/3394947925 wait complete. 2024-03-30T00:58:55.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.225+0000 7f6ed951cb80 1 -- start start 2024-03-30T00:58:55.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.225+0000 7f6ed951cb80 1 Processor -- start 2024-03-30T00:58:55.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.226+0000 7f6ed951cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916c40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.226+0000 7f6ed951cb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.226+0000 7f6ed951cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.226+0000 7f6ed951cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c9159ed80 con 0x560c916bb9a0 2024-03-30T00:58:55.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.226+0000 7f6ed951cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c9159ea20 con 0x560c916ba1f0 2024-03-30T00:58:55.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.226+0000 7f6ed951cb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c914866a0 con 0x560c916bab50 2024-03-30T00:58:55.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.226+0000 7f6ed951cb80 10 client.0 _ll_register_callbacks cb 0x560c915a8470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:55.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.226+0000 7f6ebe000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.226+0000 7f6ebf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.226+0000 7f6ebe000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916c45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:35540/0 (socket says 172.21.15.116:35540) 2024-03-30T00:58:55.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.226+0000 7f6ebe000700 1 -- 172.21.15.116:0/1566256184 learned_addr learned my addr 172.21.15.116:0/1566256184 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:55.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.227+0000 7f6ebea00700 1 --2- 172.21.15.116:0/1566256184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916c40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.227+0000 7f6ebe000700 1 -- 172.21.15.116:0/1566256184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 msgr2=0x560c916c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:55.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.227+0000 7f6ebe000700 1 --2- 172.21.15.116:0/1566256184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916c40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.227+0000 7f6ebe000700 1 -- 172.21.15.116:0/1566256184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 msgr2=0x560c916cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:55.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.227+0000 7f6ebe000700 1 --2- 172.21.15.116:0/1566256184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.227+0000 7f6ebe000700 1 -- 172.21.15.116:0/1566256184 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eac012580 con 0x560c916bab50 2024-03-30T00:58:55.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.227+0000 7f6ebea00700 1 --2- 172.21.15.116:0/1566256184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916c40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:58:55.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.227+0000 7f6ebe000700 1 --2- 172.21.15.116:0/1566256184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916c45f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6eac00a7a0 tx=0x7f6eac004a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8264d5034a60963 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:55.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.227+0000 7f6ea8c00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T00:58:55.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.228+0000 7f6ea8c00700 1 -- 172.21.15.116:0/1566256184 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac00a310 con 0x560c916bab50 2024-03-30T00:58:55.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.228+0000 7f6ed951cb80 1 -- 172.21.15.116:0/1566256184 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560c915c3380 con 0x560c916bab50 2024-03-30T00:58:55.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.228+0000 7f6ea8c00700 1 -- 172.21.15.116:0/1566256184 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6eac005430 con 0x560c916bab50 2024-03-30T00:58:55.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.228+0000 7f6ea8c00700 1 -- 172.21.15.116:0/1566256184 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6eac005ec0 con 0x560c916bab50 2024-03-30T00:58:55.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.228+0000 7f6ed951cb80 10 client.25702 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:55.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.228+0000 7f6ed951cb80 1 -- 172.21.15.116:0/1566256184 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560c91430b60 con 0x560c916bab50 2024-03-30T00:58:55.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.228+0000 7f6ea8c00700 1 -- 172.21.15.116:0/1566256184 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6eac031020 con 0x560c916bab50 2024-03-30T00:58:55.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.228+0000 7f6ea8c00700 1 -- 172.21.15.116:0/1566256184 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6eac02e070 con 0x560c916bab50 2024-03-30T00:58:55.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.229+0000 7f6ed951cb80 10 client.25702 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:55.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.229+0000 7f6ed951cb80 10 client.25702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:55.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.229+0000 7f6ed951cb80 20 client.25702 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:55.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.229+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/1566256184 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c916cfe10 0x560c916d2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.229+0000 7f6ed951cb80 4 client.25702 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:58:55.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.229+0000 7f6ed951cb80 1 -- 172.21.15.116:0/1566256184 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c915fb700 con 0x560c916cfe10 2024-03-30T00:58:55.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.229+0000 7f6ebea00700 1 --2- 172.21.15.116:0/1566256184 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c916cfe10 0x560c916d2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.230+0000 7f6ea8c00700 10 client.25702 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:55.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.230+0000 7f6ebea00700 1 --2- 172.21.15.116:0/1566256184 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c916cfe10 0x560c916d2230 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85d91f114bab02c0 server_cookie=226a5bcbdfb851ec in_seq=0 out_seq=0 2024-03-30T00:58:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.231+0000 7f6ea8c00700 1 -- 172.21.15.116:0/1566256184 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560c915fb700 con 0x560c916cfe10 2024-03-30T00:58:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.231+0000 7f6ea8c00700 10 client.25702 handle_command_reply: tid=0 2024-03-30T00:58:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.231+0000 7f6ed951cb80 1 client.25702 shutdown 2024-03-30T00:58:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.231+0000 7f6ed951cb80 1 -- 172.21.15.116:0/1566256184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 msgr2=0x560c916c45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.231+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/1566256184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916c45f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6eac00a7a0 tx=0x7f6eac004a00 comp rx=0 tx=0).stop 2024-03-30T00:58:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.231+0000 7f6ed951cb80 1 -- 172.21.15.116:0/1566256184 shutdown_connections 2024-03-30T00:58:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.231+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/1566256184 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c916cfe10 0x560c916d2230 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.231+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/1566256184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.231+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/1566256184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916c40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.231+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/1566256184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916c45f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.231+0000 7f6ed951cb80 1 -- 172.21.15.116:0/1566256184 >> 172.21.15.116:0/1566256184 conn(0x560c916106a0 msgr2=0x560c916b03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.231+0000 7f6ea8c00700 0 client.25702 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.231+0000 7f6ed951cb80 1 -- 172.21.15.116:0/1566256184 shutdown_connections 2024-03-30T00:58:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.231+0000 7f6ed951cb80 1 -- 172.21.15.116:0/1566256184 wait complete. 2024-03-30T00:58:55.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.232+0000 7f6ed951cb80 20 client.25702 trim_cache size 0 max 16384 2024-03-30T00:58:55.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.245+0000 7f6ed951cb80 1 Processor -- start 2024-03-30T00:58:55.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.245+0000 7f6ed951cb80 1 -- start start 2024-03-30T00:58:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.245+0000 7f6ed951cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.245+0000 7f6ed951cb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.245+0000 7f6ed951cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916bbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.246+0000 7f6ed951cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c916c3c20 con 0x560c916bb9a0 2024-03-30T00:58:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.246+0000 7f6ed951cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c916ad640 con 0x560c916ba1f0 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.246+0000 7f6ed951cb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c916ba990 con 0x560c916bab50 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.246+0000 7f6ebea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.246+0000 7f6ebea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916cbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43852/0 (socket says 172.21.15.116:43852) 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.246+0000 7f6ebea00700 1 -- 172.21.15.116:0/514590139 learned_addr learned my addr 172.21.15.116:0/514590139 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.246+0000 7f6ebf400700 1 --2- 172.21.15.116:0/514590139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916bbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.246+0000 7f6ebe000700 1 --2- 172.21.15.116:0/514590139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.246+0000 7f6ebea00700 1 -- 172.21.15.116:0/514590139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 msgr2=0x560c916c35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.246+0000 7f6ebea00700 1 --2- 172.21.15.116:0/514590139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916c35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.246+0000 7f6ebea00700 1 -- 172.21.15.116:0/514590139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 msgr2=0x560c916bbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.246+0000 7f6ebea00700 1 --2- 172.21.15.116:0/514590139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916bbd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.246+0000 7f6ebea00700 1 -- 172.21.15.116:0/514590139 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c915c8ea0 con 0x560c916ba1f0 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.246+0000 7f6ebe000700 1 --2- 172.21.15.116:0/514590139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.247+0000 7f6ebea00700 1 --2- 172.21.15.116:0/514590139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916cbce0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0011a50 tx=0x7f6eb0014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8be166c0c6be902b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.247+0000 7f6eb4c00700 1 -- 172.21.15.116:0/514590139 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0025040 con 0x560c916ba1f0 2024-03-30T00:58:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.247+0000 7f6eb4c00700 1 -- 172.21.15.116:0/514590139 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6eb0016070 con 0x560c916ba1f0 2024-03-30T00:58:55.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.247+0000 7f6eb4c00700 1 -- 172.21.15.116:0/514590139 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0025040 con 0x560c916ba1f0 2024-03-30T00:58:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.248+0000 7f6ed951cb80 1 -- 172.21.15.116:0/514590139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 msgr2=0x560c916cbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.248+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/514590139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916cbce0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0011a50 tx=0x7f6eb0014000 comp rx=0 tx=0).stop 2024-03-30T00:58:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.248+0000 7f6ed951cb80 1 -- 172.21.15.116:0/514590139 shutdown_connections 2024-03-30T00:58:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.248+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/514590139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916bbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.248+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/514590139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916cbce0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.248+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/514590139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.248+0000 7f6ed951cb80 1 -- 172.21.15.116:0/514590139 >> 172.21.15.116:0/514590139 conn(0x560c916106a0 msgr2=0x560c916d6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.248+0000 7f6ed951cb80 1 -- 172.21.15.116:0/514590139 shutdown_connections 2024-03-30T00:58:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.248+0000 7f6ed951cb80 1 -- 172.21.15.116:0/514590139 wait complete. 2024-03-30T00:58:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.248+0000 7f6ed951cb80 1 -- start start 2024-03-30T00:58:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.249+0000 7f6ed951cb80 1 Processor -- start 2024-03-30T00:58:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.249+0000 7f6ed951cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916caf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.249+0000 7f6ed951cb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.249+0000 7f6ed951cb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.249+0000 7f6ed951cb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c916c9bc0 con 0x560c916bb9a0 2024-03-30T00:58:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.249+0000 7f6ed951cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c916c9d30 con 0x560c916ba1f0 2024-03-30T00:58:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.249+0000 7f6ed951cb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c916c9ef0 con 0x560c916bab50 2024-03-30T00:58:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.249+0000 7f6ed951cb80 10 client.0 _ll_register_callbacks cb 0x560c915a8470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:55.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.250+0000 7f6ebea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.250+0000 7f6ebea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916caf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43860/0 (socket says 172.21.15.116:43860) 2024-03-30T00:58:55.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.250+0000 7f6ebea00700 1 -- 172.21.15.116:0/3064248261 learned_addr learned my addr 172.21.15.116:0/3064248261 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:55.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.250+0000 7f6ebe000700 1 --2- 172.21.15.116:0/3064248261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.250+0000 7f6ebf400700 1 --2- 172.21.15.116:0/3064248261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.250+0000 7f6ebe000700 1 -- 172.21.15.116:0/3064248261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 msgr2=0x560c916caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:55.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.250+0000 7f6ebe000700 1 --2- 172.21.15.116:0/3064248261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.250+0000 7f6ebe000700 1 -- 172.21.15.116:0/3064248261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 msgr2=0x560c916c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:55.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.250+0000 7f6ebe000700 1 --2- 172.21.15.116:0/3064248261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.250+0000 7f6ebe000700 1 -- 172.21.15.116:0/3064248261 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb0011c60 con 0x560c916bab50 2024-03-30T00:58:55.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.251+0000 7f6ebe000700 1 --2- 172.21.15.116:0/3064248261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916cb4c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6eac012e90 tx=0x7f6eac026510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=189d094df927f572 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:55.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.251+0000 7f6ea8c00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T00:58:55.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.251+0000 7f6ea8c00700 1 -- 172.21.15.116:0/3064248261 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac00aaf0 con 0x560c916bab50 2024-03-30T00:58:55.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.251+0000 7f6ed951cb80 1 -- 172.21.15.116:0/3064248261 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6eac018030 con 0x560c916bab50 2024-03-30T00:58:55.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.251+0000 7f6ea8c00700 1 -- 172.21.15.116:0/3064248261 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6eac032080 con 0x560c916bab50 2024-03-30T00:58:55.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.251+0000 7f6ea8c00700 1 -- 172.21.15.116:0/3064248261 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6eac005ec0 con 0x560c916bab50 2024-03-30T00:58:55.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.251+0000 7f6ed951cb80 10 client.25708 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:55.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.252+0000 7f6ed951cb80 1 -- 172.21.15.116:0/3064248261 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560c91488cd0 con 0x560c916bab50 2024-03-30T00:58:55.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.252+0000 7f6ea8c00700 1 -- 172.21.15.116:0/3064248261 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6eac018030 con 0x560c916bab50 2024-03-30T00:58:55.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.252+0000 7f6ea8c00700 1 -- 172.21.15.116:0/3064248261 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6eac034070 con 0x560c916bab50 2024-03-30T00:58:55.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.252+0000 7f6ed951cb80 10 client.25708 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:55.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.252+0000 7f6ed951cb80 10 client.25708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:55.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.252+0000 7f6ed951cb80 20 client.25708 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:55.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.252+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/3064248261 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c91724bb0 0x560c91726fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:55.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.252+0000 7f6ed951cb80 4 client.25708 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T00:58:55.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.252+0000 7f6ed951cb80 1 -- 172.21.15.116:0/3064248261 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560c91451cf0 con 0x560c91724bb0 2024-03-30T00:58:55.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.253+0000 7f6ebea00700 1 --2- 172.21.15.116:0/3064248261 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c91724bb0 0x560c91726fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:55.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.254+0000 7f6ebea00700 1 --2- 172.21.15.116:0/3064248261 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c91724bb0 0x560c91726fa0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da8a97aa97afc917 server_cookie=b05ddba302892b9 in_seq=0 out_seq=0 2024-03-30T00:58:55.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.254+0000 7f6ea8c00700 10 client.25708 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:55.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.254+0000 7f6ea8c00700 1 -- 172.21.15.116:0/3064248261 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560c91451cf0 con 0x560c91724bb0 2024-03-30T00:58:55.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.254+0000 7f6ea8c00700 10 client.25708 handle_command_reply: tid=0 2024-03-30T00:58:55.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.254+0000 7f6ed951cb80 1 client.25708 shutdown 2024-03-30T00:58:55.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.255+0000 7f6ed951cb80 1 -- 172.21.15.116:0/3064248261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 msgr2=0x560c916cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:55.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.255+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/3064248261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916cb4c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6eac012e90 tx=0x7f6eac026510 comp rx=0 tx=0).stop 2024-03-30T00:58:55.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.255+0000 7f6ed951cb80 1 -- 172.21.15.116:0/3064248261 shutdown_connections 2024-03-30T00:58:55.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.255+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/3064248261 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c91724bb0 0x560c91726fa0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.255+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/3064248261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c916bb9a0 0x560c916c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.255+0000 7f6ea8c00700 0 client.25708 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:55.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.255+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/3064248261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c916ba1f0 0x560c916caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.255+0000 7f6ed951cb80 1 --2- 172.21.15.116:0/3064248261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c916bab50 0x560c916cb4c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.255+0000 7f6ed951cb80 1 -- 172.21.15.116:0/3064248261 >> 172.21.15.116:0/3064248261 conn(0x560c916106a0 msgr2=0x560c916d1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:55.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.255+0000 7f6ed951cb80 1 -- 172.21.15.116:0/3064248261 shutdown_connections 2024-03-30T00:58:55.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.255+0000 7f6ed951cb80 1 -- 172.21.15.116:0/3064248261 wait complete. 2024-03-30T00:58:55.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.255+0000 7f6ed951cb80 20 client.25708 trim_cache size 0 max 16384 2024-03-30T00:58:55.264 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T00:58:55.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.256+0000 7f6ebfe00700 1 -- 172.21.15.116:0/4066105790 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6ea40a11a0 msgr2=0x7f6ea40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:55.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.256+0000 7f6ebfe00700 1 --2- 172.21.15.116:0/4066105790 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6ea40a11a0 0x7f6ea40a3660 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0005950 tx=0x7f6eb00058e0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.256+0000 7f6ebfe00700 1 -- 172.21.15.116:0/4066105790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ec01007a0 msgr2=0x7f6ec01a02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:55.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.256+0000 7f6ebfe00700 1 --2- 172.21.15.116:0/4066105790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ec01007a0 0x7f6ec01a02f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8009f60 tx=0x7f6eb800bd80 comp rx=0 tx=0).stop 2024-03-30T00:58:55.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.256+0000 7f6ebfe00700 1 -- 172.21.15.116:0/4066105790 shutdown_connections 2024-03-30T00:58:55.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.256+0000 7f6ebfe00700 1 --2- 172.21.15.116:0/4066105790 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6ea40a11a0 0x7f6ea40a3660 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.256+0000 7f6ebfe00700 1 --2- 172.21.15.116:0/4066105790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ec01007a0 0x7f6ec01a02f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.256+0000 7f6ebfe00700 1 --2- 172.21.15.116:0/4066105790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6ec00ff480 0x7f6ec0198d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.256+0000 7f6ebfe00700 1 --2- 172.21.15.116:0/4066105790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec00ffde0 0x7f6ec01992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:55.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.256+0000 7f6ebfe00700 1 -- 172.21.15.116:0/4066105790 >> 172.21.15.116:0/4066105790 conn(0x7f6ec00fb000 msgr2=0x7f6ec00fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:55.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.256+0000 7f6ebfe00700 1 -- 172.21.15.116:0/4066105790 shutdown_connections 2024-03-30T00:58:55.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:55.256+0000 7f6ebfe00700 1 -- 172.21.15.116:0/4066105790 wait complete. 2024-03-30T00:58:56.289 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T00:58:56.290 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T00:58:56.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.494+0000 7f599ca00700 1 -- 172.21.15.116:0/4029936234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f59981068f0 msgr2=0x7f5998108d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.494+0000 7f599ca00700 1 --2- 172.21.15.116:0/4029936234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f59981068f0 0x7f5998108d00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f599000b0a0 tx=0x7f599000b3b0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.494+0000 7f599ca00700 1 -- 172.21.15.116:0/4029936234 shutdown_connections 2024-03-30T00:58:56.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.494+0000 7f599ca00700 1 --2- 172.21.15.116:0/4029936234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5998101690 0x7f5998103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.494+0000 7f599ca00700 1 --2- 172.21.15.116:0/4029936234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f59981068f0 0x7f5998108d00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.494+0000 7f599ca00700 1 --2- 172.21.15.116:0/4029936234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5998103fc0 0x7f59981063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.494+0000 7f599ca00700 1 -- 172.21.15.116:0/4029936234 >> 172.21.15.116:0/4029936234 conn(0x7f59980fb000 msgr2=0x7f59980fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.494+0000 7f599ca00700 1 -- 172.21.15.116:0/4029936234 shutdown_connections 2024-03-30T00:58:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.494+0000 7f599ca00700 1 -- 172.21.15.116:0/4029936234 wait complete. 2024-03-30T00:58:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.495+0000 7f599ca00700 1 Processor -- start 2024-03-30T00:58:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.496+0000 7f599ca00700 1 -- start start 2024-03-30T00:58:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.496+0000 7f599ca00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5998101690 0x7f5998198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.496+0000 7f599ca00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5998103fc0 0x7f5998199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.496+0000 7f599ca00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59981068f0 0x7f59981a0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.496+0000 7f599ca00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59981a0950 con 0x7f59981068f0 2024-03-30T00:58:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.496+0000 7f599ca00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59981a0a90 con 0x7f5998101690 2024-03-30T00:58:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.496+0000 7f599ca00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59981a0c20 con 0x7f5998103fc0 2024-03-30T00:58:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.496+0000 7f5997400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5998101690 0x7f5998198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.496+0000 7f5997400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5998101690 0x7f5998198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43878/0 (socket says 172.21.15.116:43878) 2024-03-30T00:58:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.496+0000 7f5997400700 1 -- 172.21.15.116:0/1582866008 learned_addr learned my addr 172.21.15.116:0/1582866008 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.497+0000 7f5997e00700 1 --2- 172.21.15.116:0/1582866008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59981068f0 0x7f59981a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.497+0000 7f5997e00700 1 -- 172.21.15.116:0/1582866008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5998103fc0 msgr2=0x7f5998199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.497+0000 7f5997e00700 1 --2- 172.21.15.116:0/1582866008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5998103fc0 0x7f5998199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.497+0000 7f5997e00700 1 -- 172.21.15.116:0/1582866008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5998101690 msgr2=0x7f5998198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.497+0000 7f5997e00700 1 --2- 172.21.15.116:0/1582866008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5998101690 0x7f5998198e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.497+0000 7f5997e00700 1 -- 172.21.15.116:0/1582866008 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5990009d00 con 0x7f59981068f0 2024-03-30T00:58:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.497+0000 7f5997400700 1 --2- 172.21.15.116:0/1582866008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5998101690 0x7f5998198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:58:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.497+0000 7f5997e00700 1 --2- 172.21.15.116:0/1582866008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59981068f0 0x7f59981a0380 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5990007b90 tx=0x7f599000bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.498+0000 7f598fe00700 1 -- 172.21.15.116:0/1582866008 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599000e040 con 0x7f59981068f0 2024-03-30T00:58:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.498+0000 7f598fe00700 1 -- 172.21.15.116:0/1582866008 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f59900094b0 con 0x7f59981068f0 2024-03-30T00:58:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.498+0000 7f598fe00700 1 -- 172.21.15.116:0/1582866008 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599001b6a0 con 0x7f59981068f0 2024-03-30T00:58:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.498+0000 7f599ca00700 1 -- 172.21.15.116:0/1582866008 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59981a0ea0 con 0x7f59981068f0 2024-03-30T00:58:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.498+0000 7f599ca00700 1 -- 172.21.15.116:0/1582866008 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59981a1210 con 0x7f59981068f0 2024-03-30T00:58:56.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.502+0000 7f598fe00700 1 -- 172.21.15.116:0/1582866008 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f599001b800 con 0x7f59981068f0 2024-03-30T00:58:56.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.503+0000 7f59b1b92b80 1 Processor -- start 2024-03-30T00:58:56.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.503+0000 7f59b1b92b80 1 -- start start 2024-03-30T00:58:56.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.504+0000 7f59b1b92b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf419a60 0x55a6bf419e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.504+0000 7f59b1b92b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 0x55a6bf41ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.504+0000 7f59b1b92b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf41b210 0x55a6bf41b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.504+0000 7f59b1b92b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bf371a10 con 0x55a6bf41a3c0 2024-03-30T00:58:56.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.504+0000 7f59b1b92b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bf36f3d0 con 0x55a6bf41b210 2024-03-30T00:58:56.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.504+0000 7f59b1b92b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bf36acc0 con 0x55a6bf419a60 2024-03-30T00:58:56.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.504+0000 7f5997400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 0x55a6bf41ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.504+0000 7f5996a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf419a60 0x55a6bf419e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.504+0000 7f5996a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf41b210 0x55a6bf41b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.504+0000 7f5997400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 0x55a6bf41ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38014/0 (socket says 172.21.15.116:38014) 2024-03-30T00:58:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.504+0000 7f5997400700 1 -- 172.21.15.116:0/398951095 learned_addr learned my addr 172.21.15.116:0/398951095 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.505+0000 7f5997400700 1 -- 172.21.15.116:0/398951095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf419a60 msgr2=0x55a6bf419e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.505+0000 7f598fe00700 1 --2- 172.21.15.116:0/1582866008 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f597c0a1120 0x7f597c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.505+0000 7f5997400700 1 --2- 172.21.15.116:0/398951095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf419a60 0x55a6bf419e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.505+0000 7f5997400700 1 -- 172.21.15.116:0/398951095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf41b210 msgr2=0x55a6bf41b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.505+0000 7f5997400700 1 --2- 172.21.15.116:0/398951095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf41b210 0x55a6bf41b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.505+0000 7f5997400700 1 -- 172.21.15.116:0/398951095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6bf1ec430 con 0x55a6bf41a3c0 2024-03-30T00:58:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.505+0000 7f598fe00700 1 -- 172.21.15.116:0/1582866008 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f59900cd980 con 0x7f59981068f0 2024-03-30T00:58:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.505+0000 7f5997e00700 1 --2- 172.21.15.116:0/1582866008 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f597c0a1120 0x7f597c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.505+0000 7f5997400700 1 --2- 172.21.15.116:0/398951095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 0x55a6bf41ac30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5988011810 tx=0x7f5988011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1107d093ecf654 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.506+0000 7f5983e00700 1 -- 172.21.15.116:0/398951095 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598800f480 con 0x55a6bf41a3c0 2024-03-30T00:58:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.506+0000 7f5983e00700 1 -- 172.21.15.116:0/398951095 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f598800faa0 con 0x55a6bf41a3c0 2024-03-30T00:58:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.506+0000 7f5983e00700 1 -- 172.21.15.116:0/398951095 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988019a30 con 0x55a6bf41a3c0 2024-03-30T00:58:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.507+0000 7f59b1b92b80 1 -- 172.21.15.116:0/398951095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 msgr2=0x55a6bf41ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.507+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/398951095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 0x55a6bf41ac30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5988011810 tx=0x7f5988011b20 comp rx=0 tx=0).stop 2024-03-30T00:58:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.507+0000 7f5997e00700 1 --2- 172.21.15.116:0/1582866008 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f597c0a1120 0x7f597c0a35e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f599009aa80 tx=0x7f5990006230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.507+0000 7f59b1b92b80 1 -- 172.21.15.116:0/398951095 shutdown_connections 2024-03-30T00:58:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.507+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/398951095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf419a60 0x55a6bf419e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.507+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/398951095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf41b210 0x55a6bf41b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.507+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/398951095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 0x55a6bf41ac30 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.507+0000 7f59b1b92b80 1 -- 172.21.15.116:0/398951095 >> 172.21.15.116:0/398951095 conn(0x55a6bf378720 msgr2=0x55a6bf414b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.507+0000 7f59b1b92b80 1 -- 172.21.15.116:0/398951095 shutdown_connections 2024-03-30T00:58:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.507+0000 7f59b1b92b80 1 -- 172.21.15.116:0/398951095 wait complete. 2024-03-30T00:58:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.507+0000 7f59b1b92b80 1 -- start start 2024-03-30T00:58:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.508+0000 7f59b1b92b80 1 Processor -- start 2024-03-30T00:58:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.508+0000 7f59b1b92b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf419a60 0x55a6bf4238c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f59b1b92b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf41a3c0 0x55a6bf423e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f59b1b92b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41b210 0x55a6bf42ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f59b1b92b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bf306e00 con 0x55a6bf41b210 2024-03-30T00:58:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f59b1b92b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bf306aa0 con 0x55a6bf419a60 2024-03-30T00:58:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f59b1b92b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bf1ee720 con 0x55a6bf41a3c0 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f59b1b92b80 10 client.0 _ll_register_callbacks cb 0x55a6bf3104f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f5997400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf41a3c0 0x55a6bf423e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f5997400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf41a3c0 0x55a6bf423e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:35580/0 (socket says 172.21.15.116:35580) 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f5997400700 1 -- 172.21.15.116:0/357181656 learned_addr learned my addr 172.21.15.116:0/357181656 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f5996a00700 1 --2- 172.21.15.116:0/357181656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41b210 0x55a6bf42ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f5997400700 1 -- 172.21.15.116:0/357181656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf419a60 msgr2=0x55a6bf4238c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f5997400700 1 --2- 172.21.15.116:0/357181656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf419a60 0x55a6bf4238c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f5997400700 1 -- 172.21.15.116:0/357181656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41b210 msgr2=0x55a6bf42ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f5997400700 1 --2- 172.21.15.116:0/357181656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41b210 0x55a6bf42ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.509+0000 7f5997400700 1 -- 172.21.15.116:0/357181656 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5988011440 con 0x55a6bf41a3c0 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.510+0000 7f5996a00700 1 --2- 172.21.15.116:0/357181656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41b210 0x55a6bf42ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.510+0000 7f5997400700 1 --2- 172.21.15.116:0/357181656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf41a3c0 0x55a6bf423e00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f59880117e0 tx=0x7f598800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6209e13a651d7592 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.510+0000 7f5980c00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.510+0000 7f5980c00700 1 -- 172.21.15.116:0/357181656 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988005ce0 con 0x55a6bf41a3c0 2024-03-30T00:58:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.510+0000 7f59b1b92b80 1 -- 172.21.15.116:0/357181656 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a6bf32b400 con 0x55a6bf41a3c0 2024-03-30T00:58:56.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.510+0000 7f5980c00700 1 -- 172.21.15.116:0/357181656 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5988005250 con 0x55a6bf41a3c0 2024-03-30T00:58:56.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.511+0000 7f5980c00700 1 -- 172.21.15.116:0/357181656 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5988005410 con 0x55a6bf41a3c0 2024-03-30T00:58:56.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.511+0000 7f59b1b92b80 10 client.25717 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:56.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.511+0000 7f59b1b92b80 1 -- 172.21.15.116:0/357181656 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a6bf198be0 con 0x55a6bf41a3c0 2024-03-30T00:58:56.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.511+0000 7f5980c00700 1 -- 172.21.15.116:0/357181656 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a6bf32b400 con 0x55a6bf41a3c0 2024-03-30T00:58:56.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.511+0000 7f5980c00700 1 -- 172.21.15.116:0/357181656 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5988004830 con 0x55a6bf41a3c0 2024-03-30T00:58:56.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.512+0000 7f59b1b92b80 10 client.25717 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:56.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.512+0000 7f59b1b92b80 10 client.25717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:56.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.512+0000 7f59b1b92b80 20 client.25717 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:56.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.512+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/357181656 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a6bf42f550 0x55a6bf431970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.512+0000 7f59b1b92b80 4 client.25717 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:58:56.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.512+0000 7f59b1b92b80 1 -- 172.21.15.116:0/357181656 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a6bf363780 con 0x55a6bf42f550 2024-03-30T00:58:56.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.512+0000 7f5996a00700 1 --2- 172.21.15.116:0/357181656 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a6bf42f550 0x55a6bf431970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.512+0000 7f5996a00700 1 --2- 172.21.15.116:0/357181656 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a6bf42f550 0x55a6bf431970 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b44540798c4fb3 server_cookie=8b0cd008402a947c in_seq=0 out_seq=0 2024-03-30T00:58:56.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.512+0000 7f5980c00700 10 client.25717 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:56.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.513+0000 7f5980c00700 1 -- 172.21.15.116:0/357181656 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a6bf363780 con 0x55a6bf42f550 2024-03-30T00:58:56.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.513+0000 7f5980c00700 10 client.25717 handle_command_reply: tid=0 2024-03-30T00:58:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.513+0000 7f59b1b92b80 1 client.25717 shutdown 2024-03-30T00:58:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.514+0000 7f59b1b92b80 1 -- 172.21.15.116:0/357181656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf41a3c0 msgr2=0x55a6bf423e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.514+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/357181656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf41a3c0 0x55a6bf423e00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f59880117e0 tx=0x7f598800f690 comp rx=0 tx=0).stop 2024-03-30T00:58:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.514+0000 7f59b1b92b80 1 -- 172.21.15.116:0/357181656 shutdown_connections 2024-03-30T00:58:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.514+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/357181656 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a6bf42f550 0x55a6bf431970 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.514+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/357181656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41b210 0x55a6bf42ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.514+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/357181656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf419a60 0x55a6bf4238c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.514+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/357181656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf41a3c0 0x55a6bf423e00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.514+0000 7f5980c00700 0 client.25717 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.514+0000 7f59b1b92b80 1 -- 172.21.15.116:0/357181656 >> 172.21.15.116:0/357181656 conn(0x55a6bf378720 msgr2=0x55a6bf418440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.514+0000 7f59b1b92b80 1 -- 172.21.15.116:0/357181656 shutdown_connections 2024-03-30T00:58:56.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.514+0000 7f59b1b92b80 1 -- 172.21.15.116:0/357181656 wait complete. 2024-03-30T00:58:56.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.514+0000 7f59b1b92b80 20 client.25717 trim_cache size 0 max 16384 2024-03-30T00:58:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.533+0000 7f59b1b92b80 1 Processor -- start 2024-03-30T00:58:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.533+0000 7f59b1b92b80 1 -- start start 2024-03-30T00:58:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.533+0000 7f59b1b92b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf419a60 0x55a6bf4238c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.534+0000 7f59b1b92b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 0x55a6bf423e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.534+0000 7f59b1b92b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf41b210 0x55a6bf42ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.534+0000 7f59b1b92b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bf415a20 con 0x55a6bf41a3c0 2024-03-30T00:58:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.534+0000 7f59b1b92b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bf41a850 con 0x55a6bf41b210 2024-03-30T00:58:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.534+0000 7f59b1b92b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bf42bfc0 con 0x55a6bf419a60 2024-03-30T00:58:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.534+0000 7f5997400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 0x55a6bf423e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.534+0000 7f5996a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf419a60 0x55a6bf4238c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.534+0000 7f5996a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf419a60 0x55a6bf4238c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:35592/0 (socket says 172.21.15.116:35592) 2024-03-30T00:58:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.534+0000 7f5996a00700 1 -- 172.21.15.116:0/3667864073 learned_addr learned my addr 172.21.15.116:0/3667864073 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.534+0000 7f5996a00700 1 --2- 172.21.15.116:0/3667864073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf41b210 0x55a6bf42ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.535+0000 7f5996a00700 1 -- 172.21.15.116:0/3667864073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf41b210 msgr2=0x55a6bf42ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.535+0000 7f5996a00700 1 --2- 172.21.15.116:0/3667864073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf41b210 0x55a6bf42ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.535+0000 7f5996a00700 1 -- 172.21.15.116:0/3667864073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 msgr2=0x55a6bf423e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.535+0000 7f5996a00700 1 --2- 172.21.15.116:0/3667864073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 0x55a6bf423e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.535+0000 7f5996a00700 1 -- 172.21.15.116:0/3667864073 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6bf330f20 con 0x55a6bf419a60 2024-03-30T00:58:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.535+0000 7f5997400700 1 --2- 172.21.15.116:0/3667864073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 0x55a6bf423e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:58:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.535+0000 7f5996a00700 1 --2- 172.21.15.116:0/3667864073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf419a60 0x55a6bf4238c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55a6bf432250 tx=0x7f5984004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34d3d09d28fb5dde server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.535+0000 7f598d200700 1 -- 172.21.15.116:0/3667864073 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984018040 con 0x55a6bf419a60 2024-03-30T00:58:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.535+0000 7f598d200700 1 -- 172.21.15.116:0/3667864073 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f59840044e0 con 0x55a6bf419a60 2024-03-30T00:58:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.535+0000 7f598d200700 1 -- 172.21.15.116:0/3667864073 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984005260 con 0x55a6bf419a60 2024-03-30T00:58:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.536+0000 7f59b1b92b80 1 -- 172.21.15.116:0/3667864073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf419a60 msgr2=0x55a6bf4238c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.536+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/3667864073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf419a60 0x55a6bf4238c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55a6bf432250 tx=0x7f5984004750 comp rx=0 tx=0).stop 2024-03-30T00:58:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.536+0000 7f59b1b92b80 1 -- 172.21.15.116:0/3667864073 shutdown_connections 2024-03-30T00:58:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.536+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/3667864073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf419a60 0x55a6bf4238c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.536+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/3667864073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf41b210 0x55a6bf42ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.536+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/3667864073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 0x55a6bf423e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.536+0000 7f59b1b92b80 1 -- 172.21.15.116:0/3667864073 >> 172.21.15.116:0/3667864073 conn(0x55a6bf378720 msgr2=0x55a6bf422dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.536+0000 7f59b1b92b80 1 -- 172.21.15.116:0/3667864073 shutdown_connections 2024-03-30T00:58:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.536+0000 7f59b1b92b80 1 -- 172.21.15.116:0/3667864073 wait complete. 2024-03-30T00:58:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.537+0000 7f59b1b92b80 1 -- start start 2024-03-30T00:58:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.537+0000 7f59b1b92b80 1 Processor -- start 2024-03-30T00:58:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.538+0000 7f59b1b92b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf419a60 0x55a6bf42a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.538+0000 7f59b1b92b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 0x55a6bf42a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.538+0000 7f59b1b92b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf41b210 0x55a6bf48bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.538+0000 7f59b1b92b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bf48c2c0 con 0x55a6bf41a3c0 2024-03-30T00:58:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.538+0000 7f59b1b92b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bf48c430 con 0x55a6bf419a60 2024-03-30T00:58:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.538+0000 7f59b1b92b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bf48c5f0 con 0x55a6bf41b210 2024-03-30T00:58:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.538+0000 7f59b1b92b80 10 client.0 _ll_register_callbacks cb 0x55a6bf3104f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.538+0000 7f5996a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf419a60 0x55a6bf42a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.538+0000 7f5996a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf419a60 0x55a6bf42a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43916/0 (socket says 172.21.15.116:43916) 2024-03-30T00:58:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.538+0000 7f5996a00700 1 -- 172.21.15.116:0/1244441029 learned_addr learned my addr 172.21.15.116:0/1244441029 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.538+0000 7f5996a00700 1 --2- 172.21.15.116:0/1244441029 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf41b210 0x55a6bf48bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.539+0000 7f5996a00700 1 -- 172.21.15.116:0/1244441029 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf41b210 msgr2=0x55a6bf48bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.539+0000 7f5996a00700 1 --2- 172.21.15.116:0/1244441029 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf41b210 0x55a6bf48bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.539+0000 7f5996a00700 1 -- 172.21.15.116:0/1244441029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 msgr2=0x55a6bf42a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.539+0000 7f5996a00700 1 --2- 172.21.15.116:0/1244441029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 0x55a6bf42a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.539+0000 7f5996a00700 1 -- 172.21.15.116:0/1244441029 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6bf198be0 con 0x55a6bf419a60 2024-03-30T00:58:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.539+0000 7f5996a00700 1 --2- 172.21.15.116:0/1244441029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf419a60 0x55a6bf42a090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5984005e30 tx=0x7f5984022040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f499491ca4bc84c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.539+0000 7f5980c00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T00:58:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.539+0000 7f5980c00700 1 -- 172.21.15.116:0/1244441029 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984018070 con 0x55a6bf419a60 2024-03-30T00:58:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.539+0000 7f5980c00700 1 -- 172.21.15.116:0/1244441029 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5984004f90 con 0x55a6bf419a60 2024-03-30T00:58:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.539+0000 7f59b1b92b80 1 -- 172.21.15.116:0/1244441029 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f598800faa0 con 0x55a6bf419a60 2024-03-30T00:58:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.540+0000 7f5980c00700 1 -- 172.21.15.116:0/1244441029 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5984004c70 con 0x55a6bf419a60 2024-03-30T00:58:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.540+0000 7f59b1b92b80 10 client.25742 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.540+0000 7f59b1b92b80 1 -- 172.21.15.116:0/1244441029 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f5984025040 con 0x55a6bf419a60 2024-03-30T00:58:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.540+0000 7f5980c00700 1 -- 172.21.15.116:0/1244441029 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5984034020 con 0x55a6bf419a60 2024-03-30T00:58:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.540+0000 7f5980c00700 1 -- 172.21.15.116:0/1244441029 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5984003b40 con 0x55a6bf419a60 2024-03-30T00:58:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.540+0000 7f59b1b92b80 10 client.25742 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.540+0000 7f59b1b92b80 10 client.25742 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.540+0000 7f59b1b92b80 20 client.25742 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.540+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/1244441029 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a6bf48dfe0 0x55a6bf490400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.540+0000 7f59b1b92b80 4 client.25742 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T00:58:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.540+0000 7f59b1b92b80 1 -- 172.21.15.116:0/1244441029 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a6bf1b9d70 con 0x55a6bf48dfe0 2024-03-30T00:58:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.541+0000 7f5997400700 1 --2- 172.21.15.116:0/1244441029 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a6bf48dfe0 0x55a6bf490400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.541+0000 7f5997400700 1 --2- 172.21.15.116:0/1244441029 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a6bf48dfe0 0x55a6bf490400 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f652b2ef8583c80 server_cookie=2aedb7de0dce6f81 in_seq=0 out_seq=0 2024-03-30T00:58:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.541+0000 7f5980c00700 10 client.25742 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.542+0000 7f5980c00700 1 -- 172.21.15.116:0/1244441029 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a6bf1b9d70 con 0x55a6bf48dfe0 2024-03-30T00:58:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.542+0000 7f5980c00700 10 client.25742 handle_command_reply: tid=0 2024-03-30T00:58:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.542+0000 7f59b1b92b80 1 client.25742 shutdown 2024-03-30T00:58:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.543+0000 7f59b1b92b80 1 -- 172.21.15.116:0/1244441029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf419a60 msgr2=0x55a6bf42a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.543+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/1244441029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf419a60 0x55a6bf42a090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5984005e30 tx=0x7f5984022040 comp rx=0 tx=0).stop 2024-03-30T00:58:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.543+0000 7f59b1b92b80 1 -- 172.21.15.116:0/1244441029 shutdown_connections 2024-03-30T00:58:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.543+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/1244441029 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55a6bf48dfe0 0x55a6bf490400 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.543+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/1244441029 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a6bf41b210 0x55a6bf48bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.543+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/1244441029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a6bf419a60 0x55a6bf42a090 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.543+0000 7f59b1b92b80 1 --2- 172.21.15.116:0/1244441029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6bf41a3c0 0x55a6bf42a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.543+0000 7f59b1b92b80 1 -- 172.21.15.116:0/1244441029 >> 172.21.15.116:0/1244441029 conn(0x55a6bf378720 msgr2=0x55a6bf423860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.546+0000 7f5980c00700 0 client.25742 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.546+0000 7f59b1b92b80 1 -- 172.21.15.116:0/1244441029 shutdown_connections 2024-03-30T00:58:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.546+0000 7f59b1b92b80 1 -- 172.21.15.116:0/1244441029 wait complete. 2024-03-30T00:58:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.546+0000 7f59b1b92b80 20 client.25742 trim_cache size 0 max 16384 2024-03-30T00:58:56.552 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:58:56.552 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-30T00:58:56.552 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "cc78c2e3-be6f-43f3-9016-01a4a184ecd2", 2024-03-30T00:58:56.552 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-30T00:58:56.552 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:58:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.546+0000 7f599ca00700 1 -- 172.21.15.116:0/1582866008 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f597c0a1120 msgr2=0x7f597c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.546+0000 7f599ca00700 1 --2- 172.21.15.116:0/1582866008 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f597c0a1120 0x7f597c0a35e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f599009aa80 tx=0x7f5990006230 comp rx=0 tx=0).stop 2024-03-30T00:58:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.547+0000 7f599ca00700 1 -- 172.21.15.116:0/1582866008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59981068f0 msgr2=0x7f59981a0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.547+0000 7f599ca00700 1 --2- 172.21.15.116:0/1582866008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59981068f0 0x7f59981a0380 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5990007b90 tx=0x7f599000bce0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.547+0000 7f599ca00700 1 -- 172.21.15.116:0/1582866008 shutdown_connections 2024-03-30T00:58:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.547+0000 7f599ca00700 1 --2- 172.21.15.116:0/1582866008 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f597c0a1120 0x7f597c0a35e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.547+0000 7f599ca00700 1 --2- 172.21.15.116:0/1582866008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59981068f0 0x7f59981a0380 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.547+0000 7f599ca00700 1 --2- 172.21.15.116:0/1582866008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5998101690 0x7f5998198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.547+0000 7f599ca00700 1 --2- 172.21.15.116:0/1582866008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5998103fc0 0x7f5998199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.547+0000 7f599ca00700 1 -- 172.21.15.116:0/1582866008 >> 172.21.15.116:0/1582866008 conn(0x7f59980fb000 msgr2=0x7f5998100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.547+0000 7f599ca00700 1 -- 172.21.15.116:0/1582866008 shutdown_connections 2024-03-30T00:58:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.547+0000 7f599ca00700 1 -- 172.21.15.116:0/1582866008 wait complete. 2024-03-30T00:58:56.580 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T00:58:56.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:56 smithi116 ceph-mon[38086]: pgmap v323: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-30T00:58:56.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.800+0000 7f05a0800700 1 -- 172.21.15.116:0/4066206831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05981017e0 msgr2=0x7f0598103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.800+0000 7f05a0800700 1 --2- 172.21.15.116:0/4066206831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05981017e0 0x7f0598103bd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f058c0098f0 tx=0x7f058c009c00 comp rx=0 tx=0).stop 2024-03-30T00:58:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.800+0000 7f05a0800700 1 -- 172.21.15.116:0/4066206831 shutdown_connections 2024-03-30T00:58:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.800+0000 7f05a0800700 1 --2- 172.21.15.116:0/4066206831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05981017e0 0x7f0598103bd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.800+0000 7f05a0800700 1 --2- 172.21.15.116:0/4066206831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0598106a40 0x7f0598108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.800+0000 7f05a0800700 1 --2- 172.21.15.116:0/4066206831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0598104110 0x7f0598106500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.800+0000 7f05a0800700 1 -- 172.21.15.116:0/4066206831 >> 172.21.15.116:0/4066206831 conn(0x7f05980fb1d0 msgr2=0x7f05980fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.800+0000 7f05a0800700 1 -- 172.21.15.116:0/4066206831 shutdown_connections 2024-03-30T00:58:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.800+0000 7f05a0800700 1 -- 172.21.15.116:0/4066206831 wait complete. 2024-03-30T00:58:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.802+0000 7f05a0800700 1 Processor -- start 2024-03-30T00:58:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.802+0000 7f05a0800700 1 -- start start 2024-03-30T00:58:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.803+0000 7f05a0800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05981017e0 0x7f0598196cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.803+0000 7f05a0800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0598104110 0x7f0598197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.803+0000 7f05a0800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0598106a40 0x7f05981a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.803+0000 7f05a0800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0598197950 con 0x7f0598104110 2024-03-30T00:58:56.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.803+0000 7f05a0800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05981a0b80 con 0x7f05981017e0 2024-03-30T00:58:56.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.803+0000 7f05a0800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05981a0d40 con 0x7f0598106a40 2024-03-30T00:58:56.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.803+0000 7f059d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0598104110 0x7f0598197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.803+0000 7f059de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05981017e0 0x7f0598196cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.803+0000 7f059e800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0598106a40 0x7f05981a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.803+0000 7f059de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05981017e0 0x7f0598196cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43938/0 (socket says 172.21.15.116:43938) 2024-03-30T00:58:56.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.803+0000 7f059de00700 1 -- 172.21.15.116:0/1201978528 learned_addr learned my addr 172.21.15.116:0/1201978528 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:56.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.803+0000 7f059e800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0598106a40 0x7f05981a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:35616/0 (socket says 172.21.15.116:35616) 2024-03-30T00:58:56.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.804+0000 7f059e800700 1 -- 172.21.15.116:0/1201978528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05981017e0 msgr2=0x7f0598196cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.804+0000 7f059e800700 1 --2- 172.21.15.116:0/1201978528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05981017e0 0x7f0598196cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.804+0000 7f059e800700 1 -- 172.21.15.116:0/1201978528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0598104110 msgr2=0x7f0598197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.804+0000 7f059e800700 1 --2- 172.21.15.116:0/1201978528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0598104110 0x7f0598197230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.804+0000 7f059e800700 1 -- 172.21.15.116:0/1201978528 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c009520 con 0x7f0598106a40 2024-03-30T00:58:56.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.804+0000 7f059de00700 1 --2- 172.21.15.116:0/1201978528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05981017e0 0x7f0598196cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:58:56.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.804+0000 7f059d400700 1 --2- 172.21.15.116:0/1201978528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0598104110 0x7f0598197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:58:56.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.804+0000 7f059e800700 1 --2- 172.21.15.116:0/1201978528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0598106a40 0x7f05981a0640 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0594007f10 tx=0x7f059400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:56.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.805+0000 7f0592a00700 1 -- 172.21.15.116:0/1201978528 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059400dbd0 con 0x7f0598106a40 2024-03-30T00:58:56.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.805+0000 7f0592a00700 1 -- 172.21.15.116:0/1201978528 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f059400f040 con 0x7f0598106a40 2024-03-30T00:58:56.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.805+0000 7f05a0800700 1 -- 172.21.15.116:0/1201978528 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059819c3d0 con 0x7f0598106a40 2024-03-30T00:58:56.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.805+0000 7f0592a00700 1 -- 172.21.15.116:0/1201978528 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05940059c0 con 0x7f0598106a40 2024-03-30T00:58:56.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.805+0000 7f05a0800700 1 -- 172.21.15.116:0/1201978528 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059819c820 con 0x7f0598106a40 2024-03-30T00:58:56.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.809+0000 7f0592a00700 1 -- 172.21.15.116:0/1201978528 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f0594004ad0 con 0x7f0598106a40 2024-03-30T00:58:56.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.811+0000 7f05b4329b80 1 Processor -- start 2024-03-30T00:58:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.811+0000 7f05b4329b80 1 -- start start 2024-03-30T00:58:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.811+0000 7f0592a00700 1 --2- 172.21.15.116:0/1201978528 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f05800a11a0 0x7f05800a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.811+0000 7f05b4329b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f0717d0 0x55b61f071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.811+0000 7f05b4329b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f0729f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.812+0000 7f05b4329b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 0x55b61f073450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.812+0000 7f05b4329b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b61efc9990 con 0x55b61f0717d0 2024-03-30T00:58:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.812+0000 7f05b4329b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b61efc7350 con 0x55b61f072130 2024-03-30T00:58:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.812+0000 7f05b4329b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b61efc2c40 con 0x55b61f072fd0 2024-03-30T00:58:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.812+0000 7f0592a00700 1 -- 172.21.15.116:0/1201978528 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f05940cc390 con 0x7f0598106a40 2024-03-30T00:58:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.812+0000 7f059de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f0717d0 0x55b61f071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.812+0000 7f059de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f0717d0 0x55b61f071bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38050/0 (socket says 172.21.15.116:38050) 2024-03-30T00:58:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.812+0000 7f059de00700 1 -- 172.21.15.116:0/3315655214 learned_addr learned my addr 172.21.15.116:0/3315655214 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.812+0000 7f059e800700 1 --2- 172.21.15.116:0/3315655214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 0x55b61f073450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.812+0000 7f059d400700 1 --2- 172.21.15.116:0/3315655214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f0729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.813+0000 7f059d400700 1 --2- 172.21.15.116:0/1201978528 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f05800a11a0 0x7f05800a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.813+0000 7f059de00700 1 -- 172.21.15.116:0/3315655214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 msgr2=0x55b61f073450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.813+0000 7f059de00700 1 --2- 172.21.15.116:0/3315655214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 0x55b61f073450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.813+0000 7f059de00700 1 -- 172.21.15.116:0/3315655214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 msgr2=0x55b61f0729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.813+0000 7f059de00700 1 --2- 172.21.15.116:0/3315655214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f0729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.813+0000 7f059de00700 1 -- 172.21.15.116:0/3315655214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b61ee443b0 con 0x55b61f0717d0 2024-03-30T00:58:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.813+0000 7f059de00700 1 --2- 172.21.15.116:0/3315655214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f0717d0 0x55b61f071bf0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f058c00fb70 tx=0x7f058c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d41d1b123684ed server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.814+0000 7f0587000700 1 -- 172.21.15.116:0/3315655214 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c01c050 con 0x55b61f0717d0 2024-03-30T00:58:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.814+0000 7f059d400700 1 --2- 172.21.15.116:0/1201978528 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f05800a11a0 0x7f05800a3660 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0588005950 tx=0x7f0588009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.814+0000 7f0587000700 1 -- 172.21.15.116:0/3315655214 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f058c01d040 con 0x55b61f0717d0 2024-03-30T00:58:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.814+0000 7f0587000700 1 -- 172.21.15.116:0/3315655214 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c005540 con 0x55b61f0717d0 2024-03-30T00:58:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.815+0000 7f05b4329b80 1 -- 172.21.15.116:0/3315655214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f0717d0 msgr2=0x55b61f071bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.815+0000 7f05b4329b80 1 --2- 172.21.15.116:0/3315655214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f0717d0 0x55b61f071bf0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f058c00fb70 tx=0x7f058c005dc0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.815+0000 7f05b4329b80 1 -- 172.21.15.116:0/3315655214 shutdown_connections 2024-03-30T00:58:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.815+0000 7f05b4329b80 1 --2- 172.21.15.116:0/3315655214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 0x55b61f073450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.815+0000 7f05b4329b80 1 --2- 172.21.15.116:0/3315655214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f0717d0 0x55b61f071bf0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.815+0000 7f05b4329b80 1 --2- 172.21.15.116:0/3315655214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f0729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.815+0000 7f05b4329b80 1 -- 172.21.15.116:0/3315655214 >> 172.21.15.116:0/3315655214 conn(0x55b61efd06a0 msgr2=0x55b61f06c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.815+0000 7f05b4329b80 1 -- 172.21.15.116:0/3315655214 shutdown_connections 2024-03-30T00:58:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.815+0000 7f05b4329b80 1 -- 172.21.15.116:0/3315655214 wait complete. 2024-03-30T00:58:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.816+0000 7f05b4329b80 1 -- start start 2024-03-30T00:58:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.816+0000 7f05b4329b80 1 Processor -- start 2024-03-30T00:58:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.817+0000 7f05b4329b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f07b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.817+0000 7f05b4329b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 0x55b61f07bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.817+0000 7f05b4329b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f080d20 0x55b61f083110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.817+0000 7f05b4329b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b61ef5ed80 con 0x55b61f080d20 2024-03-30T00:58:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.817+0000 7f05b4329b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b61ef5ea20 con 0x55b61f072130 2024-03-30T00:58:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.817+0000 7f05b4329b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b61ee466a0 con 0x55b61f072fd0 2024-03-30T00:58:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.817+0000 7f05b4329b80 10 client.0 _ll_register_callbacks cb 0x55b61ef68470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.817+0000 7f059de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f07b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.817+0000 7f059de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f07b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43956/0 (socket says 172.21.15.116:43956) 2024-03-30T00:58:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.817+0000 7f059de00700 1 -- 172.21.15.116:0/4188566571 learned_addr learned my addr 172.21.15.116:0/4188566571 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.817+0000 7f059e800700 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f080d20 0x55b61f083110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.817+0000 7f059d400700 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 0x55b61f07bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.818+0000 7f059de00700 1 -- 172.21.15.116:0/4188566571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 msgr2=0x55b61f07bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.818+0000 7f059de00700 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 0x55b61f07bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.818+0000 7f059de00700 1 -- 172.21.15.116:0/4188566571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f080d20 msgr2=0x55b61f083110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.818+0000 7f059de00700 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f080d20 0x55b61f083110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.818+0000 7f059de00700 1 -- 172.21.15.116:0/4188566571 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0598109aa0 con 0x55b61f072130 2024-03-30T00:58:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.818+0000 7f059d400700 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 0x55b61f07bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:58:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.818+0000 7f059e800700 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f080d20 0x55b61f083110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:58:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.818+0000 7f059de00700 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f07b800 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f058c005e90 tx=0x7f058c0054f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c87dba214a9a0eae server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.818+0000 7f057b400700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T00:58:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.818+0000 7f057b400700 1 -- 172.21.15.116:0/4188566571 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c01c070 con 0x55b61f072130 2024-03-30T00:58:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.818+0000 7f057b400700 1 -- 172.21.15.116:0/4188566571 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f058c00b970 con 0x55b61f072130 2024-03-30T00:58:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.818+0000 7f05b4329b80 1 -- 172.21.15.116:0/4188566571 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b61ef83380 con 0x55b61f072130 2024-03-30T00:58:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.819+0000 7f057b400700 1 -- 172.21.15.116:0/4188566571 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f058c00bb30 con 0x55b61f072130 2024-03-30T00:58:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.819+0000 7f05b4329b80 10 client.25754 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.819+0000 7f05b4329b80 1 -- 172.21.15.116:0/4188566571 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b61edf0b60 con 0x55b61f072130 2024-03-30T00:58:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.819+0000 7f057b400700 1 -- 172.21.15.116:0/4188566571 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b61ef83380 con 0x55b61f072130 2024-03-30T00:58:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.819+0000 7f057b400700 1 -- 172.21.15.116:0/4188566571 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f058c015020 con 0x55b61f072130 2024-03-30T00:58:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.820+0000 7f05b4329b80 10 client.25754 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.820+0000 7f05b4329b80 10 client.25754 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.820+0000 7f05b4329b80 20 client.25754 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.820+0000 7f05b4329b80 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b61f087460 0x55b61f089850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.820+0000 7f05b4329b80 4 client.25754 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:58:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.820+0000 7f05b4329b80 1 -- 172.21.15.116:0/4188566571 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b61efbb700 con 0x55b61f087460 2024-03-30T00:58:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.820+0000 7f059d400700 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b61f087460 0x55b61f089850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.820+0000 7f059d400700 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b61f087460 0x55b61f089850 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcfae2ae4288451a server_cookie=cbe393bce465cd43 in_seq=0 out_seq=0 2024-03-30T00:58:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.820+0000 7f057b400700 10 client.25754 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.821+0000 7f057b400700 1 -- 172.21.15.116:0/4188566571 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b61efbb700 con 0x55b61f087460 2024-03-30T00:58:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.821+0000 7f057b400700 10 client.25754 handle_command_reply: tid=0 2024-03-30T00:58:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.821+0000 7f05b4329b80 1 client.25754 shutdown 2024-03-30T00:58:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.821+0000 7f05b4329b80 1 -- 172.21.15.116:0/4188566571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 msgr2=0x55b61f07b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.821+0000 7f05b4329b80 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f07b800 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f058c005e90 tx=0x7f058c0054f0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.821+0000 7f05b4329b80 1 -- 172.21.15.116:0/4188566571 shutdown_connections 2024-03-30T00:58:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.821+0000 7f05b4329b80 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b61f087460 0x55b61f089850 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.821+0000 7f05b4329b80 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f080d20 0x55b61f083110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.821+0000 7f05b4329b80 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f07b800 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.821+0000 7f057b400700 0 client.25754 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.821+0000 7f05b4329b80 1 --2- 172.21.15.116:0/4188566571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 0x55b61f07bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.821+0000 7f05b4329b80 1 -- 172.21.15.116:0/4188566571 >> 172.21.15.116:0/4188566571 conn(0x55b61efd06a0 msgr2=0x55b61f070170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.822+0000 7f05b4329b80 1 -- 172.21.15.116:0/4188566571 shutdown_connections 2024-03-30T00:58:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.822+0000 7f05b4329b80 1 -- 172.21.15.116:0/4188566571 wait complete. 2024-03-30T00:58:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.822+0000 7f05b4329b80 20 client.25754 trim_cache size 0 max 16384 2024-03-30T00:58:56.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.839+0000 7f05b4329b80 1 Processor -- start 2024-03-30T00:58:56.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.839+0000 7f05b4329b80 1 -- start start 2024-03-30T00:58:56.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:56 smithi130 ceph-mon[47626]: pgmap v323: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-30T00:58:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.839+0000 7f05b4329b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f0733b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.840+0000 7f05b4329b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 0x55b61f097000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.840+0000 7f05b4329b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f080d20 0x55b61f0655a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.840+0000 7f05b4329b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b61f072e50 con 0x55b61f080d20 2024-03-30T00:58:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.840+0000 7f05b4329b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b61f072870 con 0x55b61f072130 2024-03-30T00:58:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.840+0000 7f05b4329b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b61f0686d0 con 0x55b61f072fd0 2024-03-30T00:58:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.840+0000 7f059de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f0733b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.840+0000 7f059de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f0733b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43970/0 (socket says 172.21.15.116:43970) 2024-03-30T00:58:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.840+0000 7f059de00700 1 -- 172.21.15.116:0/3473691385 learned_addr learned my addr 172.21.15.116:0/3473691385 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.840+0000 7f059d400700 1 --2- 172.21.15.116:0/3473691385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 0x55b61f097000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.840+0000 7f059e800700 1 --2- 172.21.15.116:0/3473691385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f080d20 0x55b61f0655a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.840+0000 7f059de00700 1 -- 172.21.15.116:0/3473691385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 msgr2=0x55b61f097000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.840+0000 7f059de00700 1 --2- 172.21.15.116:0/3473691385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 0x55b61f097000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.840+0000 7f059de00700 1 -- 172.21.15.116:0/3473691385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f080d20 msgr2=0x55b61f0655a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.841+0000 7f059de00700 1 --2- 172.21.15.116:0/3473691385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f080d20 0x55b61f0655a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.841+0000 7f059de00700 1 -- 172.21.15.116:0/3473691385 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b61ef88ea0 con 0x55b61f072130 2024-03-30T00:58:56.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.841+0000 7f059de00700 1 --2- 172.21.15.116:0/3473691385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f0733b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f058c0057e0 tx=0x7f058c005810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=638806f9e0e55ef5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:56.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.841+0000 7f0587a00700 1 -- 172.21.15.116:0/3473691385 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c01c040 con 0x55b61f072130 2024-03-30T00:58:56.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.841+0000 7f0587a00700 1 -- 172.21.15.116:0/3473691385 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f058c005120 con 0x55b61f072130 2024-03-30T00:58:56.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.841+0000 7f0587a00700 1 -- 172.21.15.116:0/3473691385 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c02c690 con 0x55b61f072130 2024-03-30T00:58:56.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.842+0000 7f05b4329b80 1 -- 172.21.15.116:0/3473691385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 msgr2=0x55b61f0733b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.842+0000 7f05b4329b80 1 --2- 172.21.15.116:0/3473691385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f0733b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f058c0057e0 tx=0x7f058c005810 comp rx=0 tx=0).stop 2024-03-30T00:58:56.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.842+0000 7f05b4329b80 1 -- 172.21.15.116:0/3473691385 shutdown_connections 2024-03-30T00:58:56.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.842+0000 7f05b4329b80 1 --2- 172.21.15.116:0/3473691385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f080d20 0x55b61f0655a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.842+0000 7f05b4329b80 1 --2- 172.21.15.116:0/3473691385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f072130 0x55b61f0733b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.842+0000 7f05b4329b80 1 --2- 172.21.15.116:0/3473691385 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072fd0 0x55b61f097000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.842+0000 7f05b4329b80 1 -- 172.21.15.116:0/3473691385 >> 172.21.15.116:0/3473691385 conn(0x55b61efd06a0 msgr2=0x55b61f08e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.842+0000 7f05b4329b80 1 -- 172.21.15.116:0/3473691385 shutdown_connections 2024-03-30T00:58:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.842+0000 7f05b4329b80 1 -- 172.21.15.116:0/3473691385 wait complete. 2024-03-30T00:58:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.843+0000 7f05b4329b80 1 -- start start 2024-03-30T00:58:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.843+0000 7f05b4329b80 1 Processor -- start 2024-03-30T00:58:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.844+0000 7f05b4329b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072130 0x55b61f0d3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.844+0000 7f05b4329b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f072fd0 0x55b61f0d40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.844+0000 7f05b4329b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f080d20 0x55b61f0daa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.844+0000 7f05b4329b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b61f0dafd0 con 0x55b61f072fd0 2024-03-30T00:58:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.844+0000 7f05b4329b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b61f0db110 con 0x55b61f080d20 2024-03-30T00:58:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.844+0000 7f05b4329b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b61f0ec2e0 con 0x55b61f072130 2024-03-30T00:58:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.844+0000 7f05b4329b80 10 client.0 _ll_register_callbacks cb 0x55b61ef68470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:58:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.845+0000 7f059d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f072fd0 0x55b61f0d40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.845+0000 7f059d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f072fd0 0x55b61f0d40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38092/0 (socket says 172.21.15.116:38092) 2024-03-30T00:58:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.845+0000 7f059d400700 1 -- 172.21.15.116:0/3415705245 learned_addr learned my addr 172.21.15.116:0/3415705245 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:58:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.845+0000 7f059de00700 1 --2- 172.21.15.116:0/3415705245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072130 0x55b61f0d3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.845+0000 7f059e800700 1 --2- 172.21.15.116:0/3415705245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f080d20 0x55b61f0daa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.845+0000 7f059d400700 1 -- 172.21.15.116:0/3415705245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072130 msgr2=0x55b61f0d3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.845+0000 7f059d400700 1 --2- 172.21.15.116:0/3415705245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072130 0x55b61f0d3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.845+0000 7f059d400700 1 -- 172.21.15.116:0/3415705245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f080d20 msgr2=0x55b61f0daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:58:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.845+0000 7f059d400700 1 --2- 172.21.15.116:0/3415705245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f080d20 0x55b61f0daa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.845+0000 7f059d400700 1 -- 172.21.15.116:0/3415705245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b61edf0b60 con 0x55b61f072fd0 2024-03-30T00:58:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.845+0000 7f059e800700 1 --2- 172.21.15.116:0/3415705245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f080d20 0x55b61f0daa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T00:58:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.846+0000 7f059d400700 1 --2- 172.21.15.116:0/3415705245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f072fd0 0x55b61f0d40d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0588009c10 tx=0x7f05880047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=973d3be80d5d6fbd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:58:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.846+0000 7f057b400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:58:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.846+0000 7f057b400700 1 -- 172.21.15.116:0/3415705245 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058802a520 con 0x55b61f072fd0 2024-03-30T00:58:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.846+0000 7f057b400700 1 -- 172.21.15.116:0/3415705245 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f058802a910 con 0x55b61f072fd0 2024-03-30T00:58:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.846+0000 7f05b4329b80 1 -- 172.21.15.116:0/3415705245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f058c01d040 con 0x55b61f072fd0 2024-03-30T00:58:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.847+0000 7f057b400700 1 -- 172.21.15.116:0/3415705245 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f058802aad0 con 0x55b61f072fd0 2024-03-30T00:58:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.847+0000 7f05b4329b80 10 client.15690 fetch_fsmap learned FSMap version 23 2024-03-30T00:58:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.847+0000 7f05b4329b80 1 -- 172.21.15.116:0/3415705245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b61ee48cd0 con 0x55b61f072fd0 2024-03-30T00:58:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.847+0000 7f057b400700 1 -- 172.21.15.116:0/3415705245 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f058c01d040 con 0x55b61f072fd0 2024-03-30T00:58:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.847+0000 7f057b400700 1 -- 172.21.15.116:0/3415705245 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f058802a040 con 0x55b61f072fd0 2024-03-30T00:58:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.847+0000 7f05b4329b80 10 client.15690 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:58:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.847+0000 7f05b4329b80 10 client.15690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:58:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.847+0000 7f05b4329b80 20 client.15690 populate_metadata read hostname 'smithi116' 2024-03-30T00:58:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.847+0000 7f05b4329b80 1 --2- 172.21.15.116:0/3415705245 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b61f0eddc0 0x55b61f0f01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:58:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.847+0000 7f05b4329b80 4 client.15690 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T00:58:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.847+0000 7f05b4329b80 1 -- 172.21.15.116:0/3415705245 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b61ee11cf0 con 0x55b61f0eddc0 2024-03-30T00:58:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.848+0000 7f059de00700 1 --2- 172.21.15.116:0/3415705245 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b61f0eddc0 0x55b61f0f01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:58:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.848+0000 7f059de00700 1 --2- 172.21.15.116:0/3415705245 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b61f0eddc0 0x55b61f0f01e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc5813b4a38e71c5 server_cookie=f4f2ce698dd37463 in_seq=0 out_seq=0 2024-03-30T00:58:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.848+0000 7f057b400700 10 client.15690 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.848+0000 7f057b400700 1 -- 172.21.15.116:0/3415705245 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55b61ee11cf0 con 0x55b61f0eddc0 2024-03-30T00:58:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.848+0000 7f057b400700 10 client.15690 handle_command_reply: tid=0 2024-03-30T00:58:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.848+0000 7f05b4329b80 1 client.15690 shutdown 2024-03-30T00:58:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.849+0000 7f05b4329b80 1 -- 172.21.15.116:0/3415705245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f072fd0 msgr2=0x55b61f0d40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.849+0000 7f05b4329b80 1 --2- 172.21.15.116:0/3415705245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f072fd0 0x55b61f0d40d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0588009c10 tx=0x7f05880047c0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.849+0000 7f05b4329b80 1 -- 172.21.15.116:0/3415705245 shutdown_connections 2024-03-30T00:58:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.849+0000 7f05b4329b80 1 --2- 172.21.15.116:0/3415705245 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55b61f0eddc0 0x55b61f0f01e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.849+0000 7f05b4329b80 1 --2- 172.21.15.116:0/3415705245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b61f072130 0x55b61f0d3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.849+0000 7f05b4329b80 1 --2- 172.21.15.116:0/3415705245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b61f080d20 0x55b61f0daa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.849+0000 7f057b400700 0 client.15690 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:58:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.849+0000 7f05b4329b80 1 --2- 172.21.15.116:0/3415705245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b61f072fd0 0x55b61f0d40d0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.849+0000 7f05b4329b80 1 -- 172.21.15.116:0/3415705245 >> 172.21.15.116:0/3415705245 conn(0x55b61efd06a0 msgr2=0x55b61f0655a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.849+0000 7f05b4329b80 1 -- 172.21.15.116:0/3415705245 shutdown_connections 2024-03-30T00:58:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.849+0000 7f05b4329b80 1 -- 172.21.15.116:0/3415705245 wait complete. 2024-03-30T00:58:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.849+0000 7f05b4329b80 20 client.15690 trim_cache size 0 max 16384 2024-03-30T00:58:56.860 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:58:56.860 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T00:58:56.860 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-30T00:58:56.860 INFO:teuthology.orchestra.run.smithi116.stdout: "cc78c2e3-be6f-43f3-9016-01a4a184ecd2": { 2024-03-30T00:58:56.860 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-30T00:58:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "cc78c2e3-be6f-43f3-9016-01a4a184ecd2", 2024-03-30T00:58:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-30T00:58:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T00:58:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T00:58:56.861 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:58:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.850+0000 7f05a0800700 1 -- 172.21.15.116:0/1201978528 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f05800a11a0 msgr2=0x7f05800a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.850+0000 7f05a0800700 1 --2- 172.21.15.116:0/1201978528 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f05800a11a0 0x7f05800a3660 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0588005950 tx=0x7f0588009310 comp rx=0 tx=0).stop 2024-03-30T00:58:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.850+0000 7f05a0800700 1 -- 172.21.15.116:0/1201978528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0598106a40 msgr2=0x7f05981a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:58:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.850+0000 7f05a0800700 1 --2- 172.21.15.116:0/1201978528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0598106a40 0x7f05981a0640 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0594007f10 tx=0x7f059400d3b0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.850+0000 7f05a0800700 1 -- 172.21.15.116:0/1201978528 shutdown_connections 2024-03-30T00:58:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.850+0000 7f05a0800700 1 --2- 172.21.15.116:0/1201978528 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f05800a11a0 0x7f05800a3660 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.850+0000 7f05a0800700 1 --2- 172.21.15.116:0/1201978528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0598106a40 0x7f05981a0640 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.850+0000 7f05a0800700 1 --2- 172.21.15.116:0/1201978528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05981017e0 0x7f0598196cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.850+0000 7f05a0800700 1 --2- 172.21.15.116:0/1201978528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0598104110 0x7f0598197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:58:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.850+0000 7f05a0800700 1 -- 172.21.15.116:0/1201978528 >> 172.21.15.116:0/1201978528 conn(0x7f05980fb1d0 msgr2=0x7f05981036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:58:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.850+0000 7f05a0800700 1 -- 172.21.15.116:0/1201978528 shutdown_connections 2024-03-30T00:58:56.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:58:56.850+0000 7f05a0800700 1 -- 172.21.15.116:0/1201978528 wait complete. 2024-03-30T00:58:56.876 INFO:tasks.cephfs.filesystem:scrub status for tag:cc78c2e3-be6f-43f3-9016-01a4a184ecd2 - {'path': '/', 'tag': 'cc78c2e3-be6f-43f3-9016-01a4a184ecd2', 'options': 'recursive,force'} 2024-03-30T00:58:57.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:56 smithi181 ceph-mon[47622]: pgmap v323: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-30T00:58:57.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:57 smithi130 ceph-mon[47626]: scrub queued for path: / 2024-03-30T00:58:57.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:57 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T00:58:57.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:57 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T00:58:57.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:57 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T00:58:57.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:57 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T00:58:57.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:57 smithi116 ceph-mon[38086]: scrub queued for path: / 2024-03-30T00:58:57.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:57 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T00:58:57.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:57 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T00:58:57.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:57 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T00:58:57.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:57 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T00:58:58.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:57 smithi181 ceph-mon[47622]: scrub queued for path: / 2024-03-30T00:58:58.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:57 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T00:58:58.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:57 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T00:58:58.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:57 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T00:58:58.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:57 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T00:58:58.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:58:58 smithi130 ceph-mon[47626]: pgmap v324: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-30T00:58:58.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:58:58 smithi116 ceph-mon[38086]: pgmap v324: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-30T00:58:59.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:58:58 smithi181 ceph-mon[47622]: pgmap v324: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-30T00:59:00.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:00 smithi130 ceph-mon[47626]: pgmap v325: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-30T00:59:00.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:00 smithi116 ceph-mon[38086]: pgmap v325: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-30T00:59:01.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:00 smithi181 ceph-mon[47622]: pgmap v325: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-30T00:59:02.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:02 smithi130 ceph-mon[47626]: pgmap v326: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-30T00:59:02.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:02 smithi116 ceph-mon[38086]: pgmap v326: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-30T00:59:03.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:02 smithi181 ceph-mon[47622]: pgmap v326: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-30T00:59:03.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:03 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:59:03.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:03 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:59:04.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:03 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T00:59:04.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:04 smithi130 ceph-mon[47626]: pgmap v327: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-30T00:59:04.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:04 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:59:04.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:04 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:59:04.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:04 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:59:04.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:04 smithi116 ceph-mon[38086]: pgmap v327: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-30T00:59:04.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:04 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:59:04.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:04 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:59:04.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:04 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:59:05.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:04 smithi181 ceph-mon[47622]: pgmap v327: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-30T00:59:05.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:04 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T00:59:05.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:04 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T00:59:05.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:04 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T00:59:06.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:06 smithi130 ceph-mon[47626]: pgmap v328: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-30T00:59:06.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:06 smithi116 ceph-mon[38086]: pgmap v328: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-30T00:59:07.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:06 smithi181 ceph-mon[47622]: pgmap v328: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-30T00:59:08.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:08 smithi130 ceph-mon[47626]: pgmap v329: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-30T00:59:08.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:08 smithi116 ceph-mon[38086]: pgmap v329: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-30T00:59:09.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:08 smithi181 ceph-mon[47622]: pgmap v329: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-30T00:59:10.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:10 smithi130 ceph-mon[47626]: pgmap v330: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-30T00:59:10.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:10 smithi116 ceph-mon[38086]: pgmap v330: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-30T00:59:11.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:10 smithi181 ceph-mon[47622]: pgmap v330: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-30T00:59:12.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:12 smithi130 ceph-mon[47626]: pgmap v331: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-30T00:59:12.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:12 smithi116 ceph-mon[38086]: pgmap v331: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-30T00:59:13.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:12 smithi181 ceph-mon[47622]: pgmap v331: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-30T00:59:13.825 INFO:tasks.workunit.client.0.smithi116.stderr:Updating files: 63% (7845/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-30T00:59:14.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:14 smithi130 ceph-mon[47626]: pgmap v332: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:59:14.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:14 smithi116 ceph-mon[38086]: pgmap v332: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:59:14.946 INFO:tasks.workunit.client.0.smithi116.stderr:Note: switching to '470100a4ccbcf7769052bf09a5413881ccf1b97f'. 2024-03-30T00:59:14.946 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-30T00:59:14.946 INFO:tasks.workunit.client.0.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-30T00:59:14.947 INFO:tasks.workunit.client.0.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-30T00:59:14.947 INFO:tasks.workunit.client.0.smithi116.stderr:state without impacting any branches by switching back to a branch. 2024-03-30T00:59:14.947 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-30T00:59:14.947 INFO:tasks.workunit.client.0.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-30T00:59:14.947 INFO:tasks.workunit.client.0.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-30T00:59:14.947 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-30T00:59:14.947 INFO:tasks.workunit.client.0.smithi116.stderr: git switch -c 2024-03-30T00:59:14.947 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-30T00:59:14.947 INFO:tasks.workunit.client.0.smithi116.stderr:Or undo this operation with: 2024-03-30T00:59:14.947 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-30T00:59:14.947 INFO:tasks.workunit.client.0.smithi116.stderr: git switch - 2024-03-30T00:59:14.947 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-30T00:59:14.947 INFO:tasks.workunit.client.0.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-30T00:59:14.947 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-30T00:59:14.948 INFO:tasks.workunit.client.0.smithi116.stderr:HEAD is now at 470100a4ccb DNM: trigger builds again 2024-03-30T00:59:14.964 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-30T00:59:15.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:14 smithi181 ceph-mon[47622]: pgmap v332: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:59:15.026 INFO:tasks.workunit.client.0.smithi116.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-30T00:59:15.031 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-30T00:59:15.031 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-30T00:59:15.649 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-30T00:59:15.760 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-30T00:59:15.824 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-30T00:59:15.828 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-30T00:59:15.828 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-30T00:59:15.894 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-30T00:59:15.901 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T00:59:15.901 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-30T00:59:15.942 INFO:tasks.workunit:Running workunits matching suites/blogbench.sh on client.0... 2024-03-30T00:59:15.944 INFO:tasks.workunit:Running workunit suites/blogbench.sh... 2024-03-30T00:59:15.944 DEBUG:teuthology.orchestra.run.smithi116:workunit test suites/blogbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=470100a4ccbcf7769052bf09a5413881ccf1b97f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh 2024-03-30T00:59:16.037 INFO:tasks.workunit.client.0.smithi116.stdout:getting blogbench 2024-03-30T00:59:16.080 INFO:tasks.workunit.client.0.smithi116.stderr:--2024-03-30 00:59:16-- http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-03-30T00:59:16.081 INFO:tasks.workunit.client.0.smithi116.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-30T00:59:16.102 INFO:tasks.workunit.client.0.smithi116.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-30T00:59:16.125 INFO:tasks.workunit.client.0.smithi116.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-30T00:59:16.125 INFO:tasks.workunit.client.0.smithi116.stderr:Length: 103355 (101K) [application/octet-stream] 2024-03-30T00:59:16.127 INFO:tasks.workunit.client.0.smithi116.stderr:Saving to: ‘blogbench-1.0.tar.bz2’ 2024-03-30T00:59:16.131 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-30T00:59:16.167 INFO:tasks.workunit.client.0.smithi116.stderr: 0K .......... .......... .......... .......... .......... 49% 1.26M 0s 2024-03-30T00:59:16.187 INFO:tasks.workunit.client.0.smithi116.stderr: 50K .......... .......... .......... .......... .......... 99% 2.39M 0s 2024-03-30T00:59:16.187 INFO:tasks.workunit.client.0.smithi116.stderr: 100K 100% 1779G=0.06s 2024-03-30T00:59:16.187 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-30T00:59:16.187 INFO:tasks.workunit.client.0.smithi116.stderr:2024-03-30 00:59:16 (1.66 MB/s) - ‘blogbench-1.0.tar.bz2’ saved [103355/103355] 2024-03-30T00:59:16.187 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-30T00:59:16.223 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/ 2024-03-30T00:59:16.225 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/m4/ 2024-03-30T00:59:16.226 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/m4/acx_pthread.m4 2024-03-30T00:59:16.227 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/man/ 2024-03-30T00:59:16.228 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/man/blogbench.8 2024-03-30T00:59:16.229 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/man/Makefile.am 2024-03-30T00:59:16.231 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/man/Makefile.in 2024-03-30T00:59:16.232 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/ 2024-03-30T00:59:16.233 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/mysnprintf.c 2024-03-30T00:59:16.234 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/mysnprintf.h 2024-03-30T00:59:16.235 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/blogbench.c 2024-03-30T00:59:16.236 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/blogbench.h 2024-03-30T00:59:16.237 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/fakesnprintf.c 2024-03-30T00:59:16.238 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/fakesnprintf.h 2024-03-30T00:59:16.239 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/blogbench_p.h 2024-03-30T00:59:16.240 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/bsd-getopt_long.c 2024-03-30T00:59:16.241 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/bsd-getopt_long.h 2024-03-30T00:59:16.242 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/Makefile.am 2024-03-30T00:59:16.243 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/Makefile.in 2024-03-30T00:59:16.244 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/process.c 2024-03-30T00:59:16.245 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/globals.h 2024-03-30T00:59:16.246 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/commenter.c 2024-03-30T00:59:16.247 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/rewriter.c 2024-03-30T00:59:16.248 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/reader.c 2024-03-30T00:59:16.249 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/writer.c 2024-03-30T00:59:16.250 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/src/helpers.c 2024-03-30T00:59:16.251 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/NEWS 2024-03-30T00:59:16.253 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/compile 2024-03-30T00:59:16.254 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/depcomp 2024-03-30T00:59:16.256 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/aclocal.m4 2024-03-30T00:59:16.257 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/README 2024-03-30T00:59:16.258 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/configure 2024-03-30T00:59:16.263 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/configure.ac 2024-03-30T00:59:16.263 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/config.guess 2024-03-30T00:59:16.265 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/install-sh 2024-03-30T00:59:16.266 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/config.sub 2024-03-30T00:59:16.267 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/missing 2024-03-30T00:59:16.268 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/mkinstalldirs 2024-03-30T00:59:16.270 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/Makefile.am 2024-03-30T00:59:16.271 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/Makefile.in 2024-03-30T00:59:16.272 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/config.h.in 2024-03-30T00:59:16.273 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/AUTHORS 2024-03-30T00:59:16.275 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/INSTALL 2024-03-30T00:59:16.276 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/ChangeLog 2024-03-30T00:59:16.277 INFO:tasks.workunit.client.0.smithi116.stdout:blogbench-1.0/COPYING 2024-03-30T00:59:16.279 INFO:tasks.workunit.client.0.smithi116.stdout:making blogbench 2024-03-30T00:59:16.512 INFO:tasks.workunit.client.0.smithi116.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-30T00:59:16.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:16 smithi130 ceph-mon[47626]: pgmap v333: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-30T00:59:16.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:16 smithi116 ceph-mon[38086]: pgmap v333: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-30T00:59:17.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:16 smithi181 ceph-mon[47622]: pgmap v333: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-30T00:59:17.526 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether build environment is sane... yes 2024-03-30T00:59:17.544 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gawk... gawk 2024-03-30T00:59:17.595 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether make sets $(MAKE)... yes 2024-03-30T00:59:17.616 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc... gcc 2024-03-30T00:59:17.708 INFO:tasks.workunit.client.0.smithi116.stdout:checking for C compiler default output file name... a.out 2024-03-30T00:59:17.712 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether the C compiler works... yes 2024-03-30T00:59:17.716 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether we are cross compiling... no 2024-03-30T00:59:17.775 INFO:tasks.workunit.client.0.smithi116.stdout:checking for suffix of executables... 2024-03-30T00:59:17.839 INFO:tasks.workunit.client.0.smithi116.stdout:checking for suffix of object files... o 2024-03-30T00:59:17.901 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether we are using the GNU C compiler... yes 2024-03-30T00:59:17.966 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether gcc accepts -g... yes 2024-03-30T00:59:18.042 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc option to accept ANSI C... none needed 2024-03-30T00:59:18.108 INFO:tasks.workunit.client.0.smithi116.stdout:checking for style of include used by make... GNU 2024-03-30T00:59:18.253 INFO:tasks.workunit.client.0.smithi116.stdout:checking dependency style of gcc... gcc3 2024-03-30T00:59:18.253 INFO:tasks.workunit.client.0.smithi116.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-30T00:59:18.325 INFO:tasks.workunit.client.0.smithi116.stdout:checking how to run the C preprocessor... gcc -E 2024-03-30T00:59:18.398 INFO:tasks.workunit.client.0.smithi116.stdout:checking for egrep... grep -E 2024-03-30T00:59:18.415 INFO:tasks.workunit.client.0.smithi116.stdout:checking for AIX... no 2024-03-30T00:59:18.515 INFO:tasks.workunit.client.0.smithi116.stdout:checking for library containing strerror... none required 2024-03-30T00:59:18.738 INFO:tasks.workunit.client.0.smithi116.stdout:checking for ANSI C header files... yes 2024-03-30T00:59:18.809 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-30T00:59:18.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:18 smithi130 ceph-mon[47626]: pgmap v334: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:59:18.893 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/types.h... yes 2024-03-30T00:59:18.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:18 smithi116 ceph-mon[38086]: pgmap v334: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:59:18.990 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/stat.h... yes 2024-03-30T00:59:19.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:18 smithi181 ceph-mon[47622]: pgmap v334: 129 pgs: 129 active+clean; 657 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-30T00:59:19.081 INFO:tasks.workunit.client.0.smithi116.stdout:checking for stdlib.h... yes 2024-03-30T00:59:19.171 INFO:tasks.workunit.client.0.smithi116.stdout:checking for string.h... yes 2024-03-30T00:59:19.262 INFO:tasks.workunit.client.0.smithi116.stdout:checking for memory.h... yes 2024-03-30T00:59:19.354 INFO:tasks.workunit.client.0.smithi116.stdout:checking for strings.h... yes 2024-03-30T00:59:19.447 INFO:tasks.workunit.client.0.smithi116.stdout:checking for inttypes.h... yes 2024-03-30T00:59:19.538 INFO:tasks.workunit.client.0.smithi116.stdout:checking for stdint.h... yes 2024-03-30T00:59:19.630 INFO:tasks.workunit.client.0.smithi116.stdout:checking for unistd.h... yes 2024-03-30T00:59:19.653 INFO:tasks.workunit.client.0.smithi116.stdout:checking for memory.h... (cached) yes 2024-03-30T00:59:19.668 INFO:tasks.workunit.client.0.smithi116.stdout:checking for strings.h... (cached) yes 2024-03-30T00:59:19.684 INFO:tasks.workunit.client.0.smithi116.stdout:checking for unistd.h... (cached) yes 2024-03-30T00:59:19.769 INFO:tasks.workunit.client.0.smithi116.stdout:checking locale.h usability... yes 2024-03-30T00:59:19.803 INFO:tasks.workunit.client.0.smithi116.stdout:checking locale.h presence... yes 2024-03-30T00:59:19.805 INFO:tasks.workunit.client.0.smithi116.stdout:checking for locale.h... yes 2024-03-30T00:59:19.832 INFO:tasks.workunit.client.0.smithi116.stdout:checking for stdlib.h... (cached) yes 2024-03-30T00:59:19.847 INFO:tasks.workunit.client.0.smithi116.stdout:checking for string.h... (cached) yes 2024-03-30T00:59:19.946 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/socket.h usability... yes 2024-03-30T00:59:19.983 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/socket.h presence... yes 2024-03-30T00:59:19.985 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/socket.h... yes 2024-03-30T00:59:20.078 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/time.h usability... yes 2024-03-30T00:59:20.112 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/time.h presence... yes 2024-03-30T00:59:20.114 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/time.h... yes 2024-03-30T00:59:20.208 INFO:tasks.workunit.client.0.smithi116.stdout:checking fcntl.h usability... yes 2024-03-30T00:59:20.241 INFO:tasks.workunit.client.0.smithi116.stdout:checking fcntl.h presence... yes 2024-03-30T00:59:20.244 INFO:tasks.workunit.client.0.smithi116.stdout:checking for fcntl.h... yes 2024-03-30T00:59:20.341 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/fcntl.h usability... yes 2024-03-30T00:59:20.376 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/fcntl.h presence... yes 2024-03-30T00:59:20.378 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/fcntl.h... yes 2024-03-30T00:59:20.512 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/param.h usability... yes 2024-03-30T00:59:20.552 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/param.h presence... yes 2024-03-30T00:59:20.554 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/param.h... yes 2024-03-30T00:59:20.632 INFO:tasks.workunit.client.0.smithi116.stdout:checking ioctl.h usability... no 2024-03-30T00:59:20.668 INFO:tasks.workunit.client.0.smithi116.stdout:checking ioctl.h presence... no 2024-03-30T00:59:20.670 INFO:tasks.workunit.client.0.smithi116.stdout:checking for ioctl.h... no 2024-03-30T00:59:20.757 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/ioctl.h usability... yes 2024-03-30T00:59:20.791 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/ioctl.h presence... yes 2024-03-30T00:59:20.793 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/ioctl.h... yes 2024-03-30T00:59:20.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:20 smithi130 ceph-mon[47626]: pgmap v335: 129 pgs: 129 active+clean; 657 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:59:20.883 INFO:tasks.workunit.client.0.smithi116.stdout:checking return type of signal handlers... void 2024-03-30T00:59:20.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:20 smithi116 ceph-mon[38086]: pgmap v335: 129 pgs: 129 active+clean; 657 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:59:20.962 INFO:tasks.workunit.client.0.smithi116.stdout:checking for size_t... yes 2024-03-30T00:59:21.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:20 smithi181 ceph-mon[47622]: pgmap v335: 129 pgs: 129 active+clean; 657 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-30T00:59:21.031 INFO:tasks.workunit.client.0.smithi116.stdout:checking for dev_t... yes 2024-03-30T00:59:21.107 INFO:tasks.workunit.client.0.smithi116.stdout:checking for short... yes 2024-03-30T00:59:21.221 INFO:tasks.workunit.client.0.smithi116.stdout:checking size of short... 2 2024-03-30T00:59:21.301 INFO:tasks.workunit.client.0.smithi116.stdout:checking for int... yes 2024-03-30T00:59:21.405 INFO:tasks.workunit.client.0.smithi116.stdout:checking size of int... 4 2024-03-30T00:59:21.520 INFO:tasks.workunit.client.0.smithi116.stdout:checking for long... yes 2024-03-30T00:59:21.636 INFO:tasks.workunit.client.0.smithi116.stdout:checking size of long... 8 2024-03-30T00:59:21.715 INFO:tasks.workunit.client.0.smithi116.stdout:checking for long long... yes 2024-03-30T00:59:21.820 INFO:tasks.workunit.client.0.smithi116.stdout:checking size of long long... 8 2024-03-30T00:59:22.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:21 smithi181 ceph-mon[47622]: pgmap v336: 129 pgs: 129 active+clean; 1.6 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 81 KiB/s wr, 9 op/s 2024-03-30T00:59:22.005 INFO:tasks.workunit.client.0.smithi116.stdout:checking for ssize_t... yes 2024-03-30T00:59:22.006 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:21 smithi116 ceph-mon[38086]: pgmap v336: 129 pgs: 129 active+clean; 1.6 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 81 KiB/s wr, 9 op/s 2024-03-30T00:59:22.006 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sig_atomic_t... yes 2024-03-30T00:59:22.081 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether gcc needs -traditional... no 2024-03-30T00:59:22.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:21 smithi130 ceph-mon[47626]: pgmap v336: 129 pgs: 129 active+clean; 1.6 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 81 KiB/s wr, 9 op/s 2024-03-30T00:59:22.146 INFO:tasks.workunit.client.0.smithi116.stdout:checking for an ANSI C-conforming const... yes 2024-03-30T00:59:22.198 INFO:tasks.workunit.client.0.smithi116.stdout:checking for inline... inline 2024-03-30T00:59:22.292 INFO:tasks.workunit.client.0.smithi116.stdout:checking for working alloca.h... yes 2024-03-30T00:59:22.378 INFO:tasks.workunit.client.0.smithi116.stdout:checking for alloca... yes 2024-03-30T00:59:22.482 INFO:tasks.workunit.client.0.smithi116.stdout:checking for working memcmp... yes 2024-03-30T00:59:22.584 INFO:tasks.workunit.client.0.smithi116.stdout:checking for strftime... yes 2024-03-30T00:59:22.706 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether lstat dereferences a symlink specified with a trailing slash... yes 2024-03-30T00:59:22.808 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether stat accepts an empty string... no 2024-03-30T00:59:22.910 INFO:tasks.workunit.client.0.smithi116.stdout:checking for vprintf... yes 2024-03-30T00:59:23.013 INFO:tasks.workunit.client.0.smithi116.stdout:checking for _doprnt... no 2024-03-30T00:59:23.068 INFO:tasks.workunit.client.0.smithi116.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-30T00:59:23.100 INFO:tasks.workunit.client.0.smithi116.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-30T00:59:23.211 INFO:tasks.workunit.client.0.smithi116.stdout:checking for the pthreads library -lpthreads... no 2024-03-30T00:59:23.299 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether pthreads work without any flags... no 2024-03-30T00:59:23.331 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether pthreads work with -Kthread... no 2024-03-30T00:59:23.365 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether pthreads work with -kthread... no 2024-03-30T00:59:23.440 INFO:tasks.workunit.client.0.smithi116.stdout:checking for the pthreads library -llthread... no 2024-03-30T00:59:23.534 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether pthreads work with -pthread... yes 2024-03-30T00:59:23.618 INFO:tasks.workunit.client.0.smithi116.stdout:checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE 2024-03-30T00:59:23.618 INFO:tasks.workunit.client.0.smithi116.stdout:checking if more special flags are required for pthreads... no 2024-03-30T00:59:23.618 INFO:tasks.workunit.client.0.smithi116.stdout:checking for cc_r... gcc 2024-03-30T00:59:23.750 INFO:tasks.workunit.client.0.smithi116.stdout:checking for setrlimit... yes 2024-03-30T00:59:23.856 INFO:tasks.workunit.client.0.smithi116.stdout:checking for setproctitle... no 2024-03-30T00:59:23.958 INFO:tasks.workunit.client.0.smithi116.stdout:checking for getopt_long... yes 2024-03-30T00:59:24.067 INFO:tasks.workunit.client.0.smithi116.stdout:checking for strtoull... yes 2024-03-30T00:59:24.175 INFO:tasks.workunit.client.0.smithi116.stdout:checking for strtoq... yes 2024-03-30T00:59:24.283 INFO:tasks.workunit.client.0.smithi116.stdout:checking for snprintf... yes 2024-03-30T00:59:24.392 INFO:tasks.workunit.client.0.smithi116.stdout:checking for vsnprintf... yes 2024-03-30T00:59:24.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:24 smithi181 ceph-mon[47622]: pgmap v337: 129 pgs: 129 active+clean; 1.7 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 91 KiB/s wr, 10 op/s 2024-03-30T00:59:24.502 INFO:tasks.workunit.client.0.smithi116.stdout:checking for vfprintf... yes 2024-03-30T00:59:24.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:24 smithi130 ceph-mon[47626]: pgmap v337: 129 pgs: 129 active+clean; 1.7 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 91 KiB/s wr, 10 op/s 2024-03-30T00:59:24.611 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gethostname... yes 2024-03-30T00:59:24.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:24 smithi116 ceph-mon[38086]: pgmap v337: 129 pgs: 129 active+clean; 1.7 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 91 KiB/s wr, 10 op/s 2024-03-30T00:59:24.720 INFO:tasks.workunit.client.0.smithi116.stdout:checking for setlocale... yes 2024-03-30T00:59:24.831 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sync... yes 2024-03-30T00:59:24.962 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether snprintf is C99 conformant... done 2024-03-30T00:59:24.985 INFO:tasks.workunit.client.0.smithi116.stdout:configure: creating ./config.status 2024-03-30T00:59:25.291 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: creating Makefile 2024-03-30T00:59:25.332 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: creating src/Makefile 2024-03-30T00:59:25.375 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: creating man/Makefile 2024-03-30T00:59:25.397 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: creating config.h 2024-03-30T00:59:25.486 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: executing depfiles commands 2024-03-30T00:59:25.801 INFO:tasks.workunit.client.0.smithi116.stdout:make all-recursive 2024-03-30T00:59:25.803 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-30T00:59:25.809 INFO:tasks.workunit.client.0.smithi116.stdout:Making all in src 2024-03-30T00:59:25.818 INFO:tasks.workunit.client.0.smithi116.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-03-30T00:59:25.819 INFO:tasks.workunit.client.0.smithi116.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT blogbench.o -MD -MP -MF ".deps/blogbench.Tpo" -c -o blogbench.o blogbench.c; \ 2024-03-30T00:59:25.819 INFO:tasks.workunit.client.0.smithi116.stdout:then mv -f ".deps/blogbench.Tpo" ".deps/blogbench.Po"; else rm -f ".deps/blogbench.Tpo"; exit 1; fi 2024-03-30T00:59:25.931 INFO:tasks.workunit.client.0.smithi116.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT process.o -MD -MP -MF ".deps/process.Tpo" -c -o process.o process.c; \ 2024-03-30T00:59:25.932 INFO:tasks.workunit.client.0.smithi116.stdout:then mv -f ".deps/process.Tpo" ".deps/process.Po"; else rm -f ".deps/process.Tpo"; exit 1; fi 2024-03-30T00:59:26.021 INFO:tasks.workunit.client.0.smithi116.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT writer.o -MD -MP -MF ".deps/writer.Tpo" -c -o writer.o writer.c; \ 2024-03-30T00:59:26.021 INFO:tasks.workunit.client.0.smithi116.stdout:then mv -f ".deps/writer.Tpo" ".deps/writer.Po"; else rm -f ".deps/writer.Tpo"; exit 1; fi 2024-03-30T00:59:26.099 INFO:tasks.workunit.client.0.smithi116.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT rewriter.o -MD -MP -MF ".deps/rewriter.Tpo" -c -o rewriter.o rewriter.c; \ 2024-03-30T00:59:26.099 INFO:tasks.workunit.client.0.smithi116.stdout:then mv -f ".deps/rewriter.Tpo" ".deps/rewriter.Po"; else rm -f ".deps/rewriter.Tpo"; exit 1; fi 2024-03-30T00:59:26.175 INFO:tasks.workunit.client.0.smithi116.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT reader.o -MD -MP -MF ".deps/reader.Tpo" -c -o reader.o reader.c; \ 2024-03-30T00:59:26.175 INFO:tasks.workunit.client.0.smithi116.stdout:then mv -f ".deps/reader.Tpo" ".deps/reader.Po"; else rm -f ".deps/reader.Tpo"; exit 1; fi 2024-03-30T00:59:26.249 INFO:tasks.workunit.client.0.smithi116.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT commenter.o -MD -MP -MF ".deps/commenter.Tpo" -c -o commenter.o commenter.c; \ 2024-03-30T00:59:26.249 INFO:tasks.workunit.client.0.smithi116.stdout:then mv -f ".deps/commenter.Tpo" ".deps/commenter.Po"; else rm -f ".deps/commenter.Tpo"; exit 1; fi 2024-03-30T00:59:26.320 INFO:tasks.workunit.client.0.smithi116.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT helpers.o -MD -MP -MF ".deps/helpers.Tpo" -c -o helpers.o helpers.c; \ 2024-03-30T00:59:26.321 INFO:tasks.workunit.client.0.smithi116.stdout:then mv -f ".deps/helpers.Tpo" ".deps/helpers.Po"; else rm -f ".deps/helpers.Tpo"; exit 1; fi 2024-03-30T00:59:26.417 INFO:tasks.workunit.client.0.smithi116.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT bsd-getopt_long.o -MD -MP -MF ".deps/bsd-getopt_long.Tpo" -c -o bsd-getopt_long.o bsd-getopt_long.c; \ 2024-03-30T00:59:26.418 INFO:tasks.workunit.client.0.smithi116.stdout:then mv -f ".deps/bsd-getopt_long.Tpo" ".deps/bsd-getopt_long.Po"; else rm -f ".deps/bsd-getopt_long.Tpo"; exit 1; fi 2024-03-30T00:59:26.474 INFO:tasks.workunit.client.0.smithi116.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT fakesnprintf.o -MD -MP -MF ".deps/fakesnprintf.Tpo" -c -o fakesnprintf.o fakesnprintf.c; \ 2024-03-30T00:59:26.474 INFO:tasks.workunit.client.0.smithi116.stdout:then mv -f ".deps/fakesnprintf.Tpo" ".deps/fakesnprintf.Po"; else rm -f ".deps/fakesnprintf.Tpo"; exit 1; fi 2024-03-30T00:59:26.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:26 smithi181 ceph-mon[47622]: pgmap v338: 129 pgs: 129 active+clean; 3.5 MiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 262 KiB/s wr, 27 op/s 2024-03-30T00:59:26.530 INFO:tasks.workunit.client.0.smithi116.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -pthread -MT mysnprintf.o -MD -MP -MF ".deps/mysnprintf.Tpo" -c -o mysnprintf.o mysnprintf.c; \ 2024-03-30T00:59:26.530 INFO:tasks.workunit.client.0.smithi116.stdout:then mv -f ".deps/mysnprintf.Tpo" ".deps/mysnprintf.Po"; else rm -f ".deps/mysnprintf.Tpo"; exit 1; fi 2024-03-30T00:59:26.591 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -g -O2 -pthread -o blogbench blogbench.o process.o writer.o rewriter.o reader.o commenter.o helpers.o bsd-getopt_long.o fakesnprintf.o mysnprintf.o 2024-03-30T00:59:26.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:26 smithi130 ceph-mon[47626]: pgmap v338: 129 pgs: 129 active+clean; 3.5 MiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 262 KiB/s wr, 27 op/s 2024-03-30T00:59:26.623 INFO:tasks.workunit.client.0.smithi116.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-03-30T00:59:26.624 INFO:tasks.workunit.client.0.smithi116.stdout:Making all in man 2024-03-30T00:59:26.658 INFO:tasks.workunit.client.0.smithi116.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-03-30T00:59:26.658 INFO:tasks.workunit.client.0.smithi116.stdout:make[2]: Nothing to be done for 'all'. 2024-03-30T00:59:26.658 INFO:tasks.workunit.client.0.smithi116.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-03-30T00:59:26.661 INFO:tasks.workunit.client.0.smithi116.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-30T00:59:26.663 INFO:tasks.workunit.client.0.smithi116.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-30T00:59:26.664 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-30T00:59:26.668 INFO:tasks.workunit.client.0.smithi116.stdout:running blogbench 2024-03-30T00:59:26.670 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-03-30T00:59:26.670 INFO:tasks.workunit.client.0.smithi116.stdout:Frequency = 10 secs 2024-03-30T00:59:26.670 INFO:tasks.workunit.client.0.smithi116.stdout:Scratch dir = [blogtest_in] 2024-03-30T00:59:26.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:26 smithi116 ceph-mon[38086]: pgmap v338: 129 pgs: 129 active+clean; 3.5 MiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 262 KiB/s wr, 27 op/s 2024-03-30T00:59:26.877 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T00:59:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.213+0000 7f0d9f200700 1 -- 172.21.15.116:0/1875289652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0da0103fb0 msgr2=0x7f0da01063a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.213+0000 7f0d9f200700 1 --2- 172.21.15.116:0/1875289652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0da0103fb0 0x7f0da01063a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c009990 tx=0x7f0d8c009ca0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.213+0000 7f0d9f200700 1 -- 172.21.15.116:0/1875289652 shutdown_connections 2024-03-30T00:59:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.213+0000 7f0d9f200700 1 --2- 172.21.15.116:0/1875289652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0da01068e0 0x7f0da0108d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.213+0000 7f0d9f200700 1 --2- 172.21.15.116:0/1875289652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0da0101680 0x7f0da0103a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.213+0000 7f0d9f200700 1 --2- 172.21.15.116:0/1875289652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0da0103fb0 0x7f0da01063a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.213+0000 7f0d9f200700 1 -- 172.21.15.116:0/1875289652 >> 172.21.15.116:0/1875289652 conn(0x7f0da00fb070 msgr2=0x7f0da00fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.213+0000 7f0d9f200700 1 -- 172.21.15.116:0/1875289652 shutdown_connections 2024-03-30T00:59:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.213+0000 7f0d9f200700 1 -- 172.21.15.116:0/1875289652 wait complete. 2024-03-30T00:59:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.214+0000 7f0d9f200700 1 Processor -- start 2024-03-30T00:59:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.214+0000 7f0d9f200700 1 -- start start 2024-03-30T00:59:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.214+0000 7f0d9f200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0da0101680 0x7f0da0198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.214+0000 7f0d9f200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0da0103fb0 0x7f0da01992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.214+0000 7f0d9f200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0da01068e0 0x7f0da01a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.214+0000 7f0d9f200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da01a0890 con 0x7f0da01068e0 2024-03-30T00:59:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.214+0000 7f0d9f200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da01a0a00 con 0x7f0da0101680 2024-03-30T00:59:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.214+0000 7f0d9f200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da01a0bc0 con 0x7f0da0103fb0 2024-03-30T00:59:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.215+0000 7f0d9e800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0da01068e0 0x7f0da01a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.215+0000 7f0d9e800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0da01068e0 0x7f0da01a02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59870/0 (socket says 172.21.15.116:59870) 2024-03-30T00:59:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.215+0000 7f0d9e800700 1 -- 172.21.15.116:0/327316278 learned_addr learned my addr 172.21.15.116:0/327316278 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.215+0000 7f0d9d400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0da0103fb0 0x7f0da01992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.215+0000 7f0d9de00700 1 --2- 172.21.15.116:0/327316278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0da0101680 0x7f0da0198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.215+0000 7f0d9d400700 1 -- 172.21.15.116:0/327316278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0da0101680 msgr2=0x7f0da0198da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.215+0000 7f0d9d400700 1 --2- 172.21.15.116:0/327316278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0da0101680 0x7f0da0198da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.215+0000 7f0d9d400700 1 -- 172.21.15.116:0/327316278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0da01068e0 msgr2=0x7f0da01a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.215+0000 7f0d9d400700 1 --2- 172.21.15.116:0/327316278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0da01068e0 0x7f0da01a02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.215+0000 7f0d9d400700 1 -- 172.21.15.116:0/327316278 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c0095f0 con 0x7f0da0103fb0 2024-03-30T00:59:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.215+0000 7f0d9e800700 1 --2- 172.21.15.116:0/327316278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0da01068e0 0x7f0da01a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:59:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.215+0000 7f0d9d400700 1 --2- 172.21.15.116:0/327316278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0da0103fb0 0x7f0da01992e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00b680 tx=0x7f0d8c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.216+0000 7f0d96a00700 1 -- 172.21.15.116:0/327316278 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c01d070 con 0x7f0da0103fb0 2024-03-30T00:59:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.216+0000 7f0d96a00700 1 -- 172.21.15.116:0/327316278 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d8c021410 con 0x7f0da0103fb0 2024-03-30T00:59:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.216+0000 7f0d9f200700 1 -- 172.21.15.116:0/327316278 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da01a0e40 con 0x7f0da0103fb0 2024-03-30T00:59:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.216+0000 7f0d96a00700 1 -- 172.21.15.116:0/327316278 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c017650 con 0x7f0da0103fb0 2024-03-30T00:59:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.216+0000 7f0d9f200700 1 -- 172.21.15.116:0/327316278 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da01a1200 con 0x7f0da0103fb0 2024-03-30T00:59:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.220+0000 7f0d96a00700 1 -- 172.21.15.116:0/327316278 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f0d8c00fb10 con 0x7f0da0103fb0 2024-03-30T00:59:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.221+0000 7f0db87bab80 1 Processor -- start 2024-03-30T00:59:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.221+0000 7f0db87bab80 1 -- start start 2024-03-30T00:59:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.221+0000 7f0db87bab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a23e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.221+0000 7f0db87bab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a24340 0x561fb3a24bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.221+0000 7f0db87bab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a25190 0x561fb3a25610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.221+0000 7f0db87bab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fb397b990 con 0x561fb3a239e0 2024-03-30T00:59:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.221+0000 7f0db87bab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fb3979350 con 0x561fb3a24340 2024-03-30T00:59:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.221+0000 7f0db87bab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fb3974c40 con 0x561fb3a25190 2024-03-30T00:59:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d9e800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a23e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d96a00700 1 --2- 172.21.15.116:0/327316278 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0d840a11a0 0x7f0d840a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d9e800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a23e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59874/0 (socket says 172.21.15.116:59874) 2024-03-30T00:59:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d9de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a24340 0x561fb3a24bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d9e800700 1 -- 172.21.15.116:0/1336373254 learned_addr learned my addr 172.21.15.116:0/1336373254 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d9d400700 1 --2- 172.21.15.116:0/1336373254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a25190 0x561fb3a25610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d96a00700 1 -- 172.21.15.116:0/327316278 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0d8c0cdd70 con 0x7f0da0103fb0 2024-03-30T00:59:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d9de00700 1 -- 172.21.15.116:0/1336373254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a25190 msgr2=0x561fb3a25610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d9e800700 1 --2- 172.21.15.116:0/327316278 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0d840a11a0 0x7f0d840a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d9de00700 1 --2- 172.21.15.116:0/1336373254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a25190 0x561fb3a25610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d9de00700 1 -- 172.21.15.116:0/1336373254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 msgr2=0x561fb3a23e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d9de00700 1 --2- 172.21.15.116:0/1336373254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a23e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d9de00700 1 -- 172.21.15.116:0/1336373254 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fb37f63b0 con 0x561fb3a24340 2024-03-30T00:59:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d9d400700 1 --2- 172.21.15.116:0/1336373254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a25190 0x561fb3a25610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:59:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.222+0000 7f0d9de00700 1 --2- 172.21.15.116:0/1336373254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a24340 0x561fb3a24bb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009c80 tx=0x7f0d90009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e09636bcc972007e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.223+0000 7f0d8b000700 1 -- 172.21.15.116:0/1336373254 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d90011cb0 con 0x561fb3a24340 2024-03-30T00:59:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.223+0000 7f0d8b000700 1 -- 172.21.15.116:0/1336373254 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d90011e10 con 0x561fb3a24340 2024-03-30T00:59:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.223+0000 7f0d8b000700 1 -- 172.21.15.116:0/1336373254 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d90005280 con 0x561fb3a24340 2024-03-30T00:59:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.223+0000 7f0db87bab80 1 -- 172.21.15.116:0/1336373254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a24340 msgr2=0x561fb3a24bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.223+0000 7f0d9e800700 1 --2- 172.21.15.116:0/327316278 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0d840a11a0 0x7f0d840a3660 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0d98007b60 tx=0x7f0d98004b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.223+0000 7f0db87bab80 1 --2- 172.21.15.116:0/1336373254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a24340 0x561fb3a24bb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009c80 tx=0x7f0d90009f90 comp rx=0 tx=0).stop 2024-03-30T00:59:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.223+0000 7f0db87bab80 1 -- 172.21.15.116:0/1336373254 shutdown_connections 2024-03-30T00:59:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.223+0000 7f0db87bab80 1 --2- 172.21.15.116:0/1336373254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a25190 0x561fb3a25610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.223+0000 7f0db87bab80 1 --2- 172.21.15.116:0/1336373254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a23e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.223+0000 7f0db87bab80 1 --2- 172.21.15.116:0/1336373254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a24340 0x561fb3a24bb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.223+0000 7f0db87bab80 1 -- 172.21.15.116:0/1336373254 >> 172.21.15.116:0/1336373254 conn(0x561fb39826a0 msgr2=0x561fb3a1ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.223+0000 7f0db87bab80 1 -- 172.21.15.116:0/1336373254 shutdown_connections 2024-03-30T00:59:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.223+0000 7f0db87bab80 1 -- 172.21.15.116:0/1336373254 wait complete. 2024-03-30T00:59:27.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.224+0000 7f0db87bab80 1 -- start start 2024-03-30T00:59:27.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.224+0000 7f0db87bab80 1 Processor -- start 2024-03-30T00:59:27.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0db87bab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a239e0 0x561fb3a2d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0db87bab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a24340 0x561fb3a2dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0db87bab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 0x561fb3a34dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0db87bab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fb3910d80 con 0x561fb3a24340 2024-03-30T00:59:27.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0db87bab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fb3910a20 con 0x561fb3a25190 2024-03-30T00:59:27.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0db87bab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fb37f86a0 con 0x561fb3a239e0 2024-03-30T00:59:27.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0db87bab80 10 client.0 _ll_register_callbacks cb 0x561fb391a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:59:27.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0d9d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a24340 0x561fb3a2dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0d9d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a24340 0x561fb3a2dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59890/0 (socket says 172.21.15.116:59890) 2024-03-30T00:59:27.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0d9e800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 0x561fb3a34dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0d9d400700 1 -- 172.21.15.116:0/3777675074 learned_addr learned my addr 172.21.15.116:0/3777675074 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0d9de00700 1 --2- 172.21.15.116:0/3777675074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a239e0 0x561fb3a2d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0d9d400700 1 -- 172.21.15.116:0/3777675074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a239e0 msgr2=0x561fb3a2d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0d9d400700 1 --2- 172.21.15.116:0/3777675074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a239e0 0x561fb3a2d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0d9d400700 1 -- 172.21.15.116:0/3777675074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 msgr2=0x561fb3a34dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0d9d400700 1 --2- 172.21.15.116:0/3777675074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 0x561fb3a34dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0d9d400700 1 -- 172.21.15.116:0/3777675074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d900098e0 con 0x561fb3a24340 2024-03-30T00:59:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.225+0000 7f0d9e800700 1 --2- 172.21.15.116:0/3777675074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 0x561fb3a34dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:59:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0d9d400700 1 --2- 172.21.15.116:0/3777675074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a24340 0x561fb3a2dd80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c097950 tx=0x7f0d8c06a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abebd1845e641b57 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0d7f400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:59:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0d7f400700 1 -- 172.21.15.116:0/3777675074 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c09aa20 con 0x561fb3a24340 2024-03-30T00:59:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0d7f400700 1 -- 172.21.15.116:0/3777675074 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0d8c0668d0 con 0x561fb3a24340 2024-03-30T00:59:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0db87bab80 1 -- 172.21.15.116:0/3777675074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561fb3935380 con 0x561fb3a24340 2024-03-30T00:59:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0d7f400700 1 -- 172.21.15.116:0/3777675074 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d8c09a840 con 0x561fb3a24340 2024-03-30T00:59:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0db87bab80 10 client.15702 fetch_fsmap learned FSMap version 23 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0db87bab80 1 -- 172.21.15.116:0/3777675074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561fb37a2b60 con 0x561fb3a24340 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0d7f400700 1 -- 172.21.15.116:0/3777675074 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561fb3935380 con 0x561fb3a24340 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0d7f400700 1 -- 172.21.15.116:0/3777675074 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0d8c0aed20 con 0x561fb3a24340 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0db87bab80 10 client.15702 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0db87bab80 10 client.15702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0db87bab80 20 client.15702 populate_metadata read hostname 'smithi116' 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0db87bab80 1 --2- 172.21.15.116:0/3777675074 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561fb3a39230 0x561fb3a3b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0db87bab80 4 client.15702 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.226+0000 7f0db87bab80 1 -- 172.21.15.116:0/3777675074 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561fb396d700 con 0x561fb3a39230 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.227+0000 7f0d9de00700 1 --2- 172.21.15.116:0/3777675074 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561fb3a39230 0x561fb3a3b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.227+0000 7f0d7f400700 10 client.15702 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.227+0000 7f0d9de00700 1 --2- 172.21.15.116:0/3777675074 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561fb3a39230 0x561fb3a3b620 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79789abd540d781f server_cookie=b69d4f2348fad345 in_seq=0 out_seq=0 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0d7f400700 1 -- 172.21.15.116:0/3777675074 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561fb396d700 con 0x561fb3a39230 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0d7f400700 10 client.15702 handle_command_reply: tid=0 2024-03-30T00:59:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0db87bab80 1 client.15702 shutdown 2024-03-30T00:59:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0db87bab80 1 -- 172.21.15.116:0/3777675074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a24340 msgr2=0x561fb3a2dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0db87bab80 1 --2- 172.21.15.116:0/3777675074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a24340 0x561fb3a2dd80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c097950 tx=0x7f0d8c06a1f0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0db87bab80 1 -- 172.21.15.116:0/3777675074 shutdown_connections 2024-03-30T00:59:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0db87bab80 1 --2- 172.21.15.116:0/3777675074 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561fb3a39230 0x561fb3a3b620 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0db87bab80 1 --2- 172.21.15.116:0/3777675074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a239e0 0x561fb3a2d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0db87bab80 1 --2- 172.21.15.116:0/3777675074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 0x561fb3a34dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0d7f400700 0 client.15702 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0db87bab80 1 --2- 172.21.15.116:0/3777675074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a24340 0x561fb3a2dd80 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0db87bab80 1 -- 172.21.15.116:0/3777675074 >> 172.21.15.116:0/3777675074 conn(0x561fb39826a0 msgr2=0x561fb3a223c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0db87bab80 1 -- 172.21.15.116:0/3777675074 shutdown_connections 2024-03-30T00:59:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0db87bab80 1 -- 172.21.15.116:0/3777675074 wait complete. 2024-03-30T00:59:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.228+0000 7f0db87bab80 20 client.15702 trim_cache size 0 max 16384 2024-03-30T00:59:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.246+0000 7f0db87bab80 1 Processor -- start 2024-03-30T00:59:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.246+0000 7f0db87bab80 1 -- start start 2024-03-30T00:59:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.246+0000 7f0db87bab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a2cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.246+0000 7f0db87bab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a24340 0x561fb3a34d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.246+0000 7f0db87bab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 0x561fb3a2b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.246+0000 7f0db87bab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fb3a35f40 con 0x561fb3a239e0 2024-03-30T00:59:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.246+0000 7f0db87bab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fb3a2c370 con 0x561fb3a25190 2024-03-30T00:59:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.246+0000 7f0db87bab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fb3a1f640 con 0x561fb3a24340 2024-03-30T00:59:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.246+0000 7f0d9de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a2cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.246+0000 7f0d9de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a2cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59896/0 (socket says 172.21.15.116:59896) 2024-03-30T00:59:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.246+0000 7f0d9de00700 1 -- 172.21.15.116:0/4041269529 learned_addr learned my addr 172.21.15.116:0/4041269529 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.246+0000 7f0d9e800700 1 --2- 172.21.15.116:0/4041269529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 0x561fb3a2b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.246+0000 7f0d9d400700 1 --2- 172.21.15.116:0/4041269529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a24340 0x561fb3a34d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.247+0000 7f0d9de00700 1 -- 172.21.15.116:0/4041269529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a24340 msgr2=0x561fb3a34d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.247+0000 7f0d9de00700 1 --2- 172.21.15.116:0/4041269529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a24340 0x561fb3a34d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.247+0000 7f0d9de00700 1 -- 172.21.15.116:0/4041269529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 msgr2=0x561fb3a2b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.247+0000 7f0d9de00700 1 --2- 172.21.15.116:0/4041269529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 0x561fb3a2b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.247+0000 7f0d9de00700 1 -- 172.21.15.116:0/4041269529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fb393aea0 con 0x561fb3a239e0 2024-03-30T00:59:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.247+0000 7f0d9de00700 1 --2- 172.21.15.116:0/4041269529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a2cd50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0d900049c0 tx=0x7f0d90012fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1519492009bada8b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.247+0000 7f0d8ba00700 1 -- 172.21.15.116:0/4041269529 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9002c070 con 0x561fb3a239e0 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.247+0000 7f0d8ba00700 1 -- 172.21.15.116:0/4041269529 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d90011ab0 con 0x561fb3a239e0 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.247+0000 7f0d8ba00700 1 -- 172.21.15.116:0/4041269529 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d90028540 con 0x561fb3a239e0 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.248+0000 7f0db87bab80 1 -- 172.21.15.116:0/4041269529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 msgr2=0x561fb3a2cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.248+0000 7f0db87bab80 1 --2- 172.21.15.116:0/4041269529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a2cd50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0d900049c0 tx=0x7f0d90012fd0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.248+0000 7f0db87bab80 1 -- 172.21.15.116:0/4041269529 shutdown_connections 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.248+0000 7f0db87bab80 1 --2- 172.21.15.116:0/4041269529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 0x561fb3a2b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.248+0000 7f0db87bab80 1 --2- 172.21.15.116:0/4041269529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a2cd50 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.248+0000 7f0db87bab80 1 --2- 172.21.15.116:0/4041269529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a24340 0x561fb3a34d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.248+0000 7f0db87bab80 1 -- 172.21.15.116:0/4041269529 >> 172.21.15.116:0/4041269529 conn(0x561fb39826a0 msgr2=0x561fb3a2c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.248+0000 7f0db87bab80 1 -- 172.21.15.116:0/4041269529 shutdown_connections 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.248+0000 7f0db87bab80 1 -- 172.21.15.116:0/4041269529 wait complete. 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.248+0000 7f0db87bab80 1 -- start start 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.249+0000 7f0db87bab80 1 Processor -- start 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.249+0000 7f0db87bab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a347f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.249+0000 7f0db87bab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a24340 0x561fb3a34d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.249+0000 7f0db87bab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 0x561fb3a32f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.249+0000 7f0db87bab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fb3a33620 con 0x561fb3a239e0 2024-03-30T00:59:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.249+0000 7f0db87bab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fb3a33790 con 0x561fb3a25190 2024-03-30T00:59:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.249+0000 7f0db87bab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fb3a33950 con 0x561fb3a24340 2024-03-30T00:59:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.249+0000 7f0db87bab80 10 client.0 _ll_register_callbacks cb 0x561fb391a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:59:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.249+0000 7f0d9de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a347f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.249+0000 7f0d9de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a347f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59898/0 (socket says 172.21.15.116:59898) 2024-03-30T00:59:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.249+0000 7f0d9de00700 1 -- 172.21.15.116:0/483949692 learned_addr learned my addr 172.21.15.116:0/483949692 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.249+0000 7f0d9e800700 1 --2- 172.21.15.116:0/483949692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 0x561fb3a32f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.249+0000 7f0d9d400700 1 --2- 172.21.15.116:0/483949692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a24340 0x561fb3a34d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.250+0000 7f0d9de00700 1 -- 172.21.15.116:0/483949692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a24340 msgr2=0x561fb3a34d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.250+0000 7f0d9de00700 1 --2- 172.21.15.116:0/483949692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a24340 0x561fb3a34d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.250+0000 7f0d9de00700 1 -- 172.21.15.116:0/483949692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 msgr2=0x561fb3a32f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.250+0000 7f0d9de00700 1 --2- 172.21.15.116:0/483949692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 0x561fb3a32f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.250+0000 7f0d9de00700 1 -- 172.21.15.116:0/483949692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9001d040 con 0x561fb3a239e0 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.250+0000 7f0d9e800700 1 --2- 172.21.15.116:0/483949692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 0x561fb3a32f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.250+0000 7f0d9de00700 1 --2- 172.21.15.116:0/483949692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a347f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009290 tx=0x7f0d900092c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=360df0c1c23b98d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.250+0000 7f0d7f400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.250+0000 7f0d7f400700 1 -- 172.21.15.116:0/483949692 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9002c070 con 0x561fb3a239e0 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.250+0000 7f0d7f400700 1 -- 172.21.15.116:0/483949692 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0d90011c20 con 0x561fb3a239e0 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.250+0000 7f0db87bab80 1 -- 172.21.15.116:0/483949692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0d90011e10 con 0x561fb3a239e0 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.250+0000 7f0d7f400700 1 -- 172.21.15.116:0/483949692 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d900284c0 con 0x561fb3a239e0 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.250+0000 7f0db87bab80 10 client.15708 fetch_fsmap learned FSMap version 23 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.250+0000 7f0db87bab80 1 -- 172.21.15.116:0/483949692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561fb37facd0 con 0x561fb3a239e0 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.251+0000 7f0d7f400700 1 -- 172.21.15.116:0/483949692 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d90011e10 con 0x561fb3a239e0 2024-03-30T00:59:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.251+0000 7f0d7f400700 1 -- 172.21.15.116:0/483949692 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0d90003680 con 0x561fb3a239e0 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.251+0000 7f0db87bab80 10 client.15708 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.251+0000 7f0db87bab80 10 client.15708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.251+0000 7f0db87bab80 20 client.15708 populate_metadata read hostname 'smithi116' 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.251+0000 7f0db87bab80 1 --2- 172.21.15.116:0/483949692 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561fb3a8e0d0 0x561fb3a904f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.251+0000 7f0db87bab80 4 client.15708 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.251+0000 7f0db87bab80 1 -- 172.21.15.116:0/483949692 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561fb37c3cf0 con 0x561fb3a8e0d0 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.251+0000 7f0d9d400700 1 --2- 172.21.15.116:0/483949692 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561fb3a8e0d0 0x561fb3a904f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0d9d400700 1 --2- 172.21.15.116:0/483949692 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561fb3a8e0d0 0x561fb3a904f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be8a7826ef8580e9 server_cookie=2bbf2958f76b2ee6 in_seq=0 out_seq=0 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0d7f400700 10 client.15708 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0d7f400700 1 -- 172.21.15.116:0/483949692 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561fb37c3cf0 con 0x561fb3a8e0d0 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0d7f400700 10 client.15708 handle_command_reply: tid=0 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0db87bab80 1 client.15708 shutdown 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0db87bab80 1 -- 172.21.15.116:0/483949692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 msgr2=0x561fb3a347f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0db87bab80 1 --2- 172.21.15.116:0/483949692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a347f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009290 tx=0x7f0d900092c0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0db87bab80 1 -- 172.21.15.116:0/483949692 shutdown_connections 2024-03-30T00:59:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0db87bab80 1 --2- 172.21.15.116:0/483949692 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561fb3a8e0d0 0x561fb3a904f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0db87bab80 1 --2- 172.21.15.116:0/483949692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fb3a25190 0x561fb3a32f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0db87bab80 1 --2- 172.21.15.116:0/483949692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fb3a239e0 0x561fb3a347f0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0db87bab80 1 --2- 172.21.15.116:0/483949692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561fb3a24340 0x561fb3a34d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0d7f400700 0 client.15708 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0db87bab80 1 -- 172.21.15.116:0/483949692 >> 172.21.15.116:0/483949692 conn(0x561fb39826a0 msgr2=0x561fb3a2cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0db87bab80 1 -- 172.21.15.116:0/483949692 shutdown_connections 2024-03-30T00:59:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.252+0000 7f0db87bab80 1 -- 172.21.15.116:0/483949692 wait complete. 2024-03-30T00:59:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.253+0000 7f0db87bab80 20 client.15708 trim_cache size 0 max 16384 2024-03-30T00:59:27.265 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:59:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-30T00:59:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-30T00:59:27.266 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:59:27.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.253+0000 7f0d9f200700 1 -- 172.21.15.116:0/327316278 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0d840a11a0 msgr2=0x7f0d840a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.253+0000 7f0d9f200700 1 --2- 172.21.15.116:0/327316278 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0d840a11a0 0x7f0d840a3660 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0d98007b60 tx=0x7f0d98004b40 comp rx=0 tx=0).stop 2024-03-30T00:59:27.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.253+0000 7f0d9f200700 1 -- 172.21.15.116:0/327316278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0da0103fb0 msgr2=0x7f0da01992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.253+0000 7f0d9f200700 1 --2- 172.21.15.116:0/327316278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0da0103fb0 0x7f0da01992e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00b680 tx=0x7f0d8c004730 comp rx=0 tx=0).stop 2024-03-30T00:59:27.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.253+0000 7f0d9f200700 1 -- 172.21.15.116:0/327316278 shutdown_connections 2024-03-30T00:59:27.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.253+0000 7f0d9f200700 1 --2- 172.21.15.116:0/327316278 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0d840a11a0 0x7f0d840a3660 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.253+0000 7f0d9f200700 1 --2- 172.21.15.116:0/327316278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0da01068e0 0x7f0da01a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.253+0000 7f0d9f200700 1 --2- 172.21.15.116:0/327316278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0da0101680 0x7f0da0198da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.253+0000 7f0d9f200700 1 --2- 172.21.15.116:0/327316278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0da0103fb0 0x7f0da01992e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.253+0000 7f0d9f200700 1 -- 172.21.15.116:0/327316278 >> 172.21.15.116:0/327316278 conn(0x7f0da00fb070 msgr2=0x7f0da0100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:27.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.253+0000 7f0d9f200700 1 -- 172.21.15.116:0/327316278 shutdown_connections 2024-03-30T00:59:27.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.253+0000 7f0d9f200700 1 -- 172.21.15.116:0/327316278 wait complete. 2024-03-30T00:59:27.277 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T00:59:27.278 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T00:59:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.464+0000 7fcf12800700 1 -- 172.21.15.116:0/2852860823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf0c0737b0 msgr2=0x7fcf0c073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.464+0000 7fcf12800700 1 --2- 172.21.15.116:0/2852860823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf0c0737b0 0x7fcf0c073c30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcef8009990 tx=0x7fcef8009ca0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.464+0000 7fcf12800700 1 -- 172.21.15.116:0/2852860823 shutdown_connections 2024-03-30T00:59:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.464+0000 7fcf12800700 1 --2- 172.21.15.116:0/2852860823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf0c108b60 0x7fcf0c10afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.464+0000 7fcf12800700 1 --2- 172.21.15.116:0/2852860823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf0c074d90 0x7fcf0c0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.464+0000 7fcf12800700 1 --2- 172.21.15.116:0/2852860823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf0c0737b0 0x7fcf0c073c30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.464+0000 7fcf12800700 1 -- 172.21.15.116:0/2852860823 >> 172.21.15.116:0/2852860823 conn(0x7fcf0c0fc200 msgr2=0x7fcf0c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.465+0000 7fcf12800700 1 -- 172.21.15.116:0/2852860823 shutdown_connections 2024-03-30T00:59:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.465+0000 7fcf12800700 1 -- 172.21.15.116:0/2852860823 wait complete. 2024-03-30T00:59:27.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.465+0000 7fcf12800700 1 Processor -- start 2024-03-30T00:59:27.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.466+0000 7fcf12800700 1 -- start start 2024-03-30T00:59:27.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.466+0000 7fcf12800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf0c0737b0 0x7fcf0c198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.466+0000 7fcf12800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf0c074d90 0x7fcf0c199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.466+0000 7fcf12800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf0c108b60 0x7fcf0c1a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.466+0000 7fcf12800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c1a0930 con 0x7fcf0c108b60 2024-03-30T00:59:27.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.466+0000 7fcf12800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c1a0a70 con 0x7fcf0c074d90 2024-03-30T00:59:27.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.466+0000 7fcf12800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c1a0c00 con 0x7fcf0c0737b0 2024-03-30T00:59:27.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.466+0000 7fcf0be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf0c108b60 0x7fcf0c1a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.466+0000 7fcf0be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf0c108b60 0x7fcf0c1a0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59922/0 (socket says 172.21.15.116:59922) 2024-03-30T00:59:27.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.466+0000 7fcf0be00700 1 -- 172.21.15.116:0/2305724629 learned_addr learned my addr 172.21.15.116:0/2305724629 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:27.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.466+0000 7fcf0aa00700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf0c074d90 0x7fcf0c199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.466+0000 7fcf0b400700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf0c0737b0 0x7fcf0c198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.467+0000 7fcf0be00700 1 -- 172.21.15.116:0/2305724629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf0c0737b0 msgr2=0x7fcf0c198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.467+0000 7fcf0be00700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf0c0737b0 0x7fcf0c198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.467+0000 7fcf0be00700 1 -- 172.21.15.116:0/2305724629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf0c074d90 msgr2=0x7fcf0c199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.467+0000 7fcf0be00700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf0c074d90 0x7fcf0c199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.467+0000 7fcf0be00700 1 -- 172.21.15.116:0/2305724629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef80095f0 con 0x7fcf0c108b60 2024-03-30T00:59:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.467+0000 7fcf0aa00700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf0c074d90 0x7fcf0c199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:59:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.467+0000 7fcf0be00700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf0c108b60 0x7fcf0c1a0360 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fcf04007ee0 tx=0x7fcf0400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.467+0000 7fcf03e00700 1 -- 172.21.15.116:0/2305724629 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0400dd20 con 0x7fcf0c108b60 2024-03-30T00:59:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.467+0000 7fcf03e00700 1 -- 172.21.15.116:0/2305724629 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf0400f040 con 0x7fcf0c108b60 2024-03-30T00:59:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.467+0000 7fcf12800700 1 -- 172.21.15.116:0/2305724629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0c1a0ee0 con 0x7fcf0c108b60 2024-03-30T00:59:27.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.467+0000 7fcf03e00700 1 -- 172.21.15.116:0/2305724629 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf040048a0 con 0x7fcf0c108b60 2024-03-30T00:59:27.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.467+0000 7fcf12800700 1 -- 172.21.15.116:0/2305724629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0c1a1360 con 0x7fcf0c108b60 2024-03-30T00:59:27.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.468+0000 7fcf12800700 1 -- 172.21.15.116:0/2305724629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0c107970 con 0x7fcf0c108b60 2024-03-30T00:59:27.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.470+0000 7fcf03e00700 1 -- 172.21.15.116:0/2305724629 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fcf04003a00 con 0x7fcf0c108b60 2024-03-30T00:59:27.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.471+0000 7fcf03e00700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcef00a1120 0x7fcef00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.472+0000 7fcf03e00700 1 -- 172.21.15.116:0/2305724629 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcf040cc460 con 0x7fcf0c108b60 2024-03-30T00:59:27.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.472+0000 7fcf0b400700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcef00a1120 0x7fcef00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.472+0000 7fcf0b400700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcef00a1120 0x7fcef00a35e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcefc009c60 tx=0x7fcefc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:27.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.476+0000 7fcf03e00700 1 -- 172.21.15.116:0/2305724629 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcf04095880 con 0x7fcf0c108b60 2024-03-30T00:59:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.729+0000 7fcf12800700 1 -- 172.21.15.116:0/2305724629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf0c067a10 con 0x7fcf0c108b60 2024-03-30T00:59:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.729+0000 7fcf03e00700 1 -- 172.21.15.116:0/2305724629 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10886 (secure 0 0 0) 0x7fcf04098ea0 con 0x7fcf0c108b60 2024-03-30T00:59:27.736 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T00:59:27.736 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:57:40.442321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T00:59:27.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.732+0000 7fcf12800700 1 -- 172.21.15.116:0/2305724629 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcef00a1120 msgr2=0x7fcef00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.732+0000 7fcf12800700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcef00a1120 0x7fcef00a35e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcefc009c60 tx=0x7fcefc009260 comp rx=0 tx=0).stop 2024-03-30T00:59:27.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.732+0000 7fcf12800700 1 -- 172.21.15.116:0/2305724629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf0c108b60 msgr2=0x7fcf0c1a0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.732+0000 7fcf12800700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf0c108b60 0x7fcf0c1a0360 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fcf04007ee0 tx=0x7fcf0400d3b0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.732+0000 7fcf12800700 1 -- 172.21.15.116:0/2305724629 shutdown_connections 2024-03-30T00:59:27.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.732+0000 7fcf12800700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcef00a1120 0x7fcef00a35e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.732+0000 7fcf12800700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf0c0737b0 0x7fcf0c198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.732+0000 7fcf12800700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf0c108b60 0x7fcf0c1a0360 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.732+0000 7fcf12800700 1 --2- 172.21.15.116:0/2305724629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf0c074d90 0x7fcf0c199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.732+0000 7fcf12800700 1 -- 172.21.15.116:0/2305724629 >> 172.21.15.116:0/2305724629 conn(0x7fcf0c0fc200 msgr2=0x7fcf0c104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:27.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.732+0000 7fcf12800700 1 -- 172.21.15.116:0/2305724629 shutdown_connections 2024-03-30T00:59:27.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.732+0000 7fcf12800700 1 -- 172.21.15.116:0/2305724629 wait complete. 2024-03-30T00:59:27.740 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 23 2024-03-30T00:59:27.759 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T00:59:27.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.965+0000 7f1481400700 1 -- 172.21.15.116:0/725497933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f147c074df0 msgr2=0x7f147c073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.965+0000 7f1481400700 1 --2- 172.21.15.116:0/725497933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f147c074df0 0x7f147c073240 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f146c009990 tx=0x7f146c009ca0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.965+0000 7f1481400700 1 -- 172.21.15.116:0/725497933 shutdown_connections 2024-03-30T00:59:27.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.965+0000 7f1481400700 1 --2- 172.21.15.116:0/725497933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f147c108c40 0x7f147c10b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.965+0000 7f1481400700 1 --2- 172.21.15.116:0/725497933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f147c074df0 0x7f147c073240 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.965+0000 7f1481400700 1 --2- 172.21.15.116:0/725497933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f147c073810 0x7f147c073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.965+0000 7f1481400700 1 -- 172.21.15.116:0/725497933 >> 172.21.15.116:0/725497933 conn(0x7f147c0fc2f0 msgr2=0x7f147c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:27.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.965+0000 7f1481400700 1 -- 172.21.15.116:0/725497933 shutdown_connections 2024-03-30T00:59:27.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.965+0000 7f1481400700 1 -- 172.21.15.116:0/725497933 wait complete. 2024-03-30T00:59:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.966+0000 7f1481400700 1 Processor -- start 2024-03-30T00:59:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.966+0000 7f1481400700 1 -- start start 2024-03-30T00:59:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f1481400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f147c073810 0x7f147c198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f1481400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f147c074df0 0x7f147c199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f1481400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f147c108c40 0x7f147c1a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f1481400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c1a0a00 con 0x7f147c073810 2024-03-30T00:59:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f1481400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c1a0b40 con 0x7f147c108c40 2024-03-30T00:59:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f1481400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c1a0cd0 con 0x7f147c074df0 2024-03-30T00:59:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f147a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f147c073810 0x7f147c198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f147a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f147c073810 0x7f147c198f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59944/0 (socket says 172.21.15.116:59944) 2024-03-30T00:59:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f147a800700 1 -- 172.21.15.116:0/3812520825 learned_addr learned my addr 172.21.15.116:0/3812520825 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f1479e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f147c074df0 0x7f147c199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f147b200700 1 --2- 172.21.15.116:0/3812520825 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f147c108c40 0x7f147c1a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f147a800700 1 -- 172.21.15.116:0/3812520825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f147c074df0 msgr2=0x7f147c199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f147a800700 1 --2- 172.21.15.116:0/3812520825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f147c074df0 0x7f147c199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f147a800700 1 -- 172.21.15.116:0/3812520825 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f147c108c40 msgr2=0x7f147c1a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f147a800700 1 --2- 172.21.15.116:0/3812520825 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f147c108c40 0x7f147c1a0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f147a800700 1 -- 172.21.15.116:0/3812520825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c0095f0 con 0x7f147c073810 2024-03-30T00:59:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.967+0000 7f1479e00700 1 --2- 172.21.15.116:0/3812520825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f147c074df0 0x7f147c199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:59:27.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.968+0000 7f147a800700 1 --2- 172.21.15.116:0/3812520825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f147c073810 0x7f147c198f20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f146c0098e0 tx=0x7f146c005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:27.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.968+0000 7f1473400700 1 -- 172.21.15.116:0/3812520825 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c01d070 con 0x7f147c073810 2024-03-30T00:59:27.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.968+0000 7f1481400700 1 -- 172.21.15.116:0/3812520825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147c1a0f50 con 0x7f147c073810 2024-03-30T00:59:27.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.968+0000 7f1473400700 1 -- 172.21.15.116:0/3812520825 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f146c022470 con 0x7f147c073810 2024-03-30T00:59:27.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.968+0000 7f1473400700 1 -- 172.21.15.116:0/3812520825 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c017650 con 0x7f147c073810 2024-03-30T00:59:27.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.968+0000 7f1481400700 1 -- 172.21.15.116:0/3812520825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c1a1310 con 0x7f147c073810 2024-03-30T00:59:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.971+0000 7f1473400700 1 -- 172.21.15.116:0/3812520825 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f146c0225e0 con 0x7f147c073810 2024-03-30T00:59:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.972+0000 7f1473400700 1 --2- 172.21.15.116:0/3812520825 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14600a11a0 0x7f14600a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.972+0000 7f1494f6ab80 1 Processor -- start 2024-03-30T00:59:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.973+0000 7f1494f6ab80 1 -- start start 2024-03-30T00:59:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.973+0000 7f1473400700 1 -- 172.21.15.116:0/3812520825 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f146c0cdef0 con 0x7f147c073810 2024-03-30T00:59:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.973+0000 7f1479e00700 1 --2- 172.21.15.116:0/3812520825 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14600a11a0 0x7f14600a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.973+0000 7f1494f6ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245d7d0 0x55600245dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.973+0000 7f1494f6ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245e130 0x55600245e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.973+0000 7f1494f6ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 0x55600245f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.973+0000 7f1494f6ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560023b5990 con 0x55600245d7d0 2024-03-30T00:59:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.973+0000 7f1494f6ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560023b3350 con 0x55600245efd0 2024-03-30T00:59:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.973+0000 7f1494f6ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560023aec40 con 0x55600245e130 2024-03-30T00:59:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.973+0000 7f147a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245d7d0 0x55600245dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.973+0000 7f147a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245d7d0 0x55600245dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59948/0 (socket says 172.21.15.116:59948) 2024-03-30T00:59:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.973+0000 7f147a800700 1 -- 172.21.15.116:0/3926214381 learned_addr learned my addr 172.21.15.116:0/3926214381 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.973+0000 7f147b200700 1 --2- 172.21.15.116:0/3926214381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 0x55600245f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.973+0000 7f1479e00700 1 --2- 172.21.15.116:0/3926214381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245e130 0x55600245e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.974+0000 7f1479e00700 1 --2- 172.21.15.116:0/3812520825 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14600a11a0 0x7f14600a3660 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f147c074b10 tx=0x7f1468011830 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.974+0000 7f147b200700 1 -- 172.21.15.116:0/3926214381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245e130 msgr2=0x55600245e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.974+0000 7f147b200700 1 --2- 172.21.15.116:0/3926214381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245e130 0x55600245e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.974+0000 7f147b200700 1 -- 172.21.15.116:0/3926214381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245d7d0 msgr2=0x55600245dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.974+0000 7f147b200700 1 --2- 172.21.15.116:0/3926214381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245d7d0 0x55600245dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.974+0000 7f147b200700 1 -- 172.21.15.116:0/3926214381 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560022303b0 con 0x55600245efd0 2024-03-30T00:59:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.974+0000 7f147a800700 1 --2- 172.21.15.116:0/3926214381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245d7d0 0x55600245dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:59:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.974+0000 7f147b200700 1 --2- 172.21.15.116:0/3926214381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 0x55600245f450 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1474011580 tx=0x7f1474009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6ee8ed6740fd656 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.974+0000 7f1467400700 1 -- 172.21.15.116:0/3926214381 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147400e040 con 0x55600245efd0 2024-03-30T00:59:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.974+0000 7f1467400700 1 -- 172.21.15.116:0/3926214381 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f147400f040 con 0x55600245efd0 2024-03-30T00:59:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.974+0000 7f1467400700 1 -- 172.21.15.116:0/3926214381 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14740044f0 con 0x55600245efd0 2024-03-30T00:59:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.975+0000 7f1494f6ab80 1 -- 172.21.15.116:0/3926214381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 msgr2=0x55600245f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.975+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/3926214381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 0x55600245f450 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1474011580 tx=0x7f1474009920 comp rx=0 tx=0).stop 2024-03-30T00:59:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.975+0000 7f1494f6ab80 1 -- 172.21.15.116:0/3926214381 shutdown_connections 2024-03-30T00:59:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.975+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/3926214381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 0x55600245f450 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.975+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/3926214381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245d7d0 0x55600245dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.975+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/3926214381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245e130 0x55600245e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.975+0000 7f1494f6ab80 1 -- 172.21.15.116:0/3926214381 >> 172.21.15.116:0/3926214381 conn(0x5560023bc6a0 msgr2=0x5560024589a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.975+0000 7f1494f6ab80 1 -- 172.21.15.116:0/3926214381 shutdown_connections 2024-03-30T00:59:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.975+0000 7f1494f6ab80 1 -- 172.21.15.116:0/3926214381 wait complete. 2024-03-30T00:59:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.976+0000 7f1494f6ab80 1 -- start start 2024-03-30T00:59:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.976+0000 7f1494f6ab80 1 Processor -- start 2024-03-30T00:59:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.976+0000 7f1494f6ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245d7d0 0x556002467710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.976+0000 7f1494f6ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245e130 0x556002467c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.976+0000 7f1494f6ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 0x55600246ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.976+0000 7f1494f6ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55600234ad80 con 0x55600245e130 2024-03-30T00:59:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.976+0000 7f1494f6ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55600234aa20 con 0x55600245efd0 2024-03-30T00:59:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.976+0000 7f1494f6ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560022326a0 con 0x55600245d7d0 2024-03-30T00:59:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.976+0000 7f1494f6ab80 10 client.0 _ll_register_callbacks cb 0x556002354470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:59:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f1479e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245e130 0x556002467c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f1479e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245e130 0x556002467c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59952/0 (socket says 172.21.15.116:59952) 2024-03-30T00:59:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f1479e00700 1 -- 172.21.15.116:0/156853688 learned_addr learned my addr 172.21.15.116:0/156853688 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f147b200700 1 --2- 172.21.15.116:0/156853688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 0x55600246ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f147a800700 1 --2- 172.21.15.116:0/156853688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245d7d0 0x556002467710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f1479e00700 1 -- 172.21.15.116:0/156853688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245d7d0 msgr2=0x556002467710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f1479e00700 1 --2- 172.21.15.116:0/156853688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245d7d0 0x556002467710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f1479e00700 1 -- 172.21.15.116:0/156853688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 msgr2=0x55600246ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f1479e00700 1 --2- 172.21.15.116:0/156853688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 0x55600246ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f1479e00700 1 -- 172.21.15.116:0/156853688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c1a1310 con 0x55600245e130 2024-03-30T00:59:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f1479e00700 1 --2- 172.21.15.116:0/156853688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245e130 0x556002467c50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f146800fb20 tx=0x7f14680047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1925c28397ae57d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f145be00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:59:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f145be00700 1 -- 172.21.15.116:0/156853688 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468022070 con 0x55600245e130 2024-03-30T00:59:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f145be00700 1 -- 172.21.15.116:0/156853688 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f146801da60 con 0x55600245e130 2024-03-30T00:59:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.977+0000 7f1494f6ab80 1 -- 172.21.15.116:0/156853688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55600236f380 con 0x55600245e130 2024-03-30T00:59:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.978+0000 7f145be00700 1 -- 172.21.15.116:0/156853688 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f146801dc60 con 0x55600245e130 2024-03-30T00:59:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.978+0000 7f1494f6ab80 10 client.15726 fetch_fsmap learned FSMap version 23 2024-03-30T00:59:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.978+0000 7f1494f6ab80 1 -- 172.21.15.116:0/156853688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f1474011230 con 0x55600245e130 2024-03-30T00:59:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.978+0000 7f145be00700 1 -- 172.21.15.116:0/156853688 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55600236f380 con 0x55600245e130 2024-03-30T00:59:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.978+0000 7f145be00700 1 -- 172.21.15.116:0/156853688 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f146800e3b0 con 0x55600245e130 2024-03-30T00:59:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.978+0000 7f1494f6ab80 10 client.15726 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:59:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.978+0000 7f1494f6ab80 10 client.15726 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T00:59:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.978+0000 7f1494f6ab80 20 client.15726 populate_metadata read hostname 'smithi116' 2024-03-30T00:59:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.978+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/156853688 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x556002473580 0x5560024759a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.978+0000 7f1494f6ab80 4 client.15726 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:59:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.978+0000 7f1494f6ab80 1 -- 172.21.15.116:0/156853688 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5560023a7700 con 0x556002473580 2024-03-30T00:59:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.979+0000 7f147a800700 1 --2- 172.21.15.116:0/156853688 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x556002473580 0x5560024759a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.979+0000 7f147a800700 1 --2- 172.21.15.116:0/156853688 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x556002473580 0x5560024759a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1bcfda7466df215b server_cookie=dc69fdff73a09cef in_seq=0 out_seq=0 2024-03-30T00:59:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.979+0000 7f145be00700 10 client.15726 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T00:59:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.980+0000 7f145be00700 1 -- 172.21.15.116:0/156853688 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5560023a7700 con 0x556002473580 2024-03-30T00:59:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.980+0000 7f145be00700 10 client.15726 handle_command_reply: tid=0 2024-03-30T00:59:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.980+0000 7f1494f6ab80 1 client.15726 shutdown 2024-03-30T00:59:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.980+0000 7f1494f6ab80 1 -- 172.21.15.116:0/156853688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245e130 msgr2=0x556002467c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.980+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/156853688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245e130 0x556002467c50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f146800fb20 tx=0x7f14680047c0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.980+0000 7f1494f6ab80 1 -- 172.21.15.116:0/156853688 shutdown_connections 2024-03-30T00:59:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.980+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/156853688 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x556002473580 0x5560024759a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.980+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/156853688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245d7d0 0x556002467710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.980+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/156853688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 0x55600246ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.980+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/156853688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245e130 0x556002467c50 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:27.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.980+0000 7f1494f6ab80 1 -- 172.21.15.116:0/156853688 >> 172.21.15.116:0/156853688 conn(0x5560023bc6a0 msgr2=0x55600245c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:27.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.980+0000 7f145be00700 0 client.15726 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T00:59:27.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.981+0000 7f1494f6ab80 1 -- 172.21.15.116:0/156853688 shutdown_connections 2024-03-30T00:59:27.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.981+0000 7f1494f6ab80 1 -- 172.21.15.116:0/156853688 wait complete. 2024-03-30T00:59:27.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.981+0000 7f1494f6ab80 20 client.15726 trim_cache size 0 max 16384 2024-03-30T00:59:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.994+0000 7f1494f6ab80 1 Processor -- start 2024-03-30T00:59:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.994+0000 7f1494f6ab80 1 -- start start 2024-03-30T00:59:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f1494f6ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245d7d0 0x556002466c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f1494f6ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245e130 0x55600246f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f1494f6ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 0x55600246ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f1494f6ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556002467280 con 0x55600245d7d0 2024-03-30T00:59:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f1494f6ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55600245df70 con 0x55600245efd0 2024-03-30T00:59:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f1494f6ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55600245d370 con 0x55600245e130 2024-03-30T00:59:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f147a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245d7d0 0x556002466c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f147a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245d7d0 0x556002466c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59964/0 (socket says 172.21.15.116:59964) 2024-03-30T00:59:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f147a800700 1 -- 172.21.15.116:0/3609259903 learned_addr learned my addr 172.21.15.116:0/3609259903 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f1479e00700 1 --2- 172.21.15.116:0/3609259903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245e130 0x55600246f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f147b200700 1 --2- 172.21.15.116:0/3609259903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 0x55600246ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f147a800700 1 -- 172.21.15.116:0/3609259903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245e130 msgr2=0x55600246f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f147a800700 1 --2- 172.21.15.116:0/3609259903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245e130 0x55600246f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f147a800700 1 -- 172.21.15.116:0/3609259903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 msgr2=0x55600246ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f147a800700 1 --2- 172.21.15.116:0/3609259903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 0x55600246ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.995+0000 7f147a800700 1 -- 172.21.15.116:0/3609259903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560021dcb60 con 0x55600245d7d0 2024-03-30T00:59:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.996+0000 7f147a800700 1 --2- 172.21.15.116:0/3609259903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245d7d0 0x556002466c20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f146c09adf0 tx=0x7f146c0a1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327347e4131d2af4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.996+0000 7f1467e00700 1 -- 172.21.15.116:0/3609259903 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c0a34e0 con 0x55600245d7d0 2024-03-30T00:59:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.996+0000 7f1467e00700 1 -- 172.21.15.116:0/3609259903 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f146c099230 con 0x55600245d7d0 2024-03-30T00:59:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.996+0000 7f1467e00700 1 -- 172.21.15.116:0/3609259903 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c09b350 con 0x55600245d7d0 2024-03-30T00:59:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.997+0000 7f1494f6ab80 1 -- 172.21.15.116:0/3609259903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245d7d0 msgr2=0x556002466c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.997+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/3609259903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245d7d0 0x556002466c20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f146c09adf0 tx=0x7f146c0a1040 comp rx=0 tx=0).stop 2024-03-30T00:59:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.997+0000 7f1494f6ab80 1 -- 172.21.15.116:0/3609259903 shutdown_connections 2024-03-30T00:59:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.997+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/3609259903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245efd0 0x55600246ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.997+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/3609259903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245d7d0 0x556002466c20 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.997+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/3609259903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245e130 0x55600246f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.997+0000 7f1494f6ab80 1 -- 172.21.15.116:0/3609259903 >> 172.21.15.116:0/3609259903 conn(0x5560023bc6a0 msgr2=0x5560024655d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.997+0000 7f1494f6ab80 1 -- 172.21.15.116:0/3609259903 shutdown_connections 2024-03-30T00:59:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.997+0000 7f1494f6ab80 1 -- 172.21.15.116:0/3609259903 wait complete. 2024-03-30T00:59:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.997+0000 7f1494f6ab80 1 -- start start 2024-03-30T00:59:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.998+0000 7f1494f6ab80 1 Processor -- start 2024-03-30T00:59:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.998+0000 7f1494f6ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245d7d0 0x55600246c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.998+0000 7f1494f6ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245e130 0x55600246ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.998+0000 7f1494f6ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245efd0 0x55600246ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.998+0000 7f1494f6ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55600246e310 con 0x55600245efd0 2024-03-30T00:59:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.998+0000 7f1494f6ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55600246e480 con 0x55600245e130 2024-03-30T00:59:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.998+0000 7f1494f6ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55600246e640 con 0x55600245d7d0 2024-03-30T00:59:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.998+0000 7f1494f6ab80 10 client.0 _ll_register_callbacks cb 0x7f146800f8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:59:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f1479e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245e130 0x55600246ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f1479e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245e130 0x55600246ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43846/0 (socket says 172.21.15.116:43846) 2024-03-30T00:59:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f147a800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245d7d0 0x55600246c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f1479e00700 1 -- 172.21.15.116:0/1722806348 learned_addr learned my addr 172.21.15.116:0/1722806348 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f147b200700 1 --2- 172.21.15.116:0/1722806348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245efd0 0x55600246ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f1479e00700 1 -- 172.21.15.116:0/1722806348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245d7d0 msgr2=0x55600246c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f1479e00700 1 --2- 172.21.15.116:0/1722806348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245d7d0 0x55600246c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f1479e00700 1 -- 172.21.15.116:0/1722806348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245efd0 msgr2=0x55600246ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f1479e00700 1 --2- 172.21.15.116:0/1722806348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245efd0 0x55600246ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f1479e00700 1 -- 172.21.15.116:0/1722806348 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c1a0f50 con 0x55600245e130 2024-03-30T00:59:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f1479e00700 1 --2- 172.21.15.116:0/1722806348 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245e130 0x55600246ceb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f146800e790 tx=0x7f146800eb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13af6b71f4b0bacc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f145be00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T00:59:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f145be00700 1 -- 172.21.15.116:0/1722806348 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468022070 con 0x55600245e130 2024-03-30T00:59:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f1494f6ab80 1 -- 172.21.15.116:0/1722806348 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f147400f040 con 0x55600245e130 2024-03-30T00:59:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:27.999+0000 7f145be00700 1 -- 172.21.15.116:0/1722806348 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f146800ecc0 con 0x55600245e130 2024-03-30T00:59:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.000+0000 7f145be00700 1 -- 172.21.15.116:0/1722806348 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f146800ee80 con 0x55600245e130 2024-03-30T00:59:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.000+0000 7f1494f6ab80 10 client.25799 fetch_fsmap learned FSMap version 23 2024-03-30T00:59:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.000+0000 7f1494f6ab80 1 -- 172.21.15.116:0/1722806348 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556002374ea0 con 0x55600245e130 2024-03-30T00:59:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.000+0000 7f145be00700 1 -- 172.21.15.116:0/1722806348 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f147400f040 con 0x55600245e130 2024-03-30T00:59:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.000+0000 7f145be00700 1 -- 172.21.15.116:0/1722806348 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1468005020 con 0x55600245e130 2024-03-30T00:59:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.000+0000 7f1494f6ab80 10 client.25799 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:59:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.000+0000 7f1494f6ab80 10 client.25799 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T00:59:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.000+0000 7f1494f6ab80 20 client.25799 populate_metadata read hostname 'smithi116' 2024-03-30T00:59:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.000+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/1722806348 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5560024c8110 0x5560024ca530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.000+0000 7f1494f6ab80 4 client.25799 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T00:59:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.000+0000 7f1494f6ab80 1 -- 172.21.15.116:0/1722806348 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5560021fdcf0 con 0x5560024c8110 2024-03-30T00:59:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.001+0000 7f147a800700 1 --2- 172.21.15.116:0/1722806348 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5560024c8110 0x5560024ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.001+0000 7f145be00700 10 client.25799 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T00:59:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.001+0000 7f147a800700 1 --2- 172.21.15.116:0/1722806348 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5560024c8110 0x5560024ca530 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3c011cb16a01e0a8 server_cookie=38469f80e809db50 in_seq=0 out_seq=0 2024-03-30T00:59:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.001+0000 7f145be00700 1 -- 172.21.15.116:0/1722806348 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5560021fdcf0 con 0x5560024c8110 2024-03-30T00:59:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.001+0000 7f145be00700 10 client.25799 handle_command_reply: tid=0 2024-03-30T00:59:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.001+0000 7f1494f6ab80 1 client.25799 shutdown 2024-03-30T00:59:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1494f6ab80 1 -- 172.21.15.116:0/1722806348 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245e130 msgr2=0x55600246ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/1722806348 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245e130 0x55600246ceb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f146800e790 tx=0x7f146800eb60 comp rx=0 tx=0).stop 2024-03-30T00:59:28.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1494f6ab80 1 -- 172.21.15.116:0/1722806348 shutdown_connections 2024-03-30T00:59:28.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/1722806348 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5560024c8110 0x5560024ca530 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/1722806348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55600245d7d0 0x55600246c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/1722806348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55600245efd0 0x55600246ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f145be00700 0 client.25799 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T00:59:28.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1494f6ab80 1 --2- 172.21.15.116:0/1722806348 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55600245e130 0x55600246ceb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1494f6ab80 1 -- 172.21.15.116:0/1722806348 >> 172.21.15.116:0/1722806348 conn(0x5560023bc6a0 msgr2=0x556002465560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1494f6ab80 1 -- 172.21.15.116:0/1722806348 shutdown_connections 2024-03-30T00:59:28.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1494f6ab80 1 -- 172.21.15.116:0/1722806348 wait complete. 2024-03-30T00:59:28.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1494f6ab80 20 client.25799 trim_cache size 0 max 16384 2024-03-30T00:59:28.013 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T00:59:28.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1481400700 1 -- 172.21.15.116:0/3812520825 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14600a11a0 msgr2=0x7f14600a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1481400700 1 --2- 172.21.15.116:0/3812520825 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14600a11a0 0x7f14600a3660 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f147c074b10 tx=0x7f1468011830 comp rx=0 tx=0).stop 2024-03-30T00:59:28.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1481400700 1 -- 172.21.15.116:0/3812520825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f147c073810 msgr2=0x7f147c198f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.002+0000 7f1481400700 1 --2- 172.21.15.116:0/3812520825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f147c073810 0x7f147c198f20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f146c0098e0 tx=0x7f146c005720 comp rx=0 tx=0).stop 2024-03-30T00:59:28.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.003+0000 7f1481400700 1 -- 172.21.15.116:0/3812520825 shutdown_connections 2024-03-30T00:59:28.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.003+0000 7f1481400700 1 --2- 172.21.15.116:0/3812520825 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14600a11a0 0x7f14600a3660 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.003+0000 7f1481400700 1 --2- 172.21.15.116:0/3812520825 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f147c108c40 0x7f147c1a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.003+0000 7f1481400700 1 --2- 172.21.15.116:0/3812520825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f147c073810 0x7f147c198f20 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.003+0000 7f1481400700 1 --2- 172.21.15.116:0/3812520825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f147c074df0 0x7f147c199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.003+0000 7f1481400700 1 -- 172.21.15.116:0/3812520825 >> 172.21.15.116:0/3812520825 conn(0x7f147c0fc2f0 msgr2=0x7f147c104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.003+0000 7f1481400700 1 -- 172.21.15.116:0/3812520825 shutdown_connections 2024-03-30T00:59:28.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.003+0000 7f1481400700 1 -- 172.21.15.116:0/3812520825 wait complete. 2024-03-30T00:59:28.027 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T00:59:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.223+0000 7fe5f5c00700 1 -- 172.21.15.116:0/4021742968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5f0100d80 msgr2=0x7fe5f01011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.223+0000 7fe5f5c00700 1 --2- 172.21.15.116:0/4021742968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5f0100d80 0x7fe5f01011a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009920 tx=0x7fe5e0009c30 comp rx=0 tx=0).stop 2024-03-30T00:59:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.223+0000 7fe5f5c00700 1 -- 172.21.15.116:0/4021742968 shutdown_connections 2024-03-30T00:59:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.223+0000 7fe5f5c00700 1 --2- 172.21.15.116:0/4021742968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f0102910 0x7fe5f01091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.223+0000 7fe5f5c00700 1 --2- 172.21.15.116:0/4021742968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5f0100d80 0x7fe5f01011a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.223+0000 7fe5f5c00700 1 --2- 172.21.15.116:0/4021742968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5f0101f70 0x7fe5f01023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.223+0000 7fe5f5c00700 1 -- 172.21.15.116:0/4021742968 >> 172.21.15.116:0/4021742968 conn(0x7fe5f00fc2f0 msgr2=0x7fe5f00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.223+0000 7fe5f5c00700 1 -- 172.21.15.116:0/4021742968 shutdown_connections 2024-03-30T00:59:28.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.223+0000 7fe5f5c00700 1 -- 172.21.15.116:0/4021742968 wait complete. 2024-03-30T00:59:28.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.224+0000 7fe5f5c00700 1 Processor -- start 2024-03-30T00:59:28.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.224+0000 7fe5f5c00700 1 -- start start 2024-03-30T00:59:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5f5c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5f0100d80 0x7fe5f0074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5f5c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5f0101f70 0x7fe5f00731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5f5c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f0102910 0x7fe5f00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5f5c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f0073de0 con 0x7fe5f0102910 2024-03-30T00:59:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5f5c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f0073f50 con 0x7fe5f0101f70 2024-03-30T00:59:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5f5c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f01a0e90 con 0x7fe5f0100d80 2024-03-30T00:59:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5ef400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f0102910 0x7fe5f00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5ee000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5f0101f70 0x7fe5f00731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5ef400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f0102910 0x7fe5f00736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59990/0 (socket says 172.21.15.116:59990) 2024-03-30T00:59:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5ef400700 1 -- 172.21.15.116:0/1690433369 learned_addr learned my addr 172.21.15.116:0/1690433369 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5eea00700 1 --2- 172.21.15.116:0/1690433369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5f0100d80 0x7fe5f0074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5ef400700 1 -- 172.21.15.116:0/1690433369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5f0100d80 msgr2=0x7fe5f0074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5ef400700 1 --2- 172.21.15.116:0/1690433369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5f0100d80 0x7fe5f0074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5ef400700 1 -- 172.21.15.116:0/1690433369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5f0101f70 msgr2=0x7fe5f00731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5ef400700 1 --2- 172.21.15.116:0/1690433369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5f0101f70 0x7fe5f00731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.225+0000 7fe5ef400700 1 -- 172.21.15.116:0/1690433369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e0009520 con 0x7fe5f0102910 2024-03-30T00:59:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.226+0000 7fe5ef400700 1 --2- 172.21.15.116:0/1690433369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f0102910 0x7fe5f00736e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800c110 tx=0x7fe5e800c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.226+0000 7fe5e7400700 1 -- 172.21.15.116:0/1690433369 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8017070 con 0x7fe5f0102910 2024-03-30T00:59:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.226+0000 7fe5e7400700 1 -- 172.21.15.116:0/1690433369 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5e800e050 con 0x7fe5f0102910 2024-03-30T00:59:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.226+0000 7fe5f5c00700 1 -- 172.21.15.116:0/1690433369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f01a1090 con 0x7fe5f0102910 2024-03-30T00:59:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.226+0000 7fe5e7400700 1 -- 172.21.15.116:0/1690433369 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e80078f0 con 0x7fe5f0102910 2024-03-30T00:59:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.226+0000 7fe5f5c00700 1 -- 172.21.15.116:0/1690433369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f01a15a0 con 0x7fe5f0102910 2024-03-30T00:59:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.230+0000 7fe5e7400700 1 -- 172.21.15.116:0/1690433369 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fe5e8007a50 con 0x7fe5f0102910 2024-03-30T00:59:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.231+0000 7fe609694b80 1 Processor -- start 2024-03-30T00:59:28.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.231+0000 7fe609694b80 1 -- start start 2024-03-30T00:59:28.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe609694b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc8967d0 0x555bcc896bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe609694b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897130 0x555bcc8979f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe609694b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc897fd0 0x555bcc898450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe609694b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bcc7ee990 con 0x555bcc897130 2024-03-30T00:59:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe609694b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bcc7ec350 con 0x555bcc897fd0 2024-03-30T00:59:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe609694b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bcc7e7c40 con 0x555bcc8967d0 2024-03-30T00:59:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe5eea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897130 0x555bcc8979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe5eea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897130 0x555bcc8979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60000/0 (socket says 172.21.15.116:60000) 2024-03-30T00:59:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe5eea00700 1 -- 172.21.15.116:0/3448079320 learned_addr learned my addr 172.21.15.116:0/3448079320 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe5e7400700 1 --2- 172.21.15.116:0/1690433369 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe5d40a1120 0x7fe5d40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe5ee000700 1 --2- 172.21.15.116:0/3448079320 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc8967d0 0x555bcc896bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe5ee000700 1 --2- 172.21.15.116:0/3448079320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc897fd0 0x555bcc898450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe5e7400700 1 -- 172.21.15.116:0/1690433369 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe5e80cc5d0 con 0x7fe5f0102910 2024-03-30T00:59:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe5eea00700 1 -- 172.21.15.116:0/3448079320 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc8967d0 msgr2=0x555bcc896bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe5eea00700 1 --2- 172.21.15.116:0/3448079320 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc8967d0 0x555bcc896bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe5eea00700 1 -- 172.21.15.116:0/3448079320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc897fd0 msgr2=0x555bcc898450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe5eea00700 1 --2- 172.21.15.116:0/3448079320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc897fd0 0x555bcc898450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.232+0000 7fe5eea00700 1 -- 172.21.15.116:0/3448079320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555bcc6693b0 con 0x555bcc897130 2024-03-30T00:59:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.233+0000 7fe5ef400700 1 --2- 172.21.15.116:0/1690433369 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe5d40a1120 0x7fe5d40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.233+0000 7fe5eea00700 1 --2- 172.21.15.116:0/3448079320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897130 0x555bcc8979f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000fa40 tx=0x7fe5e0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcd2079b8e9dd5f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.233+0000 7fe5dbe00700 1 -- 172.21.15.116:0/3448079320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e000f6e0 con 0x555bcc897130 2024-03-30T00:59:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.233+0000 7fe5dbe00700 1 -- 172.21.15.116:0/3448079320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5e001d040 con 0x555bcc897130 2024-03-30T00:59:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.233+0000 7fe5dbe00700 1 -- 172.21.15.116:0/3448079320 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0005680 con 0x555bcc897130 2024-03-30T00:59:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.234+0000 7fe609694b80 1 -- 172.21.15.116:0/3448079320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897130 msgr2=0x555bcc8979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.234+0000 7fe5ef400700 1 --2- 172.21.15.116:0/1690433369 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe5d40a1120 0x7fe5d40a35e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80a1660 tx=0x7fe5e8009da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.234+0000 7fe609694b80 1 --2- 172.21.15.116:0/3448079320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897130 0x555bcc8979f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000fa40 tx=0x7fe5e0009f80 comp rx=0 tx=0).stop 2024-03-30T00:59:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.234+0000 7fe609694b80 1 -- 172.21.15.116:0/3448079320 shutdown_connections 2024-03-30T00:59:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.234+0000 7fe609694b80 1 --2- 172.21.15.116:0/3448079320 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc8967d0 0x555bcc896bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.234+0000 7fe609694b80 1 --2- 172.21.15.116:0/3448079320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc897fd0 0x555bcc898450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.234+0000 7fe609694b80 1 --2- 172.21.15.116:0/3448079320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897130 0x555bcc8979f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.234+0000 7fe609694b80 1 -- 172.21.15.116:0/3448079320 >> 172.21.15.116:0/3448079320 conn(0x555bcc7f56a0 msgr2=0x555bcc8919a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.234+0000 7fe609694b80 1 -- 172.21.15.116:0/3448079320 shutdown_connections 2024-03-30T00:59:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.234+0000 7fe609694b80 1 -- 172.21.15.116:0/3448079320 wait complete. 2024-03-30T00:59:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.235+0000 7fe609694b80 1 -- start start 2024-03-30T00:59:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.235+0000 7fe609694b80 1 Processor -- start 2024-03-30T00:59:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.235+0000 7fe609694b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc8967d0 0x555bcc8a0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe609694b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897130 0x555bcc8a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe609694b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897fd0 0x555bcc8a7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe609694b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bcc783d80 con 0x555bcc897130 2024-03-30T00:59:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe609694b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bcc783a20 con 0x555bcc8967d0 2024-03-30T00:59:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe609694b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bcc66b6a0 con 0x555bcc897fd0 2024-03-30T00:59:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe609694b80 10 client.0 _ll_register_callbacks cb 0x555bcc78d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:59:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe5eea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897130 0x555bcc8a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe5eea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897130 0x555bcc8a0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60002/0 (socket says 172.21.15.116:60002) 2024-03-30T00:59:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe5eea00700 1 -- 172.21.15.116:0/1020843322 learned_addr learned my addr 172.21.15.116:0/1020843322 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe5ee000700 1 --2- 172.21.15.116:0/1020843322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc8967d0 0x555bcc8a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe5ee000700 1 --2- 172.21.15.116:0/1020843322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897fd0 0x555bcc8a7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe5eea00700 1 -- 172.21.15.116:0/1020843322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897fd0 msgr2=0x555bcc8a7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe5eea00700 1 --2- 172.21.15.116:0/1020843322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897fd0 0x555bcc8a7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe5eea00700 1 -- 172.21.15.116:0/1020843322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc8967d0 msgr2=0x555bcc8a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe5eea00700 1 --2- 172.21.15.116:0/1020843322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc8967d0 0x555bcc8a0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe5eea00700 1 -- 172.21.15.116:0/1020843322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f0109b10 con 0x555bcc897130 2024-03-30T00:59:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe5eea00700 1 --2- 172.21.15.116:0/1020843322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897130 0x555bcc8a0aa0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0005850 tx=0x7fe5e000f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ac433aaa356799 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.236+0000 7fe5d8c00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:59:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.237+0000 7fe5d8c00700 1 -- 172.21.15.116:0/1020843322 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e000b970 con 0x555bcc897130 2024-03-30T00:59:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.237+0000 7fe5d8c00700 1 -- 172.21.15.116:0/1020843322 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe5e000bad0 con 0x555bcc897130 2024-03-30T00:59:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.237+0000 7fe609694b80 1 -- 172.21.15.116:0/1020843322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555bcc7a8380 con 0x555bcc897130 2024-03-30T00:59:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.237+0000 7fe5d8c00700 1 -- 172.21.15.116:0/1020843322 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5e0005920 con 0x555bcc897130 2024-03-30T00:59:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.237+0000 7fe609694b80 10 client.15744 fetch_fsmap learned FSMap version 23 2024-03-30T00:59:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.237+0000 7fe609694b80 1 -- 172.21.15.116:0/1020843322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555bcc615b60 con 0x555bcc897130 2024-03-30T00:59:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.237+0000 7fe5d8c00700 1 -- 172.21.15.116:0/1020843322 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555bcc7a8380 con 0x555bcc897130 2024-03-30T00:59:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.238+0000 7fe5d8c00700 1 -- 172.21.15.116:0/1020843322 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe5e0028070 con 0x555bcc897130 2024-03-30T00:59:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.238+0000 7fe609694b80 10 client.15744 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:59:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.238+0000 7fe609694b80 10 client.15744 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T00:59:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.238+0000 7fe609694b80 20 client.15744 populate_metadata read hostname 'smithi116' 2024-03-30T00:59:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.238+0000 7fe609694b80 1 --2- 172.21.15.116:0/1020843322 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x555bcc8abe90 0x555bcc8ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.238+0000 7fe609694b80 4 client.15744 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:59:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.238+0000 7fe609694b80 1 -- 172.21.15.116:0/1020843322 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555bcc7e0700 con 0x555bcc8abe90 2024-03-30T00:59:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.238+0000 7fe5ee000700 1 --2- 172.21.15.116:0/1020843322 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x555bcc8abe90 0x555bcc8ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.238+0000 7fe5ee000700 1 --2- 172.21.15.116:0/1020843322 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x555bcc8abe90 0x555bcc8ae2b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9cc47ac5dba13b95 server_cookie=8427223539d3556b in_seq=0 out_seq=0 2024-03-30T00:59:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.238+0000 7fe5d8c00700 10 client.15744 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T00:59:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.239+0000 7fe5d8c00700 1 -- 172.21.15.116:0/1020843322 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555bcc7e0700 con 0x555bcc8abe90 2024-03-30T00:59:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.239+0000 7fe5d8c00700 10 client.15744 handle_command_reply: tid=0 2024-03-30T00:59:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.239+0000 7fe609694b80 1 client.15744 shutdown 2024-03-30T00:59:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.239+0000 7fe609694b80 1 -- 172.21.15.116:0/1020843322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897130 msgr2=0x555bcc8a0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.239+0000 7fe609694b80 1 --2- 172.21.15.116:0/1020843322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897130 0x555bcc8a0aa0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0005850 tx=0x7fe5e000f710 comp rx=0 tx=0).stop 2024-03-30T00:59:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.239+0000 7fe609694b80 1 -- 172.21.15.116:0/1020843322 shutdown_connections 2024-03-30T00:59:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.239+0000 7fe609694b80 1 --2- 172.21.15.116:0/1020843322 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x555bcc8abe90 0x555bcc8ae2b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.239+0000 7fe609694b80 1 --2- 172.21.15.116:0/1020843322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897fd0 0x555bcc8a7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.239+0000 7fe609694b80 1 --2- 172.21.15.116:0/1020843322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc8967d0 0x555bcc8a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.239+0000 7fe5d8c00700 0 client.15744 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T00:59:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.239+0000 7fe609694b80 1 --2- 172.21.15.116:0/1020843322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897130 0x555bcc8a0aa0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.239+0000 7fe609694b80 1 -- 172.21.15.116:0/1020843322 >> 172.21.15.116:0/1020843322 conn(0x555bcc7f56a0 msgr2=0x555bcc895170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.240+0000 7fe609694b80 1 -- 172.21.15.116:0/1020843322 shutdown_connections 2024-03-30T00:59:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.240+0000 7fe609694b80 1 -- 172.21.15.116:0/1020843322 wait complete. 2024-03-30T00:59:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.240+0000 7fe609694b80 20 client.15744 trim_cache size 0 max 16384 2024-03-30T00:59:28.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.253+0000 7fe609694b80 1 Processor -- start 2024-03-30T00:59:28.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.253+0000 7fe609694b80 1 -- start start 2024-03-30T00:59:28.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.253+0000 7fe609694b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc8967d0 0x555bcc8a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.253+0000 7fe609694b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc897130 0x555bcc89fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.253+0000 7fe609694b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897fd0 0x555bcc8a7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.253+0000 7fe609694b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bcc897e80 con 0x555bcc8967d0 2024-03-30T00:59:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.253+0000 7fe609694b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bcc8ae6a0 con 0x555bcc897130 2024-03-30T00:59:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.253+0000 7fe609694b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bcc8a00d0 con 0x555bcc897fd0 2024-03-30T00:59:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.254+0000 7fe5ee000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc8967d0 0x555bcc8a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.254+0000 7fe5eea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc897130 0x555bcc89fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.254+0000 7fe5ee000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc8967d0 0x555bcc8a8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60010/0 (socket says 172.21.15.116:60010) 2024-03-30T00:59:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.254+0000 7fe5eea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc897130 0x555bcc89fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43892/0 (socket says 172.21.15.116:43892) 2024-03-30T00:59:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.254+0000 7fe5ee000700 1 -- 172.21.15.116:0/1448273350 learned_addr learned my addr 172.21.15.116:0/1448273350 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.254+0000 7fe5ee000700 1 --2- 172.21.15.116:0/1448273350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897fd0 0x555bcc8a7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.254+0000 7fe5eea00700 1 -- 172.21.15.116:0/1448273350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897fd0 msgr2=0x555bcc8a7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.254+0000 7fe5eea00700 1 --2- 172.21.15.116:0/1448273350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897fd0 0x555bcc8a7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.254+0000 7fe5eea00700 1 -- 172.21.15.116:0/1448273350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc8967d0 msgr2=0x555bcc8a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.254+0000 7fe5eea00700 1 --2- 172.21.15.116:0/1448273350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc8967d0 0x555bcc8a8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.254+0000 7fe5eea00700 1 -- 172.21.15.116:0/1448273350 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555bcc7adea0 con 0x555bcc897130 2024-03-30T00:59:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.254+0000 7fe5ee000700 1 --2- 172.21.15.116:0/1448273350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897fd0 0x555bcc8a7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T00:59:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.254+0000 7fe5eea00700 1 --2- 172.21.15.116:0/1448273350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc897130 0x555bcc89fa70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0034c70 tx=0x7fe5e002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf199880d672e980 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.254+0000 7fe5e4c00700 1 -- 172.21.15.116:0/1448273350 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e000f780 con 0x555bcc897130 2024-03-30T00:59:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.255+0000 7fe5e4c00700 1 -- 172.21.15.116:0/1448273350 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5e00058e0 con 0x555bcc897130 2024-03-30T00:59:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.255+0000 7fe5e4c00700 1 -- 172.21.15.116:0/1448273350 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0034610 con 0x555bcc897130 2024-03-30T00:59:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.255+0000 7fe609694b80 1 -- 172.21.15.116:0/1448273350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc897130 msgr2=0x555bcc89fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.255+0000 7fe609694b80 1 --2- 172.21.15.116:0/1448273350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc897130 0x555bcc89fa70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0034c70 tx=0x7fe5e002b840 comp rx=0 tx=0).stop 2024-03-30T00:59:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.255+0000 7fe609694b80 1 -- 172.21.15.116:0/1448273350 shutdown_connections 2024-03-30T00:59:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.255+0000 7fe609694b80 1 --2- 172.21.15.116:0/1448273350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897fd0 0x555bcc8a7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.255+0000 7fe609694b80 1 --2- 172.21.15.116:0/1448273350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc8967d0 0x555bcc8a8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.255+0000 7fe609694b80 1 --2- 172.21.15.116:0/1448273350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc897130 0x555bcc89fa70 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.255+0000 7fe609694b80 1 -- 172.21.15.116:0/1448273350 >> 172.21.15.116:0/1448273350 conn(0x555bcc7f56a0 msgr2=0x555bcc88af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.255+0000 7fe609694b80 1 -- 172.21.15.116:0/1448273350 shutdown_connections 2024-03-30T00:59:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.255+0000 7fe609694b80 1 -- 172.21.15.116:0/1448273350 wait complete. 2024-03-30T00:59:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.256+0000 7fe609694b80 1 -- start start 2024-03-30T00:59:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.256+0000 7fe609694b80 1 Processor -- start 2024-03-30T00:59:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.256+0000 7fe609694b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc8967d0 0x555bcc8a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.256+0000 7fe609694b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897130 0x555bcc8a6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.256+0000 7fe609694b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897fd0 0x555bcc8a7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.256+0000 7fe609694b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bcc8a75a0 con 0x555bcc897fd0 2024-03-30T00:59:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.256+0000 7fe609694b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bcc8a7710 con 0x555bcc8967d0 2024-03-30T00:59:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.256+0000 7fe609694b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bcc8a78d0 con 0x555bcc897130 2024-03-30T00:59:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.256+0000 7fe609694b80 10 client.0 _ll_register_callbacks cb 0x555bcc78d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:59:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.256+0000 7fe5ee000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc8967d0 0x555bcc8a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe5ee000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc8967d0 0x555bcc8a5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43906/0 (socket says 172.21.15.116:43906) 2024-03-30T00:59:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe5ee000700 1 -- 172.21.15.116:0/2070158934 learned_addr learned my addr 172.21.15.116:0/2070158934 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe5eea00700 1 --2- 172.21.15.116:0/2070158934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897130 0x555bcc8a6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe5ee000700 1 --2- 172.21.15.116:0/2070158934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897fd0 0x555bcc8a7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe5eea00700 1 -- 172.21.15.116:0/2070158934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc8967d0 msgr2=0x555bcc8a5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe5eea00700 1 --2- 172.21.15.116:0/2070158934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc8967d0 0x555bcc8a5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe5eea00700 1 -- 172.21.15.116:0/2070158934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897fd0 msgr2=0x555bcc8a7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe5eea00700 1 --2- 172.21.15.116:0/2070158934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897fd0 0x555bcc8a7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe5eea00700 1 -- 172.21.15.116:0/2070158934 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555bcc615b60 con 0x555bcc897130 2024-03-30T00:59:28.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe5ee000700 1 --2- 172.21.15.116:0/2070158934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc8967d0 0x555bcc8a5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:59:28.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe5eea00700 1 --2- 172.21.15.116:0/2070158934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897130 0x555bcc8a6050 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x555bcc89efd0 tx=0x7fe5e000bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20d63f19a038eeda server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:28.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe5d8c00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T00:59:28.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe5d8c00700 1 -- 172.21.15.116:0/2070158934 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e000b770 con 0x555bcc897130 2024-03-30T00:59:28.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe609694b80 1 -- 172.21.15.116:0/2070158934 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5e001d040 con 0x555bcc897130 2024-03-30T00:59:28.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.257+0000 7fe5d8c00700 1 -- 172.21.15.116:0/2070158934 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe5e0000e40 con 0x555bcc897130 2024-03-30T00:59:28.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.258+0000 7fe5d8c00700 1 -- 172.21.15.116:0/2070158934 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5e000b8d0 con 0x555bcc897130 2024-03-30T00:59:28.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.258+0000 7fe609694b80 10 client.25798 fetch_fsmap learned FSMap version 23 2024-03-30T00:59:28.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.258+0000 7fe609694b80 1 -- 172.21.15.116:0/2070158934 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555bcc66dcd0 con 0x555bcc897130 2024-03-30T00:59:28.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.258+0000 7fe5d8c00700 1 -- 172.21.15.116:0/2070158934 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5e001d040 con 0x555bcc897130 2024-03-30T00:59:28.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.258+0000 7fe5d8c00700 1 -- 172.21.15.116:0/2070158934 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe5e0009950 con 0x555bcc897130 2024-03-30T00:59:28.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.258+0000 7fe609694b80 10 client.25798 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:59:28.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.258+0000 7fe609694b80 10 client.25798 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T00:59:28.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.258+0000 7fe609694b80 20 client.25798 populate_metadata read hostname 'smithi116' 2024-03-30T00:59:28.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.258+0000 7fe609694b80 1 --2- 172.21.15.116:0/2070158934 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x555bcc900820 0x555bcc902c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.258+0000 7fe609694b80 4 client.25798 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T00:59:28.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.258+0000 7fe609694b80 1 -- 172.21.15.116:0/2070158934 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555bcc636cf0 con 0x555bcc900820 2024-03-30T00:59:28.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe5ee000700 1 --2- 172.21.15.116:0/2070158934 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x555bcc900820 0x555bcc902c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe5ee000700 1 --2- 172.21.15.116:0/2070158934 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x555bcc900820 0x555bcc902c10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=580b767050ed2eee server_cookie=fa524940f74942b1 in_seq=0 out_seq=0 2024-03-30T00:59:28.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe5d8c00700 10 client.25798 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T00:59:28.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe5d8c00700 1 -- 172.21.15.116:0/2070158934 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555bcc636cf0 con 0x555bcc900820 2024-03-30T00:59:28.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe5d8c00700 10 client.25798 handle_command_reply: tid=0 2024-03-30T00:59:28.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe609694b80 1 client.25798 shutdown 2024-03-30T00:59:28.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe609694b80 1 -- 172.21.15.116:0/2070158934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897130 msgr2=0x555bcc8a6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe609694b80 1 --2- 172.21.15.116:0/2070158934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897130 0x555bcc8a6050 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x555bcc89efd0 tx=0x7fe5e000bf70 comp rx=0 tx=0).stop 2024-03-30T00:59:28.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe609694b80 1 -- 172.21.15.116:0/2070158934 shutdown_connections 2024-03-30T00:59:28.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe609694b80 1 --2- 172.21.15.116:0/2070158934 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x555bcc900820 0x555bcc902c10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe609694b80 1 --2- 172.21.15.116:0/2070158934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555bcc897fd0 0x555bcc8a7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe609694b80 1 --2- 172.21.15.116:0/2070158934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555bcc8967d0 0x555bcc8a5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe609694b80 1 --2- 172.21.15.116:0/2070158934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555bcc897130 0x555bcc8a6050 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe5d8c00700 0 client.25798 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T00:59:28.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.259+0000 7fe609694b80 1 -- 172.21.15.116:0/2070158934 >> 172.21.15.116:0/2070158934 conn(0x555bcc7f56a0 msgr2=0x555bcc89b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.260+0000 7fe609694b80 1 -- 172.21.15.116:0/2070158934 shutdown_connections 2024-03-30T00:59:28.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.260+0000 7fe609694b80 1 -- 172.21.15.116:0/2070158934 wait complete. 2024-03-30T00:59:28.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.260+0000 7fe609694b80 20 client.25798 trim_cache size 0 max 16384 2024-03-30T00:59:28.276 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T00:59:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.260+0000 7fe5f5c00700 1 -- 172.21.15.116:0/1690433369 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe5d40a1120 msgr2=0x7fe5d40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.260+0000 7fe5f5c00700 1 --2- 172.21.15.116:0/1690433369 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe5d40a1120 0x7fe5d40a35e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80a1660 tx=0x7fe5e8009da0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.260+0000 7fe5f5c00700 1 -- 172.21.15.116:0/1690433369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f0102910 msgr2=0x7fe5f00736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.260+0000 7fe5f5c00700 1 --2- 172.21.15.116:0/1690433369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f0102910 0x7fe5f00736e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800c110 tx=0x7fe5e800c420 comp rx=0 tx=0).stop 2024-03-30T00:59:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.260+0000 7fe5f5c00700 1 -- 172.21.15.116:0/1690433369 shutdown_connections 2024-03-30T00:59:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.260+0000 7fe5f5c00700 1 --2- 172.21.15.116:0/1690433369 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe5d40a1120 0x7fe5d40a35e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.260+0000 7fe5f5c00700 1 --2- 172.21.15.116:0/1690433369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5f0100d80 0x7fe5f0074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.260+0000 7fe5f5c00700 1 --2- 172.21.15.116:0/1690433369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f0102910 0x7fe5f00736e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.260+0000 7fe5f5c00700 1 --2- 172.21.15.116:0/1690433369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5f0101f70 0x7fe5f00731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.260+0000 7fe5f5c00700 1 -- 172.21.15.116:0/1690433369 >> 172.21.15.116:0/1690433369 conn(0x7fe5f00fc2f0 msgr2=0x7fe5f0105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.261+0000 7fe5f5c00700 1 -- 172.21.15.116:0/1690433369 shutdown_connections 2024-03-30T00:59:28.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.261+0000 7fe5f5c00700 1 -- 172.21.15.116:0/1690433369 wait complete. 2024-03-30T00:59:28.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:28 smithi116 ceph-mon[38086]: pgmap v339: 129 pgs: 129 active+clean; 3.8 MiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 307 KiB/s wr, 33 op/s 2024-03-30T00:59:28.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:28 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2305724629' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:59:28.285 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T00:59:28.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.475+0000 7f1393c00700 1 -- 172.21.15.116:0/3696504553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f138c100d80 msgr2=0x7f138c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.475+0000 7f1393c00700 1 --2- 172.21.15.116:0/3696504553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f138c100d80 0x7f138c1011a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f13800099c0 tx=0x7f1380009cd0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.475+0000 7f1393c00700 1 -- 172.21.15.116:0/3696504553 shutdown_connections 2024-03-30T00:59:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.475+0000 7f1393c00700 1 --2- 172.21.15.116:0/3696504553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c102910 0x7f138c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.475+0000 7f1393c00700 1 --2- 172.21.15.116:0/3696504553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f138c100d80 0x7f138c1011a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.475+0000 7f1393c00700 1 --2- 172.21.15.116:0/3696504553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f138c101f70 0x7f138c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.475+0000 7f1393c00700 1 -- 172.21.15.116:0/3696504553 >> 172.21.15.116:0/3696504553 conn(0x7f138c0fc2f0 msgr2=0x7f138c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.475+0000 7f1393c00700 1 -- 172.21.15.116:0/3696504553 shutdown_connections 2024-03-30T00:59:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.475+0000 7f1393c00700 1 -- 172.21.15.116:0/3696504553 wait complete. 2024-03-30T00:59:28.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.476+0000 7f1393c00700 1 Processor -- start 2024-03-30T00:59:28.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.476+0000 7f1393c00700 1 -- start start 2024-03-30T00:59:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1393c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c100d80 0x7f138c198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1393c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f138c101f70 0x7f138c199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1393c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f138c102910 0x7f138c1a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1393c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c1a0a10 con 0x7f138c100d80 2024-03-30T00:59:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1393c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c1a0b50 con 0x7f138c102910 2024-03-30T00:59:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1393c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c1a0ce0 con 0x7f138c101f70 2024-03-30T00:59:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1391200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c100d80 0x7f138c198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1391200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c100d80 0x7f138c198ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60044/0 (socket says 172.21.15.116:60044) 2024-03-30T00:59:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1391200700 1 -- 172.21.15.116:0/3072818141 learned_addr learned my addr 172.21.15.116:0/3072818141 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1391c00700 1 --2- 172.21.15.116:0/3072818141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f138c102910 0x7f138c1a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f138be00700 1 --2- 172.21.15.116:0/3072818141 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f138c101f70 0x7f138c199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1391200700 1 -- 172.21.15.116:0/3072818141 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f138c101f70 msgr2=0x7f138c199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1391200700 1 --2- 172.21.15.116:0/3072818141 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f138c101f70 0x7f138c199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1391200700 1 -- 172.21.15.116:0/3072818141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f138c102910 msgr2=0x7f138c1a0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1391200700 1 --2- 172.21.15.116:0/3072818141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f138c102910 0x7f138c1a0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.477+0000 7f1391200700 1 -- 172.21.15.116:0/3072818141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13800095f0 con 0x7f138c100d80 2024-03-30T00:59:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.478+0000 7f1391200700 1 --2- 172.21.15.116:0/3072818141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c100d80 0x7f138c198ee0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1380003730 tx=0x7f138000f720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.478+0000 7f1389600700 1 -- 172.21.15.116:0/3072818141 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138001d070 con 0x7f138c100d80 2024-03-30T00:59:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.478+0000 7f1389600700 1 -- 172.21.15.116:0/3072818141 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13800059a0 con 0x7f138c100d80 2024-03-30T00:59:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.478+0000 7f1393c00700 1 -- 172.21.15.116:0/3072818141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138c1a0f60 con 0x7f138c100d80 2024-03-30T00:59:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.478+0000 7f1389600700 1 -- 172.21.15.116:0/3072818141 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1380004b30 con 0x7f138c100d80 2024-03-30T00:59:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.478+0000 7f1393c00700 1 -- 172.21.15.116:0/3072818141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138c1a1320 con 0x7f138c100d80 2024-03-30T00:59:28.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.481+0000 7f1389600700 1 -- 172.21.15.116:0/3072818141 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f138000fc10 con 0x7f138c100d80 2024-03-30T00:59:28.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.482+0000 7f13a76c8b80 1 Processor -- start 2024-03-30T00:59:28.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.482+0000 7f1389600700 1 --2- 172.21.15.116:0/3072818141 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13740a1120 0x7f13740a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.482+0000 7f13a76c8b80 1 -- start start 2024-03-30T00:59:28.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.482+0000 7f1389600700 1 -- 172.21.15.116:0/3072818141 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f13800cd950 con 0x7f138c100d80 2024-03-30T00:59:28.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f1391c00700 1 --2- 172.21.15.116:0/3072818141 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13740a1120 0x7f13740a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f13a76c8b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c6117d0 0x560c8c611bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f13a76c8b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c612130 0x560c8c6129f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f13a76c8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612fd0 0x560c8c613450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f13a76c8b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c8c569990 con 0x560c8c6117d0 2024-03-30T00:59:28.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f13a76c8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c8c567350 con 0x560c8c612fd0 2024-03-30T00:59:28.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f13a76c8b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c8c562c40 con 0x560c8c612130 2024-03-30T00:59:28.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f1391200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c6117d0 0x560c8c611bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f138be00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c612130 0x560c8c6129f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f1391200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c6117d0 0x560c8c611bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60050/0 (socket says 172.21.15.116:60050) 2024-03-30T00:59:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f138be00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c612130 0x560c8c6129f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59604/0 (socket says 172.21.15.116:59604) 2024-03-30T00:59:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f1391200700 1 -- 172.21.15.116:0/2347587786 learned_addr learned my addr 172.21.15.116:0/2347587786 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f1391c00700 1 --2- 172.21.15.116:0/2347587786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612fd0 0x560c8c613450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f138be00700 1 -- 172.21.15.116:0/2347587786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612fd0 msgr2=0x560c8c613450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f138be00700 1 --2- 172.21.15.116:0/2347587786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612fd0 0x560c8c613450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f138be00700 1 -- 172.21.15.116:0/2347587786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c6117d0 msgr2=0x560c8c611bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f138be00700 1 --2- 172.21.15.116:0/2347587786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c6117d0 0x560c8c611bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f138be00700 1 -- 172.21.15.116:0/2347587786 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c8c3e43b0 con 0x560c8c612130 2024-03-30T00:59:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.483+0000 7f1391200700 1 --2- 172.21.15.116:0/2347587786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c6117d0 0x560c8c611bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:59:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.484+0000 7f1391c00700 1 --2- 172.21.15.116:0/3072818141 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13740a1120 0x7f13740a35e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f13840093c0 tx=0x7f1384009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.484+0000 7f138be00700 1 --2- 172.21.15.116:0/2347587786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c612130 0x560c8c6129f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1378009c00 tx=0x7f1378009f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=552e65d75d8b6a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.484+0000 7f137e000700 1 -- 172.21.15.116:0/2347587786 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1378011b70 con 0x560c8c612130 2024-03-30T00:59:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.484+0000 7f137e000700 1 -- 172.21.15.116:0/2347587786 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1378011cd0 con 0x560c8c612130 2024-03-30T00:59:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.484+0000 7f137e000700 1 -- 172.21.15.116:0/2347587786 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137800f680 con 0x560c8c612130 2024-03-30T00:59:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.485+0000 7f13a76c8b80 1 -- 172.21.15.116:0/2347587786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c612130 msgr2=0x560c8c6129f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.485+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/2347587786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c612130 0x560c8c6129f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1378009c00 tx=0x7f1378009f10 comp rx=0 tx=0).stop 2024-03-30T00:59:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.485+0000 7f13a76c8b80 1 -- 172.21.15.116:0/2347587786 shutdown_connections 2024-03-30T00:59:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.485+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/2347587786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612fd0 0x560c8c613450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.485+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/2347587786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c6117d0 0x560c8c611bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.485+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/2347587786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c612130 0x560c8c6129f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.485+0000 7f13a76c8b80 1 -- 172.21.15.116:0/2347587786 >> 172.21.15.116:0/2347587786 conn(0x560c8c5706a0 msgr2=0x560c8c60c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.485+0000 7f13a76c8b80 1 -- 172.21.15.116:0/2347587786 shutdown_connections 2024-03-30T00:59:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.485+0000 7f13a76c8b80 1 -- 172.21.15.116:0/2347587786 wait complete. 2024-03-30T00:59:28.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.485+0000 7f13a76c8b80 1 -- start start 2024-03-30T00:59:28.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.486+0000 7f13a76c8b80 1 Processor -- start 2024-03-30T00:59:28.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.486+0000 7f13a76c8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c6117d0 0x560c8c61b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.486+0000 7f13a76c8b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c612130 0x560c8c61bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.486+0000 7f13a76c8b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612fd0 0x560c8c622be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.486+0000 7f13a76c8b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c8c4fed80 con 0x560c8c612fd0 2024-03-30T00:59:28.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.486+0000 7f13a76c8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c8c4fea20 con 0x560c8c6117d0 2024-03-30T00:59:28.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.486+0000 7f13a76c8b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c8c3e66a0 con 0x560c8c612130 2024-03-30T00:59:28.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.486+0000 7f13a76c8b80 10 client.0 _ll_register_callbacks cb 0x560c8c508470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:59:28.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.486+0000 7f1391200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c6117d0 0x560c8c61b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.486+0000 7f1391200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c6117d0 0x560c8c61b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43948/0 (socket says 172.21.15.116:43948) 2024-03-30T00:59:28.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.486+0000 7f1391200700 1 -- 172.21.15.116:0/2383698634 learned_addr learned my addr 172.21.15.116:0/2383698634 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:28.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.486+0000 7f138be00700 1 --2- 172.21.15.116:0/2383698634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c612130 0x560c8c61bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.486+0000 7f1391c00700 1 --2- 172.21.15.116:0/2383698634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612fd0 0x560c8c622be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.487+0000 7f1391200700 1 -- 172.21.15.116:0/2383698634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c612130 msgr2=0x560c8c61bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.487+0000 7f1391200700 1 --2- 172.21.15.116:0/2383698634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c612130 0x560c8c61bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.487+0000 7f1391200700 1 -- 172.21.15.116:0/2383698634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612fd0 msgr2=0x560c8c622be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.487+0000 7f1391200700 1 --2- 172.21.15.116:0/2383698634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612fd0 0x560c8c622be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.487+0000 7f1391200700 1 -- 172.21.15.116:0/2383698634 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138c1a1320 con 0x560c8c6117d0 2024-03-30T00:59:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.487+0000 7f1391200700 1 --2- 172.21.15.116:0/2383698634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c6117d0 0x560c8c61b660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1380009d70 tx=0x7f138009b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e7e4f295308c48 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.487+0000 7f1372a00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T00:59:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.487+0000 7f1372a00700 1 -- 172.21.15.116:0/2383698634 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13800a2c20 con 0x560c8c6117d0 2024-03-30T00:59:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.487+0000 7f1372a00700 1 -- 172.21.15.116:0/2383698634 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f138009aac0 con 0x560c8c6117d0 2024-03-30T00:59:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.487+0000 7f13a76c8b80 1 -- 172.21.15.116:0/2383698634 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560c8c523380 con 0x560c8c6117d0 2024-03-30T00:59:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.487+0000 7f1372a00700 1 -- 172.21.15.116:0/2383698634 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f138009e8f0 con 0x560c8c6117d0 2024-03-30T00:59:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.487+0000 7f13a76c8b80 10 client.25829 fetch_fsmap learned FSMap version 23 2024-03-30T00:59:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.487+0000 7f13a76c8b80 1 -- 172.21.15.116:0/2383698634 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f1378009830 con 0x560c8c6117d0 2024-03-30T00:59:28.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.488+0000 7f1372a00700 1 -- 172.21.15.116:0/2383698634 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13800b7070 con 0x560c8c6117d0 2024-03-30T00:59:28.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.488+0000 7f1372a00700 1 -- 172.21.15.116:0/2383698634 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1380099a20 con 0x560c8c6117d0 2024-03-30T00:59:28.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.488+0000 7f13a76c8b80 10 client.25829 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:59:28.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.488+0000 7f13a76c8b80 10 client.25829 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:59:28.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.488+0000 7f13a76c8b80 20 client.25829 populate_metadata read hostname 'smithi116' 2024-03-30T00:59:28.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.488+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/2383698634 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c8c627030 0x560c8c629450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.488+0000 7f13a76c8b80 4 client.25829 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:59:28.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.488+0000 7f13a76c8b80 1 -- 172.21.15.116:0/2383698634 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c8c55b700 con 0x560c8c627030 2024-03-30T00:59:28.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.488+0000 7f138be00700 1 --2- 172.21.15.116:0/2383698634 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c8c627030 0x560c8c629450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.489+0000 7f1372a00700 10 client.25829 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:28.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.489+0000 7f138be00700 1 --2- 172.21.15.116:0/2383698634 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c8c627030 0x560c8c629450 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc0d18becf76566b server_cookie=472ae18f03d2afef in_seq=0 out_seq=0 2024-03-30T00:59:28.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.489+0000 7f1372a00700 1 -- 172.21.15.116:0/2383698634 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560c8c55b700 con 0x560c8c627030 2024-03-30T00:59:28.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.489+0000 7f1372a00700 10 client.25829 handle_command_reply: tid=0 2024-03-30T00:59:28.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.489+0000 7f13a76c8b80 1 client.25829 shutdown 2024-03-30T00:59:28.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.490+0000 7f13a76c8b80 1 -- 172.21.15.116:0/2383698634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c6117d0 msgr2=0x560c8c61b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.490+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/2383698634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c6117d0 0x560c8c61b660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1380009d70 tx=0x7f138009b000 comp rx=0 tx=0).stop 2024-03-30T00:59:28.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.490+0000 7f13a76c8b80 1 -- 172.21.15.116:0/2383698634 shutdown_connections 2024-03-30T00:59:28.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.490+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/2383698634 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c8c627030 0x560c8c629450 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.490+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/2383698634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612fd0 0x560c8c622be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.490+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/2383698634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c6117d0 0x560c8c61b660 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.490+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/2383698634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c612130 0x560c8c61bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.490+0000 7f1372a00700 0 client.25829 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:28.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.490+0000 7f13a76c8b80 1 -- 172.21.15.116:0/2383698634 >> 172.21.15.116:0/2383698634 conn(0x560c8c5706a0 msgr2=0x560c8c610170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.490+0000 7f13a76c8b80 1 -- 172.21.15.116:0/2383698634 shutdown_connections 2024-03-30T00:59:28.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.490+0000 7f13a76c8b80 1 -- 172.21.15.116:0/2383698634 wait complete. 2024-03-30T00:59:28.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.490+0000 7f13a76c8b80 20 client.25829 trim_cache size 0 max 16384 2024-03-30T00:59:28.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:28 smithi181 ceph-mon[47622]: pgmap v339: 129 pgs: 129 active+clean; 3.8 MiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 307 KiB/s wr, 33 op/s 2024-03-30T00:59:28.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:28 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2305724629' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:59:28.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.503+0000 7f13a76c8b80 1 Processor -- start 2024-03-30T00:59:28.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.503+0000 7f13a76c8b80 1 -- start start 2024-03-30T00:59:28.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.503+0000 7f13a76c8b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c6117d0 0x560c8c623290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.503+0000 7f13a76c8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612130 0x560c8c622ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.503+0000 7f13a76c8b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612fd0 0x560c8c6194c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.504+0000 7f13a76c8b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c8c60d660 con 0x560c8c612fd0 2024-03-30T00:59:28.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.504+0000 7f13a76c8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c8c60d8a0 con 0x560c8c612130 2024-03-30T00:59:28.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.504+0000 7f13a76c8b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c8c608480 con 0x560c8c6117d0 2024-03-30T00:59:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.504+0000 7f1391c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612fd0 0x560c8c6194c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.504+0000 7f1391c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612fd0 0x560c8c6194c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60074/0 (socket says 172.21.15.116:60074) 2024-03-30T00:59:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.504+0000 7f1391c00700 1 -- 172.21.15.116:0/1911333584 learned_addr learned my addr 172.21.15.116:0/1911333584 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.504+0000 7f138be00700 1 --2- 172.21.15.116:0/1911333584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612130 0x560c8c622ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.504+0000 7f1391200700 1 --2- 172.21.15.116:0/1911333584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c6117d0 0x560c8c623290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.504+0000 7f1391c00700 1 -- 172.21.15.116:0/1911333584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c6117d0 msgr2=0x560c8c623290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.504+0000 7f1391c00700 1 --2- 172.21.15.116:0/1911333584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c6117d0 0x560c8c623290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.504+0000 7f1391c00700 1 -- 172.21.15.116:0/1911333584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612130 msgr2=0x560c8c622ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.504+0000 7f1391c00700 1 --2- 172.21.15.116:0/1911333584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612130 0x560c8c622ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.504+0000 7f1391c00700 1 -- 172.21.15.116:0/1911333584 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c8c390b60 con 0x560c8c612fd0 2024-03-30T00:59:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.504+0000 7f1391c00700 1 --2- 172.21.15.116:0/1911333584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612fd0 0x560c8c6194c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f1384017790 tx=0x7f138401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839c31c1f89efa9f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.505+0000 7f137ea00700 1 -- 172.21.15.116:0/1911333584 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138400e050 con 0x560c8c612fd0 2024-03-30T00:59:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.505+0000 7f137ea00700 1 -- 172.21.15.116:0/1911333584 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f138400f040 con 0x560c8c612fd0 2024-03-30T00:59:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.505+0000 7f137ea00700 1 -- 172.21.15.116:0/1911333584 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1384018970 con 0x560c8c612fd0 2024-03-30T00:59:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.505+0000 7f13a76c8b80 1 -- 172.21.15.116:0/1911333584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612fd0 msgr2=0x560c8c6194c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.505+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/1911333584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612fd0 0x560c8c6194c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f1384017790 tx=0x7f138401b000 comp rx=0 tx=0).stop 2024-03-30T00:59:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.505+0000 7f13a76c8b80 1 -- 172.21.15.116:0/1911333584 shutdown_connections 2024-03-30T00:59:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.505+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/1911333584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c6117d0 0x560c8c623290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.505+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/1911333584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612fd0 0x560c8c6194c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.505+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/1911333584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612130 0x560c8c622ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.505+0000 7f13a76c8b80 1 -- 172.21.15.116:0/1911333584 >> 172.21.15.116:0/1911333584 conn(0x560c8c5706a0 msgr2=0x560c8c6055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.505+0000 7f13a76c8b80 1 -- 172.21.15.116:0/1911333584 shutdown_connections 2024-03-30T00:59:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.506+0000 7f13a76c8b80 1 -- 172.21.15.116:0/1911333584 wait complete. 2024-03-30T00:59:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.506+0000 7f13a76c8b80 1 -- start start 2024-03-30T00:59:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.506+0000 7f13a76c8b80 1 Processor -- start 2024-03-30T00:59:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.506+0000 7f13a76c8b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c6117d0 0x560c8c6210b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.506+0000 7f13a76c8b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612130 0x560c8c6215f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.506+0000 7f13a76c8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612fd0 0x560c8c621fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.506+0000 7f13a76c8b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c8c6224e0 con 0x560c8c612130 2024-03-30T00:59:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.506+0000 7f13a76c8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c8c622650 con 0x560c8c612fd0 2024-03-30T00:59:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.506+0000 7f13a76c8b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c8c622810 con 0x560c8c6117d0 2024-03-30T00:59:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.506+0000 7f13a76c8b80 10 client.0 _ll_register_callbacks cb 0x560c8c508470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:59:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f1391200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c6117d0 0x560c8c6210b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f138be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612130 0x560c8c6215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f1391c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612fd0 0x560c8c621fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f1391c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612fd0 0x560c8c621fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43972/0 (socket says 172.21.15.116:43972) 2024-03-30T00:59:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f1391c00700 1 -- 172.21.15.116:0/4256383669 learned_addr learned my addr 172.21.15.116:0/4256383669 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:28.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f1391200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c6117d0 0x560c8c6210b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59630/0 (socket says 172.21.15.116:59630) 2024-03-30T00:59:28.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f1391c00700 1 -- 172.21.15.116:0/4256383669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c6117d0 msgr2=0x560c8c6210b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f1391c00700 1 --2- 172.21.15.116:0/4256383669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c6117d0 0x560c8c6210b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f1391c00700 1 -- 172.21.15.116:0/4256383669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612130 msgr2=0x560c8c6215f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:28.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f1391c00700 1 --2- 172.21.15.116:0/4256383669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612130 0x560c8c6215f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f1391c00700 1 -- 172.21.15.116:0/4256383669 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1384017440 con 0x560c8c612fd0 2024-03-30T00:59:28.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f1391200700 1 --2- 172.21.15.116:0/4256383669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c6117d0 0x560c8c6210b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:59:28.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f138be00700 1 --2- 172.21.15.116:0/4256383669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612130 0x560c8c6215f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:59:28.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f1391c00700 1 --2- 172.21.15.116:0/4256383669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612fd0 0x560c8c621fa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f13840189a0 tx=0x7f13840189d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4abd449af4d27911 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:28.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.507+0000 7f1372a00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T00:59:28.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.508+0000 7f1372a00700 1 -- 172.21.15.116:0/4256383669 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1384004a90 con 0x560c8c612fd0 2024-03-30T00:59:28.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.508+0000 7f1372a00700 1 -- 172.21.15.116:0/4256383669 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1384004bf0 con 0x560c8c612fd0 2024-03-30T00:59:28.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.508+0000 7f13a76c8b80 1 -- 172.21.15.116:0/4256383669 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1378011cd0 con 0x560c8c612fd0 2024-03-30T00:59:28.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.508+0000 7f1372a00700 1 -- 172.21.15.116:0/4256383669 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1384004db0 con 0x560c8c612fd0 2024-03-30T00:59:28.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.508+0000 7f13a76c8b80 10 client.25835 fetch_fsmap learned FSMap version 23 2024-03-30T00:59:28.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.508+0000 7f13a76c8b80 1 -- 172.21.15.116:0/4256383669 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560c8c528ea0 con 0x560c8c612fd0 2024-03-30T00:59:28.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.508+0000 7f1372a00700 1 -- 172.21.15.116:0/4256383669 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1378011cd0 con 0x560c8c612fd0 2024-03-30T00:59:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.508+0000 7f1372a00700 1 -- 172.21.15.116:0/4256383669 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1384024460 con 0x560c8c612fd0 2024-03-30T00:59:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.508+0000 7f13a76c8b80 10 client.25835 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:59:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.508+0000 7f13a76c8b80 10 client.25835 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:59:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.508+0000 7f13a76c8b80 20 client.25835 populate_metadata read hostname 'smithi116' 2024-03-30T00:59:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.509+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/4256383669 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c8c67be90 0x560c8c67e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.509+0000 7f13a76c8b80 4 client.25835 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T00:59:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.509+0000 7f13a76c8b80 1 -- 172.21.15.116:0/4256383669 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560c8c3b1cf0 con 0x560c8c67be90 2024-03-30T00:59:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.509+0000 7f1391200700 1 --2- 172.21.15.116:0/4256383669 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c8c67be90 0x560c8c67e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.509+0000 7f1372a00700 10 client.25835 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.509+0000 7f1391200700 1 --2- 172.21.15.116:0/4256383669 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c8c67be90 0x560c8c67e2b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b44e5af03733ecc server_cookie=b753e8796e3a28dc in_seq=0 out_seq=0 2024-03-30T00:59:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.509+0000 7f1372a00700 1 -- 172.21.15.116:0/4256383669 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560c8c3b1cf0 con 0x560c8c67be90 2024-03-30T00:59:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.509+0000 7f1372a00700 10 client.25835 handle_command_reply: tid=0 2024-03-30T00:59:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.509+0000 7f13a76c8b80 1 client.25835 shutdown 2024-03-30T00:59:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f13a76c8b80 1 -- 172.21.15.116:0/4256383669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612fd0 msgr2=0x560c8c621fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/4256383669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612fd0 0x560c8c621fa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f13840189a0 tx=0x7f13840189d0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f13a76c8b80 1 -- 172.21.15.116:0/4256383669 shutdown_connections 2024-03-30T00:59:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/4256383669 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x560c8c67be90 0x560c8c67e2b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/4256383669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560c8c6117d0 0x560c8c6210b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/4256383669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560c8c612fd0 0x560c8c621fa0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f1372a00700 0 client.25835 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f13a76c8b80 1 --2- 172.21.15.116:0/4256383669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c8c612130 0x560c8c6215f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f13a76c8b80 1 -- 172.21.15.116:0/4256383669 >> 172.21.15.116:0/4256383669 conn(0x560c8c5706a0 msgr2=0x560c8c61ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f13a76c8b80 1 -- 172.21.15.116:0/4256383669 shutdown_connections 2024-03-30T00:59:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f13a76c8b80 1 -- 172.21.15.116:0/4256383669 wait complete. 2024-03-30T00:59:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f13a76c8b80 20 client.25835 trim_cache size 0 max 16384 2024-03-30T00:59:28.523 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T00:59:28.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f1393c00700 1 -- 172.21.15.116:0/3072818141 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13740a1120 msgr2=0x7f13740a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f1393c00700 1 --2- 172.21.15.116:0/3072818141 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13740a1120 0x7f13740a35e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f13840093c0 tx=0x7f1384009920 comp rx=0 tx=0).stop 2024-03-30T00:59:28.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f1393c00700 1 -- 172.21.15.116:0/3072818141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c100d80 msgr2=0x7f138c198ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:28.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.510+0000 7f1393c00700 1 --2- 172.21.15.116:0/3072818141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c100d80 0x7f138c198ee0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1380003730 tx=0x7f138000f720 comp rx=0 tx=0).stop 2024-03-30T00:59:28.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.511+0000 7f1393c00700 1 -- 172.21.15.116:0/3072818141 shutdown_connections 2024-03-30T00:59:28.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.511+0000 7f1393c00700 1 --2- 172.21.15.116:0/3072818141 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13740a1120 0x7f13740a35e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.511+0000 7f1393c00700 1 --2- 172.21.15.116:0/3072818141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f138c102910 0x7f138c1a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.511+0000 7f1393c00700 1 --2- 172.21.15.116:0/3072818141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c100d80 0x7f138c198ee0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.511+0000 7f1393c00700 1 --2- 172.21.15.116:0/3072818141 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f138c101f70 0x7f138c199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.511+0000 7f1393c00700 1 -- 172.21.15.116:0/3072818141 >> 172.21.15.116:0/3072818141 conn(0x7f138c0fc2f0 msgr2=0x7f138c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.511+0000 7f1393c00700 1 -- 172.21.15.116:0/3072818141 shutdown_connections 2024-03-30T00:59:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:28.511+0000 7f1393c00700 1 -- 172.21.15.116:0/3072818141 wait complete. 2024-03-30T00:59:28.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:28 smithi130 ceph-mon[47626]: pgmap v339: 129 pgs: 129 active+clean; 3.8 MiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 307 KiB/s wr, 33 op/s 2024-03-30T00:59:28.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:28 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2305724629' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T00:59:28.944 INFO:tasks.workunit.client.0.smithi116.stdout:Spawning 3 writers... 2024-03-30T00:59:28.945 INFO:tasks.workunit.client.0.smithi116.stdout:Spawning 1 rewriters... 2024-03-30T00:59:28.945 INFO:tasks.workunit.client.0.smithi116.stdout:Spawning 5 commenters... 2024-03-30T00:59:28.945 INFO:tasks.workunit.client.0.smithi116.stdout:Spawning 100 readers... 2024-03-30T00:59:28.951 INFO:tasks.workunit.client.0.smithi116.stdout:Benchmarking for 30 iterations. 2024-03-30T00:59:28.951 INFO:tasks.workunit.client.0.smithi116.stdout:The test will run during 5 minutes. 2024-03-30T00:59:28.951 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-03-30T00:59:28.951 INFO:tasks.workunit.client.0.smithi116.stdout: Nb blogs R articles W articles R pictures W pictures R comments W comments 2024-03-30T00:59:29.536 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T00:59:29.536 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T00:59:29.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.738+0000 7fce61000700 1 -- 172.21.15.116:0/4036940289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce5c100d80 msgr2=0x7fce5c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:29.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.738+0000 7fce61000700 1 --2- 172.21.15.116:0/4036940289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce5c100d80 0x7fce5c1011a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fce4c0099c0 tx=0x7fce4c009cd0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.738+0000 7fce61000700 1 -- 172.21.15.116:0/4036940289 shutdown_connections 2024-03-30T00:59:29.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.738+0000 7fce61000700 1 --2- 172.21.15.116:0/4036940289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce5c102910 0x7fce5c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.738+0000 7fce61000700 1 --2- 172.21.15.116:0/4036940289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce5c100d80 0x7fce5c1011a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.738+0000 7fce61000700 1 --2- 172.21.15.116:0/4036940289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce5c101f70 0x7fce5c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.738+0000 7fce61000700 1 -- 172.21.15.116:0/4036940289 >> 172.21.15.116:0/4036940289 conn(0x7fce5c0fc2f0 msgr2=0x7fce5c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:29.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.738+0000 7fce61000700 1 -- 172.21.15.116:0/4036940289 shutdown_connections 2024-03-30T00:59:29.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.738+0000 7fce61000700 1 -- 172.21.15.116:0/4036940289 wait complete. 2024-03-30T00:59:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.740+0000 7fce61000700 1 Processor -- start 2024-03-30T00:59:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.740+0000 7fce61000700 1 -- start start 2024-03-30T00:59:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.740+0000 7fce61000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce5c100d80 0x7fce5c196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.740+0000 7fce61000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce5c101f70 0x7fce5c1972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.740+0000 7fce61000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce5c102910 0x7fce5c19e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.740+0000 7fce61000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c19e870 con 0x7fce5c102910 2024-03-30T00:59:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.740+0000 7fce61000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c19e9e0 con 0x7fce5c100d80 2024-03-30T00:59:29.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.740+0000 7fce61000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c19eba0 con 0x7fce5c101f70 2024-03-30T00:59:29.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.741+0000 7fce59e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce5c100d80 0x7fce5c196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.741+0000 7fce59e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce5c100d80 0x7fce5c196d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:43994/0 (socket says 172.21.15.116:43994) 2024-03-30T00:59:29.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.741+0000 7fce59e00700 1 -- 172.21.15.116:0/1470364201 learned_addr learned my addr 172.21.15.116:0/1470364201 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:29.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.741+0000 7fce5a800700 1 --2- 172.21.15.116:0/1470364201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce5c102910 0x7fce5c19e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.741+0000 7fce59400700 1 --2- 172.21.15.116:0/1470364201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce5c101f70 0x7fce5c1972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.741+0000 7fce59e00700 1 -- 172.21.15.116:0/1470364201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce5c101f70 msgr2=0x7fce5c1972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.741+0000 7fce59e00700 1 --2- 172.21.15.116:0/1470364201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce5c101f70 0x7fce5c1972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.741+0000 7fce59e00700 1 -- 172.21.15.116:0/1470364201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce5c102910 msgr2=0x7fce5c19e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.741+0000 7fce59e00700 1 --2- 172.21.15.116:0/1470364201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce5c102910 0x7fce5c19e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.741+0000 7fce59e00700 1 -- 172.21.15.116:0/1470364201 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4c0095f0 con 0x7fce5c100d80 2024-03-30T00:59:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.741+0000 7fce59e00700 1 --2- 172.21.15.116:0/1470364201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce5c100d80 0x7fce5c196d80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fce4c003730 tx=0x7fce4c00fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.742+0000 7fce52a00700 1 -- 172.21.15.116:0/1470364201 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c01d070 con 0x7fce5c100d80 2024-03-30T00:59:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.742+0000 7fce61000700 1 -- 172.21.15.116:0/1470364201 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5c19ee20 con 0x7fce5c100d80 2024-03-30T00:59:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.742+0000 7fce52a00700 1 -- 172.21.15.116:0/1470364201 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce4c00bda0 con 0x7fce5c100d80 2024-03-30T00:59:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.742+0000 7fce52a00700 1 -- 172.21.15.116:0/1470364201 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c004b30 con 0x7fce5c100d80 2024-03-30T00:59:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.742+0000 7fce61000700 1 -- 172.21.15.116:0/1470364201 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5c19f1e0 con 0x7fce5c100d80 2024-03-30T00:59:29.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.745+0000 7fce52a00700 1 -- 172.21.15.116:0/1470364201 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fce4c004c90 con 0x7fce5c100d80 2024-03-30T00:59:29.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.746+0000 7fce74b43b80 1 Processor -- start 2024-03-30T00:59:29.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.746+0000 7fce74b43b80 1 -- start start 2024-03-30T00:59:29.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.746+0000 7fce52a00700 1 --2- 172.21.15.116:0/1470364201 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fce400a1120 0x7fce400a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.746+0000 7fce52a00700 1 -- 172.21.15.116:0/1470364201 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fce4c0cda20 con 0x7fce5c100d80 2024-03-30T00:59:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.746+0000 7fce5a800700 1 --2- 172.21.15.116:0/1470364201 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fce400a1120 0x7fce400a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.746+0000 7fce74b43b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f0850 0x55574f2f0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce74b43b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f11b0 0x55574f2f1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce74b43b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 0x55574f2f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce74b43b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574f248a10 con 0x55574f2f0850 2024-03-30T00:59:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce74b43b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574f2463d0 con 0x55574f2f2050 2024-03-30T00:59:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce74b43b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574f241cc0 con 0x55574f2f11b0 2024-03-30T00:59:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce5a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 0x55574f2f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce59e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f0850 0x55574f2f0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce59400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f11b0 0x55574f2f1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce5a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 0x55574f2f24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:44002/0 (socket says 172.21.15.116:44002) 2024-03-30T00:59:29.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce59400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f11b0 0x55574f2f1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59646/0 (socket says 172.21.15.116:59646) 2024-03-30T00:59:29.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce5a800700 1 -- 172.21.15.116:0/3676015323 learned_addr learned my addr 172.21.15.116:0/3676015323 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce59e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f0850 0x55574f2f0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60110/0 (socket says 172.21.15.116:60110) 2024-03-30T00:59:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce59400700 1 -- 172.21.15.116:0/3676015323 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 msgr2=0x55574f2f24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce59400700 1 --2- 172.21.15.116:0/3676015323 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 0x55574f2f24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce59400700 1 -- 172.21.15.116:0/3676015323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f0850 msgr2=0x55574f2f0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce59400700 1 --2- 172.21.15.116:0/3676015323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f0850 0x55574f2f0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce59400700 1 -- 172.21.15.116:0/3676015323 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55574f0c3430 con 0x55574f2f11b0 2024-03-30T00:59:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce59e00700 1 --2- 172.21.15.116:0/3676015323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f0850 0x55574f2f0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:59:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.747+0000 7fce5a800700 1 --2- 172.21.15.116:0/1470364201 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fce400a1120 0x7fce400a35e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fce5400f930 tx=0x7fce5400fcf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.748+0000 7fce59400700 1 --2- 172.21.15.116:0/3676015323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f11b0 0x55574f2f1a70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fce48007c70 tx=0x7fce48007f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5c39f10a4f433e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.748+0000 7fce47400700 1 -- 172.21.15.116:0/3676015323 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce48004310 con 0x55574f2f11b0 2024-03-30T00:59:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.748+0000 7fce47400700 1 -- 172.21.15.116:0/3676015323 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce48010e20 con 0x55574f2f11b0 2024-03-30T00:59:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.748+0000 7fce47400700 1 -- 172.21.15.116:0/3676015323 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce48005020 con 0x55574f2f11b0 2024-03-30T00:59:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.749+0000 7fce74b43b80 1 -- 172.21.15.116:0/3676015323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f11b0 msgr2=0x55574f2f1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.749+0000 7fce74b43b80 1 --2- 172.21.15.116:0/3676015323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f11b0 0x55574f2f1a70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fce48007c70 tx=0x7fce48007f80 comp rx=0 tx=0).stop 2024-03-30T00:59:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.749+0000 7fce74b43b80 1 -- 172.21.15.116:0/3676015323 shutdown_connections 2024-03-30T00:59:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.749+0000 7fce74b43b80 1 --2- 172.21.15.116:0/3676015323 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 0x55574f2f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.749+0000 7fce74b43b80 1 --2- 172.21.15.116:0/3676015323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f0850 0x55574f2f0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.749+0000 7fce74b43b80 1 --2- 172.21.15.116:0/3676015323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f11b0 0x55574f2f1a70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.749+0000 7fce74b43b80 1 -- 172.21.15.116:0/3676015323 >> 172.21.15.116:0/3676015323 conn(0x55574f24f720 msgr2=0x55574f2eba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.749+0000 7fce74b43b80 1 -- 172.21.15.116:0/3676015323 shutdown_connections 2024-03-30T00:59:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.749+0000 7fce74b43b80 1 -- 172.21.15.116:0/3676015323 wait complete. 2024-03-30T00:59:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.749+0000 7fce74b43b80 1 -- start start 2024-03-30T00:59:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.750+0000 7fce74b43b80 1 Processor -- start 2024-03-30T00:59:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.750+0000 7fce74b43b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f0850 0x55574f2feaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.750+0000 7fce74b43b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f11b0 0x55574f2ff030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.750+0000 7fce74b43b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 0x55574f306070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.750+0000 7fce74b43b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574f1dde00 con 0x55574f2f11b0 2024-03-30T00:59:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.750+0000 7fce74b43b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574f1ddaa0 con 0x55574f2f2050 2024-03-30T00:59:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.750+0000 7fce74b43b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574f0c5720 con 0x55574f2f0850 2024-03-30T00:59:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.750+0000 7fce74b43b80 10 client.0 _ll_register_callbacks cb 0x55574f1e74f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:59:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.750+0000 7fce59e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f0850 0x55574f2feaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.750+0000 7fce59e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f0850 0x55574f2feaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59654/0 (socket says 172.21.15.116:59654) 2024-03-30T00:59:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.750+0000 7fce59e00700 1 -- 172.21.15.116:0/2711151242 learned_addr learned my addr 172.21.15.116:0/2711151242 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.750+0000 7fce59400700 1 --2- 172.21.15.116:0/2711151242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f11b0 0x55574f2ff030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.751+0000 7fce5a800700 1 --2- 172.21.15.116:0/2711151242 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 0x55574f306070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.751+0000 7fce59e00700 1 -- 172.21.15.116:0/2711151242 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 msgr2=0x55574f306070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.751+0000 7fce59e00700 1 --2- 172.21.15.116:0/2711151242 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 0x55574f306070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.751+0000 7fce59e00700 1 -- 172.21.15.116:0/2711151242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f11b0 msgr2=0x55574f2ff030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.751+0000 7fce59e00700 1 --2- 172.21.15.116:0/2711151242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f11b0 0x55574f2ff030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.751+0000 7fce59e00700 1 -- 172.21.15.116:0/2711151242 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5c19f1e0 con 0x55574f2f0850 2024-03-30T00:59:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.751+0000 7fce59e00700 1 --2- 172.21.15.116:0/2711151242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f0850 0x55574f2feaf0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fce4c09ab20 tx=0x7fce4c09b0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fbeb9e326423d03 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.751+0000 7fce3be00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T00:59:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.751+0000 7fce3be00700 1 -- 172.21.15.116:0/2711151242 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c0a2cf0 con 0x55574f2f0850 2024-03-30T00:59:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.751+0000 7fce3be00700 1 -- 172.21.15.116:0/2711151242 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fce4c09b290 con 0x55574f2f0850 2024-03-30T00:59:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.751+0000 7fce74b43b80 1 -- 172.21.15.116:0/2711151242 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55574f202400 con 0x55574f2f0850 2024-03-30T00:59:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.752+0000 7fce3be00700 1 -- 172.21.15.116:0/2711151242 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce4c09ab90 con 0x55574f2f0850 2024-03-30T00:59:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.752+0000 7fce74b43b80 10 client.25828 fetch_fsmap learned FSMap version 23 2024-03-30T00:59:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.752+0000 7fce74b43b80 1 -- 172.21.15.116:0/2711151242 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fce480078a0 con 0x55574f2f0850 2024-03-30T00:59:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.752+0000 7fce3be00700 1 -- 172.21.15.116:0/2711151242 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce4c064020 con 0x55574f2f0850 2024-03-30T00:59:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.752+0000 7fce3be00700 1 -- 172.21.15.116:0/2711151242 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fce4c09bba0 con 0x55574f2f0850 2024-03-30T00:59:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.752+0000 7fce74b43b80 10 client.25828 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:59:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.752+0000 7fce74b43b80 10 client.25828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:59:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.752+0000 7fce74b43b80 20 client.25828 populate_metadata read hostname 'smithi116' 2024-03-30T00:59:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.752+0000 7fce74b43b80 1 --2- 172.21.15.116:0/2711151242 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55574f30a490 0x55574f30c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.752+0000 7fce74b43b80 4 client.25828 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:59:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.752+0000 7fce74b43b80 1 -- 172.21.15.116:0/2711151242 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55574f23a780 con 0x55574f30a490 2024-03-30T00:59:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.753+0000 7fce59400700 1 --2- 172.21.15.116:0/2711151242 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55574f30a490 0x55574f30c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.753+0000 7fce3be00700 10 client.25828 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.753+0000 7fce59400700 1 --2- 172.21.15.116:0/2711151242 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55574f30a490 0x55574f30c8b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f69343a2223141e3 server_cookie=9290e44202b222f6 in_seq=0 out_seq=0 2024-03-30T00:59:29.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.758+0000 7fce3be00700 1 -- 172.21.15.116:0/2711151242 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55574f23a780 con 0x55574f30a490 2024-03-30T00:59:29.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.758+0000 7fce3be00700 10 client.25828 handle_command_reply: tid=0 2024-03-30T00:59:29.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.759+0000 7fce74b43b80 1 client.25828 shutdown 2024-03-30T00:59:29.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.759+0000 7fce74b43b80 1 -- 172.21.15.116:0/2711151242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f0850 msgr2=0x55574f2feaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:29.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.759+0000 7fce74b43b80 1 --2- 172.21.15.116:0/2711151242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f0850 0x55574f2feaf0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fce4c09ab20 tx=0x7fce4c09b0d0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.759+0000 7fce74b43b80 1 -- 172.21.15.116:0/2711151242 shutdown_connections 2024-03-30T00:59:29.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.759+0000 7fce74b43b80 1 --2- 172.21.15.116:0/2711151242 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55574f30a490 0x55574f30c8b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.759+0000 7fce74b43b80 1 --2- 172.21.15.116:0/2711151242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f0850 0x55574f2feaf0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.759+0000 7fce74b43b80 1 --2- 172.21.15.116:0/2711151242 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 0x55574f306070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.759+0000 7fce74b43b80 1 --2- 172.21.15.116:0/2711151242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f11b0 0x55574f2ff030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.759+0000 7fce3be00700 0 client.25828 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:29.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.759+0000 7fce74b43b80 1 -- 172.21.15.116:0/2711151242 >> 172.21.15.116:0/2711151242 conn(0x55574f24f720 msgr2=0x55574f2ef1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:29.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.759+0000 7fce74b43b80 1 -- 172.21.15.116:0/2711151242 shutdown_connections 2024-03-30T00:59:29.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.759+0000 7fce74b43b80 1 -- 172.21.15.116:0/2711151242 wait complete. 2024-03-30T00:59:29.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.759+0000 7fce74b43b80 20 client.25828 trim_cache size 0 max 16384 2024-03-30T00:59:29.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.777+0000 7fce74b43b80 1 Processor -- start 2024-03-30T00:59:29.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce74b43b80 1 -- start start 2024-03-30T00:59:29.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce74b43b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f0850 0x55574f2eaf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce74b43b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f11b0 0x55574f306030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce74b43b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 0x55574f2ef1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce74b43b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574f2ec6e0 con 0x55574f2f11b0 2024-03-30T00:59:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce74b43b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574f2ec920 con 0x55574f2f2050 2024-03-30T00:59:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce74b43b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574f2e7500 con 0x55574f2f0850 2024-03-30T00:59:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce59400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f11b0 0x55574f306030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce5a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 0x55574f2ef1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce59400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f11b0 0x55574f306030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60138/0 (socket says 172.21.15.116:60138) 2024-03-30T00:59:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce59400700 1 -- 172.21.15.116:0/2999593720 learned_addr learned my addr 172.21.15.116:0/2999593720 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:29.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce59e00700 1 --2- 172.21.15.116:0/2999593720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f0850 0x55574f2eaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce59400700 1 -- 172.21.15.116:0/2999593720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f0850 msgr2=0x55574f2eaf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:29.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce59400700 1 --2- 172.21.15.116:0/2999593720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f0850 0x55574f2eaf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce59400700 1 -- 172.21.15.116:0/2999593720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 msgr2=0x55574f2ef1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:29.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.778+0000 7fce59400700 1 --2- 172.21.15.116:0/2999593720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 0x55574f2ef1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.779+0000 7fce59400700 1 -- 172.21.15.116:0/2999593720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55574f3114e0 con 0x55574f2f11b0 2024-03-30T00:59:29.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.779+0000 7fce59e00700 1 --2- 172.21.15.116:0/2999593720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f0850 0x55574f2eaf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:59:29.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.779+0000 7fce5a800700 1 --2- 172.21.15.116:0/2999593720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 0x55574f2ef1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:59:29.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.779+0000 7fce59400700 1 --2- 172.21.15.116:0/2999593720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f11b0 0x55574f306030 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fce48005fd0 tx=0x7fce48000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e893e8fceafe90d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:29.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.779+0000 7fce47e00700 1 -- 172.21.15.116:0/2999593720 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4801f070 con 0x55574f2f11b0 2024-03-30T00:59:29.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.779+0000 7fce47e00700 1 -- 172.21.15.116:0/2999593720 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce48020040 con 0x55574f2f11b0 2024-03-30T00:59:29.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.779+0000 7fce47e00700 1 -- 172.21.15.116:0/2999593720 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce480045d0 con 0x55574f2f11b0 2024-03-30T00:59:29.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.780+0000 7fce74b43b80 1 -- 172.21.15.116:0/2999593720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f11b0 msgr2=0x55574f306030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:29.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.780+0000 7fce74b43b80 1 --2- 172.21.15.116:0/2999593720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f11b0 0x55574f306030 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fce48005fd0 tx=0x7fce48000bc0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.780+0000 7fce74b43b80 1 -- 172.21.15.116:0/2999593720 shutdown_connections 2024-03-30T00:59:29.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.780+0000 7fce74b43b80 1 --2- 172.21.15.116:0/2999593720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f0850 0x55574f2eaf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.780+0000 7fce74b43b80 1 --2- 172.21.15.116:0/2999593720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f2050 0x55574f2ef1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.780+0000 7fce74b43b80 1 --2- 172.21.15.116:0/2999593720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f11b0 0x55574f306030 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.780+0000 7fce74b43b80 1 -- 172.21.15.116:0/2999593720 >> 172.21.15.116:0/2999593720 conn(0x55574f24f720 msgr2=0x55574f2f8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:29.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.780+0000 7fce74b43b80 1 -- 172.21.15.116:0/2999593720 shutdown_connections 2024-03-30T00:59:29.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.780+0000 7fce74b43b80 1 -- 172.21.15.116:0/2999593720 wait complete. 2024-03-30T00:59:29.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.780+0000 7fce74b43b80 1 -- start start 2024-03-30T00:59:29.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.781+0000 7fce74b43b80 1 Processor -- start 2024-03-30T00:59:29.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.781+0000 7fce74b43b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f0850 0x55574f3053f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.781+0000 7fce74b43b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f11b0 0x55574f305930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.781+0000 7fce74b43b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f2050 0x55574f35e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.781+0000 7fce74b43b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574f35eaa0 con 0x55574f2f0850 2024-03-30T00:59:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.781+0000 7fce74b43b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574f35ec10 con 0x55574f2f11b0 2024-03-30T00:59:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.781+0000 7fce74b43b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574f35edd0 con 0x55574f2f2050 2024-03-30T00:59:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.781+0000 7fce74b43b80 10 client.0 _ll_register_callbacks cb 0x55574f1e74f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:59:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.781+0000 7fce59e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f0850 0x55574f3053f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.781+0000 7fce59400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f11b0 0x55574f305930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.781+0000 7fce59e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f0850 0x55574f3053f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60146/0 (socket says 172.21.15.116:60146) 2024-03-30T00:59:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.781+0000 7fce59e00700 1 -- 172.21.15.116:0/1452394249 learned_addr learned my addr 172.21.15.116:0/1452394249 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.782+0000 7fce5a800700 1 --2- 172.21.15.116:0/1452394249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f2050 0x55574f35e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.782+0000 7fce59e00700 1 -- 172.21.15.116:0/1452394249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f2050 msgr2=0x55574f35e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.782+0000 7fce59e00700 1 --2- 172.21.15.116:0/1452394249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f2050 0x55574f35e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.782+0000 7fce59e00700 1 -- 172.21.15.116:0/1452394249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f11b0 msgr2=0x55574f305930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.782+0000 7fce59e00700 1 --2- 172.21.15.116:0/1452394249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f11b0 0x55574f305930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.782+0000 7fce59e00700 1 -- 172.21.15.116:0/1452394249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce54009040 con 0x55574f2f0850 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.782+0000 7fce59400700 1 --2- 172.21.15.116:0/1452394249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f11b0 0x55574f305930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.782+0000 7fce59e00700 1 --2- 172.21.15.116:0/1452394249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f0850 0x55574f3053f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fce4c0a2a50 tx=0x7fce4c00fee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0223ac8edfda18e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.782+0000 7fce3be00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.782+0000 7fce3be00700 1 -- 172.21.15.116:0/1452394249 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c0b3480 con 0x55574f2f0850 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.782+0000 7fce3be00700 1 -- 172.21.15.116:0/1452394249 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fce4c065080 con 0x55574f2f0850 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.782+0000 7fce74b43b80 1 -- 172.21.15.116:0/1452394249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce48010e20 con 0x55574f2f0850 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.782+0000 7fce3be00700 1 -- 172.21.15.116:0/1452394249 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce4c09b290 con 0x55574f2f0850 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.782+0000 7fce74b43b80 10 client.15786 fetch_fsmap learned FSMap version 23 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.783+0000 7fce74b43b80 1 -- 172.21.15.116:0/1452394249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55574f0c3430 con 0x55574f2f0850 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.783+0000 7fce3be00700 1 -- 172.21.15.116:0/1452394249 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce4c07d020 con 0x55574f2f0850 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.783+0000 7fce3be00700 1 -- 172.21.15.116:0/1452394249 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fce4c0662b0 con 0x55574f2f0850 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.783+0000 7fce74b43b80 10 client.15786 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:59:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.783+0000 7fce74b43b80 10 client.15786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:59:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.783+0000 7fce74b43b80 20 client.15786 populate_metadata read hostname 'smithi116' 2024-03-30T00:59:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.783+0000 7fce74b43b80 1 --2- 172.21.15.116:0/1452394249 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55574f360400 0x55574f362820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.783+0000 7fce74b43b80 4 client.15786 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T00:59:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.783+0000 7fce74b43b80 1 -- 172.21.15.116:0/1452394249 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55574f090d70 con 0x55574f360400 2024-03-30T00:59:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.783+0000 7fce59400700 1 --2- 172.21.15.116:0/1452394249 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55574f360400 0x55574f362820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.784+0000 7fce3be00700 10 client.15786 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:29.784+0000 7fce59400700 1 --2- 172.21.15.116:0/1452394249 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55574f360400 0x55574f362820 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56f5fa60823b2131 server_cookie=29dad2ff133a290 in_seq=0 out_seq=0 2024-03-30T00:59:30.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce3be00700 1 -- 172.21.15.116:0/1452394249 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55574f090d70 con 0x55574f360400 2024-03-30T00:59:30.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce3be00700 10 client.15786 handle_command_reply: tid=0 2024-03-30T00:59:30.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce74b43b80 1 client.15786 shutdown 2024-03-30T00:59:30.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce74b43b80 1 -- 172.21.15.116:0/1452394249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f0850 msgr2=0x55574f3053f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:30.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce74b43b80 1 --2- 172.21.15.116:0/1452394249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f0850 0x55574f3053f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fce4c0a2a50 tx=0x7fce4c00fee0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce74b43b80 1 -- 172.21.15.116:0/1452394249 shutdown_connections 2024-03-30T00:59:30.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce74b43b80 1 --2- 172.21.15.116:0/1452394249 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55574f360400 0x55574f362820 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce74b43b80 1 --2- 172.21.15.116:0/1452394249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55574f2f2050 0x55574f35e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce74b43b80 1 --2- 172.21.15.116:0/1452394249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55574f2f0850 0x55574f3053f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce74b43b80 1 --2- 172.21.15.116:0/1452394249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55574f2f11b0 0x55574f305930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce3be00700 0 client.15786 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:30.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce74b43b80 1 -- 172.21.15.116:0/1452394249 >> 172.21.15.116:0/1452394249 conn(0x55574f24f720 msgr2=0x55574f2f8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:30.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce74b43b80 1 -- 172.21.15.116:0/1452394249 shutdown_connections 2024-03-30T00:59:30.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce74b43b80 1 -- 172.21.15.116:0/1452394249 wait complete. 2024-03-30T00:59:30.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.337+0000 7fce74b43b80 20 client.15786 trim_cache size 0 max 16384 2024-03-30T00:59:30.345 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:59:30.345 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-30T00:59:30.345 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "38a16ed4-3eba-4ac3-85f4-17229e993d03", 2024-03-30T00:59:30.346 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-30T00:59:30.346 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:59:30.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.338+0000 7fce61000700 1 -- 172.21.15.116:0/1470364201 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fce400a1120 msgr2=0x7fce400a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:30.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.338+0000 7fce61000700 1 --2- 172.21.15.116:0/1470364201 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fce400a1120 0x7fce400a35e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fce5400f930 tx=0x7fce5400fcf0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.338+0000 7fce61000700 1 -- 172.21.15.116:0/1470364201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce5c100d80 msgr2=0x7fce5c196d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:30.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.338+0000 7fce61000700 1 --2- 172.21.15.116:0/1470364201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce5c100d80 0x7fce5c196d80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fce4c003730 tx=0x7fce4c00fa00 comp rx=0 tx=0).stop 2024-03-30T00:59:30.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.338+0000 7fce61000700 1 -- 172.21.15.116:0/1470364201 shutdown_connections 2024-03-30T00:59:30.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.338+0000 7fce61000700 1 --2- 172.21.15.116:0/1470364201 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fce400a1120 0x7fce400a35e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.338+0000 7fce61000700 1 --2- 172.21.15.116:0/1470364201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce5c102910 0x7fce5c19e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.338+0000 7fce61000700 1 --2- 172.21.15.116:0/1470364201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce5c100d80 0x7fce5c196d80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.338+0000 7fce61000700 1 --2- 172.21.15.116:0/1470364201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce5c101f70 0x7fce5c1972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.338+0000 7fce61000700 1 -- 172.21.15.116:0/1470364201 >> 172.21.15.116:0/1470364201 conn(0x7fce5c0fc2f0 msgr2=0x7fce5c105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:30.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.338+0000 7fce61000700 1 -- 172.21.15.116:0/1470364201 shutdown_connections 2024-03-30T00:59:30.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.339+0000 7fce61000700 1 -- 172.21.15.116:0/1470364201 wait complete. 2024-03-30T00:59:30.363 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T00:59:30.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:30 smithi181 ceph-mon[47622]: pgmap v340: 129 pgs: 129 active+clean; 4.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 341 KiB/s wr, 40 op/s 2024-03-30T00:59:30.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.557+0000 7f7372000700 1 -- 172.21.15.116:0/1097067868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736c073810 msgr2=0x7f736c073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:30.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.557+0000 7f7372000700 1 --2- 172.21.15.116:0/1097067868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736c073810 0x7f736c073c90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7358009990 tx=0x7f7358009ca0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.557+0000 7f7372000700 1 -- 172.21.15.116:0/1097067868 shutdown_connections 2024-03-30T00:59:30.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.557+0000 7f7372000700 1 --2- 172.21.15.116:0/1097067868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f736c108c40 0x7f736c10b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.557+0000 7f7372000700 1 --2- 172.21.15.116:0/1097067868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f736c074df0 0x7f736c073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.557+0000 7f7372000700 1 --2- 172.21.15.116:0/1097067868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736c073810 0x7f736c073c90 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.558+0000 7f7372000700 1 -- 172.21.15.116:0/1097067868 >> 172.21.15.116:0/1097067868 conn(0x7f736c0fc2f0 msgr2=0x7f736c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:30.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.558+0000 7f7372000700 1 -- 172.21.15.116:0/1097067868 shutdown_connections 2024-03-30T00:59:30.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.558+0000 7f7372000700 1 -- 172.21.15.116:0/1097067868 wait complete. 2024-03-30T00:59:30.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.559+0000 7f7372000700 1 Processor -- start 2024-03-30T00:59:30.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.559+0000 7f7372000700 1 -- start start 2024-03-30T00:59:30.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.559+0000 7f7372000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f736c073810 0x7f736c198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.559+0000 7f7372000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736c074df0 0x7f736c1994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.559+0000 7f7372000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f736c108c40 0x7f736c1a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.559+0000 7f7372000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c1a0a90 con 0x7f736c074df0 2024-03-30T00:59:30.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.559+0000 7f7372000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c1a0bd0 con 0x7f736c108c40 2024-03-30T00:59:30.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.559+0000 7f7372000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c1a0d60 con 0x7f736c073810 2024-03-30T00:59:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.559+0000 7f736aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736c074df0 0x7f736c1994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.559+0000 7f736aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736c074df0 0x7f736c1994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60162/0 (socket says 172.21.15.116:60162) 2024-03-30T00:59:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.559+0000 7f736aa00700 1 -- 172.21.15.116:0/674105780 learned_addr learned my addr 172.21.15.116:0/674105780 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.559+0000 7f736be00700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f736c108c40 0x7f736c1a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.559+0000 7f736b400700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f736c073810 0x7f736c198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.560+0000 7f736aa00700 1 -- 172.21.15.116:0/674105780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f736c073810 msgr2=0x7f736c198f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.560+0000 7f736aa00700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f736c073810 0x7f736c198f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.560+0000 7f736aa00700 1 -- 172.21.15.116:0/674105780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f736c108c40 msgr2=0x7f736c1a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.560+0000 7f736aa00700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f736c108c40 0x7f736c1a04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.560+0000 7f736aa00700 1 -- 172.21.15.116:0/674105780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c009520 con 0x7f736c074df0 2024-03-30T00:59:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.560+0000 7f736be00700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f736c108c40 0x7f736c1a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T00:59:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.560+0000 7f736aa00700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736c074df0 0x7f736c1994a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7358000c00 tx=0x7f7358005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:30.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.560+0000 7f7363e00700 1 -- 172.21.15.116:0/674105780 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735801d070 con 0x7f736c074df0 2024-03-30T00:59:30.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.560+0000 7f7363e00700 1 -- 172.21.15.116:0/674105780 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7358021410 con 0x7f736c074df0 2024-03-30T00:59:30.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.560+0000 7f7363e00700 1 -- 172.21.15.116:0/674105780 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7358017650 con 0x7f736c074df0 2024-03-30T00:59:30.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.560+0000 7f7372000700 1 -- 172.21.15.116:0/674105780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73580095f0 con 0x7f736c074df0 2024-03-30T00:59:30.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.561+0000 7f7372000700 1 -- 172.21.15.116:0/674105780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f736c1a11c0 con 0x7f736c074df0 2024-03-30T00:59:30.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.563+0000 7f7363e00700 1 -- 172.21.15.116:0/674105780 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f7358021580 con 0x7f736c074df0 2024-03-30T00:59:30.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.564+0000 7f7385b7ab80 1 Processor -- start 2024-03-30T00:59:30.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.565+0000 7f7385b7ab80 1 -- start start 2024-03-30T00:59:30.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.565+0000 7f7363e00700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f73500a1120 0x7f73500a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.565+0000 7f7363e00700 1 -- 172.21.15.116:0/674105780 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f73580cdfe0 con 0x7f736c074df0 2024-03-30T00:59:30.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.565+0000 7f7385b7ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a89dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.565+0000 7f7385b7ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89e130 0x56091a89e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.565+0000 7f7385b7ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89efd0 0x56091a89f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.565+0000 7f7385b7ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091a7f5990 con 0x56091a89efd0 2024-03-30T00:59:30.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.565+0000 7f7385b7ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091a7f3350 con 0x56091a89d7d0 2024-03-30T00:59:30.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.565+0000 7f7385b7ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091a7eec40 con 0x56091a89e130 2024-03-30T00:59:30.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.565+0000 7f736be00700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f73500a1120 0x7f73500a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.565+0000 7f736b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a89dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.565+0000 7f736b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a89dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:44054/0 (socket says 172.21.15.116:44054) 2024-03-30T00:59:30.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.565+0000 7f736b400700 1 -- 172.21.15.116:0/3359392499 learned_addr learned my addr 172.21.15.116:0/3359392499 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:30.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.565+0000 7f736be00700 1 --2- 172.21.15.116:0/3359392499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89efd0 0x56091a89f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.566+0000 7f736aa00700 1 --2- 172.21.15.116:0/3359392499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89e130 0x56091a89e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.566+0000 7f736b400700 1 -- 172.21.15.116:0/3359392499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89e130 msgr2=0x56091a89e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:30.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.566+0000 7f736b400700 1 --2- 172.21.15.116:0/3359392499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89e130 0x56091a89e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.566+0000 7f736b400700 1 -- 172.21.15.116:0/3359392499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89efd0 msgr2=0x56091a89f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:30.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.566+0000 7f736b400700 1 --2- 172.21.15.116:0/3359392499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89efd0 0x56091a89f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.566+0000 7f736b400700 1 -- 172.21.15.116:0/3359392499 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56091a6703b0 con 0x56091a89d7d0 2024-03-30T00:59:30.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.566+0000 7f736b400700 1 --2- 172.21.15.116:0/3359392499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a89dbf0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f735c009f10 tx=0x7f735c011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49d0389ce32a06b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:30.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.566+0000 7f736be00700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f73500a1120 0x7f73500a35e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7364007f40 tx=0x7f7364007580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:30.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.566+0000 7f7357e00700 1 -- 172.21.15.116:0/3359392499 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c019630 con 0x56091a89d7d0 2024-03-30T00:59:30.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.566+0000 7f7357e00700 1 -- 172.21.15.116:0/3359392499 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f735c019c50 con 0x56091a89d7d0 2024-03-30T00:59:30.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.566+0000 7f7357e00700 1 -- 172.21.15.116:0/3359392499 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c00ec10 con 0x56091a89d7d0 2024-03-30T00:59:30.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.567+0000 7f7385b7ab80 1 -- 172.21.15.116:0/3359392499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 msgr2=0x56091a89dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:30.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.567+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/3359392499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a89dbf0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f735c009f10 tx=0x7f735c011720 comp rx=0 tx=0).stop 2024-03-30T00:59:30.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.567+0000 7f7385b7ab80 1 -- 172.21.15.116:0/3359392499 shutdown_connections 2024-03-30T00:59:30.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.567+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/3359392499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89efd0 0x56091a89f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.567+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/3359392499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a89dbf0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.567+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/3359392499 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89e130 0x56091a89e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.567+0000 7f7385b7ab80 1 -- 172.21.15.116:0/3359392499 >> 172.21.15.116:0/3359392499 conn(0x56091a7fc6a0 msgr2=0x56091a8989a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:30.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.567+0000 7f7385b7ab80 1 -- 172.21.15.116:0/3359392499 shutdown_connections 2024-03-30T00:59:30.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.567+0000 7f7385b7ab80 1 -- 172.21.15.116:0/3359392499 wait complete. 2024-03-30T00:59:30.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.568+0000 7f7385b7ab80 1 -- start start 2024-03-30T00:59:30.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.568+0000 7f7385b7ab80 1 Processor -- start 2024-03-30T00:59:30.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.568+0000 7f7385b7ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89d7d0 0x56091a8a7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.568+0000 7f7385b7ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89e130 0x56091a8a7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.568+0000 7f7385b7ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89efd0 0x56091a8aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.568+0000 7f7385b7ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091a78ad80 con 0x56091a89d7d0 2024-03-30T00:59:30.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.568+0000 7f7385b7ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091a78aa20 con 0x56091a89e130 2024-03-30T00:59:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.568+0000 7f7385b7ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091a6726a0 con 0x56091a89efd0 2024-03-30T00:59:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.568+0000 7f7385b7ab80 10 client.0 _ll_register_callbacks cb 0x56091a794470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:59:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.568+0000 7f736aa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89e130 0x56091a8a7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.568+0000 7f736b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89d7d0 0x56091a8a7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.568+0000 7f736aa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89e130 0x56091a8a7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:44058/0 (socket says 172.21.15.116:44058) 2024-03-30T00:59:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.568+0000 7f736aa00700 1 -- 172.21.15.116:0/590159213 learned_addr learned my addr 172.21.15.116:0/590159213 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.569+0000 7f736aa00700 1 -- 172.21.15.116:0/590159213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89efd0 msgr2=0x56091a8aebb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T00:59:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.569+0000 7f736aa00700 1 --2- 172.21.15.116:0/590159213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89efd0 0x56091a8aebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.569+0000 7f736aa00700 1 -- 172.21.15.116:0/590159213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89d7d0 msgr2=0x56091a8a7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.569+0000 7f736aa00700 1 --2- 172.21.15.116:0/590159213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89d7d0 0x56091a8a7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.569+0000 7f736aa00700 1 -- 172.21.15.116:0/590159213 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f735c009b70 con 0x56091a89e130 2024-03-30T00:59:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.569+0000 7f736aa00700 1 --2- 172.21.15.116:0/590159213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89e130 0x56091a8a7b70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7358097c20 tx=0x7f73580a1390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69a493edc146ffc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.569+0000 7f7354c00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T00:59:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.569+0000 7f7354c00700 1 -- 172.21.15.116:0/590159213 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735809a8a0 con 0x56091a89e130 2024-03-30T00:59:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.569+0000 7f7354c00700 1 -- 172.21.15.116:0/590159213 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f735809b8b0 con 0x56091a89e130 2024-03-30T00:59:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.569+0000 7f7385b7ab80 1 -- 172.21.15.116:0/590159213 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56091a7af380 con 0x56091a89e130 2024-03-30T00:59:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.569+0000 7f7354c00700 1 -- 172.21.15.116:0/590159213 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f735809b1b0 con 0x56091a89e130 2024-03-30T00:59:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.569+0000 7f7385b7ab80 10 client.25865 fetch_fsmap learned FSMap version 23 2024-03-30T00:59:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.569+0000 7f7385b7ab80 1 -- 172.21.15.116:0/590159213 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56091a61cb60 con 0x56091a89e130 2024-03-30T00:59:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.570+0000 7f7354c00700 1 -- 172.21.15.116:0/590159213 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56091a7af380 con 0x56091a89e130 2024-03-30T00:59:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.570+0000 7f7354c00700 1 -- 172.21.15.116:0/590159213 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7358096fa0 con 0x56091a89e130 2024-03-30T00:59:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.570+0000 7f7385b7ab80 10 client.25865 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:59:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.570+0000 7f7385b7ab80 10 client.25865 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:59:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.570+0000 7f7385b7ab80 20 client.25865 populate_metadata read hostname 'smithi116' 2024-03-30T00:59:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.570+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/590159213 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56091a8b30b0 0x56091a8b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.570+0000 7f7385b7ab80 4 client.25865 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T00:59:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.570+0000 7f7385b7ab80 1 -- 172.21.15.116:0/590159213 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56091a7e7700 con 0x56091a8b30b0 2024-03-30T00:59:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.570+0000 7f736b400700 1 --2- 172.21.15.116:0/590159213 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56091a8b30b0 0x56091a8b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.571+0000 7f736b400700 1 --2- 172.21.15.116:0/590159213 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56091a8b30b0 0x56091a8b54a0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90ecbf110a374e1d server_cookie=b2740a7876e97492 in_seq=0 out_seq=0 2024-03-30T00:59:30.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.571+0000 7f7354c00700 10 client.25865 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.574+0000 7f7354c00700 1 -- 172.21.15.116:0/590159213 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56091a7e7700 con 0x56091a8b30b0 2024-03-30T00:59:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.574+0000 7f7354c00700 10 client.25865 handle_command_reply: tid=0 2024-03-30T00:59:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.574+0000 7f7385b7ab80 1 client.25865 shutdown 2024-03-30T00:59:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.574+0000 7f7385b7ab80 1 -- 172.21.15.116:0/590159213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89e130 msgr2=0x56091a8a7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.574+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/590159213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89e130 0x56091a8a7b70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7358097c20 tx=0x7f73580a1390 comp rx=0 tx=0).stop 2024-03-30T00:59:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.574+0000 7f7385b7ab80 1 -- 172.21.15.116:0/590159213 shutdown_connections 2024-03-30T00:59:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.574+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/590159213 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56091a8b30b0 0x56091a8b54a0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.574+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/590159213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89efd0 0x56091a8aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.574+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/590159213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89d7d0 0x56091a8a7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.574+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/590159213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89e130 0x56091a8a7b70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.574+0000 7f7354c00700 0 client.25865 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.574+0000 7f7385b7ab80 1 -- 172.21.15.116:0/590159213 >> 172.21.15.116:0/590159213 conn(0x56091a7fc6a0 msgr2=0x56091a89c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.575+0000 7f7385b7ab80 1 -- 172.21.15.116:0/590159213 shutdown_connections 2024-03-30T00:59:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.575+0000 7f7385b7ab80 1 -- 172.21.15.116:0/590159213 wait complete. 2024-03-30T00:59:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.575+0000 7f7385b7ab80 20 client.25865 trim_cache size 0 max 16384 2024-03-30T00:59:30.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:30 smithi130 ceph-mon[47626]: pgmap v340: 129 pgs: 129 active+clean; 4.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 341 KiB/s wr, 40 op/s 2024-03-30T00:59:30.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.592+0000 7f7385b7ab80 1 Processor -- start 2024-03-30T00:59:30.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.592+0000 7f7385b7ab80 1 -- start start 2024-03-30T00:59:30.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f7385b7ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a89f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f7385b7ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89e130 0x56091a8aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f7385b7ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89efd0 0x56091a8a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f7385b7ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091a89d470 con 0x56091a89e130 2024-03-30T00:59:30.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f7385b7ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091a8994b0 con 0x56091a89d7d0 2024-03-30T00:59:30.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f7385b7ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091a8a8530 con 0x56091a89efd0 2024-03-30T00:59:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f736b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a89f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f736aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89e130 0x56091a8aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f736b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a89f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:44060/0 (socket says 172.21.15.116:44060) 2024-03-30T00:59:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f736b400700 1 -- 172.21.15.116:0/3713383671 learned_addr learned my addr 172.21.15.116:0/3713383671 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f736be00700 1 --2- 172.21.15.116:0/3713383671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89efd0 0x56091a8a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f736b400700 1 -- 172.21.15.116:0/3713383671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89efd0 msgr2=0x56091a8a75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f736b400700 1 --2- 172.21.15.116:0/3713383671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89efd0 0x56091a8a75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f736b400700 1 -- 172.21.15.116:0/3713383671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89e130 msgr2=0x56091a8aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f736b400700 1 --2- 172.21.15.116:0/3713383671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89e130 0x56091a8aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.593+0000 7f736b400700 1 -- 172.21.15.116:0/3713383671 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56091a7b4ea0 con 0x56091a89d7d0 2024-03-30T00:59:30.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.594+0000 7f736b400700 1 --2- 172.21.15.116:0/3713383671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a89f3b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f735c011a70 tx=0x7f735c005e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebe34997fe5fd209 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:30.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.594+0000 7f7361200700 1 -- 172.21.15.116:0/3713383671 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c00e820 con 0x56091a89d7d0 2024-03-30T00:59:30.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.594+0000 7f7361200700 1 -- 172.21.15.116:0/3713383671 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f735c0198f0 con 0x56091a89d7d0 2024-03-30T00:59:30.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.594+0000 7f7361200700 1 -- 172.21.15.116:0/3713383671 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c0052a0 con 0x56091a89d7d0 2024-03-30T00:59:30.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.595+0000 7f7385b7ab80 1 -- 172.21.15.116:0/3713383671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 msgr2=0x56091a89f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:30.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.595+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/3713383671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a89f3b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f735c011a70 tx=0x7f735c005e20 comp rx=0 tx=0).stop 2024-03-30T00:59:30.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.595+0000 7f7385b7ab80 1 -- 172.21.15.116:0/3713383671 shutdown_connections 2024-03-30T00:59:30.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.595+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/3713383671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89efd0 0x56091a8a75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.595+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/3713383671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a89f3b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.595+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/3713383671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89e130 0x56091a8aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.595+0000 7f7385b7ab80 1 -- 172.21.15.116:0/3713383671 >> 172.21.15.116:0/3713383671 conn(0x56091a7fc6a0 msgr2=0x56091a8915a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:30.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.595+0000 7f7385b7ab80 1 -- 172.21.15.116:0/3713383671 shutdown_connections 2024-03-30T00:59:30.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.595+0000 7f7385b7ab80 1 -- 172.21.15.116:0/3713383671 wait complete. 2024-03-30T00:59:30.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.595+0000 7f7385b7ab80 1 -- start start 2024-03-30T00:59:30.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.595+0000 7f7385b7ab80 1 Processor -- start 2024-03-30T00:59:30.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.596+0000 7f7385b7ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a8ae840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.596+0000 7f7385b7ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89e130 0x56091a8acea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.596+0000 7f7385b7ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89efd0 0x56091a8ad3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.596+0000 7f7385b7ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091a8ad920 con 0x56091a89efd0 2024-03-30T00:59:30.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.596+0000 7f7385b7ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091a8ada90 con 0x56091a89d7d0 2024-03-30T00:59:30.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.596+0000 7f7385b7ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091a8adc50 con 0x56091a89e130 2024-03-30T00:59:30.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.596+0000 7f7385b7ab80 10 client.0 _ll_register_callbacks cb 0x56091a794470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T00:59:30.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.596+0000 7f736b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a8ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.596+0000 7f736b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a8ae840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:44068/0 (socket says 172.21.15.116:44068) 2024-03-30T00:59:30.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.596+0000 7f736b400700 1 -- 172.21.15.116:0/540796028 learned_addr learned my addr 172.21.15.116:0/540796028 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T00:59:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.596+0000 7f736be00700 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89efd0 0x56091a8ad3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.596+0000 7f736aa00700 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89e130 0x56091a8acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f736b400700 1 -- 172.21.15.116:0/540796028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89e130 msgr2=0x56091a8acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f736b400700 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89e130 0x56091a8acea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f736b400700 1 -- 172.21.15.116:0/540796028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89efd0 msgr2=0x56091a8ad3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T00:59:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f736b400700 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89efd0 0x56091a8ad3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f736b400700 1 -- 172.21.15.116:0/540796028 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56091a6703b0 con 0x56091a89d7d0 2024-03-30T00:59:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f736be00700 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89efd0 0x56091a8ad3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:59:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f736aa00700 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89e130 0x56091a8acea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T00:59:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f736b400700 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a8ae840 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f735c000c00 tx=0x7f735c011aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd98fbf6172bcf05 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T00:59:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f7354c00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T00:59:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f7354c00700 1 -- 172.21.15.116:0/540796028 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c019630 con 0x56091a89d7d0 2024-03-30T00:59:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f7354c00700 1 -- 172.21.15.116:0/540796028 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f735c034030 con 0x56091a89d7d0 2024-03-30T00:59:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f7385b7ab80 1 -- 172.21.15.116:0/540796028 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f735c019c50 con 0x56091a89d7d0 2024-03-30T00:59:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f7354c00700 1 -- 172.21.15.116:0/540796028 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f735c019790 con 0x56091a89d7d0 2024-03-30T00:59:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f7385b7ab80 10 client.25871 fetch_fsmap learned FSMap version 23 2024-03-30T00:59:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.597+0000 7f7385b7ab80 1 -- 172.21.15.116:0/540796028 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56091a674cd0 con 0x56091a89d7d0 2024-03-30T00:59:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.598+0000 7f7354c00700 1 -- 172.21.15.116:0/540796028 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f735c041020 con 0x56091a89d7d0 2024-03-30T00:59:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.598+0000 7f7354c00700 1 -- 172.21.15.116:0/540796028 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f735c0038f0 con 0x56091a89d7d0 2024-03-30T00:59:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.598+0000 7f7385b7ab80 10 client.25871 fetch_fsmap finished waiting for FSMap version 23 2024-03-30T00:59:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.598+0000 7f7385b7ab80 10 client.25871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T00:59:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.598+0000 7f7385b7ab80 20 client.25871 populate_metadata read hostname 'smithi116' 2024-03-30T00:59:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.598+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56091a908220 0x56091a90a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T00:59:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.598+0000 7f7385b7ab80 4 client.25871 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T00:59:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.598+0000 7f7385b7ab80 1 -- 172.21.15.116:0/540796028 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56091a63dcf0 con 0x56091a908220 2024-03-30T00:59:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.598+0000 7f736aa00700 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56091a908220 0x56091a90a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T00:59:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.599+0000 7f7354c00700 10 client.25871 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.599+0000 7f736aa00700 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56091a908220 0x56091a90a610 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2570c43186feb483 server_cookie=f13f24410bd18762 in_seq=0 out_seq=0 2024-03-30T00:59:30.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.602+0000 7f7354c00700 1 -- 172.21.15.116:0/540796028 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x56091a63dcf0 con 0x56091a908220 2024-03-30T00:59:30.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.602+0000 7f7354c00700 10 client.25871 handle_command_reply: tid=0 2024-03-30T00:59:30.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.602+0000 7f7385b7ab80 1 client.25871 shutdown 2024-03-30T00:59:30.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.602+0000 7f7385b7ab80 1 -- 172.21.15.116:0/540796028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 msgr2=0x56091a8ae840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:30.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.602+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a8ae840 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f735c000c00 tx=0x7f735c011aa0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.602+0000 7f7385b7ab80 1 -- 172.21.15.116:0/540796028 shutdown_connections 2024-03-30T00:59:30.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.602+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56091a908220 0x56091a90a610 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.602+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56091a89efd0 0x56091a8ad3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.602+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56091a89d7d0 0x56091a8ae840 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.602+0000 7f7385b7ab80 1 --2- 172.21.15.116:0/540796028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56091a89e130 0x56091a8acea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.602+0000 7f7354c00700 0 client.25871 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T00:59:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.602+0000 7f7385b7ab80 1 -- 172.21.15.116:0/540796028 >> 172.21.15.116:0/540796028 conn(0x56091a7fc6a0 msgr2=0x56091a8a75d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.602+0000 7f7385b7ab80 1 -- 172.21.15.116:0/540796028 shutdown_connections 2024-03-30T00:59:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.602+0000 7f7385b7ab80 1 -- 172.21.15.116:0/540796028 wait complete. 2024-03-30T00:59:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.603+0000 7f7385b7ab80 20 client.25871 trim_cache size 0 max 16384 2024-03-30T00:59:30.612 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T00:59:30.612 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (3 inodes in the stack)", 2024-03-30T00:59:30.612 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-30T00:59:30.612 INFO:teuthology.orchestra.run.smithi116.stdout: "38a16ed4-3eba-4ac3-85f4-17229e993d03": { 2024-03-30T00:59:30.613 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-30T00:59:30.613 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "38a16ed4-3eba-4ac3-85f4-17229e993d03", 2024-03-30T00:59:30.613 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-30T00:59:30.613 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T00:59:30.613 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T00:59:30.613 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T00:59:30.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.603+0000 7f7372000700 1 -- 172.21.15.116:0/674105780 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f73500a1120 msgr2=0x7f73500a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:30.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.603+0000 7f7372000700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f73500a1120 0x7f73500a35e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7364007f40 tx=0x7f7364007580 comp rx=0 tx=0).stop 2024-03-30T00:59:30.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.603+0000 7f7372000700 1 -- 172.21.15.116:0/674105780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736c074df0 msgr2=0x7f736c1994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T00:59:30.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.603+0000 7f7372000700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736c074df0 0x7f736c1994a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7358000c00 tx=0x7f7358005740 comp rx=0 tx=0).stop 2024-03-30T00:59:30.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.603+0000 7f7372000700 1 -- 172.21.15.116:0/674105780 shutdown_connections 2024-03-30T00:59:30.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.603+0000 7f7372000700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f73500a1120 0x7f73500a35e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.603+0000 7f7372000700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f736c073810 0x7f736c198f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.603+0000 7f7372000700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f736c108c40 0x7f736c1a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.603+0000 7f7372000700 1 --2- 172.21.15.116:0/674105780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736c074df0 0x7f736c1994a0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T00:59:30.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.603+0000 7f7372000700 1 -- 172.21.15.116:0/674105780 >> 172.21.15.116:0/674105780 conn(0x7f736c0fc2f0 msgr2=0x7f736c104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T00:59:30.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.603+0000 7f7372000700 1 -- 172.21.15.116:0/674105780 shutdown_connections 2024-03-30T00:59:30.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T00:59:30.603+0000 7f7372000700 1 -- 172.21.15.116:0/674105780 wait complete. 2024-03-30T00:59:30.626 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:30 smithi116 ceph-mon[38086]: pgmap v340: 129 pgs: 129 active+clean; 4.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 341 KiB/s wr, 40 op/s 2024-03-30T00:59:30.628 INFO:tasks.cephfs.filesystem:scrub status for tag:38a16ed4-3eba-4ac3-85f4-17229e993d03 - {'path': '/', 'tag': '38a16ed4-3eba-4ac3-85f4-17229e993d03', 'options': 'recursive,force'} 2024-03-30T00:59:31.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:31 smithi181 ceph-mon[47622]: scrub queued for path: / 2024-03-30T00:59:31.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:31 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T00:59:31.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:31 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T00:59:31.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:31 smithi130 ceph-mon[47626]: scrub queued for path: / 2024-03-30T00:59:31.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:31 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T00:59:31.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:31 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T00:59:31.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:31 smithi116 ceph-mon[38086]: scrub queued for path: / 2024-03-30T00:59:31.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:31 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T00:59:31.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:31 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T00:59:32.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:32 smithi181 ceph-mon[47622]: pgmap v341: 129 pgs: 129 active+clean; 6.9 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 616 KiB/s wr, 57 op/s 2024-03-30T00:59:32.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:32 smithi130 ceph-mon[47626]: pgmap v341: 129 pgs: 129 active+clean; 6.9 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 616 KiB/s wr, 57 op/s 2024-03-30T00:59:32.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:32 smithi116 ceph-mon[38086]: pgmap v341: 129 pgs: 129 active+clean; 6.9 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 616 KiB/s wr, 57 op/s 2024-03-30T00:59:34.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:34 smithi130 ceph-mon[47626]: pgmap v342: 129 pgs: 129 active+clean; 6.9 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 555 KiB/s wr, 57 op/s 2024-03-30T00:59:34.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:34 smithi116 ceph-mon[38086]: pgmap v342: 129 pgs: 129 active+clean; 6.9 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 555 KiB/s wr, 57 op/s 2024-03-30T00:59:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:34 smithi181 ceph-mon[47622]: pgmap v342: 129 pgs: 129 active+clean; 6.9 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 555 KiB/s wr, 57 op/s 2024-03-30T00:59:35.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:35 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T00:59:35.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:35 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T00:59:35.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:35 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T00:59:36.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:36 smithi130 ceph-mon[47626]: pgmap v343: 129 pgs: 129 active+clean; 14 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1.2 MiB/s wr, 84 op/s 2024-03-30T00:59:36.682 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:36 smithi116 ceph-mon[38086]: pgmap v343: 129 pgs: 129 active+clean; 14 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1.2 MiB/s wr, 84 op/s 2024-03-30T00:59:36.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:36 smithi181 ceph-mon[47622]: pgmap v343: 129 pgs: 129 active+clean; 14 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1.2 MiB/s wr, 84 op/s 2024-03-30T00:59:37.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:37 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T00:59:37.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:37 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T00:59:37.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:37 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T00:59:38.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:38 smithi130 ceph-mon[47626]: pgmap v344: 129 pgs: 129 active+clean; 29 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 108 op/s 2024-03-30T00:59:38.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:38 smithi130 ceph-mon[47626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T00:59:38.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:38 smithi116 ceph-mon[38086]: pgmap v344: 129 pgs: 129 active+clean; 29 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 108 op/s 2024-03-30T00:59:38.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:38 smithi116 ceph-mon[38086]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T00:59:38.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:38 smithi181 ceph-mon[47622]: pgmap v344: 129 pgs: 129 active+clean; 29 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 108 op/s 2024-03-30T00:59:38.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:38 smithi181 ceph-mon[47622]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T00:59:38.951 INFO:tasks.workunit.client.0.smithi116.stdout: 17 278890 1059 146810 987 55368 1097 2024-03-30T00:59:40.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:40 smithi130 ceph-mon[47626]: pgmap v345: 129 pgs: 129 active+clean; 34 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 128 op/s 2024-03-30T00:59:40.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:40 smithi116 ceph-mon[38086]: pgmap v345: 129 pgs: 129 active+clean; 34 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 128 op/s 2024-03-30T00:59:40.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:40 smithi181 ceph-mon[47622]: pgmap v345: 129 pgs: 129 active+clean; 34 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 128 op/s 2024-03-30T00:59:42.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:42 smithi130 ceph-mon[47626]: pgmap v346: 129 pgs: 129 active+clean; 52 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 196 op/s 2024-03-30T00:59:42.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:42 smithi116 ceph-mon[38086]: pgmap v346: 129 pgs: 129 active+clean; 52 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 196 op/s 2024-03-30T00:59:42.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:42 smithi181 ceph-mon[47622]: pgmap v346: 129 pgs: 129 active+clean; 52 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 196 op/s 2024-03-30T00:59:43.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:43 smithi130 ceph-mon[47626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T00:59:43.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:43 smithi116 ceph-mon[38086]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T00:59:43.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:43 smithi181 ceph-mon[47622]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T00:59:44.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:44 smithi130 ceph-mon[47626]: pgmap v347: 129 pgs: 129 active+clean; 55 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 204 op/s 2024-03-30T00:59:44.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:44 smithi116 ceph-mon[38086]: pgmap v347: 129 pgs: 129 active+clean; 55 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 204 op/s 2024-03-30T00:59:44.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:44 smithi181 ceph-mon[47622]: pgmap v347: 129 pgs: 129 active+clean; 55 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 204 op/s 2024-03-30T00:59:46.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:46 smithi130 ceph-mon[47626]: pgmap v348: 129 pgs: 129 active+clean; 59 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 292 op/s 2024-03-30T00:59:46.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:46 smithi116 ceph-mon[38086]: pgmap v348: 129 pgs: 129 active+clean; 59 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 292 op/s 2024-03-30T00:59:46.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:46 smithi181 ceph-mon[47622]: pgmap v348: 129 pgs: 129 active+clean; 59 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 292 op/s 2024-03-30T00:59:48.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:48 smithi130 ceph-mon[47626]: pgmap v349: 129 pgs: 129 active+clean; 65 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.8 MiB/s wr, 314 op/s 2024-03-30T00:59:48.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:48 smithi116 ceph-mon[38086]: pgmap v349: 129 pgs: 129 active+clean; 65 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.8 MiB/s wr, 314 op/s 2024-03-30T00:59:48.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:48 smithi181 ceph-mon[47622]: pgmap v349: 129 pgs: 129 active+clean; 65 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.8 MiB/s wr, 314 op/s 2024-03-30T00:59:48.951 INFO:tasks.workunit.client.0.smithi116.stdout: 29 478248 700 327435 629 127745 1404 2024-03-30T00:59:50.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:50 smithi130 ceph-mon[47626]: pgmap v350: 129 pgs: 129 active+clean; 65 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.6 MiB/s wr, 306 op/s 2024-03-30T00:59:50.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:50 smithi116 ceph-mon[38086]: pgmap v350: 129 pgs: 129 active+clean; 65 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.6 MiB/s wr, 306 op/s 2024-03-30T00:59:50.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:50 smithi181 ceph-mon[47622]: pgmap v350: 129 pgs: 129 active+clean; 65 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.6 MiB/s wr, 306 op/s 2024-03-30T00:59:52.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:52 smithi116 ceph-mon[38086]: pgmap v351: 129 pgs: 129 active+clean; 70 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 310 op/s 2024-03-30T00:59:52.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:52 smithi181 ceph-mon[47622]: pgmap v351: 129 pgs: 129 active+clean; 70 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 310 op/s 2024-03-30T00:59:52.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:52 smithi130 ceph-mon[47626]: pgmap v351: 129 pgs: 129 active+clean; 70 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 310 op/s 2024-03-30T00:59:54.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:54 smithi116 ceph-mon[38086]: pgmap v352: 129 pgs: 129 active+clean; 74 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 245 op/s 2024-03-30T00:59:54.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:54 smithi181 ceph-mon[47622]: pgmap v352: 129 pgs: 129 active+clean; 74 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 245 op/s 2024-03-30T00:59:54.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:54 smithi130 ceph-mon[47626]: pgmap v352: 129 pgs: 129 active+clean; 74 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.3 MiB/s wr, 245 op/s 2024-03-30T00:59:56.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:56 smithi116 ceph-mon[38086]: pgmap v353: 129 pgs: 129 active+clean; 82 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.7 MiB/s wr, 243 op/s 2024-03-30T00:59:56.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:56 smithi181 ceph-mon[47622]: pgmap v353: 129 pgs: 129 active+clean; 82 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.7 MiB/s wr, 243 op/s 2024-03-30T00:59:56.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:56 smithi130 ceph-mon[47626]: pgmap v353: 129 pgs: 129 active+clean; 82 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.7 MiB/s wr, 243 op/s 2024-03-30T00:59:58.690 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 00:59:58 smithi116 ceph-mon[38086]: pgmap v354: 129 pgs: 129 active+clean; 82 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.3 MiB/s wr, 169 op/s 2024-03-30T00:59:58.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 00:59:58 smithi181 ceph-mon[47622]: pgmap v354: 129 pgs: 129 active+clean; 82 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.3 MiB/s wr, 169 op/s 2024-03-30T00:59:58.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 00:59:58 smithi130 ceph-mon[47626]: pgmap v354: 129 pgs: 129 active+clean; 82 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.3 MiB/s wr, 169 op/s 2024-03-30T00:59:58.962 INFO:tasks.workunit.client.0.smithi116.stdout: 47 305411 862 224960 1035 120012 1545 2024-03-30T01:00:00.629 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T01:00:00.701 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:00 smithi116 ceph-mon[38086]: pgmap v355: 129 pgs: 129 active+clean; 86 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 140 op/s 2024-03-30T01:00:00.701 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:00 smithi116 ceph-mon[38086]: overall HEALTH_OK 2024-03-30T01:00:00.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:00 smithi181 ceph-mon[47622]: pgmap v355: 129 pgs: 129 active+clean; 86 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 140 op/s 2024-03-30T01:00:00.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:00 smithi181 ceph-mon[47622]: overall HEALTH_OK 2024-03-30T01:00:00.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:00 smithi130 ceph-mon[47626]: pgmap v355: 129 pgs: 129 active+clean; 86 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 140 op/s 2024-03-30T01:00:00.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:00 smithi130 ceph-mon[47626]: overall HEALTH_OK 2024-03-30T01:00:02.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.080+0000 7f4691400700 1 -- 172.21.15.116:0/2912899354 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f468c108970 msgr2=0x7f468c071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:02.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.080+0000 7f4691400700 1 --2- 172.21.15.116:0/2912899354 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f468c108970 0x7f468c071df0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f467c0098f0 tx=0x7f467c009c00 comp rx=0 tx=0).stop 2024-03-30T01:00:02.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.081+0000 7f4691400700 1 -- 172.21.15.116:0/2912899354 shutdown_connections 2024-03-30T01:00:02.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.081+0000 7f4691400700 1 --2- 172.21.15.116:0/2912899354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f468c10f0f0 0x7f468c1114e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.081+0000 7f4691400700 1 --2- 172.21.15.116:0/2912899354 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f468c108970 0x7f468c071df0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.081+0000 7f4691400700 1 --2- 172.21.15.116:0/2912899354 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f468c072330 0x7f468c0727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.081+0000 7f4691400700 1 -- 172.21.15.116:0/2912899354 >> 172.21.15.116:0/2912899354 conn(0x7f468c06dc60 msgr2=0x7f468c0700a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:02.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.082+0000 7f4691400700 1 -- 172.21.15.116:0/2912899354 shutdown_connections 2024-03-30T01:00:02.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.084+0000 7f4691400700 1 -- 172.21.15.116:0/2912899354 wait complete. 2024-03-30T01:00:02.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.089+0000 7f4691400700 1 Processor -- start 2024-03-30T01:00:02.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.091+0000 7f4691400700 1 -- start start 2024-03-30T01:00:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.092+0000 7f4691400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f468c072330 0x7f468c10da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.092+0000 7f468a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f468c072330 0x7f468c10da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.092+0000 7f468a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f468c072330 0x7f468c10da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:53118/0 (socket says 172.21.15.116:53118) 2024-03-30T01:00:02.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.093+0000 7f468a800700 1 -- 172.21.15.116:0/1265330887 learned_addr learned my addr 172.21.15.116:0/1265330887 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:02.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.093+0000 7f4691400700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f468c108970 0x7f468c10df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.093+0000 7f4691400700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f468c10f0f0 0x7f468c1ad120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.093+0000 7f4691400700 1 -- 172.21.15.116:0/1265330887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c10e6c0 con 0x7f468c108970 2024-03-30T01:00:02.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.093+0000 7f4691400700 1 -- 172.21.15.116:0/1265330887 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c1ad690 con 0x7f468c072330 2024-03-30T01:00:02.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.093+0000 7f4691400700 1 -- 172.21.15.116:0/1265330887 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c1ad850 con 0x7f468c10f0f0 2024-03-30T01:00:02.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.094+0000 7f468a800700 1 -- 172.21.15.116:0/1265330887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f468c10f0f0 msgr2=0x7f468c1ad120 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T01:00:02.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.094+0000 7f468a800700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f468c10f0f0 0x7f468c1ad120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.094+0000 7f4689e00700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f468c108970 0x7f468c10df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.094+0000 7f468a800700 1 -- 172.21.15.116:0/1265330887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f468c108970 msgr2=0x7f468c10df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:02.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.094+0000 7f468a800700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f468c108970 0x7f468c10df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.094+0000 7f468a800700 1 -- 172.21.15.116:0/1265330887 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4684009d00 con 0x7f468c072330 2024-03-30T01:00:02.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.094+0000 7f4689e00700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f468c108970 0x7f468c10df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:02.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.094+0000 7f468a800700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f468c072330 0x7f468c10da30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f467c009230 tx=0x7f467c00b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:02.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.095+0000 7f4683400700 1 -- 172.21.15.116:0/1265330887 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c01c070 con 0x7f468c072330 2024-03-30T01:00:02.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.095+0000 7f4683400700 1 -- 172.21.15.116:0/1265330887 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f467c00f7d0 con 0x7f468c072330 2024-03-30T01:00:02.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.095+0000 7f4683400700 1 -- 172.21.15.116:0/1265330887 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c004aa0 con 0x7f468c072330 2024-03-30T01:00:02.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.095+0000 7f4691400700 1 -- 172.21.15.116:0/1265330887 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467c009520 con 0x7f468c072330 2024-03-30T01:00:02.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.096+0000 7f4691400700 1 -- 172.21.15.116:0/1265330887 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468c1add10 con 0x7f468c072330 2024-03-30T01:00:02.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.104+0000 7f4683400700 1 -- 172.21.15.116:0/1265330887 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f467c020070 con 0x7f468c072330 2024-03-30T01:00:02.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.111+0000 7f4683400700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46700a12a0 0x7f46700a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.111+0000 7f4689e00700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46700a12a0 0x7f46700a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.114+0000 7f4683400700 1 -- 172.21.15.116:0/1265330887 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f467c0cd600 con 0x7f468c072330 2024-03-30T01:00:02.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.115+0000 7f4689e00700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46700a12a0 0x7f46700a3760 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4678005fd0 tx=0x7f4678005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.123+0000 7f46a4fb5b80 1 Processor -- start 2024-03-30T01:00:02.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.127+0000 7f46a4fb5b80 1 -- start start 2024-03-30T01:00:02.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.128+0000 7f46a4fb5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6bc9e0 0x556dcd6bce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.128+0000 7f46a4fb5b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bd340 0x556dcd6bdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.128+0000 7f468a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6bc9e0 0x556dcd6bce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.128+0000 7f46a4fb5b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6be190 0x556dcd6be610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.128+0000 7f468a800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6bc9e0 0x556dcd6bce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:53122/0 (socket says 172.21.15.116:53122) 2024-03-30T01:00:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.128+0000 7f468a800700 1 -- 172.21.15.116:0/1111153086 learned_addr learned my addr 172.21.15.116:0/1111153086 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.128+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/1111153086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcd614990 con 0x556dcd6bd340 2024-03-30T01:00:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.128+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/1111153086 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcd612350 con 0x556dcd6bc9e0 2024-03-30T01:00:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.128+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/1111153086 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcd60dc40 con 0x556dcd6be190 2024-03-30T01:00:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.129+0000 7f4689e00700 1 --2- 172.21.15.116:0/1111153086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bd340 0x556dcd6bdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.130+0000 7f468b200700 1 --2- 172.21.15.116:0/1111153086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6be190 0x556dcd6be610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.130+0000 7f468a800700 1 -- 172.21.15.116:0/1111153086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6be190 msgr2=0x556dcd6be610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.130+0000 7f468a800700 1 --2- 172.21.15.116:0/1111153086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6be190 0x556dcd6be610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.130+0000 7f468a800700 1 -- 172.21.15.116:0/1111153086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bd340 msgr2=0x556dcd6bdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.130+0000 7f468a800700 1 --2- 172.21.15.116:0/1111153086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bd340 0x556dcd6bdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.130+0000 7f468a800700 1 -- 172.21.15.116:0/1111153086 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556dcd48f3b0 con 0x556dcd6bc9e0 2024-03-30T01:00:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.134+0000 7f468a800700 1 --2- 172.21.15.116:0/1111153086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6bc9e0 0x556dcd6bce00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f467c00f9a0 tx=0x7f467c005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e120b19ace74d14 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.134+0000 7f4677400700 1 -- 172.21.15.116:0/1111153086 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c09aed0 con 0x556dcd6bc9e0 2024-03-30T01:00:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.134+0000 7f4677400700 1 -- 172.21.15.116:0/1111153086 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f467c09a7d0 con 0x556dcd6bc9e0 2024-03-30T01:00:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.134+0000 7f4677400700 1 -- 172.21.15.116:0/1111153086 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c098c00 con 0x556dcd6bc9e0 2024-03-30T01:00:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.136+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/1111153086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6bc9e0 msgr2=0x556dcd6bce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.137+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/1111153086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6bc9e0 0x556dcd6bce00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f467c00f9a0 tx=0x7f467c005750 comp rx=0 tx=0).stop 2024-03-30T01:00:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.137+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/1111153086 shutdown_connections 2024-03-30T01:00:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.137+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/1111153086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6be190 0x556dcd6be610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.137+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/1111153086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6bc9e0 0x556dcd6bce00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.137+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/1111153086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bd340 0x556dcd6bdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.137+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/1111153086 >> 172.21.15.116:0/1111153086 conn(0x556dcd61b6a0 msgr2=0x556dcd6b7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.141+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/1111153086 shutdown_connections 2024-03-30T01:00:02.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.141+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/1111153086 wait complete. 2024-03-30T01:00:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.142+0000 7f46a4fb5b80 1 -- start start 2024-03-30T01:00:02.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.142+0000 7f46a4fb5b80 1 Processor -- start 2024-03-30T01:00:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.144+0000 7f46a4fb5b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bc9e0 0x556dcd6c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.144+0000 7f46a4fb5b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bd340 0x556dcd6c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.144+0000 7f468a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bc9e0 0x556dcd6c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.145+0000 7f468a800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bc9e0 0x556dcd6c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48324/0 (socket says 172.21.15.116:48324) 2024-03-30T01:00:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.145+0000 7f468a800700 1 -- 172.21.15.116:0/2097148688 learned_addr learned my addr 172.21.15.116:0/2097148688 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.145+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/2097148688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6be190 0x556dcd6cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.146+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2097148688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcd5a9d80 con 0x556dcd6bc9e0 2024-03-30T01:00:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.146+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2097148688 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcd5a9a20 con 0x556dcd6be190 2024-03-30T01:00:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.146+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2097148688 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcd4916a0 con 0x556dcd6bd340 2024-03-30T01:00:02.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.146+0000 7f46a4fb5b80 10 client.0 _ll_register_callbacks cb 0x556dcd5b3470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:02.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.147+0000 7f468a800700 1 -- 172.21.15.116:0/2097148688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bd340 msgr2=0x556dcd6c6a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:00:02.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.147+0000 7f468a800700 1 --2- 172.21.15.116:0/2097148688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bd340 0x556dcd6c6a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.147+0000 7f468a800700 1 -- 172.21.15.116:0/2097148688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6be190 msgr2=0x556dcd6cda70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:00:02.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.147+0000 7f468a800700 1 --2- 172.21.15.116:0/2097148688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6be190 0x556dcd6cda70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.147+0000 7f468a800700 1 -- 172.21.15.116:0/2097148688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468c1add10 con 0x556dcd6bc9e0 2024-03-30T01:00:02.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.148+0000 7f468a800700 1 --2- 172.21.15.116:0/2097148688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bc9e0 0x556dcd6c64f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f467c00fa40 tx=0x7f467c00bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d4a601b46d69dd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:02.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.149+0000 7f466be00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:00:02.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.149+0000 7f466be00700 1 -- 172.21.15.116:0/2097148688 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c0a2bc0 con 0x556dcd6bc9e0 2024-03-30T01:00:02.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.149+0000 7f466be00700 1 -- 172.21.15.116:0/2097148688 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f467c09aa30 con 0x556dcd6bc9e0 2024-03-30T01:00:02.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.150+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2097148688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556dcd5ce380 con 0x556dcd6bc9e0 2024-03-30T01:00:02.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.153+0000 7f466be00700 1 -- 172.21.15.116:0/2097148688 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f467c09abf0 con 0x556dcd6bc9e0 2024-03-30T01:00:02.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.153+0000 7f46a4fb5b80 10 client.15813 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:02.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.154+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2097148688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556dcd43bb60 con 0x556dcd6bc9e0 2024-03-30T01:00:02.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.154+0000 7f466be00700 1 -- 172.21.15.116:0/2097148688 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556dcd5ce380 con 0x556dcd6bc9e0 2024-03-30T01:00:02.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.155+0000 7f466be00700 1 -- 172.21.15.116:0/2097148688 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f467c0cc030 con 0x556dcd6bc9e0 2024-03-30T01:00:02.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.155+0000 7f46a4fb5b80 10 client.15813 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:02.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.155+0000 7f46a4fb5b80 10 client.15813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:00:02.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.155+0000 7f46a4fb5b80 20 client.15813 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:02.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.155+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/2097148688 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556dcd6d1dd0 0x556dcd6d41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.155+0000 7f46a4fb5b80 4 client.15813 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:00:02.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.155+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2097148688 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556dcd606700 con 0x556dcd6d1dd0 2024-03-30T01:00:02.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.156+0000 7f4689e00700 1 --2- 172.21.15.116:0/2097148688 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556dcd6d1dd0 0x556dcd6d41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.158+0000 7f4689e00700 1 --2- 172.21.15.116:0/2097148688 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556dcd6d1dd0 0x556dcd6d41f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b27fd93f0522cd7f server_cookie=3f942702b0ca9462 in_seq=0 out_seq=0 2024-03-30T01:00:02.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.159+0000 7f466be00700 10 client.15813 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:02.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.163+0000 7f466be00700 1 -- 172.21.15.116:0/2097148688 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556dcd606700 con 0x556dcd6d1dd0 2024-03-30T01:00:02.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.163+0000 7f466be00700 10 client.15813 handle_command_reply: tid=0 2024-03-30T01:00:02.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.164+0000 7f46a4fb5b80 1 client.15813 shutdown 2024-03-30T01:00:02.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.164+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2097148688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bc9e0 msgr2=0x556dcd6c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:02.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.164+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/2097148688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bc9e0 0x556dcd6c64f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f467c00fa40 tx=0x7f467c00bb50 comp rx=0 tx=0).stop 2024-03-30T01:00:02.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.165+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2097148688 shutdown_connections 2024-03-30T01:00:02.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.165+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/2097148688 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556dcd6d1dd0 0x556dcd6d41f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.165+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/2097148688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6be190 0x556dcd6cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.165+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/2097148688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bc9e0 0x556dcd6c64f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.165+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/2097148688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bd340 0x556dcd6c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.165+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2097148688 >> 172.21.15.116:0/2097148688 conn(0x556dcd61b6a0 msgr2=0x556dcd6bb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:02.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.165+0000 7f466be00700 0 client.15813 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:02.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.166+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2097148688 shutdown_connections 2024-03-30T01:00:02.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.168+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2097148688 wait complete. 2024-03-30T01:00:02.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.169+0000 7f46a4fb5b80 20 client.15813 trim_cache size 0 max 16384 2024-03-30T01:00:02.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.221+0000 7f46a4fb5b80 1 Processor -- start 2024-03-30T01:00:02.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.222+0000 7f46a4fb5b80 1 -- start start 2024-03-30T01:00:02.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.222+0000 7f46a4fb5b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bc9e0 0x556dcd6bcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.222+0000 7f46a4fb5b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bd340 0x556dcd6cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.222+0000 7f46a4fb5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6be190 0x556dcd6c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.222+0000 7f46a4fb5b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcd6cec10 con 0x556dcd6bd340 2024-03-30T01:00:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.222+0000 7f46a4fb5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcd6bd800 con 0x556dcd6be190 2024-03-30T01:00:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.222+0000 7f46a4fb5b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcd6b8690 con 0x556dcd6bc9e0 2024-03-30T01:00:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.222+0000 7f4689e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bd340 0x556dcd6cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.222+0000 7f468a800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bc9e0 0x556dcd6bcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.222+0000 7f468a800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bc9e0 0x556dcd6bcdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:33416/0 (socket says 172.21.15.116:33416) 2024-03-30T01:00:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.222+0000 7f468a800700 1 -- 172.21.15.116:0/1140481302 learned_addr learned my addr 172.21.15.116:0/1140481302 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.223+0000 7f468b200700 1 --2- 172.21.15.116:0/1140481302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6be190 0x556dcd6c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.223+0000 7f468b200700 1 -- 172.21.15.116:0/1140481302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bc9e0 msgr2=0x556dcd6bcdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:02.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.225+0000 7f468b200700 1 --2- 172.21.15.116:0/1140481302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bc9e0 0x556dcd6bcdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.225+0000 7f468b200700 1 -- 172.21.15.116:0/1140481302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bd340 msgr2=0x556dcd6cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:02.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.225+0000 7f468b200700 1 --2- 172.21.15.116:0/1140481302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bd340 0x556dcd6cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.225+0000 7f468b200700 1 -- 172.21.15.116:0/1140481302 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556dcd5d3ea0 con 0x556dcd6be190 2024-03-30T01:00:02.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.225+0000 7f4689e00700 1 --2- 172.21.15.116:0/1140481302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bd340 0x556dcd6cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:00:02.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.225+0000 7f468b200700 1 --2- 172.21.15.116:0/1140481302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6be190 0x556dcd6c5a00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x556dcd6b8060 tx=0x7f46840099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79cd4117bda5de1f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:02.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.229+0000 7f4677e00700 1 -- 172.21.15.116:0/1140481302 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684019070 con 0x556dcd6be190 2024-03-30T01:00:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.229+0000 7f4677e00700 1 -- 172.21.15.116:0/1140481302 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f468400f050 con 0x556dcd6be190 2024-03-30T01:00:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.230+0000 7f4677e00700 1 -- 172.21.15.116:0/1140481302 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46840156d0 con 0x556dcd6be190 2024-03-30T01:00:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.231+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/1140481302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6be190 msgr2=0x556dcd6c5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.231+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/1140481302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6be190 0x556dcd6c5a00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x556dcd6b8060 tx=0x7f46840099e0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.231+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/1140481302 shutdown_connections 2024-03-30T01:00:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.231+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/1140481302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bc9e0 0x556dcd6bcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.231+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/1140481302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6be190 0x556dcd6c5a00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.231+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/1140481302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6bd340 0x556dcd6cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.231+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/1140481302 >> 172.21.15.116:0/1140481302 conn(0x556dcd61b6a0 msgr2=0x556dcd6c5200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.231+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/1140481302 shutdown_connections 2024-03-30T01:00:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.231+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/1140481302 wait complete. 2024-03-30T01:00:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.232+0000 7f46a4fb5b80 1 -- start start 2024-03-30T01:00:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.233+0000 7f46a4fb5b80 1 Processor -- start 2024-03-30T01:00:02.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.233+0000 7f46a4fb5b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bc9e0 0x556dcd6cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.233+0000 7f46a4fb5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6bd340 0x556dcd6cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.234+0000 7f46a4fb5b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6be190 0x556dcd6ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.234+0000 7f4689e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6bd340 0x556dcd6cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.234+0000 7f4689e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6bd340 0x556dcd6cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:53144/0 (socket says 172.21.15.116:53144) 2024-03-30T01:00:02.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.234+0000 7f4689e00700 1 -- 172.21.15.116:0/2456528773 learned_addr learned my addr 172.21.15.116:0/2456528773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:02.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.234+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2456528773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcd6cd420 con 0x556dcd6be190 2024-03-30T01:00:02.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.234+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2456528773 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcd6cd590 con 0x556dcd6bd340 2024-03-30T01:00:02.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.234+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2456528773 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcd6cd750 con 0x556dcd6bc9e0 2024-03-30T01:00:02.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.235+0000 7f46a4fb5b80 10 client.0 _ll_register_callbacks cb 0x556dcd5b3470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:02.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.235+0000 7f468a800700 1 --2- 172.21.15.116:0/2456528773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bc9e0 0x556dcd6cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.235+0000 7f468b200700 1 --2- 172.21.15.116:0/2456528773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6be190 0x556dcd6ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.236+0000 7f4689e00700 1 -- 172.21.15.116:0/2456528773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bc9e0 msgr2=0x556dcd6cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:02.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.236+0000 7f4689e00700 1 --2- 172.21.15.116:0/2456528773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bc9e0 0x556dcd6cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.236+0000 7f4689e00700 1 -- 172.21.15.116:0/2456528773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6be190 msgr2=0x556dcd6ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:02.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.236+0000 7f4689e00700 1 --2- 172.21.15.116:0/2456528773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6be190 0x556dcd6ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.236+0000 7f4689e00700 1 -- 172.21.15.116:0/2456528773 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46840091b0 con 0x556dcd6bd340 2024-03-30T01:00:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.236+0000 7f4689e00700 1 --2- 172.21.15.116:0/2456528773 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6bd340 0x556dcd6cbfc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4678009410 tx=0x7f46780047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=258c50f4a7fd2dfd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.237+0000 7f466be00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T01:00:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.237+0000 7f466be00700 1 -- 172.21.15.116:0/2456528773 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678024040 con 0x556dcd6bd340 2024-03-30T01:00:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.237+0000 7f466be00700 1 -- 172.21.15.116:0/2456528773 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4678018040 con 0x556dcd6bd340 2024-03-30T01:00:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.237+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2456528773 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f467c09a7d0 con 0x556dcd6bd340 2024-03-30T01:00:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.238+0000 7f466be00700 1 -- 172.21.15.116:0/2456528773 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4678026580 con 0x556dcd6bd340 2024-03-30T01:00:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.238+0000 7f46a4fb5b80 10 client.25886 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.238+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2456528773 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556dcd493cd0 con 0x556dcd6bd340 2024-03-30T01:00:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.239+0000 7f466be00700 1 -- 172.21.15.116:0/2456528773 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f467c09a7d0 con 0x556dcd6bd340 2024-03-30T01:00:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.240+0000 7f466be00700 1 -- 172.21.15.116:0/2456528773 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f4678011ad0 con 0x556dcd6bd340 2024-03-30T01:00:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.240+0000 7f46a4fb5b80 10 client.25886 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.240+0000 7f46a4fb5b80 10 client.25886 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:00:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.240+0000 7f46a4fb5b80 20 client.25886 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.240+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/2456528773 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556dcd726dd0 0x556dcd7291f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.240+0000 7f46a4fb5b80 4 client.25886 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T01:00:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.240+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2456528773 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556dcd45ccf0 con 0x556dcd726dd0 2024-03-30T01:00:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.241+0000 7f468a800700 1 --2- 172.21.15.116:0/2456528773 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556dcd726dd0 0x556dcd7291f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.242+0000 7f468a800700 1 --2- 172.21.15.116:0/2456528773 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556dcd726dd0 0x556dcd7291f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2e209121ffe3e01 server_cookie=42b21b4feb76de7d in_seq=0 out_seq=0 2024-03-30T01:00:02.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.242+0000 7f466be00700 10 client.25886 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:02.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.246+0000 7f466be00700 1 -- 172.21.15.116:0/2456528773 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556dcd45ccf0 con 0x556dcd726dd0 2024-03-30T01:00:02.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.246+0000 7f466be00700 10 client.25886 handle_command_reply: tid=0 2024-03-30T01:00:02.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.246+0000 7f46a4fb5b80 1 client.25886 shutdown 2024-03-30T01:00:02.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.247+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2456528773 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6bd340 msgr2=0x556dcd6cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:02.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.247+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/2456528773 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6bd340 0x556dcd6cbfc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4678009410 tx=0x7f46780047c0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.248+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2456528773 shutdown_connections 2024-03-30T01:00:02.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.248+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/2456528773 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556dcd726dd0 0x556dcd7291f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.248+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/2456528773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556dcd6bc9e0 0x556dcd6cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.248+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/2456528773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dcd6be190 0x556dcd6ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.248+0000 7f46a4fb5b80 1 --2- 172.21.15.116:0/2456528773 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556dcd6bd340 0x556dcd6cbfc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.248+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2456528773 >> 172.21.15.116:0/2456528773 conn(0x556dcd61b6a0 msgr2=0x556dcd6b7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:02.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.248+0000 7f466be00700 0 client.25886 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:02.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.249+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2456528773 shutdown_connections 2024-03-30T01:00:02.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.249+0000 7f46a4fb5b80 1 -- 172.21.15.116:0/2456528773 wait complete. 2024-03-30T01:00:02.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.250+0000 7f46a4fb5b80 20 client.25886 trim_cache size 0 max 16384 2024-03-30T01:00:02.256 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:00:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-30T01:00:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-30T01:00:02.256 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:00:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.252+0000 7f4691400700 1 -- 172.21.15.116:0/1265330887 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46700a12a0 msgr2=0x7f46700a3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.252+0000 7f4691400700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46700a12a0 0x7f46700a3760 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4678005fd0 tx=0x7f4678005ee0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.252+0000 7f4691400700 1 -- 172.21.15.116:0/1265330887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f468c072330 msgr2=0x7f468c10da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:02.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.252+0000 7f4691400700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f468c072330 0x7f468c10da30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f467c009230 tx=0x7f467c00b840 comp rx=0 tx=0).stop 2024-03-30T01:00:02.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.252+0000 7f4691400700 1 -- 172.21.15.116:0/1265330887 shutdown_connections 2024-03-30T01:00:02.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.252+0000 7f4691400700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46700a12a0 0x7f46700a3760 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.252+0000 7f4691400700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f468c10f0f0 0x7f468c1ad120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.252+0000 7f4691400700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f468c072330 0x7f468c10da30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.252+0000 7f4691400700 1 --2- 172.21.15.116:0/1265330887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f468c108970 0x7f468c10df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.252+0000 7f4691400700 1 -- 172.21.15.116:0/1265330887 >> 172.21.15.116:0/1265330887 conn(0x7f468c06dc60 msgr2=0x7f468c19a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:02.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.253+0000 7f4691400700 1 -- 172.21.15.116:0/1265330887 shutdown_connections 2024-03-30T01:00:02.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.253+0000 7f4691400700 1 -- 172.21.15.116:0/1265330887 wait complete. 2024-03-30T01:00:02.292 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T01:00:02.292 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T01:00:02.696 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:02 smithi116 ceph-mon[38086]: pgmap v356: 129 pgs: 129 active+clean; 100 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 169 op/s 2024-03-30T01:00:02.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:02 smithi181 ceph-mon[47622]: pgmap v356: 129 pgs: 129 active+clean; 100 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 169 op/s 2024-03-30T01:00:02.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:02 smithi130 ceph-mon[47626]: pgmap v356: 129 pgs: 129 active+clean; 100 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 169 op/s 2024-03-30T01:00:02.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.890+0000 7fc7d4a00700 1 -- 172.21.15.116:0/1512302140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7d0108440 msgr2=0x7fc7d01088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:02.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.890+0000 7fc7d4a00700 1 --2- 172.21.15.116:0/1512302140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7d0108440 0x7fc7d01088c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc01a9a0 tx=0x7fc7bc01acb0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.890+0000 7fc7d4a00700 1 -- 172.21.15.116:0/1512302140 shutdown_connections 2024-03-30T01:00:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.890+0000 7fc7d4a00700 1 --2- 172.21.15.116:0/1512302140 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d0108e00 0x7fc7d010d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.890+0000 7fc7d4a00700 1 --2- 172.21.15.116:0/1512302140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d00721b0 0x7fc7d00725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.890+0000 7fc7d4a00700 1 --2- 172.21.15.116:0/1512302140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7d0108440 0x7fc7d01088c0 secure :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc01a9a0 tx=0x7fc7bc01acb0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.890+0000 7fc7d4a00700 1 -- 172.21.15.116:0/1512302140 >> 172.21.15.116:0/1512302140 conn(0x7fc7d006dc80 msgr2=0x7fc7d00700c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.896+0000 7fc7d4a00700 1 -- 172.21.15.116:0/1512302140 shutdown_connections 2024-03-30T01:00:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.900+0000 7fc7d4a00700 1 -- 172.21.15.116:0/1512302140 wait complete. 2024-03-30T01:00:02.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.902+0000 7fc7d4a00700 1 Processor -- start 2024-03-30T01:00:02.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.907+0000 7fc7d4a00700 1 -- start start 2024-03-30T01:00:02.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.907+0000 7fc7d4a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7d0108e00 0x7fc7d019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.907+0000 7fc7d4a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d019db10 0x7fc7d01a2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.907+0000 7fc7d4a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d019df90 0x7fc7d01a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.907+0000 7fc7d4a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d01a5870 con 0x7fc7d019df90 2024-03-30T01:00:02.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.907+0000 7fc7d4a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d01a59e0 con 0x7fc7d0108e00 2024-03-30T01:00:02.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.907+0000 7fc7d4a00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d01a5ba0 con 0x7fc7d019db10 2024-03-30T01:00:02.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.908+0000 7fc7ce800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d019df90 0x7fc7d01a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.908+0000 7fc7ce800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d019df90 0x7fc7d01a50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48356/0 (socket says 172.21.15.116:48356) 2024-03-30T01:00:02.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.908+0000 7fc7ce800700 1 -- 172.21.15.116:0/2089311839 learned_addr learned my addr 172.21.15.116:0/2089311839 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:02.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.908+0000 7fc7cd400700 1 --2- 172.21.15.116:0/2089311839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d019db10 0x7fc7d01a2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.909+0000 7fc7ce800700 1 -- 172.21.15.116:0/2089311839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d019db10 msgr2=0x7fc7d01a2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:02.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.909+0000 7fc7ce800700 1 --2- 172.21.15.116:0/2089311839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d019db10 0x7fc7d01a2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.909+0000 7fc7ce800700 1 -- 172.21.15.116:0/2089311839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7d0108e00 msgr2=0x7fc7d019d5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T01:00:02.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.909+0000 7fc7ce800700 1 --2- 172.21.15.116:0/2089311839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7d0108e00 0x7fc7d019d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:02.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.909+0000 7fc7ce800700 1 -- 172.21.15.116:0/2089311839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc01a600 con 0x7fc7d019df90 2024-03-30T01:00:02.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.911+0000 7fc7ce800700 1 --2- 172.21.15.116:0/2089311839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d019df90 0x7fc7d01a50d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000e940 tx=0x7fc7c000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:02.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.911+0000 7fc7c6a00700 1 -- 172.21.15.116:0/2089311839 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c00097b0 con 0x7fc7d019df90 2024-03-30T01:00:02.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.911+0000 7fc7d4a00700 1 -- 172.21.15.116:0/2089311839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d01a5e20 con 0x7fc7d019df90 2024-03-30T01:00:02.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.912+0000 7fc7d4a00700 1 -- 172.21.15.116:0/2089311839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d01a62a0 con 0x7fc7d019df90 2024-03-30T01:00:02.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.912+0000 7fc7c6a00700 1 -- 172.21.15.116:0/2089311839 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7c000ee90 con 0x7fc7d019df90 2024-03-30T01:00:02.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.917+0000 7fc7c6a00700 1 -- 172.21.15.116:0/2089311839 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c00097b0 con 0x7fc7d019df90 2024-03-30T01:00:02.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.918+0000 7fc7d4a00700 1 -- 172.21.15.116:0/2089311839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b0005320 con 0x7fc7d019df90 2024-03-30T01:00:02.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.920+0000 7fc7c6a00700 1 -- 172.21.15.116:0/2089311839 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fc7c0005420 con 0x7fc7d019df90 2024-03-30T01:00:02.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.925+0000 7fc7c6a00700 1 --2- 172.21.15.116:0/2089311839 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc7b40a1300 0x7fc7b40a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:02.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.926+0000 7fc7c6a00700 1 -- 172.21.15.116:0/2089311839 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc7c0014070 con 0x7fc7d019df90 2024-03-30T01:00:02.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.930+0000 7fc7cde00700 1 --2- 172.21.15.116:0/2089311839 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc7b40a1300 0x7fc7b40a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.934+0000 7fc7cde00700 1 --2- 172.21.15.116:0/2089311839 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc7b40a1300 0x7fc7b40a37c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0107f60 tx=0x7fc7c8009250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:02.939+0000 7fc7c6a00700 1 -- 172.21.15.116:0/2089311839 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc7c0096f00 con 0x7fc7d019df90 2024-03-30T01:00:04.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:04 smithi116 ceph-mon[38086]: pgmap v357: 129 pgs: 129 active+clean; 123 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.8 MiB/s wr, 233 op/s 2024-03-30T01:00:04.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:04 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:00:04.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:04.689+0000 7fc7d4a00700 1 -- 172.21.15.116:0/2089311839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7b0005b50 con 0x7fc7d019df90 2024-03-30T01:00:04.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:04.691+0000 7fc7c6a00700 1 -- 172.21.15.116:0/2089311839 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10889 (secure 0 0 0) 0x7fc7c009a520 con 0x7fc7d019df90 2024-03-30T01:00:04.701 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T01:00:04.701 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:59:42.300374+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T01:00:04.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:04.708+0000 7fc7d4a00700 1 -- 172.21.15.116:0/2089311839 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc7b40a1300 msgr2=0x7fc7b40a37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:04.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:04.708+0000 7fc7d4a00700 1 --2- 172.21.15.116:0/2089311839 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc7b40a1300 0x7fc7b40a37c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0107f60 tx=0x7fc7c8009250 comp rx=0 tx=0).stop 2024-03-30T01:00:04.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:04.708+0000 7fc7d4a00700 1 -- 172.21.15.116:0/2089311839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d019df90 msgr2=0x7fc7d01a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:04.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:04.708+0000 7fc7d4a00700 1 --2- 172.21.15.116:0/2089311839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d019df90 0x7fc7d01a50d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000e940 tx=0x7fc7c000ec50 comp rx=0 tx=0).stop 2024-03-30T01:00:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:04.711+0000 7fc7d4a00700 1 -- 172.21.15.116:0/2089311839 shutdown_connections 2024-03-30T01:00:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:04.711+0000 7fc7d4a00700 1 --2- 172.21.15.116:0/2089311839 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc7b40a1300 0x7fc7b40a37c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:04.711+0000 7fc7d4a00700 1 --2- 172.21.15.116:0/2089311839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d019df90 0x7fc7d01a50d0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:04.711+0000 7fc7d4a00700 1 --2- 172.21.15.116:0/2089311839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7d0108e00 0x7fc7d019d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:04.711+0000 7fc7d4a00700 1 --2- 172.21.15.116:0/2089311839 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d019db10 0x7fc7d01a2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:04.711+0000 7fc7d4a00700 1 -- 172.21.15.116:0/2089311839 >> 172.21.15.116:0/2089311839 conn(0x7fc7d006dc80 msgr2=0x7fc7d010b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:04.714+0000 7fc7d4a00700 1 -- 172.21.15.116:0/2089311839 shutdown_connections 2024-03-30T01:00:04.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:04.720+0000 7fc7d4a00700 1 -- 172.21.15.116:0/2089311839 wait complete. 2024-03-30T01:00:04.729 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 25 2024-03-30T01:00:04.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:04 smithi181 ceph-mon[47622]: pgmap v357: 129 pgs: 129 active+clean; 123 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.8 MiB/s wr, 233 op/s 2024-03-30T01:00:04.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:04 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:00:04.830 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T01:00:04.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:04 smithi130 ceph-mon[47626]: pgmap v357: 129 pgs: 129 active+clean; 123 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.8 MiB/s wr, 233 op/s 2024-03-30T01:00:04.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:04 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:00:05.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.191+0000 7f174fa00700 1 -- 172.21.15.116:0/4244627157 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1748101820 msgr2=0x7f17481080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.191+0000 7f174fa00700 1 --2- 172.21.15.116:0/4244627157 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1748101820 0x7f17481080d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f174000d310 tx=0x7f1740009f90 comp rx=0 tx=0).stop 2024-03-30T01:00:05.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.191+0000 7f174fa00700 1 -- 172.21.15.116:0/4244627157 shutdown_connections 2024-03-30T01:00:05.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.191+0000 7f174fa00700 1 --2- 172.21.15.116:0/4244627157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17480ffc90 0x7f17481000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.191+0000 7f174fa00700 1 --2- 172.21.15.116:0/4244627157 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1748101820 0x7f17481080d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.191+0000 7f174fa00700 1 --2- 172.21.15.116:0/4244627157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1748100e80 0x7f17481012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.191+0000 7f174fa00700 1 -- 172.21.15.116:0/4244627157 >> 172.21.15.116:0/4244627157 conn(0x7f17480fb200 msgr2=0x7f17480fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:05.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.191+0000 7f174fa00700 1 -- 172.21.15.116:0/4244627157 shutdown_connections 2024-03-30T01:00:05.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.191+0000 7f174fa00700 1 -- 172.21.15.116:0/4244627157 wait complete. 2024-03-30T01:00:05.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.192+0000 7f174fa00700 1 Processor -- start 2024-03-30T01:00:05.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.193+0000 7f174fa00700 1 -- start start 2024-03-30T01:00:05.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.193+0000 7f174fa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17480ffc90 0x7f1748194bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.193+0000 7f174fa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1748100e80 0x7f1748195100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.193+0000 7f174fa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1748101820 0x7f174819c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.193+0000 7f174fa00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174819c6f0 con 0x7f1748100e80 2024-03-30T01:00:05.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.193+0000 7f174fa00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174819c830 con 0x7f17480ffc90 2024-03-30T01:00:05.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.193+0000 7f174fa00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174819c9c0 con 0x7f1748101820 2024-03-30T01:00:05.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.193+0000 7f174d000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17480ffc90 0x7f1748194bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.193+0000 7f174d000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17480ffc90 0x7f1748194bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38364/0 (socket says 172.21.15.116:38364) 2024-03-30T01:00:05.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.193+0000 7f174d000700 1 -- 172.21.15.116:0/1903428633 learned_addr learned my addr 172.21.15.116:0/1903428633 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:05.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.193+0000 7f1747e00700 1 --2- 172.21.15.116:0/1903428633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1748100e80 0x7f1748195100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.194+0000 7f174da00700 1 --2- 172.21.15.116:0/1903428633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1748101820 0x7f174819c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.194+0000 7f174d000700 1 -- 172.21.15.116:0/1903428633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1748101820 msgr2=0x7f174819c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.194+0000 7f174d000700 1 --2- 172.21.15.116:0/1903428633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1748101820 0x7f174819c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.194+0000 7f174d000700 1 -- 172.21.15.116:0/1903428633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1748100e80 msgr2=0x7f1748195100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.194+0000 7f174d000700 1 --2- 172.21.15.116:0/1903428633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1748100e80 0x7f1748195100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.194+0000 7f174d000700 1 -- 172.21.15.116:0/1903428633 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1740009c40 con 0x7f17480ffc90 2024-03-30T01:00:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.194+0000 7f174d000700 1 --2- 172.21.15.116:0/1903428633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17480ffc90 0x7f1748194bc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f173c00e970 tx=0x7f173c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.194+0000 7f1745600700 1 -- 172.21.15.116:0/1903428633 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c009790 con 0x7f17480ffc90 2024-03-30T01:00:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.195+0000 7f1745600700 1 -- 172.21.15.116:0/1903428633 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f173c00c7c0 con 0x7f17480ffc90 2024-03-30T01:00:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.195+0000 7f174fa00700 1 -- 172.21.15.116:0/1903428633 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f174819cca0 con 0x7f17480ffc90 2024-03-30T01:00:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.195+0000 7f1745600700 1 -- 172.21.15.116:0/1903428633 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c003c20 con 0x7f17480ffc90 2024-03-30T01:00:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.195+0000 7f174fa00700 1 -- 172.21.15.116:0/1903428633 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174819d1b0 con 0x7f17480ffc90 2024-03-30T01:00:05.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.200+0000 7f1745600700 1 -- 172.21.15.116:0/1903428633 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f173c00c930 con 0x7f17480ffc90 2024-03-30T01:00:05.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.201+0000 7f17635e8b80 1 Processor -- start 2024-03-30T01:00:05.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.201+0000 7f17635e8b80 1 -- start start 2024-03-30T01:00:05.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.202+0000 7f1745600700 1 --2- 172.21.15.116:0/1903428633 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f17300a11a0 0x7f17300a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.202+0000 7f1745600700 1 -- 172.21.15.116:0/1903428633 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f173c014070 con 0x7f17480ffc90 2024-03-30T01:00:05.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.202+0000 7f174da00700 1 --2- 172.21.15.116:0/1903428633 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f17300a11a0 0x7f17300a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.202+0000 7f17635e8b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34aaff7d0 0x55a34aaffbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.202+0000 7f17635e8b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34ab00130 0x55a34ab009f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.202+0000 7f17635e8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00fd0 0x55a34ab01450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.202+0000 7f17635e8b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34aa57990 con 0x55a34ab00130 2024-03-30T01:00:05.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.202+0000 7f17635e8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34aa55350 con 0x55a34ab00fd0 2024-03-30T01:00:05.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.202+0000 7f17635e8b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34aa50c40 con 0x55a34aaff7d0 2024-03-30T01:00:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.202+0000 7f174d000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34aaff7d0 0x55a34aaffbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.203+0000 7f174d000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34aaff7d0 0x55a34aaffbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:53014/0 (socket says 172.21.15.116:53014) 2024-03-30T01:00:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.203+0000 7f174d000700 1 -- 172.21.15.116:0/217890037 learned_addr learned my addr 172.21.15.116:0/217890037 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.203+0000 7f174da00700 1 --2- 172.21.15.116:0/217890037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00fd0 0x55a34ab01450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.203+0000 7f174d000700 1 -- 172.21.15.116:0/217890037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00fd0 msgr2=0x55a34ab01450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.203+0000 7f1747e00700 1 --2- 172.21.15.116:0/217890037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34ab00130 0x55a34ab009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.203+0000 7f174d000700 1 --2- 172.21.15.116:0/217890037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00fd0 0x55a34ab01450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.203+0000 7f174d000700 1 -- 172.21.15.116:0/217890037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34ab00130 msgr2=0x55a34ab009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.203+0000 7f174d000700 1 --2- 172.21.15.116:0/217890037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34ab00130 0x55a34ab009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.203+0000 7f174d000700 1 -- 172.21.15.116:0/217890037 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a34a8d23b0 con 0x55a34aaff7d0 2024-03-30T01:00:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.203+0000 7f174da00700 1 --2- 172.21.15.116:0/217890037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00fd0 0x55a34ab01450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.203+0000 7f1747e00700 1 --2- 172.21.15.116:0/217890037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34ab00130 0x55a34ab009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.203+0000 7f174da00700 1 --2- 172.21.15.116:0/1903428633 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f17300a11a0 0x7f17300a3660 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f174000d2e0 tx=0x7f174001c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.203+0000 7f174d000700 1 --2- 172.21.15.116:0/217890037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34aaff7d0 0x55a34aaffbf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f173c099e30 tx=0x7f173c0998d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=133a5e946d4733c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:05.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.203+0000 7f173a000700 1 -- 172.21.15.116:0/217890037 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c0b1070 con 0x55a34aaff7d0 2024-03-30T01:00:05.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.204+0000 7f173a000700 1 -- 172.21.15.116:0/217890037 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f173c0a2320 con 0x55a34aaff7d0 2024-03-30T01:00:05.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.204+0000 7f173a000700 1 -- 172.21.15.116:0/217890037 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c003e80 con 0x55a34aaff7d0 2024-03-30T01:00:05.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.204+0000 7f17635e8b80 1 -- 172.21.15.116:0/217890037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34aaff7d0 msgr2=0x55a34aaffbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.204+0000 7f17635e8b80 1 --2- 172.21.15.116:0/217890037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34aaff7d0 0x55a34aaffbf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f173c099e30 tx=0x7f173c0998d0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.204+0000 7f17635e8b80 1 -- 172.21.15.116:0/217890037 shutdown_connections 2024-03-30T01:00:05.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.204+0000 7f17635e8b80 1 --2- 172.21.15.116:0/217890037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34aaff7d0 0x55a34aaffbf0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.204+0000 7f17635e8b80 1 --2- 172.21.15.116:0/217890037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00fd0 0x55a34ab01450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.204+0000 7f17635e8b80 1 --2- 172.21.15.116:0/217890037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34ab00130 0x55a34ab009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.204+0000 7f17635e8b80 1 -- 172.21.15.116:0/217890037 >> 172.21.15.116:0/217890037 conn(0x55a34aa5e6a0 msgr2=0x55a34aafa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:05.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.204+0000 7f17635e8b80 1 -- 172.21.15.116:0/217890037 shutdown_connections 2024-03-30T01:00:05.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.205+0000 7f17635e8b80 1 -- 172.21.15.116:0/217890037 wait complete. 2024-03-30T01:00:05.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.205+0000 7f17635e8b80 1 -- start start 2024-03-30T01:00:05.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.205+0000 7f17635e8b80 1 Processor -- start 2024-03-30T01:00:05.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.206+0000 7f17635e8b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab0da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.206+0000 7f17635e8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 0x55a34ab0df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.206+0000 7f17635e8b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 0x55a34ab14fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.206+0000 7f17635e8b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34a9ecd80 con 0x55a34aaff7d0 2024-03-30T01:00:05.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.206+0000 7f17635e8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34a9eca20 con 0x55a34ab00130 2024-03-30T01:00:05.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.206+0000 7f17635e8b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34a8d46a0 con 0x55a34ab00fd0 2024-03-30T01:00:05.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.206+0000 7f17635e8b80 10 client.0 _ll_register_callbacks cb 0x55a34a9f6470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:05.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.206+0000 7f174d000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab0da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.206+0000 7f174da00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 0x55a34ab14fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.207+0000 7f174da00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 0x55a34ab14fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:53030/0 (socket says 172.21.15.116:53030) 2024-03-30T01:00:05.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.206+0000 7f1747e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 0x55a34ab0df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.207+0000 7f174da00700 1 -- 172.21.15.116:0/448270867 learned_addr learned my addr 172.21.15.116:0/448270867 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.206+0000 7f174d000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab0da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60072/0 (socket says 172.21.15.116:60072) 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.207+0000 7f174da00700 1 -- 172.21.15.116:0/448270867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 msgr2=0x55a34ab0df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.207+0000 7f174da00700 1 --2- 172.21.15.116:0/448270867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 0x55a34ab0df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.207+0000 7f174da00700 1 -- 172.21.15.116:0/448270867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 msgr2=0x55a34ab0da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.207+0000 7f174da00700 1 --2- 172.21.15.116:0/448270867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab0da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.207+0000 7f174da00700 1 -- 172.21.15.116:0/448270867 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174819d1b0 con 0x55a34ab00fd0 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.207+0000 7f174da00700 1 --2- 172.21.15.116:0/448270867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 0x55a34ab14fa0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1740017000 tx=0x7f1740009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c38dcd8a173f4dd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.207+0000 7f172ea00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.208+0000 7f172ea00700 1 -- 172.21.15.116:0/448270867 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174002a070 con 0x55a34ab00fd0 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.208+0000 7f172ea00700 1 -- 172.21.15.116:0/448270867 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1740012cf0 con 0x55a34ab00fd0 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.208+0000 7f17635e8b80 1 -- 172.21.15.116:0/448270867 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a34aa11380 con 0x55a34ab00fd0 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.208+0000 7f172ea00700 1 -- 172.21.15.116:0/448270867 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1740012eb0 con 0x55a34ab00fd0 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.208+0000 7f17635e8b80 10 client.25876 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.208+0000 7f17635e8b80 1 -- 172.21.15.116:0/448270867 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a34a87eb60 con 0x55a34ab00fd0 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.208+0000 7f172ea00700 1 -- 172.21.15.116:0/448270867 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a34aa11380 con 0x55a34ab00fd0 2024-03-30T01:00:05.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.208+0000 7f172ea00700 1 -- 172.21.15.116:0/448270867 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f17400075c0 con 0x55a34ab00fd0 2024-03-30T01:00:05.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.208+0000 7f17635e8b80 10 client.25876 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:05.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.208+0000 7f17635e8b80 10 client.25876 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T01:00:05.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.208+0000 7f17635e8b80 20 client.25876 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:05.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.208+0000 7f17635e8b80 1 --2- 172.21.15.116:0/448270867 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55a34ab19840 0x55a34ab1bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.208+0000 7f17635e8b80 4 client.25876 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:00:05.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.208+0000 7f17635e8b80 1 -- 172.21.15.116:0/448270867 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a34aa49700 con 0x55a34ab19840 2024-03-30T01:00:05.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.209+0000 7f174d000700 1 --2- 172.21.15.116:0/448270867 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55a34ab19840 0x55a34ab1bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.209+0000 7f172ea00700 10 client.25876 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T01:00:05.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.209+0000 7f174d000700 1 --2- 172.21.15.116:0/448270867 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55a34ab19840 0x55a34ab1bc60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a3137ace1b7b9e03 server_cookie=c83bacd16ab286f0 in_seq=0 out_seq=0 2024-03-30T01:00:05.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.210+0000 7f172ea00700 1 -- 172.21.15.116:0/448270867 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a34aa49700 con 0x55a34ab19840 2024-03-30T01:00:05.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.210+0000 7f172ea00700 10 client.25876 handle_command_reply: tid=0 2024-03-30T01:00:05.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.210+0000 7f17635e8b80 1 client.25876 shutdown 2024-03-30T01:00:05.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.210+0000 7f17635e8b80 1 -- 172.21.15.116:0/448270867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 msgr2=0x55a34ab14fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.210+0000 7f17635e8b80 1 --2- 172.21.15.116:0/448270867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 0x55a34ab14fa0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1740017000 tx=0x7f1740009920 comp rx=0 tx=0).stop 2024-03-30T01:00:05.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.210+0000 7f17635e8b80 1 -- 172.21.15.116:0/448270867 shutdown_connections 2024-03-30T01:00:05.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.210+0000 7f17635e8b80 1 --2- 172.21.15.116:0/448270867 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55a34ab19840 0x55a34ab1bc60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.210+0000 7f17635e8b80 1 --2- 172.21.15.116:0/448270867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 0x55a34ab14fa0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.210+0000 7f17635e8b80 1 --2- 172.21.15.116:0/448270867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab0da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.210+0000 7f172ea00700 0 client.25876 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T01:00:05.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.210+0000 7f17635e8b80 1 --2- 172.21.15.116:0/448270867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 0x55a34ab0df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.210+0000 7f17635e8b80 1 -- 172.21.15.116:0/448270867 >> 172.21.15.116:0/448270867 conn(0x55a34aa5e6a0 msgr2=0x55a34aafe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:05.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.210+0000 7f17635e8b80 1 -- 172.21.15.116:0/448270867 shutdown_connections 2024-03-30T01:00:05.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.210+0000 7f17635e8b80 1 -- 172.21.15.116:0/448270867 wait complete. 2024-03-30T01:00:05.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.211+0000 7f17635e8b80 20 client.25876 trim_cache size 0 max 16384 2024-03-30T01:00:05.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.225+0000 7f17635e8b80 1 Processor -- start 2024-03-30T01:00:05.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.225+0000 7f17635e8b80 1 -- start start 2024-03-30T01:00:05.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.226+0000 7f17635e8b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab15650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.226+0000 7f17635e8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 0x55a34ab14f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.226+0000 7f17635e8b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 0x55a34ab013b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.226+0000 7f17635e8b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34ab0d590 con 0x55a34aaff7d0 2024-03-30T01:00:05.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.226+0000 7f17635e8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34aafff70 con 0x55a34ab00130 2024-03-30T01:00:05.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.226+0000 7f17635e8b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34aafb660 con 0x55a34ab00fd0 2024-03-30T01:00:05.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.226+0000 7f1747e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 0x55a34ab14f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.226+0000 7f174d000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab15650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.226+0000 7f1747e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 0x55a34ab14f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38396/0 (socket says 172.21.15.116:38396) 2024-03-30T01:00:05.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.226+0000 7f1747e00700 1 -- 172.21.15.116:0/2865019508 learned_addr learned my addr 172.21.15.116:0/2865019508 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:05.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.226+0000 7f174da00700 1 --2- 172.21.15.116:0/2865019508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 0x55a34ab013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.227+0000 7f1747e00700 1 -- 172.21.15.116:0/2865019508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 msgr2=0x55a34ab013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.227+0000 7f1747e00700 1 --2- 172.21.15.116:0/2865019508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 0x55a34ab013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.227+0000 7f1747e00700 1 -- 172.21.15.116:0/2865019508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 msgr2=0x55a34ab15650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.227+0000 7f1747e00700 1 --2- 172.21.15.116:0/2865019508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab15650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.227+0000 7f1747e00700 1 -- 172.21.15.116:0/2865019508 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a34aa16ea0 con 0x55a34ab00130 2024-03-30T01:00:05.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.227+0000 7f174d000700 1 --2- 172.21.15.116:0/2865019508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab15650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:00:05.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.227+0000 7f1747e00700 1 --2- 172.21.15.116:0/2865019508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 0x55a34ab14f60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1734009ea0 tx=0x7f1734013740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5f13d568a22ccea server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:05.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.227+0000 7f173aa00700 1 -- 172.21.15.116:0/2865019508 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734018070 con 0x55a34ab00130 2024-03-30T01:00:05.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.227+0000 7f173aa00700 1 -- 172.21.15.116:0/2865019508 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1734004510 con 0x55a34ab00130 2024-03-30T01:00:05.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.227+0000 7f173aa00700 1 -- 172.21.15.116:0/2865019508 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734003e30 con 0x55a34ab00130 2024-03-30T01:00:05.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.228+0000 7f17635e8b80 1 -- 172.21.15.116:0/2865019508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 msgr2=0x55a34ab14f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.228+0000 7f17635e8b80 1 --2- 172.21.15.116:0/2865019508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 0x55a34ab14f60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1734009ea0 tx=0x7f1734013740 comp rx=0 tx=0).stop 2024-03-30T01:00:05.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.228+0000 7f17635e8b80 1 -- 172.21.15.116:0/2865019508 shutdown_connections 2024-03-30T01:00:05.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.228+0000 7f17635e8b80 1 --2- 172.21.15.116:0/2865019508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 0x55a34ab013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.228+0000 7f17635e8b80 1 --2- 172.21.15.116:0/2865019508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab15650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.228+0000 7f17635e8b80 1 --2- 172.21.15.116:0/2865019508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 0x55a34ab14f60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.228+0000 7f17635e8b80 1 -- 172.21.15.116:0/2865019508 >> 172.21.15.116:0/2865019508 conn(0x55a34aa5e6a0 msgr2=0x55a34aaf35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:05.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.228+0000 7f17635e8b80 1 -- 172.21.15.116:0/2865019508 shutdown_connections 2024-03-30T01:00:05.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.228+0000 7f17635e8b80 1 -- 172.21.15.116:0/2865019508 wait complete. 2024-03-30T01:00:05.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.229+0000 7f17635e8b80 1 -- start start 2024-03-30T01:00:05.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.229+0000 7f17635e8b80 1 Processor -- start 2024-03-30T01:00:05.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f17635e8b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab12e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f17635e8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 0x55a34ab13380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f17635e8b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 0x55a34ab14340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f17635e8b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34ab14880 con 0x55a34aaff7d0 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f17635e8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34ab149f0 con 0x55a34ab00130 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f17635e8b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34ab14bb0 con 0x55a34ab00fd0 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f17635e8b80 10 client.0 _ll_register_callbacks cb 0x55a34a9f6470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f174d000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab12e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f174d000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab12e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60092/0 (socket says 172.21.15.116:60092) 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f174d000700 1 -- 172.21.15.116:0/221868876 learned_addr learned my addr 172.21.15.116:0/221868876 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f174da00700 1 --2- 172.21.15.116:0/221868876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 0x55a34ab14340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f174d000700 1 -- 172.21.15.116:0/221868876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 msgr2=0x55a34ab14340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f174d000700 1 --2- 172.21.15.116:0/221868876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 0x55a34ab14340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f174d000700 1 -- 172.21.15.116:0/221868876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 msgr2=0x55a34ab13380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f174d000700 1 --2- 172.21.15.116:0/221868876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 0x55a34ab13380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.230+0000 7f174d000700 1 -- 172.21.15.116:0/221868876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a34a8d23b0 con 0x55a34aaff7d0 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.231+0000 7f174d000700 1 --2- 172.21.15.116:0/221868876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab12e40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f173c099ba0 tx=0x7f173c00e8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c29cab386e24e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:05.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.231+0000 7f172ea00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.231+0000 7f172ea00700 1 -- 172.21.15.116:0/221868876 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c0b1070 con 0x55a34aaff7d0 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.231+0000 7f172ea00700 1 -- 172.21.15.116:0/221868876 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f173c00cc30 con 0x55a34aaff7d0 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.231+0000 7f17635e8b80 1 -- 172.21.15.116:0/221868876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f173c0a2320 con 0x55a34aaff7d0 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.231+0000 7f172ea00700 1 -- 172.21.15.116:0/221868876 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f173c0b82b0 con 0x55a34aaff7d0 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.231+0000 7f17635e8b80 10 client.15837 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.231+0000 7f17635e8b80 1 -- 172.21.15.116:0/221868876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f1734009b50 con 0x55a34aaff7d0 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.231+0000 7f172ea00700 1 -- 172.21.15.116:0/221868876 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f173c0a2320 con 0x55a34aaff7d0 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.231+0000 7f172ea00700 1 -- 172.21.15.116:0/221868876 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f173c0a1e90 con 0x55a34aaff7d0 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.231+0000 7f17635e8b80 10 client.15837 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.232+0000 7f17635e8b80 10 client.15837 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.232+0000 7f17635e8b80 20 client.15837 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.232+0000 7f17635e8b80 1 --2- 172.21.15.116:0/221868876 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55a34ab6e280 0x55a34ab70670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.232+0000 7f17635e8b80 4 client.15837 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.232+0000 7f17635e8b80 1 -- 172.21.15.116:0/221868876 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a34a89fcf0 con 0x55a34ab6e280 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.232+0000 7f1747e00700 1 --2- 172.21.15.116:0/221868876 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55a34ab6e280 0x55a34ab70670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.232+0000 7f172ea00700 10 client.15837 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.232+0000 7f1747e00700 1 --2- 172.21.15.116:0/221868876 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55a34ab6e280 0x55a34ab70670 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a550ac0011586ec3 server_cookie=27573527ff698137 in_seq=0 out_seq=0 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.232+0000 7f172ea00700 1 -- 172.21.15.116:0/221868876 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a34a89fcf0 con 0x55a34ab6e280 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.232+0000 7f172ea00700 10 client.15837 handle_command_reply: tid=0 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.232+0000 7f17635e8b80 1 client.15837 shutdown 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.233+0000 7f17635e8b80 1 -- 172.21.15.116:0/221868876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 msgr2=0x55a34ab12e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.233+0000 7f17635e8b80 1 --2- 172.21.15.116:0/221868876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab12e40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f173c099ba0 tx=0x7f173c00e8f0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.233+0000 7f17635e8b80 1 -- 172.21.15.116:0/221868876 shutdown_connections 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.233+0000 7f17635e8b80 1 --2- 172.21.15.116:0/221868876 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55a34ab6e280 0x55a34ab70670 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.233+0000 7f17635e8b80 1 --2- 172.21.15.116:0/221868876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a34ab00fd0 0x55a34ab14340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.233+0000 7f17635e8b80 1 --2- 172.21.15.116:0/221868876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34aaff7d0 0x55a34ab12e40 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.233+0000 7f172ea00700 0 client.15837 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.233+0000 7f17635e8b80 1 --2- 172.21.15.116:0/221868876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a34ab00130 0x55a34ab13380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.233+0000 7f17635e8b80 1 -- 172.21.15.116:0/221868876 >> 172.21.15.116:0/221868876 conn(0x55a34aa5e6a0 msgr2=0x55a34ab041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.233+0000 7f17635e8b80 1 -- 172.21.15.116:0/221868876 shutdown_connections 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.233+0000 7f17635e8b80 1 -- 172.21.15.116:0/221868876 wait complete. 2024-03-30T01:00:05.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.233+0000 7f17635e8b80 20 client.15837 trim_cache size 0 max 16384 2024-03-30T01:00:05.242 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:00:05.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.234+0000 7f174fa00700 1 -- 172.21.15.116:0/1903428633 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f17300a11a0 msgr2=0x7f17300a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.234+0000 7f174fa00700 1 --2- 172.21.15.116:0/1903428633 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f17300a11a0 0x7f17300a3660 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f174000d2e0 tx=0x7f174001c040 comp rx=0 tx=0).stop 2024-03-30T01:00:05.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.234+0000 7f174fa00700 1 -- 172.21.15.116:0/1903428633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17480ffc90 msgr2=0x7f1748194bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.234+0000 7f174fa00700 1 --2- 172.21.15.116:0/1903428633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17480ffc90 0x7f1748194bc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f173c00e970 tx=0x7f173c00ec80 comp rx=0 tx=0).stop 2024-03-30T01:00:05.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.234+0000 7f174fa00700 1 -- 172.21.15.116:0/1903428633 shutdown_connections 2024-03-30T01:00:05.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.234+0000 7f174fa00700 1 --2- 172.21.15.116:0/1903428633 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f17300a11a0 0x7f17300a3660 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.234+0000 7f174fa00700 1 --2- 172.21.15.116:0/1903428633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1748101820 0x7f174819c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.234+0000 7f174fa00700 1 --2- 172.21.15.116:0/1903428633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17480ffc90 0x7f1748194bc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.234+0000 7f174fa00700 1 --2- 172.21.15.116:0/1903428633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1748100e80 0x7f1748195100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.234+0000 7f174fa00700 1 -- 172.21.15.116:0/1903428633 >> 172.21.15.116:0/1903428633 conn(0x7f17480fb200 msgr2=0x7f1748104180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:05.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.234+0000 7f174fa00700 1 -- 172.21.15.116:0/1903428633 shutdown_connections 2024-03-30T01:00:05.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.234+0000 7f174fa00700 1 -- 172.21.15.116:0/1903428633 wait complete. 2024-03-30T01:00:05.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T01:00:05.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.557+0000 7f5f45600700 1 -- 172.21.15.116:0/4199081999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f380a4ca0 msgr2=0x7f5f380a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.558+0000 7f5f45600700 1 --2- 172.21.15.116:0/4199081999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f380a4ca0 0x7f5f380a50c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5f40064450 tx=0x7f5f4004f4c0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.563+0000 7f5f45600700 1 -- 172.21.15.116:0/4199081999 shutdown_connections 2024-03-30T01:00:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.563+0000 7f5f45600700 1 --2- 172.21.15.116:0/4199081999 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f380a6790 0x7f5f380ab030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.563+0000 7f5f45600700 1 --2- 172.21.15.116:0/4199081999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f380a4ca0 0x7f5f380a50c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.563+0000 7f5f45600700 1 --2- 172.21.15.116:0/4199081999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f380a5dd0 0x7f5f380a6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.563+0000 7f5f45600700 1 -- 172.21.15.116:0/4199081999 >> 172.21.15.116:0/4199081999 conn(0x7f5f380a0130 msgr2=0x7f5f380a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.569+0000 7f5f45600700 1 -- 172.21.15.116:0/4199081999 shutdown_connections 2024-03-30T01:00:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.570+0000 7f5f45600700 1 -- 172.21.15.116:0/4199081999 wait complete. 2024-03-30T01:00:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.572+0000 7f5f45600700 1 Processor -- start 2024-03-30T01:00:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.574+0000 7f5f45600700 1 -- start start 2024-03-30T01:00:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.575+0000 7f5f45600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f380a4ca0 0x7f5f380b1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.575+0000 7f5f45600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f380a5dd0 0x7f5f380b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.576+0000 7f5f45600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f380a6790 0x7f5f381534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.576+0000 7f5f3f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f380a5dd0 0x7f5f380b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.577+0000 7f5f45600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f380b4d90 con 0x7f5f380a5dd0 2024-03-30T01:00:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.577+0000 7f5f45600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f380b4f00 con 0x7f5f380a4ca0 2024-03-30T01:00:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.577+0000 7f5f45600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f38153a30 con 0x7f5f380a6790 2024-03-30T01:00:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.577+0000 7f5f44c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f380a6790 0x7f5f381534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.577+0000 7f5f44c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f380a6790 0x7f5f381534f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:53060/0 (socket says 172.21.15.116:53060) 2024-03-30T01:00:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.577+0000 7f5f44c00700 1 -- 172.21.15.116:0/952794182 learned_addr learned my addr 172.21.15.116:0/952794182 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.579+0000 7f5f44c00700 1 -- 172.21.15.116:0/952794182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f380a4ca0 msgr2=0x7f5f380b1810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T01:00:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.579+0000 7f5f44c00700 1 --2- 172.21.15.116:0/952794182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f380a4ca0 0x7f5f380b1810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.579+0000 7f5f44c00700 1 -- 172.21.15.116:0/952794182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f380a5dd0 msgr2=0x7f5f380b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.579+0000 7f5f44c00700 1 --2- 172.21.15.116:0/952794182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f380a5dd0 0x7f5f380b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.579+0000 7f5f44c00700 1 -- 172.21.15.116:0/952794182 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4004f330 con 0x7f5f380a6790 2024-03-30T01:00:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.580+0000 7f5f44c00700 1 --2- 172.21.15.116:0/952794182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f380a6790 0x7f5f381534f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00b580 tx=0x7f5f2c00b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.580+0000 7f5f3cc00700 1 -- 172.21.15.116:0/952794182 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c011830 con 0x7f5f380a6790 2024-03-30T01:00:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.580+0000 7f5f3cc00700 1 -- 172.21.15.116:0/952794182 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f2c011e50 con 0x7f5f380a6790 2024-03-30T01:00:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.580+0000 7f5f3cc00700 1 -- 172.21.15.116:0/952794182 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c00f2e0 con 0x7f5f380a6790 2024-03-30T01:00:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.581+0000 7f5f45600700 1 -- 172.21.15.116:0/952794182 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f38153c60 con 0x7f5f380a6790 2024-03-30T01:00:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.581+0000 7f5f45600700 1 -- 172.21.15.116:0/952794182 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f38154120 con 0x7f5f380a6790 2024-03-30T01:00:05.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.601+0000 7f5f5a67ab80 1 Processor -- start 2024-03-30T01:00:05.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.601+0000 7f5f3cc00700 1 -- 172.21.15.116:0/952794182 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f5f2c010610 con 0x7f5f380a6790 2024-03-30T01:00:05.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.602+0000 7f5f5a67ab80 1 -- start start 2024-03-30T01:00:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.604+0000 7f5f5a67ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21a6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.604+0000 7f5f5a67ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a7340 0x55d3f21a7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.604+0000 7f5f5a67ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a8190 0x55d3f21a8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.604+0000 7f5f5a67ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f20fe990 con 0x55d3f21a69e0 2024-03-30T01:00:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.604+0000 7f5f5a67ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f20fc350 con 0x55d3f21a8190 2024-03-30T01:00:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.604+0000 7f5f5a67ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f20f7c40 con 0x55d3f21a7340 2024-03-30T01:00:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.606+0000 7f5f3fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a7340 0x55d3f21a7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.606+0000 7f5f3fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a7340 0x55d3f21a7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:53070/0 (socket says 172.21.15.116:53070) 2024-03-30T01:00:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.606+0000 7f5f3fe00700 1 -- 172.21.15.116:0/2439810100 learned_addr learned my addr 172.21.15.116:0/2439810100 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.607+0000 7f5f3cc00700 1 --2- 172.21.15.116:0/952794182 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5f280a1170 0x7f5f280a3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.607+0000 7f5f3cc00700 1 -- 172.21.15.116:0/952794182 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5f2c0cc3d0 con 0x7f5f380a6790 2024-03-30T01:00:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.608+0000 7f5f44c00700 1 --2- 172.21.15.116:0/952794182 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5f280a1170 0x7f5f280a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.608+0000 7f5f3f400700 1 --2- 172.21.15.116:0/2439810100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a8190 0x55d3f21a8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.608+0000 7f5f3f400700 1 --2- 172.21.15.116:0/2439810100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21a6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.608+0000 7f5f3fe00700 1 -- 172.21.15.116:0/2439810100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a8190 msgr2=0x55d3f21a8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.608+0000 7f5f3fe00700 1 --2- 172.21.15.116:0/2439810100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a8190 0x55d3f21a8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.609+0000 7f5f3fe00700 1 -- 172.21.15.116:0/2439810100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 msgr2=0x55d3f21a6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.609+0000 7f5f3fe00700 1 --2- 172.21.15.116:0/2439810100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21a6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.609+0000 7f5f3fe00700 1 -- 172.21.15.116:0/2439810100 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3f1f793b0 con 0x55d3f21a7340 2024-03-30T01:00:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.609+0000 7f5f3f400700 1 --2- 172.21.15.116:0/2439810100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a8190 0x55d3f21a8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.610+0000 7f5f44c00700 1 --2- 172.21.15.116:0/952794182 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5f280a1170 0x7f5f280a3630 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c095a10 tx=0x7f5f2c004750 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:05.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.613+0000 7f5f3fe00700 1 --2- 172.21.15.116:0/2439810100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a7340 0x55d3f21a7bb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5f40064030 tx=0x7f5f40066b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9417acf8ec879b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:05.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.614+0000 7f5f31200700 1 -- 172.21.15.116:0/2439810100 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4007f070 con 0x55d3f21a7340 2024-03-30T01:00:05.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.615+0000 7f5f31200700 1 -- 172.21.15.116:0/2439810100 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f4006dc90 con 0x55d3f21a7340 2024-03-30T01:00:05.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.615+0000 7f5f31200700 1 -- 172.21.15.116:0/2439810100 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f40078470 con 0x55d3f21a7340 2024-03-30T01:00:05.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.616+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/2439810100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a7340 msgr2=0x55d3f21a7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.616+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/2439810100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a7340 0x55d3f21a7bb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5f40064030 tx=0x7f5f40066b30 comp rx=0 tx=0).stop 2024-03-30T01:00:05.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.616+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/2439810100 shutdown_connections 2024-03-30T01:00:05.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.616+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/2439810100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a8190 0x55d3f21a8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.616+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/2439810100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21a6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.616+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/2439810100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a7340 0x55d3f21a7bb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.618+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/2439810100 >> 172.21.15.116:0/2439810100 conn(0x55d3f21056a0 msgr2=0x55d3f21a1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.627+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/2439810100 shutdown_connections 2024-03-30T01:00:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.628+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/2439810100 wait complete. 2024-03-30T01:00:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.629+0000 7f5f5a67ab80 1 -- start start 2024-03-30T01:00:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.629+0000 7f5f5a67ab80 1 Processor -- start 2024-03-30T01:00:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.630+0000 7f5f5a67ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.631+0000 7f5f5a67ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 0x55d3f21b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.631+0000 7f5f5a67ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a8190 0x55d3f21b7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.631+0000 7f5f3f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.631+0000 7f5f3f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60150/0 (socket says 172.21.15.116:60150) 2024-03-30T01:00:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.631+0000 7f5f5a67ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f2093d80 con 0x55d3f21a69e0 2024-03-30T01:00:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.631+0000 7f5f5a67ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f2093a20 con 0x55d3f21a7340 2024-03-30T01:00:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.631+0000 7f5f5a67ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f1f7b6a0 con 0x55d3f21a8190 2024-03-30T01:00:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.631+0000 7f5f5a67ab80 10 client.0 _ll_register_callbacks cb 0x55d3f209d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.631+0000 7f5f3f400700 1 -- 172.21.15.116:0/2431171378 learned_addr learned my addr 172.21.15.116:0/2431171378 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:05.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.639+0000 7f5f3f400700 1 --2- 172.21.15.116:0/2431171378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a8190 0x55d3f21b7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.640+0000 7f5f3f400700 1 -- 172.21.15.116:0/2431171378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 msgr2=0x55d3f21b0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.640+0000 7f5f3f400700 1 --2- 172.21.15.116:0/2431171378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 0x55d3f21b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.640+0000 7f5f3f400700 1 -- 172.21.15.116:0/2431171378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 msgr2=0x55d3f21b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.640+0000 7f5f3f400700 1 --2- 172.21.15.116:0/2431171378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.640+0000 7f5f3f400700 1 -- 172.21.15.116:0/2431171378 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f380aba60 con 0x55d3f21a8190 2024-03-30T01:00:05.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.645+0000 7f5f3f400700 1 --2- 172.21.15.116:0/2431171378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a8190 0x55d3f21b7d70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5f34014ce0 tx=0x7f5f34004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5570359a9ddeec65 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:05.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.647+0000 7f5f25600700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:00:05.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.647+0000 7f5f25600700 1 -- 172.21.15.116:0/2431171378 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34014130 con 0x55d3f21a8190 2024-03-30T01:00:05.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.647+0000 7f5f25600700 1 -- 172.21.15.116:0/2431171378 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5f34014290 con 0x55d3f21a8190 2024-03-30T01:00:05.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.647+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/2431171378 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d3f20b8380 con 0x55d3f21a8190 2024-03-30T01:00:05.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.648+0000 7f5f25600700 1 -- 172.21.15.116:0/2431171378 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f34010a60 con 0x55d3f21a8190 2024-03-30T01:00:05.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.648+0000 7f5f5a67ab80 10 client.25894 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:05.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.648+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/2431171378 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d3f1f25b60 con 0x55d3f21a8190 2024-03-30T01:00:05.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.648+0000 7f5f25600700 1 -- 172.21.15.116:0/2431171378 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f34024030 con 0x55d3f21a8190 2024-03-30T01:00:05.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.651+0000 7f5f25600700 1 -- 172.21.15.116:0/2431171378 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f5f340047c0 con 0x55d3f21a8190 2024-03-30T01:00:05.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.651+0000 7f5f5a67ab80 10 client.25894 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:05.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.651+0000 7f5f5a67ab80 10 client.25894 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T01:00:05.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.651+0000 7f5f5a67ab80 20 client.25894 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:05.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.651+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/2431171378 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55d3f21bc520 0x55d3f21be940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.651+0000 7f5f5a67ab80 4 client.25894 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:00:05.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.651+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/2431171378 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d3f20f0700 con 0x55d3f21bc520 2024-03-30T01:00:05.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.651+0000 7f5f3fe00700 1 --2- 172.21.15.116:0/2431171378 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55d3f21bc520 0x55d3f21be940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.657+0000 7f5f25600700 10 client.25894 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T01:00:05.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.657+0000 7f5f3fe00700 1 --2- 172.21.15.116:0/2431171378 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55d3f21bc520 0x55d3f21be940 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=753bcdcd7d9b6666 server_cookie=4785e5623116c44a in_seq=0 out_seq=0 2024-03-30T01:00:05.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.660+0000 7f5f25600700 1 -- 172.21.15.116:0/2431171378 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d3f20f0700 con 0x55d3f21bc520 2024-03-30T01:00:05.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.660+0000 7f5f25600700 10 client.25894 handle_command_reply: tid=0 2024-03-30T01:00:05.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.660+0000 7f5f5a67ab80 1 client.25894 shutdown 2024-03-30T01:00:05.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.661+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/2431171378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a8190 msgr2=0x55d3f21b7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.662+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/2431171378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a8190 0x55d3f21b7d70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5f34014ce0 tx=0x7f5f34004750 comp rx=0 tx=0).stop 2024-03-30T01:00:05.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.662+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/2431171378 shutdown_connections 2024-03-30T01:00:05.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.662+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/2431171378 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55d3f21bc520 0x55d3f21be940 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.662+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/2431171378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a8190 0x55d3f21b7d70 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5f34014ce0 tx=0x7f5f34004750 comp rx=0 tx=0).stop 2024-03-30T01:00:05.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.662+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/2431171378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.662+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/2431171378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 0x55d3f21b0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.662+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/2431171378 >> 172.21.15.116:0/2431171378 conn(0x55d3f21056a0 msgr2=0x55d3f21a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:05.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.662+0000 7f5f25600700 0 client.25894 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T01:00:05.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.663+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/2431171378 shutdown_connections 2024-03-30T01:00:05.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.663+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/2431171378 wait complete. 2024-03-30T01:00:05.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.664+0000 7f5f5a67ab80 20 client.25894 trim_cache size 0 max 16384 2024-03-30T01:00:05.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:05 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2089311839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:00:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.706+0000 7f5f5a67ab80 1 Processor -- start 2024-03-30T01:00:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.706+0000 7f5f5a67ab80 1 -- start start 2024-03-30T01:00:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.709+0000 7f5f5a67ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a69e0 0x55d3f21a8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.710+0000 7f5f5a67ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 0x55d3f21b7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.710+0000 7f5f5a67ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21afd00 0x55d3f219b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.710+0000 7f5f5a67ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f21a2640 con 0x55d3f21afd00 2024-03-30T01:00:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.710+0000 7f5f5a67ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f21a2a80 con 0x55d3f21a7340 2024-03-30T01:00:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.710+0000 7f5f5a67ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f21b87e0 con 0x55d3f21a69e0 2024-03-30T01:00:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.711+0000 7f5f3f400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a69e0 0x55d3f21a8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.711+0000 7f5f3f400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a69e0 0x55d3f21a8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:53094/0 (socket says 172.21.15.116:53094) 2024-03-30T01:00:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.711+0000 7f5f3f400700 1 -- 172.21.15.116:0/1780220338 learned_addr learned my addr 172.21.15.116:0/1780220338 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.711+0000 7f5f3f400700 1 --2- 172.21.15.116:0/1780220338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21afd00 0x55d3f219b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.711+0000 7f5f3fe00700 1 --2- 172.21.15.116:0/1780220338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 0x55d3f21b7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.712+0000 7f5f3f400700 1 -- 172.21.15.116:0/1780220338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a69e0 msgr2=0x55d3f21a8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.712+0000 7f5f3f400700 1 --2- 172.21.15.116:0/1780220338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a69e0 0x55d3f21a8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.712+0000 7f5f3f400700 1 -- 172.21.15.116:0/1780220338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 msgr2=0x55d3f21b7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.712+0000 7f5f3f400700 1 --2- 172.21.15.116:0/1780220338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 0x55d3f21b7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.712+0000 7f5f3f400700 1 -- 172.21.15.116:0/1780220338 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3f20bdea0 con 0x55d3f21afd00 2024-03-30T01:00:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.717+0000 7f5f3f400700 1 --2- 172.21.15.116:0/1780220338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21afd00 0x55d3f219b010 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5f340148c0 tx=0x7f5f34010d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7936773468c01741 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.717+0000 7f5f31c00700 1 -- 172.21.15.116:0/1780220338 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34010a60 con 0x55d3f21afd00 2024-03-30T01:00:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.718+0000 7f5f31c00700 1 -- 172.21.15.116:0/1780220338 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f34024030 con 0x55d3f21afd00 2024-03-30T01:00:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.718+0000 7f5f31c00700 1 -- 172.21.15.116:0/1780220338 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34010a60 con 0x55d3f21afd00 2024-03-30T01:00:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.722+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/1780220338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21afd00 msgr2=0x55d3f219b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.722+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/1780220338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21afd00 0x55d3f219b010 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5f340148c0 tx=0x7f5f34010d30 comp rx=0 tx=0).stop 2024-03-30T01:00:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.722+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/1780220338 shutdown_connections 2024-03-30T01:00:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.723+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/1780220338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21a69e0 0x55d3f21a8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.723+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/1780220338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21afd00 0x55d3f219b010 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.723+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/1780220338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 0x55d3f21b7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.723+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/1780220338 >> 172.21.15.116:0/1780220338 conn(0x55d3f21056a0 msgr2=0x55d3f219a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.723+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/1780220338 shutdown_connections 2024-03-30T01:00:05.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.723+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/1780220338 wait complete. 2024-03-30T01:00:05.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.724+0000 7f5f5a67ab80 1 -- start start 2024-03-30T01:00:05.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.724+0000 7f5f5a67ab80 1 Processor -- start 2024-03-30T01:00:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.728+0000 7f5f5a67ab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21b6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.728+0000 7f5f5a67ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 0x55d3f21b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.728+0000 7f5f5a67ab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21afd00 0x55d3f21b6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.728+0000 7f5f5a67ab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f21b8100 con 0x55d3f21a69e0 2024-03-30T01:00:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.728+0000 7f5f5a67ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f2210300 con 0x55d3f21a7340 2024-03-30T01:00:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.728+0000 7f5f5a67ab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3f22104c0 con 0x55d3f21afd00 2024-03-30T01:00:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.728+0000 7f5f5a67ab80 10 client.0 _ll_register_callbacks cb 0x55d3f209d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.729+0000 7f5f3f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.729+0000 7f5f3f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21b6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60158/0 (socket says 172.21.15.116:60158) 2024-03-30T01:00:05.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.730+0000 7f5f3fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 0x55d3f21b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.732+0000 7f5f3fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 0x55d3f21b69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38456/0 (socket says 172.21.15.116:38456) 2024-03-30T01:00:05.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.732+0000 7f5f3fe00700 1 -- 172.21.15.116:0/3353146058 learned_addr learned my addr 172.21.15.116:0/3353146058 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:05.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.732+0000 7f5f3f400700 1 --2- 172.21.15.116:0/3353146058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21afd00 0x55d3f21b6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.736+0000 7f5f3f400700 1 -- 172.21.15.116:0/3353146058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21afd00 msgr2=0x55d3f21b6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.736+0000 7f5f3f400700 1 --2- 172.21.15.116:0/3353146058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21afd00 0x55d3f21b6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.736+0000 7f5f3f400700 1 -- 172.21.15.116:0/3353146058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 msgr2=0x55d3f21b69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.736+0000 7f5f3f400700 1 --2- 172.21.15.116:0/3353146058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 0x55d3f21b69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.737+0000 7f5f3f400700 1 -- 172.21.15.116:0/3353146058 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d3f1f25b60 con 0x55d3f21a69e0 2024-03-30T01:00:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.738+0000 7f5f3f400700 1 --2- 172.21.15.116:0/3353146058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21b6480 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5f340148c0 tx=0x7f5f34010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb02ea126cc9dea server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.738+0000 7f5f25600700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:00:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.740+0000 7f5f25600700 1 -- 172.21.15.116:0/3353146058 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34010ba0 con 0x55d3f21a69e0 2024-03-30T01:00:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.740+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/3353146058 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5f4006dc90 con 0x55d3f21a69e0 2024-03-30T01:00:05.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.741+0000 7f5f25600700 1 -- 172.21.15.116:0/3353146058 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5f3401f3a0 con 0x55d3f21a69e0 2024-03-30T01:00:05.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.742+0000 7f5f25600700 1 -- 172.21.15.116:0/3353146058 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f3401f560 con 0x55d3f21a69e0 2024-03-30T01:00:05.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.743+0000 7f5f5a67ab80 10 client.15855 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:05.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.743+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/3353146058 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d3f1f7dcd0 con 0x55d3f21a69e0 2024-03-30T01:00:05.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.743+0000 7f5f25600700 1 -- 172.21.15.116:0/3353146058 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f34043080 con 0x55d3f21a69e0 2024-03-30T01:00:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.745+0000 7f5f25600700 1 -- 172.21.15.116:0/3353146058 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f5f3402c810 con 0x55d3f21a69e0 2024-03-30T01:00:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.745+0000 7f5f5a67ab80 10 client.15855 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.745+0000 7f5f5a67ab80 10 client.15855 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T01:00:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.745+0000 7f5f5a67ab80 20 client.15855 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.745+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/3353146058 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55d3f2211c50 0x55d3f2214070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.745+0000 7f5f5a67ab80 4 client.15855 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:00:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.745+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/3353146058 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d3f1f46cf0 con 0x55d3f2211c50 2024-03-30T01:00:05.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:05 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2089311839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:00:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.747+0000 7f5f3fe00700 1 --2- 172.21.15.116:0/3353146058 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55d3f2211c50 0x55d3f2214070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.748+0000 7f5f3fe00700 1 --2- 172.21.15.116:0/3353146058 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55d3f2211c50 0x55d3f2214070 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ca8b4a4d534c4181 server_cookie=3488662b7bb68442 in_seq=0 out_seq=0 2024-03-30T01:00:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.748+0000 7f5f25600700 10 client.15855 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T01:00:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.756+0000 7f5f25600700 1 -- 172.21.15.116:0/3353146058 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d3f1f46cf0 con 0x55d3f2211c50 2024-03-30T01:00:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.756+0000 7f5f25600700 10 client.15855 handle_command_reply: tid=0 2024-03-30T01:00:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.757+0000 7f5f5a67ab80 1 client.15855 shutdown 2024-03-30T01:00:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.757+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/3353146058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 msgr2=0x55d3f21b6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.757+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/3353146058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21b6480 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5f340148c0 tx=0x7f5f34010da0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.757+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/3353146058 shutdown_connections 2024-03-30T01:00:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.757+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/3353146058 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55d3f2211c50 0x55d3f2214070 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.757+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/3353146058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d3f21afd00 0x55d3f21b6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.757+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/3353146058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3f21a69e0 0x55d3f21b6480 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.757+0000 7f5f5a67ab80 1 --2- 172.21.15.116:0/3353146058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3f21a7340 0x55d3f21b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.757+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/3353146058 >> 172.21.15.116:0/3353146058 conn(0x55d3f21056a0 msgr2=0x55d3f21ae5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.758+0000 7f5f25600700 0 client.15855 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T01:00:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.758+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/3353146058 shutdown_connections 2024-03-30T01:00:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.760+0000 7f5f5a67ab80 1 -- 172.21.15.116:0/3353146058 wait complete. 2024-03-30T01:00:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.760+0000 7f5f5a67ab80 20 client.15855 trim_cache size 0 max 16384 2024-03-30T01:00:05.766 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:00:05.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.763+0000 7f5f45600700 1 -- 172.21.15.116:0/952794182 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5f280a1170 msgr2=0x7f5f280a3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.763+0000 7f5f45600700 1 --2- 172.21.15.116:0/952794182 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5f280a1170 0x7f5f280a3630 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c095a10 tx=0x7f5f2c004750 comp rx=0 tx=0).stop 2024-03-30T01:00:05.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.763+0000 7f5f45600700 1 -- 172.21.15.116:0/952794182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f380a6790 msgr2=0x7f5f381534f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:05.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.763+0000 7f5f45600700 1 --2- 172.21.15.116:0/952794182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f380a6790 0x7f5f381534f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00b580 tx=0x7f5f2c00b890 comp rx=0 tx=0).stop 2024-03-30T01:00:05.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.764+0000 7f5f45600700 1 -- 172.21.15.116:0/952794182 shutdown_connections 2024-03-30T01:00:05.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.764+0000 7f5f45600700 1 --2- 172.21.15.116:0/952794182 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5f280a1170 0x7f5f280a3630 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.764+0000 7f5f45600700 1 --2- 172.21.15.116:0/952794182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f380a6790 0x7f5f381534f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.764+0000 7f5f45600700 1 --2- 172.21.15.116:0/952794182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f380a4ca0 0x7f5f380b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.764+0000 7f5f45600700 1 --2- 172.21.15.116:0/952794182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f380a5dd0 0x7f5f380b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:05.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.764+0000 7f5f45600700 1 -- 172.21.15.116:0/952794182 >> 172.21.15.116:0/952794182 conn(0x7f5f380a0130 msgr2=0x7f5f380a0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:05.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.764+0000 7f5f45600700 1 -- 172.21.15.116:0/952794182 shutdown_connections 2024-03-30T01:00:05.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:05.766+0000 7f5f45600700 1 -- 172.21.15.116:0/952794182 wait complete. 2024-03-30T01:00:05.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:05 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2089311839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:00:05.883 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T01:00:06.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:06 smithi116 ceph-mon[38086]: pgmap v358: 129 pgs: 129 active+clean; 147 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.7 MiB/s wr, 391 op/s 2024-03-30T01:00:06.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:00:06.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:00:06.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:06 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:00:06.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.427+0000 7ff2c5000700 1 -- 172.21.15.116:0/1891700722 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2b8097fd0 msgr2=0x7ff2b80983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:06.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.427+0000 7ff2c5000700 1 --2- 172.21.15.116:0/1891700722 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2b8097fd0 0x7ff2b80983f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009990 tx=0x7ff2b4009ca0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.427+0000 7ff2c5000700 1 -- 172.21.15.116:0/1891700722 shutdown_connections 2024-03-30T01:00:06.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.427+0000 7ff2c5000700 1 --2- 172.21.15.116:0/1891700722 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2b8097fd0 0x7ff2b80983f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.427+0000 7ff2c5000700 1 --2- 172.21.15.116:0/1891700722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2b8099b60 0x7ff2b80a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.427+0000 7ff2c5000700 1 --2- 172.21.15.116:0/1891700722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2b80991c0 0x7ff2b8099620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.427+0000 7ff2c5000700 1 -- 172.21.15.116:0/1891700722 >> 172.21.15.116:0/1891700722 conn(0x7ff2b8093560 msgr2=0x7ff2b8095980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:06.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.429+0000 7ff2c5000700 1 -- 172.21.15.116:0/1891700722 shutdown_connections 2024-03-30T01:00:06.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.429+0000 7ff2c5000700 1 -- 172.21.15.116:0/1891700722 wait complete. 2024-03-30T01:00:06.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.431+0000 7ff2c5000700 1 Processor -- start 2024-03-30T01:00:06.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.433+0000 7ff2c5000700 1 -- start start 2024-03-30T01:00:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.433+0000 7ff2c5000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2b8097fd0 0x7ff2b813efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.433+0000 7ff2c5000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2b80991c0 0x7ff2b813f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.433+0000 7ff2c5000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2b8099b60 0x7ff2b8146540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.433+0000 7ff2c5000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b8146b10 con 0x7ff2b80991c0 2024-03-30T01:00:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.433+0000 7ff2c5000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b8146c50 con 0x7ff2b8099b60 2024-03-30T01:00:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.433+0000 7ff2c5000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b8146de0 con 0x7ff2b8097fd0 2024-03-30T01:00:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.433+0000 7ff2bf400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2b8097fd0 0x7ff2b813efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.433+0000 7ff2bf400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2b8097fd0 0x7ff2b813efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:53126/0 (socket says 172.21.15.116:53126) 2024-03-30T01:00:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.433+0000 7ff2bf400700 1 -- 172.21.15.116:0/2741415387 learned_addr learned my addr 172.21.15.116:0/2741415387 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.433+0000 7ff2bfe00700 1 --2- 172.21.15.116:0/2741415387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2b8099b60 0x7ff2b8146540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.434+0000 7ff2bfe00700 1 -- 172.21.15.116:0/2741415387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2b8097fd0 msgr2=0x7ff2b813efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.434+0000 7ff2bfe00700 1 --2- 172.21.15.116:0/2741415387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2b8097fd0 0x7ff2b813efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.434+0000 7ff2bfe00700 1 -- 172.21.15.116:0/2741415387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2b80991c0 msgr2=0x7ff2b813f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:06.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.434+0000 7ff2bfe00700 1 --2- 172.21.15.116:0/2741415387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2b80991c0 0x7ff2b813f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.434+0000 7ff2bfe00700 1 -- 172.21.15.116:0/2741415387 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b40095f0 con 0x7ff2b8099b60 2024-03-30T01:00:06.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.434+0000 7ff2bf400700 1 --2- 172.21.15.116:0/2741415387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2b8097fd0 0x7ff2b813efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:00:06.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.434+0000 7ff2bfe00700 1 --2- 172.21.15.116:0/2741415387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2b8099b60 0x7ff2b8146540 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0066a30 tx=0x7ff2c0078710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:06.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.435+0000 7ff2b3e00700 1 -- 172.21.15.116:0/2741415387 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c006a040 con 0x7ff2b8099b60 2024-03-30T01:00:06.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.435+0000 7ff2b3e00700 1 -- 172.21.15.116:0/2741415387 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2c0070900 con 0x7ff2b8099b60 2024-03-30T01:00:06.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.435+0000 7ff2b3e00700 1 -- 172.21.15.116:0/2741415387 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c006f720 con 0x7ff2b8099b60 2024-03-30T01:00:06.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.435+0000 7ff2c5000700 1 -- 172.21.15.116:0/2741415387 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b81470c0 con 0x7ff2b8099b60 2024-03-30T01:00:06.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.435+0000 7ff2c5000700 1 -- 172.21.15.116:0/2741415387 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b8009f00 con 0x7ff2b8099b60 2024-03-30T01:00:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.464+0000 7ff2da100b80 1 Processor -- start 2024-03-30T01:00:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.465+0000 7ff2da100b80 1 -- start start 2024-03-30T01:00:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.465+0000 7ff2da100b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecaa9e0 0x56480ecaae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.465+0000 7ff2da100b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecab340 0x56480ecabbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.465+0000 7ff2da100b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecac190 0x56480ecac610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.465+0000 7ff2da100b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56480ec02990 con 0x56480ecab340 2024-03-30T01:00:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.465+0000 7ff2da100b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56480ec00350 con 0x56480ecaa9e0 2024-03-30T01:00:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.465+0000 7ff2da100b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56480ebfbc40 con 0x56480ecac190 2024-03-30T01:00:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.467+0000 7ff2bf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecab340 0x56480ecabbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.467+0000 7ff2bf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecab340 0x56480ecabbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60198/0 (socket says 172.21.15.116:60198) 2024-03-30T01:00:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.467+0000 7ff2bf400700 1 -- 172.21.15.116:0/1921162531 learned_addr learned my addr 172.21.15.116:0/1921162531 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.472+0000 7ff2bf400700 1 -- 172.21.15.116:0/1921162531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecac190 msgr2=0x56480ecac610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:00:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.472+0000 7ff2bf400700 1 --2- 172.21.15.116:0/1921162531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecac190 0x56480ecac610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.473+0000 7ff2bf400700 1 -- 172.21.15.116:0/1921162531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecaa9e0 msgr2=0x56480ecaae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.473+0000 7ff2bea00700 1 --2- 172.21.15.116:0/1921162531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecaa9e0 0x56480ecaae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.473+0000 7ff2bf400700 1 --2- 172.21.15.116:0/1921162531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecaa9e0 0x56480ecaae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.473+0000 7ff2bf400700 1 -- 172.21.15.116:0/1921162531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56480ea7d3b0 con 0x56480ecab340 2024-03-30T01:00:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.473+0000 7ff2bea00700 1 --2- 172.21.15.116:0/1921162531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecaa9e0 0x56480ecaae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.477+0000 7ff2b3e00700 1 -- 172.21.15.116:0/2741415387 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7ff2c0070440 con 0x7ff2b8099b60 2024-03-30T01:00:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.479+0000 7ff2b3e00700 1 --2- 172.21.15.116:0/2741415387 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff2a80a1270 0x7ff2a80a3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.482+0000 7ff2bf400700 1 --2- 172.21.15.116:0/1921162531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecab340 0x56480ecabbb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4005c70 tx=0x7ff2b400bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb83c908a3fb8bbb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.483+0000 7ff2bea00700 1 --2- 172.21.15.116:0/2741415387 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff2a80a1270 0x7ff2a80a3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.488+0000 7ff2b3e00700 1 -- 172.21.15.116:0/2741415387 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff2c012f8e0 con 0x7ff2b8099b60 2024-03-30T01:00:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.490+0000 7ff2bea00700 1 --2- 172.21.15.116:0/2741415387 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff2a80a1270 0x7ff2a80a3730 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac005d90 tx=0x7ff2ac005f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.504+0000 7ff2b0c00700 1 -- 172.21.15.116:0/1921162531 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b401d070 con 0x56480ecab340 2024-03-30T01:00:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.504+0000 7ff2b0c00700 1 -- 172.21.15.116:0/1921162531 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2b401e040 con 0x56480ecab340 2024-03-30T01:00:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.504+0000 7ff2b0c00700 1 -- 172.21.15.116:0/1921162531 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4023460 con 0x56480ecab340 2024-03-30T01:00:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.505+0000 7ff2da100b80 1 -- 172.21.15.116:0/1921162531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecab340 msgr2=0x56480ecabbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.505+0000 7ff2da100b80 1 --2- 172.21.15.116:0/1921162531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecab340 0x56480ecabbb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4005c70 tx=0x7ff2b400bc00 comp rx=0 tx=0).stop 2024-03-30T01:00:06.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.509+0000 7ff2da100b80 1 -- 172.21.15.116:0/1921162531 shutdown_connections 2024-03-30T01:00:06.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.509+0000 7ff2da100b80 1 --2- 172.21.15.116:0/1921162531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecac190 0x56480ecac610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.509+0000 7ff2da100b80 1 --2- 172.21.15.116:0/1921162531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecaa9e0 0x56480ecaae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.509+0000 7ff2da100b80 1 --2- 172.21.15.116:0/1921162531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecab340 0x56480ecabbb0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.509+0000 7ff2da100b80 1 -- 172.21.15.116:0/1921162531 >> 172.21.15.116:0/1921162531 conn(0x56480ec096a0 msgr2=0x56480eca5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:06.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.512+0000 7ff2da100b80 1 -- 172.21.15.116:0/1921162531 shutdown_connections 2024-03-30T01:00:06.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.512+0000 7ff2da100b80 1 -- 172.21.15.116:0/1921162531 wait complete. 2024-03-30T01:00:06.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.513+0000 7ff2da100b80 1 -- start start 2024-03-30T01:00:06.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.514+0000 7ff2da100b80 1 Processor -- start 2024-03-30T01:00:06.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.520+0000 7ff2da100b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecaa9e0 0x56480ecb47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.523+0000 7ff2bf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecaa9e0 0x56480ecb47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:06.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.523+0000 7ff2bf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecaa9e0 0x56480ecb47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60202/0 (socket says 172.21.15.116:60202) 2024-03-30T01:00:06.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.524+0000 7ff2bf400700 1 -- 172.21.15.116:0/1437154661 learned_addr learned my addr 172.21.15.116:0/1437154661 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:06.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.525+0000 7ff2da100b80 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecab340 0x56480ecb4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.527+0000 7ff2da100b80 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecac190 0x56480ecbbd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.527+0000 7ff2da100b80 1 -- 172.21.15.116:0/1437154661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56480eb97d80 con 0x56480ecaa9e0 2024-03-30T01:00:06.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.528+0000 7ff2bea00700 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecab340 0x56480ecb4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:06.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.528+0000 7ff2da100b80 1 -- 172.21.15.116:0/1437154661 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56480eb97a20 con 0x56480ecac190 2024-03-30T01:00:06.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.528+0000 7ff2da100b80 1 -- 172.21.15.116:0/1437154661 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56480ea7f6a0 con 0x56480ecab340 2024-03-30T01:00:06.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.528+0000 7ff2da100b80 10 client.0 _ll_register_callbacks cb 0x56480eba1470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:06.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.530+0000 7ff2bea00700 1 -- 172.21.15.116:0/1437154661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecac190 msgr2=0x56480ecbbd70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:00:06.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.530+0000 7ff2bea00700 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecac190 0x56480ecbbd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.530+0000 7ff2bea00700 1 -- 172.21.15.116:0/1437154661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecaa9e0 msgr2=0x56480ecb47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:06.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.530+0000 7ff2bea00700 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecaa9e0 0x56480ecb47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.530+0000 7ff2bea00700 1 -- 172.21.15.116:0/1437154661 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b80a0d60 con 0x56480ecab340 2024-03-30T01:00:06.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.530+0000 7ff2bf400700 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecaa9e0 0x56480ecb47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:06.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.530+0000 7ff2da100b80 1 -- 172.21.15.116:0/1437154661 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56480ebbc380 con 0x56480ecab340 2024-03-30T01:00:06.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.531+0000 7ff2bea00700 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecab340 0x56480ecb4d30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x56480ecab970 tx=0x7ff2ac0058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e86f0f978518c8d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:06.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.531+0000 7ff2a5600700 10 client.25909 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:00:06.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.542+0000 7ff2a5600700 1 -- 172.21.15.116:0/1437154661 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac011e90 con 0x56480ecab340 2024-03-30T01:00:06.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.542+0000 7ff2a5600700 1 -- 172.21.15.116:0/1437154661 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff2ac01e440 con 0x56480ecab340 2024-03-30T01:00:06.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.543+0000 7ff2a5600700 1 -- 172.21.15.116:0/1437154661 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2ac01e600 con 0x56480ecab340 2024-03-30T01:00:06.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.543+0000 7ff2da100b80 10 client.25909 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:06.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.543+0000 7ff2da100b80 1 -- 172.21.15.116:0/1437154661 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x56480ea29b60 con 0x56480ecab340 2024-03-30T01:00:06.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.543+0000 7ff2a5600700 1 -- 172.21.15.116:0/1437154661 <== mon.1 v2:172.21.15.181:3300/0 4 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7ff2ac01e760 con 0x56480ecab340 2024-03-30T01:00:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.546+0000 7ff2da100b80 10 client.25909 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.546+0000 7ff2da100b80 10 client.25909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:00:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.546+0000 7ff2da100b80 20 client.25909 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.546+0000 7ff2da100b80 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56480ecc0430 0x56480ecc2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.546+0000 7ff2da100b80 4 client.25909 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:00:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.546+0000 7ff2da100b80 1 -- 172.21.15.116:0/1437154661 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56480ebf4700 con 0x56480ecc0430 2024-03-30T01:00:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.559+0000 7ff2bf400700 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56480ecc0430 0x56480ecc2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:06.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.571+0000 7ff2bf400700 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56480ecc0430 0x56480ecc2850 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=496baab4517ac5f0 server_cookie=bcfbb340ed7ee2d4 in_seq=0 out_seq=0 2024-03-30T01:00:06.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.571+0000 7ff2a5600700 10 client.25909 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:06.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.574+0000 7ff2a5600700 1 -- 172.21.15.116:0/1437154661 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56480ebf4700 con 0x56480ecc0430 2024-03-30T01:00:06.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.574+0000 7ff2a5600700 10 client.25909 handle_command_reply: tid=0 2024-03-30T01:00:06.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.575+0000 7ff2da100b80 1 client.25909 shutdown 2024-03-30T01:00:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.575+0000 7ff2da100b80 1 -- 172.21.15.116:0/1437154661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecab340 msgr2=0x56480ecb4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:06.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.575+0000 7ff2da100b80 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecab340 0x56480ecb4d30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x56480ecab970 tx=0x7ff2ac0058e0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.593+0000 7ff2da100b80 1 -- 172.21.15.116:0/1437154661 shutdown_connections 2024-03-30T01:00:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.593+0000 7ff2da100b80 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56480ecc0430 0x56480ecc2850 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.593+0000 7ff2da100b80 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecac190 0x56480ecbbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.593+0000 7ff2da100b80 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecaa9e0 0x56480ecb47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.593+0000 7ff2da100b80 1 --2- 172.21.15.116:0/1437154661 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecab340 0x56480ecb4d30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.593+0000 7ff2da100b80 1 -- 172.21.15.116:0/1437154661 >> 172.21.15.116:0/1437154661 conn(0x56480ec096a0 msgr2=0x56480eca93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.593+0000 7ff2a5600700 0 client.25909 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.594+0000 7ff2da100b80 1 -- 172.21.15.116:0/1437154661 shutdown_connections 2024-03-30T01:00:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.594+0000 7ff2da100b80 1 -- 172.21.15.116:0/1437154661 wait complete. 2024-03-30T01:00:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.595+0000 7ff2da100b80 20 client.25909 trim_cache size 0 max 16384 2024-03-30T01:00:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.685+0000 7ff2da100b80 1 Processor -- start 2024-03-30T01:00:06.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.686+0000 7ff2da100b80 1 -- start start 2024-03-30T01:00:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.688+0000 7ff2da100b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecaa9e0 0x56480ecbc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.688+0000 7ff2da100b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecab340 0x56480ecbbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.688+0000 7ff2da100b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecac190 0x56480ecac570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.688+0000 7ff2da100b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56480ecbcef0 con 0x56480ecab340 2024-03-30T01:00:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.688+0000 7ff2da100b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56480eca6a80 con 0x56480ecac190 2024-03-30T01:00:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.688+0000 7ff2da100b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56480ecb59e0 con 0x56480ecaa9e0 2024-03-30T01:00:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.689+0000 7ff2bea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecab340 0x56480ecbbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.690+0000 7ff2bea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecab340 0x56480ecbbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60206/0 (socket says 172.21.15.116:60206) 2024-03-30T01:00:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.690+0000 7ff2bea00700 1 -- 172.21.15.116:0/863680969 learned_addr learned my addr 172.21.15.116:0/863680969 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.690+0000 7ff2bea00700 1 -- 172.21.15.116:0/863680969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecaa9e0 msgr2=0x56480ecbc420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:00:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.690+0000 7ff2bea00700 1 --2- 172.21.15.116:0/863680969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecaa9e0 0x56480ecbc420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.690+0000 7ff2bea00700 1 -- 172.21.15.116:0/863680969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecac190 msgr2=0x56480ecac570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:00:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.690+0000 7ff2bea00700 1 --2- 172.21.15.116:0/863680969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecac190 0x56480ecac570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.691+0000 7ff2bea00700 1 -- 172.21.15.116:0/863680969 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56480ebc1ea0 con 0x56480ecab340 2024-03-30T01:00:06.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.693+0000 7ff2bea00700 1 --2- 172.21.15.116:0/863680969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecab340 0x56480ecbbd30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0099c0 tx=0x7ff2ac01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7b3cd277c31f6a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:06.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.696+0000 7ff2b1600700 1 -- 172.21.15.116:0/863680969 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac011e10 con 0x56480ecab340 2024-03-30T01:00:06.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.696+0000 7ff2b1600700 1 -- 172.21.15.116:0/863680969 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56480ebbc380 con 0x56480ecab340 2024-03-30T01:00:06.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.696+0000 7ff2b1600700 1 -- 172.21.15.116:0/863680969 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac031a40 con 0x56480ecab340 2024-03-30T01:00:06.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.698+0000 7ff2da100b80 1 -- 172.21.15.116:0/863680969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecab340 msgr2=0x56480ecbbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:06.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.698+0000 7ff2da100b80 1 --2- 172.21.15.116:0/863680969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecab340 0x56480ecbbd30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0099c0 tx=0x7ff2ac01e000 comp rx=0 tx=0).stop 2024-03-30T01:00:06.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.716+0000 7ff2da100b80 1 -- 172.21.15.116:0/863680969 shutdown_connections 2024-03-30T01:00:06.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.716+0000 7ff2da100b80 1 --2- 172.21.15.116:0/863680969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecaa9e0 0x56480ecbc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.716+0000 7ff2da100b80 1 --2- 172.21.15.116:0/863680969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecac190 0x56480ecac570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.716+0000 7ff2da100b80 1 --2- 172.21.15.116:0/863680969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecab340 0x56480ecbbd30 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.716+0000 7ff2da100b80 1 -- 172.21.15.116:0/863680969 >> 172.21.15.116:0/863680969 conn(0x56480ec096a0 msgr2=0x56480ec9e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:06.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.722+0000 7ff2da100b80 1 -- 172.21.15.116:0/863680969 shutdown_connections 2024-03-30T01:00:06.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.722+0000 7ff2da100b80 1 -- 172.21.15.116:0/863680969 wait complete. 2024-03-30T01:00:06.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.723+0000 7ff2da100b80 1 -- start start 2024-03-30T01:00:06.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.723+0000 7ff2da100b80 1 Processor -- start 2024-03-30T01:00:06.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.724+0000 7ff2da100b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecaa9e0 0x56480ecbb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.725+0000 7ff2da100b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecab340 0x56480ecbbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.725+0000 7ff2da100b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecac190 0x56480ecb9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.725+0000 7ff2bf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecaa9e0 0x56480ecbb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:06.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.725+0000 7ff2bf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecaa9e0 0x56480ecbb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60208/0 (socket says 172.21.15.116:60208) 2024-03-30T01:00:06.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.725+0000 7ff2bf400700 1 -- 172.21.15.116:0/4293154304 learned_addr learned my addr 172.21.15.116:0/4293154304 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:06.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.726+0000 7ff2bfe00700 1 --2- 172.21.15.116:0/4293154304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecac190 0x56480ecb9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:06.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.726+0000 7ff2da100b80 1 -- 172.21.15.116:0/4293154304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56480ecba300 con 0x56480ecaa9e0 2024-03-30T01:00:06.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.726+0000 7ff2da100b80 1 -- 172.21.15.116:0/4293154304 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56480ecba470 con 0x56480ecab340 2024-03-30T01:00:06.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.726+0000 7ff2da100b80 1 -- 172.21.15.116:0/4293154304 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56480ecba630 con 0x56480ecac190 2024-03-30T01:00:06.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.726+0000 7ff2da100b80 10 client.0 _ll_register_callbacks cb 0x56480eba1470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:06.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.727+0000 7ff2bfe00700 1 -- 172.21.15.116:0/4293154304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecab340 msgr2=0x56480ecbbba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:00:06.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.727+0000 7ff2bfe00700 1 --2- 172.21.15.116:0/4293154304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecab340 0x56480ecbbba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.727+0000 7ff2bfe00700 1 -- 172.21.15.116:0/4293154304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecaa9e0 msgr2=0x56480ecbb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:06.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.727+0000 7ff2bfe00700 1 --2- 172.21.15.116:0/4293154304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecaa9e0 0x56480ecbb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.727+0000 7ff2bfe00700 1 -- 172.21.15.116:0/4293154304 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56480ea29b60 con 0x56480ecac190 2024-03-30T01:00:06.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.727+0000 7ff2bfe00700 1 --2- 172.21.15.116:0/4293154304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecac190 0x56480ecb9c70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00645f0 tx=0x7ff2c0070b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d35af03ba13b4cfc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:06.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.732+0000 7ff2a5600700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:00:06.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.732+0000 7ff2a5600700 1 -- 172.21.15.116:0/4293154304 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c00fc9c0 con 0x56480ecac190 2024-03-30T01:00:06.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.732+0000 7ff2a5600700 1 -- 172.21.15.116:0/4293154304 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff2c01007f0 con 0x56480ecac190 2024-03-30T01:00:06.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.739+0000 7ff2da100b80 1 -- 172.21.15.116:0/4293154304 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff2b401e040 con 0x56480ecac190 2024-03-30T01:00:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.743+0000 7ff2a5600700 1 -- 172.21.15.116:0/4293154304 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2c0104e40 con 0x56480ecac190 2024-03-30T01:00:06.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:06 smithi181 ceph-mon[47622]: pgmap v358: 129 pgs: 129 active+clean; 147 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.7 MiB/s wr, 391 op/s 2024-03-30T01:00:06.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:00:06.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:00:06.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:06 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:00:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.748+0000 7ff2da100b80 10 client.25912 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.748+0000 7ff2da100b80 1 -- 172.21.15.116:0/4293154304 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56480ea81cd0 con 0x56480ecac190 2024-03-30T01:00:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.749+0000 7ff2a5600700 1 -- 172.21.15.116:0/4293154304 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2b401e040 con 0x56480ecac190 2024-03-30T01:00:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.749+0000 7ff2a5600700 1 -- 172.21.15.116:0/4293154304 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7ff2c00fbfc0 con 0x56480ecac190 2024-03-30T01:00:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.750+0000 7ff2da100b80 10 client.25912 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.750+0000 7ff2da100b80 10 client.25912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:00:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.750+0000 7ff2da100b80 20 client.25912 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.750+0000 7ff2da100b80 1 --2- 172.21.15.116:0/4293154304 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56480ed15380 0x56480ed177a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.750+0000 7ff2da100b80 4 client.25912 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:00:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.750+0000 7ff2da100b80 1 -- 172.21.15.116:0/4293154304 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56480ea4acf0 con 0x56480ed15380 2024-03-30T01:00:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.761+0000 7ff2bf400700 1 --2- 172.21.15.116:0/4293154304 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56480ed15380 0x56480ed177a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.761+0000 7ff2bf400700 1 --2- 172.21.15.116:0/4293154304 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56480ed15380 0x56480ed177a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=928a175d85e1fb62 server_cookie=2ddb7a90912a119b in_seq=0 out_seq=0 2024-03-30T01:00:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.764+0000 7ff2a5600700 10 client.25912 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:06.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.770+0000 7ff2a5600700 1 -- 172.21.15.116:0/4293154304 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56480ea4acf0 con 0x56480ed15380 2024-03-30T01:00:06.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.770+0000 7ff2a5600700 10 client.25912 handle_command_reply: tid=0 2024-03-30T01:00:06.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.770+0000 7ff2da100b80 1 client.25912 shutdown 2024-03-30T01:00:06.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.772+0000 7ff2da100b80 1 -- 172.21.15.116:0/4293154304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecac190 msgr2=0x56480ecb9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:06.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.772+0000 7ff2da100b80 1 --2- 172.21.15.116:0/4293154304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecac190 0x56480ecb9c70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00645f0 tx=0x7ff2c0070b20 comp rx=0 tx=0).stop 2024-03-30T01:00:06.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.772+0000 7ff2da100b80 1 -- 172.21.15.116:0/4293154304 shutdown_connections 2024-03-30T01:00:06.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.772+0000 7ff2da100b80 1 --2- 172.21.15.116:0/4293154304 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56480ed15380 0x56480ed177a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.772+0000 7ff2da100b80 1 --2- 172.21.15.116:0/4293154304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56480ecac190 0x56480ecb9c70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.772+0000 7ff2da100b80 1 --2- 172.21.15.116:0/4293154304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56480ecaa9e0 0x56480ecbb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.772+0000 7ff2da100b80 1 --2- 172.21.15.116:0/4293154304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56480ecab340 0x56480ecbbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.772+0000 7ff2da100b80 1 -- 172.21.15.116:0/4293154304 >> 172.21.15.116:0/4293154304 conn(0x56480ec096a0 msgr2=0x56480ecaf3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:06.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.772+0000 7ff2a5600700 0 client.25912 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:06.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.776+0000 7ff2da100b80 1 -- 172.21.15.116:0/4293154304 shutdown_connections 2024-03-30T01:00:06.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.776+0000 7ff2da100b80 1 -- 172.21.15.116:0/4293154304 wait complete. 2024-03-30T01:00:06.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.789+0000 7ff2da100b80 20 client.25912 trim_cache size 0 max 16384 2024-03-30T01:00:06.799 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:00:06.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.793+0000 7ff2c5000700 1 -- 172.21.15.116:0/2741415387 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff2a80a1270 msgr2=0x7ff2a80a3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:06.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.793+0000 7ff2c5000700 1 --2- 172.21.15.116:0/2741415387 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff2a80a1270 0x7ff2a80a3730 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac005d90 tx=0x7ff2ac005f70 comp rx=0 tx=0).stop 2024-03-30T01:00:06.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.794+0000 7ff2c5000700 1 -- 172.21.15.116:0/2741415387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2b8099b60 msgr2=0x7ff2b8146540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:06.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.794+0000 7ff2c5000700 1 --2- 172.21.15.116:0/2741415387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2b8099b60 0x7ff2b8146540 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0066a30 tx=0x7ff2c0078710 comp rx=0 tx=0).stop 2024-03-30T01:00:06.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.801+0000 7ff2c5000700 1 -- 172.21.15.116:0/2741415387 shutdown_connections 2024-03-30T01:00:06.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.802+0000 7ff2c5000700 1 --2- 172.21.15.116:0/2741415387 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff2a80a1270 0x7ff2a80a3730 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.802+0000 7ff2c5000700 1 --2- 172.21.15.116:0/2741415387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2b8097fd0 0x7ff2b813efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.802+0000 7ff2c5000700 1 --2- 172.21.15.116:0/2741415387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2b8099b60 0x7ff2b8146540 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.803+0000 7ff2c5000700 1 --2- 172.21.15.116:0/2741415387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2b80991c0 0x7ff2b813f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:06.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.803+0000 7ff2c5000700 1 -- 172.21.15.116:0/2741415387 >> 172.21.15.116:0/2741415387 conn(0x7ff2b8093560 msgr2=0x7ff2b809c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:06.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.809+0000 7ff2c5000700 1 -- 172.21.15.116:0/2741415387 shutdown_connections 2024-03-30T01:00:06.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:06.809+0000 7ff2c5000700 1 -- 172.21.15.116:0/2741415387 wait complete. 2024-03-30T01:00:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:06 smithi130 ceph-mon[47626]: pgmap v358: 129 pgs: 129 active+clean; 147 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.7 MiB/s wr, 391 op/s 2024-03-30T01:00:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:00:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:00:06.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:06 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:00:07.918 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T01:00:07.919 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T01:00:08.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.118+0000 7f4710600700 1 -- 172.21.15.116:0/2809358958 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4708101820 msgr2=0x7f4708103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.118+0000 7f4710600700 1 --2- 172.21.15.116:0/2809358958 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4708101820 0x7f4708103c10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009990 tx=0x7f46fc009ca0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.118+0000 7f4710600700 1 -- 172.21.15.116:0/2809358958 shutdown_connections 2024-03-30T01:00:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.118+0000 7f4710600700 1 --2- 172.21.15.116:0/2809358958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708106a80 0x7f4708108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.118+0000 7f4710600700 1 --2- 172.21.15.116:0/2809358958 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4708101820 0x7f4708103c10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.118+0000 7f4710600700 1 --2- 172.21.15.116:0/2809358958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4708104150 0x7f4708106540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.118+0000 7f4710600700 1 -- 172.21.15.116:0/2809358958 >> 172.21.15.116:0/2809358958 conn(0x7f47080fb1d0 msgr2=0x7f47080fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.118+0000 7f4710600700 1 -- 172.21.15.116:0/2809358958 shutdown_connections 2024-03-30T01:00:08.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.118+0000 7f4710600700 1 -- 172.21.15.116:0/2809358958 wait complete. 2024-03-30T01:00:08.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.119+0000 7f4710600700 1 Processor -- start 2024-03-30T01:00:08.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.119+0000 7f4710600700 1 -- start start 2024-03-30T01:00:08.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.119+0000 7f4710600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101820 0x7f4708198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.119+0000 7f4710600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4708104150 0x7f4708199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.119+0000 7f4710600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4708106a80 0x7f47081a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.119+0000 7f4710600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47081a0a50 con 0x7f4708101820 2024-03-30T01:00:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.119+0000 7f4710600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47081a0b90 con 0x7f4708104150 2024-03-30T01:00:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.119+0000 7f4710600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47081a0d20 con 0x7f4708106a80 2024-03-30T01:00:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.120+0000 7f470dc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101820 0x7f4708198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.120+0000 7f470d200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4708104150 0x7f4708199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.120+0000 7f470dc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101820 0x7f4708198f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60228/0 (socket says 172.21.15.116:60228) 2024-03-30T01:00:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.120+0000 7f470dc00700 1 -- 172.21.15.116:0/2908287411 learned_addr learned my addr 172.21.15.116:0/2908287411 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.120+0000 7f470e600700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4708106a80 0x7f47081a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.120+0000 7f470dc00700 1 -- 172.21.15.116:0/2908287411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4708106a80 msgr2=0x7f47081a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.120+0000 7f470dc00700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4708106a80 0x7f47081a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.120+0000 7f470dc00700 1 -- 172.21.15.116:0/2908287411 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4708104150 msgr2=0x7f4708199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.120+0000 7f470dc00700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4708104150 0x7f4708199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.120+0000 7f470dc00700 1 -- 172.21.15.116:0/2908287411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc0095f0 con 0x7f4708101820 2024-03-30T01:00:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.120+0000 7f470e600700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4708106a80 0x7f47081a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:00:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.120+0000 7f470d200700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4708104150 0x7f4708199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:00:08.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.120+0000 7f470dc00700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101820 0x7f4708198f20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f46fc000c00 tx=0x7f46fc005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:08.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.121+0000 7f4706000700 1 -- 172.21.15.116:0/2908287411 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc01d070 con 0x7f4708101820 2024-03-30T01:00:08.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.121+0000 7f4706000700 1 -- 172.21.15.116:0/2908287411 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46fc021410 con 0x7f4708101820 2024-03-30T01:00:08.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.121+0000 7f4710600700 1 -- 172.21.15.116:0/2908287411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47081a0fa0 con 0x7f4708101820 2024-03-30T01:00:08.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.121+0000 7f4706000700 1 -- 172.21.15.116:0/2908287411 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc017650 con 0x7f4708101820 2024-03-30T01:00:08.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.121+0000 7f4710600700 1 -- 172.21.15.116:0/2908287411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47081a1360 con 0x7f4708101820 2024-03-30T01:00:08.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.124+0000 7f4706000700 1 -- 172.21.15.116:0/2908287411 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f46fc021810 con 0x7f4708101820 2024-03-30T01:00:08.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.125+0000 7f4706000700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46f00a11a0 0x7f46f00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.125+0000 7f4706000700 1 -- 172.21.15.116:0/2908287411 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f46fc0ce0d0 con 0x7f4708101820 2024-03-30T01:00:08.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.125+0000 7f470d200700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46f00a11a0 0x7f46f00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.125+0000 7f4724037b80 1 Processor -- start 2024-03-30T01:00:08.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.126+0000 7f4724037b80 1 -- start start 2024-03-30T01:00:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.126+0000 7f4724037b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d86850 0x55f987d86c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.126+0000 7f470d200700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46f00a11a0 0x7f46f00a3660 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f46f4010420 tx=0x7f46f40073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.126+0000 7f4724037b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 0x55f987d87a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.126+0000 7f4724037b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d88050 0x55f987d884d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.126+0000 7f4724037b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f987cdea10 con 0x55f987d86850 2024-03-30T01:00:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.126+0000 7f4724037b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f987cdc3d0 con 0x55f987d871b0 2024-03-30T01:00:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.126+0000 7f4724037b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f987cd7cc0 con 0x55f987d88050 2024-03-30T01:00:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.126+0000 7f470dc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d86850 0x55f987d86c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.126+0000 7f470d200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 0x55f987d87a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.126+0000 7f470dc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d86850 0x55f987d86c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60242/0 (socket says 172.21.15.116:60242) 2024-03-30T01:00:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.126+0000 7f470dc00700 1 -- 172.21.15.116:0/3380771314 learned_addr learned my addr 172.21.15.116:0/3380771314 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.126+0000 7f470e600700 1 --2- 172.21.15.116:0/3380771314 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d88050 0x55f987d884d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.127+0000 7f470dc00700 1 -- 172.21.15.116:0/3380771314 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d88050 msgr2=0x55f987d884d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.127+0000 7f470dc00700 1 --2- 172.21.15.116:0/3380771314 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d88050 0x55f987d884d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.127+0000 7f470dc00700 1 -- 172.21.15.116:0/3380771314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 msgr2=0x55f987d87a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.127+0000 7f470dc00700 1 --2- 172.21.15.116:0/3380771314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 0x55f987d87a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.127+0000 7f470dc00700 1 -- 172.21.15.116:0/3380771314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f987b59430 con 0x55f987d86850 2024-03-30T01:00:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.127+0000 7f470dc00700 1 --2- 172.21.15.116:0/3380771314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d86850 0x55f987d86c70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f46fc09b1d0 tx=0x7f46fc009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4241bbef4a2ceaa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.127+0000 7f46faa00700 1 -- 172.21.15.116:0/3380771314 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc09f070 con 0x55f987d86850 2024-03-30T01:00:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.127+0000 7f46faa00700 1 -- 172.21.15.116:0/3380771314 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46fc0a3630 con 0x55f987d86850 2024-03-30T01:00:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.127+0000 7f46faa00700 1 -- 172.21.15.116:0/3380771314 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc09a9e0 con 0x55f987d86850 2024-03-30T01:00:08.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.128+0000 7f4724037b80 1 -- 172.21.15.116:0/3380771314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d86850 msgr2=0x55f987d86c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.128+0000 7f4724037b80 1 --2- 172.21.15.116:0/3380771314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d86850 0x55f987d86c70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f46fc09b1d0 tx=0x7f46fc009ca0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.128+0000 7f4724037b80 1 -- 172.21.15.116:0/3380771314 shutdown_connections 2024-03-30T01:00:08.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.128+0000 7f4724037b80 1 --2- 172.21.15.116:0/3380771314 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d88050 0x55f987d884d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.128+0000 7f4724037b80 1 --2- 172.21.15.116:0/3380771314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d86850 0x55f987d86c70 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.128+0000 7f4724037b80 1 --2- 172.21.15.116:0/3380771314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 0x55f987d87a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.128+0000 7f4724037b80 1 -- 172.21.15.116:0/3380771314 >> 172.21.15.116:0/3380771314 conn(0x55f987ce5720 msgr2=0x55f987d81a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:08.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.128+0000 7f4724037b80 1 -- 172.21.15.116:0/3380771314 shutdown_connections 2024-03-30T01:00:08.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.128+0000 7f4724037b80 1 -- 172.21.15.116:0/3380771314 wait complete. 2024-03-30T01:00:08.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.128+0000 7f4724037b80 1 -- start start 2024-03-30T01:00:08.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.129+0000 7f4724037b80 1 Processor -- start 2024-03-30T01:00:08.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.129+0000 7f4724037b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d86850 0x55f987d90350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.129+0000 7f4724037b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 0x55f987d90890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.129+0000 7f4724037b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d88050 0x55f987d978d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.129+0000 7f4724037b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f987c73e00 con 0x55f987d88050 2024-03-30T01:00:08.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.129+0000 7f4724037b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f987c73aa0 con 0x55f987d871b0 2024-03-30T01:00:08.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.129+0000 7f4724037b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f987b5b720 con 0x55f987d86850 2024-03-30T01:00:08.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.129+0000 7f4724037b80 10 client.0 _ll_register_callbacks cb 0x55f987c7d4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:08.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.129+0000 7f470d200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 0x55f987d90890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.129+0000 7f470dc00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d86850 0x55f987d90350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.129+0000 7f470d200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 0x55f987d90890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38524/0 (socket says 172.21.15.116:38524) 2024-03-30T01:00:08.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.129+0000 7f470d200700 1 -- 172.21.15.116:0/3435366183 learned_addr learned my addr 172.21.15.116:0/3435366183 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:08.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.130+0000 7f470e600700 1 --2- 172.21.15.116:0/3435366183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d88050 0x55f987d978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.130+0000 7f470d200700 1 -- 172.21.15.116:0/3435366183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d86850 msgr2=0x55f987d90350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.130+0000 7f470d200700 1 --2- 172.21.15.116:0/3435366183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d86850 0x55f987d90350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.130+0000 7f470d200700 1 -- 172.21.15.116:0/3435366183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d88050 msgr2=0x55f987d978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.130+0000 7f470d200700 1 --2- 172.21.15.116:0/3435366183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d88050 0x55f987d978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.130+0000 7f470d200700 1 -- 172.21.15.116:0/3435366183 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47081a1360 con 0x55f987d871b0 2024-03-30T01:00:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.130+0000 7f470dc00700 1 --2- 172.21.15.116:0/3435366183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d86850 0x55f987d90350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:00:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.130+0000 7f470d200700 1 --2- 172.21.15.116:0/3435366183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 0x55f987d90890 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f46f4011010 tx=0x7f46f40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38b551252cf062cf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.130+0000 7f46ef400700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T01:00:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.130+0000 7f46ef400700 1 -- 172.21.15.116:0/3435366183 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4011440 con 0x55f987d871b0 2024-03-30T01:00:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.130+0000 7f4724037b80 1 -- 172.21.15.116:0/3435366183 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f987c98400 con 0x55f987d871b0 2024-03-30T01:00:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.130+0000 7f46ef400700 1 -- 172.21.15.116:0/3435366183 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f46f4017070 con 0x55f987d871b0 2024-03-30T01:00:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.131+0000 7f46ef400700 1 -- 172.21.15.116:0/3435366183 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46f40115a0 con 0x55f987d871b0 2024-03-30T01:00:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.131+0000 7f4724037b80 10 client.25925 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.131+0000 7f4724037b80 1 -- 172.21.15.116:0/3435366183 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f987b05be0 con 0x55f987d871b0 2024-03-30T01:00:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.131+0000 7f46ef400700 1 -- 172.21.15.116:0/3435366183 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f987c98400 con 0x55f987d871b0 2024-03-30T01:00:08.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.131+0000 7f46ef400700 1 -- 172.21.15.116:0/3435366183 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f46f4020740 con 0x55f987d871b0 2024-03-30T01:00:08.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.131+0000 7f4724037b80 10 client.25925 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:08.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.131+0000 7f4724037b80 10 client.25925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:00:08.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.131+0000 7f4724037b80 20 client.25925 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:08.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.131+0000 7f4724037b80 1 --2- 172.21.15.116:0/3435366183 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f987d9c110 0x55f987d9e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.131+0000 7f4724037b80 4 client.25925 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:00:08.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.131+0000 7f4724037b80 1 -- 172.21.15.116:0/3435366183 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f987cd0780 con 0x55f987d9c110 2024-03-30T01:00:08.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.132+0000 7f470dc00700 1 --2- 172.21.15.116:0/3435366183 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f987d9c110 0x55f987d9e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.132+0000 7f470dc00700 1 --2- 172.21.15.116:0/3435366183 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f987d9c110 0x55f987d9e530 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce63ab2429664899 server_cookie=225a2942912d9b42 in_seq=0 out_seq=0 2024-03-30T01:00:08.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.132+0000 7f46ef400700 10 client.25925 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:08.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.135+0000 7f46ef400700 1 -- 172.21.15.116:0/3435366183 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f987cd0780 con 0x55f987d9c110 2024-03-30T01:00:08.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.135+0000 7f46ef400700 10 client.25925 handle_command_reply: tid=0 2024-03-30T01:00:08.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.135+0000 7f4724037b80 1 client.25925 shutdown 2024-03-30T01:00:08.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.135+0000 7f4724037b80 1 -- 172.21.15.116:0/3435366183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 msgr2=0x55f987d90890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.135+0000 7f4724037b80 1 --2- 172.21.15.116:0/3435366183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 0x55f987d90890 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f46f4011010 tx=0x7f46f40047c0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.135+0000 7f4724037b80 1 -- 172.21.15.116:0/3435366183 shutdown_connections 2024-03-30T01:00:08.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.135+0000 7f4724037b80 1 --2- 172.21.15.116:0/3435366183 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f987d9c110 0x55f987d9e530 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.135+0000 7f4724037b80 1 --2- 172.21.15.116:0/3435366183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d86850 0x55f987d90350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.135+0000 7f4724037b80 1 --2- 172.21.15.116:0/3435366183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d88050 0x55f987d978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.135+0000 7f4724037b80 1 --2- 172.21.15.116:0/3435366183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 0x55f987d90890 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.135+0000 7f46ef400700 0 client.25925 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:08.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.135+0000 7f4724037b80 1 -- 172.21.15.116:0/3435366183 >> 172.21.15.116:0/3435366183 conn(0x55f987ce5720 msgr2=0x55f987d851f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:08.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.135+0000 7f4724037b80 1 -- 172.21.15.116:0/3435366183 shutdown_connections 2024-03-30T01:00:08.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.135+0000 7f4724037b80 1 -- 172.21.15.116:0/3435366183 wait complete. 2024-03-30T01:00:08.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.136+0000 7f4724037b80 20 client.25925 trim_cache size 0 max 16384 2024-03-30T01:00:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.153+0000 7f4724037b80 1 Processor -- start 2024-03-30T01:00:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.153+0000 7f4724037b80 1 -- start start 2024-03-30T01:00:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.153+0000 7f4724037b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d86850 0x55f987d8e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.153+0000 7f4724037b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 0x55f987d977f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.153+0000 7f4724037b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d88050 0x55f987d851f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.153+0000 7f4724037b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f987d98a80 con 0x55f987d88050 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.153+0000 7f4724037b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f987d8fef0 con 0x55f987d871b0 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.153+0000 7f4724037b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f987d82730 con 0x55f987d86850 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.153+0000 7f470e600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d88050 0x55f987d851f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.153+0000 7f470e600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d88050 0x55f987d851f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60258/0 (socket says 172.21.15.116:60258) 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.153+0000 7f470e600700 1 -- 172.21.15.116:0/3261585074 learned_addr learned my addr 172.21.15.116:0/3261585074 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.153+0000 7f470d200700 1 --2- 172.21.15.116:0/3261585074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 0x55f987d977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.154+0000 7f470dc00700 1 --2- 172.21.15.116:0/3261585074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d86850 0x55f987d8e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.154+0000 7f470e600700 1 -- 172.21.15.116:0/3261585074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d86850 msgr2=0x55f987d8e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.154+0000 7f470e600700 1 --2- 172.21.15.116:0/3261585074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d86850 0x55f987d8e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.154+0000 7f470e600700 1 -- 172.21.15.116:0/3261585074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 msgr2=0x55f987d977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.154+0000 7f470e600700 1 --2- 172.21.15.116:0/3261585074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 0x55f987d977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.154+0000 7f470e600700 1 -- 172.21.15.116:0/3261585074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f987c9df20 con 0x55f987d88050 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.154+0000 7f470dc00700 1 --2- 172.21.15.116:0/3261585074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d86850 0x55f987d8e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:00:08.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.154+0000 7f470e600700 1 --2- 172.21.15.116:0/3261585074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d88050 0x55f987d851f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f47000076c0 tx=0x7f470000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a5d9cd586530019 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:08.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.154+0000 7f46fb400700 1 -- 172.21.15.116:0/3261585074 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470000e040 con 0x55f987d88050 2024-03-30T01:00:08.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.154+0000 7f46fb400700 1 -- 172.21.15.116:0/3261585074 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f470000f040 con 0x55f987d88050 2024-03-30T01:00:08.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.154+0000 7f46fb400700 1 -- 172.21.15.116:0/3261585074 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4700017670 con 0x55f987d88050 2024-03-30T01:00:08.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.155+0000 7f4724037b80 1 -- 172.21.15.116:0/3261585074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d88050 msgr2=0x55f987d851f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.155+0000 7f4724037b80 1 --2- 172.21.15.116:0/3261585074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d88050 0x55f987d851f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f47000076c0 tx=0x7f470000b040 comp rx=0 tx=0).stop 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.155+0000 7f4724037b80 1 -- 172.21.15.116:0/3261585074 shutdown_connections 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.155+0000 7f4724037b80 1 --2- 172.21.15.116:0/3261585074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d86850 0x55f987d8e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.155+0000 7f4724037b80 1 --2- 172.21.15.116:0/3261585074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d88050 0x55f987d851f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.155+0000 7f4724037b80 1 --2- 172.21.15.116:0/3261585074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d871b0 0x55f987d977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.155+0000 7f4724037b80 1 -- 172.21.15.116:0/3261585074 >> 172.21.15.116:0/3261585074 conn(0x55f987ce5720 msgr2=0x55f987d7a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.155+0000 7f4724037b80 1 -- 172.21.15.116:0/3261585074 shutdown_connections 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.155+0000 7f4724037b80 1 -- 172.21.15.116:0/3261585074 wait complete. 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.155+0000 7f4724037b80 1 -- start start 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.156+0000 7f4724037b80 1 Processor -- start 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.156+0000 7f4724037b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d86850 0x55f987d96ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.156+0000 7f4724037b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d871b0 0x55f987d973e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.156+0000 7f4724037b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d88050 0x55f987df8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.156+0000 7f4724037b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f987df9120 con 0x55f987d871b0 2024-03-30T01:00:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.156+0000 7f4724037b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f987df9290 con 0x55f987d86850 2024-03-30T01:00:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.156+0000 7f4724037b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f987df9450 con 0x55f987d88050 2024-03-30T01:00:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.156+0000 7f4724037b80 10 client.0 _ll_register_callbacks cb 0x55f987c7d4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.156+0000 7f470dc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d86850 0x55f987d96ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.156+0000 7f470dc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d86850 0x55f987d96ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38548/0 (socket says 172.21.15.116:38548) 2024-03-30T01:00:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.156+0000 7f470dc00700 1 -- 172.21.15.116:0/1568468065 learned_addr learned my addr 172.21.15.116:0/1568468065 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.156+0000 7f470e600700 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d88050 0x55f987df8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.156+0000 7f470d200700 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d871b0 0x55f987d973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f470e600700 1 -- 172.21.15.116:0/1568468065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d86850 msgr2=0x55f987d96ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f470e600700 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d86850 0x55f987d96ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f470e600700 1 -- 172.21.15.116:0/1568468065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d871b0 msgr2=0x55f987d973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f470e600700 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d871b0 0x55f987d973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f470e600700 1 -- 172.21.15.116:0/1568468065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4700009370 con 0x55f987d88050 2024-03-30T01:00:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f470d200700 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d871b0 0x55f987d973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:00:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f470dc00700 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d86850 0x55f987d96ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:00:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f470e600700 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d88050 0x55f987df8be0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f47000079d0 tx=0x7f4700003c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b961212c4a15071 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f46ef400700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:00:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f46ef400700 1 -- 172.21.15.116:0/1568468065 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470000e070 con 0x55f987d88050 2024-03-30T01:00:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f4724037b80 1 -- 172.21.15.116:0/1568468065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f46fc0a3630 con 0x55f987d88050 2024-03-30T01:00:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f46ef400700 1 -- 172.21.15.116:0/1568468065 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4700004230 con 0x55f987d88050 2024-03-30T01:00:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f46ef400700 1 -- 172.21.15.116:0/1568468065 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4700024070 con 0x55f987d88050 2024-03-30T01:00:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f4724037b80 10 client.25930 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.157+0000 7f4724037b80 1 -- 172.21.15.116:0/1568468065 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f987b5dd50 con 0x55f987d88050 2024-03-30T01:00:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.158+0000 7f46ef400700 1 -- 172.21.15.116:0/1568468065 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46fc0a3630 con 0x55f987d88050 2024-03-30T01:00:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.158+0000 7f46ef400700 1 -- 172.21.15.116:0/1568468065 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f470001d030 con 0x55f987d88050 2024-03-30T01:00:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.158+0000 7f4724037b80 10 client.25930 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.158+0000 7f4724037b80 10 client.25930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:00:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.158+0000 7f4724037b80 20 client.25930 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.158+0000 7f4724037b80 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f987dfae20 0x55f987dfd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.158+0000 7f4724037b80 4 client.25930 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T01:00:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.158+0000 7f4724037b80 1 -- 172.21.15.116:0/1568468065 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f987b26d70 con 0x55f987dfae20 2024-03-30T01:00:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.158+0000 7f470dc00700 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f987dfae20 0x55f987dfd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.159+0000 7f470dc00700 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f987dfae20 0x55f987dfd240 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89c57ce4e201a20f server_cookie=8b80f2724f58c597 in_seq=0 out_seq=0 2024-03-30T01:00:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.159+0000 7f46ef400700 10 client.25930 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:08.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f46ef400700 1 -- 172.21.15.116:0/1568468065 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f987b26d70 con 0x55f987dfae20 2024-03-30T01:00:08.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f46ef400700 10 client.25930 handle_command_reply: tid=0 2024-03-30T01:00:08.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f4724037b80 1 client.25930 shutdown 2024-03-30T01:00:08.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f4724037b80 1 -- 172.21.15.116:0/1568468065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d88050 msgr2=0x55f987df8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f4724037b80 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d88050 0x55f987df8be0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f47000079d0 tx=0x7f4700003c30 comp rx=0 tx=0).stop 2024-03-30T01:00:08.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f4724037b80 1 -- 172.21.15.116:0/1568468065 shutdown_connections 2024-03-30T01:00:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f4724037b80 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f987dfae20 0x55f987dfd240 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f4724037b80 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f987d88050 0x55f987df8be0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f4724037b80 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f987d86850 0x55f987d96ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f4724037b80 1 --2- 172.21.15.116:0/1568468065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f987d871b0 0x55f987d973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f46ef400700 0 client.25930 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f4724037b80 1 -- 172.21.15.116:0/1568468065 >> 172.21.15.116:0/1568468065 conn(0x55f987ce5720 msgr2=0x55f987d8e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f4724037b80 1 -- 172.21.15.116:0/1568468065 shutdown_connections 2024-03-30T01:00:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f4724037b80 1 -- 172.21.15.116:0/1568468065 wait complete. 2024-03-30T01:00:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.419+0000 7f4724037b80 20 client.25930 trim_cache size 0 max 16384 2024-03-30T01:00:08.426 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:00:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-30T01:00:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "b51b928f-0c53-48f1-8b7e-9d812744668a", 2024-03-30T01:00:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-30T01:00:08.427 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:00:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.420+0000 7f4710600700 1 -- 172.21.15.116:0/2908287411 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46f00a11a0 msgr2=0x7f46f00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.420+0000 7f4710600700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46f00a11a0 0x7f46f00a3660 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f46f4010420 tx=0x7f46f40073d0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.420+0000 7f4710600700 1 -- 172.21.15.116:0/2908287411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101820 msgr2=0x7f4708198f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.420+0000 7f4710600700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101820 0x7f4708198f20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f46fc000c00 tx=0x7f46fc005740 comp rx=0 tx=0).stop 2024-03-30T01:00:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.420+0000 7f4710600700 1 -- 172.21.15.116:0/2908287411 shutdown_connections 2024-03-30T01:00:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.420+0000 7f4710600700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46f00a11a0 0x7f46f00a3660 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.420+0000 7f4710600700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4708106a80 0x7f47081a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.420+0000 7f4710600700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101820 0x7f4708198f20 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.420+0000 7f4710600700 1 --2- 172.21.15.116:0/2908287411 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4708104150 0x7f4708199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.420+0000 7f4710600700 1 -- 172.21.15.116:0/2908287411 >> 172.21.15.116:0/2908287411 conn(0x7f47080fb1d0 msgr2=0x7f47081010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.420+0000 7f4710600700 1 -- 172.21.15.116:0/2908287411 shutdown_connections 2024-03-30T01:00:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.420+0000 7f4710600700 1 -- 172.21.15.116:0/2908287411 wait complete. 2024-03-30T01:00:08.445 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T01:00:08.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:08 smithi116 ceph-mon[38086]: pgmap v359: 129 pgs: 129 active+clean; 178 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 9.0 MiB/s wr, 579 op/s 2024-03-30T01:00:08.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.674+0000 7f6b82400700 1 -- 172.21.15.116:0/2450794305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c102910 msgr2=0x7f6b7c1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.674+0000 7f6b82400700 1 --2- 172.21.15.116:0/2450794305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c102910 0x7f6b7c1091c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6b74009fc0 tx=0x7f6b7400b330 comp rx=0 tx=0).stop 2024-03-30T01:00:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.674+0000 7f6b82400700 1 -- 172.21.15.116:0/2450794305 shutdown_connections 2024-03-30T01:00:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.674+0000 7f6b82400700 1 --2- 172.21.15.116:0/2450794305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c102910 0x7f6b7c1091c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.674+0000 7f6b82400700 1 --2- 172.21.15.116:0/2450794305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b7c100d80 0x7f6b7c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.674+0000 7f6b82400700 1 --2- 172.21.15.116:0/2450794305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b7c101f70 0x7f6b7c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.674+0000 7f6b82400700 1 -- 172.21.15.116:0/2450794305 >> 172.21.15.116:0/2450794305 conn(0x7f6b7c0fc2f0 msgr2=0x7f6b7c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.674+0000 7f6b82400700 1 -- 172.21.15.116:0/2450794305 shutdown_connections 2024-03-30T01:00:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.674+0000 7f6b82400700 1 -- 172.21.15.116:0/2450794305 wait complete. 2024-03-30T01:00:08.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.675+0000 7f6b82400700 1 Processor -- start 2024-03-30T01:00:08.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.675+0000 7f6b82400700 1 -- start start 2024-03-30T01:00:08.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b82400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b7c100d80 0x7f6b7c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b82400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b7c101f70 0x7f6b7c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b82400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c102910 0x7f6b7c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b82400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c073e80 con 0x7f6b7c102910 2024-03-30T01:00:08.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b82400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c073ff0 con 0x7f6b7c100d80 2024-03-30T01:00:08.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b82400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c1a0e80 con 0x7f6b7c101f70 2024-03-30T01:00:08.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b7b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b7c100d80 0x7f6b7c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b7aa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b7c101f70 0x7f6b7c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b7b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b7c100d80 0x7f6b7c074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38568/0 (socket says 172.21.15.116:38568) 2024-03-30T01:00:08.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b7be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c102910 0x7f6b7c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b7b400700 1 -- 172.21.15.116:0/421898013 learned_addr learned my addr 172.21.15.116:0/421898013 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:08.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b7aa00700 1 -- 172.21.15.116:0/421898013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b7c100d80 msgr2=0x7f6b7c074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b7aa00700 1 --2- 172.21.15.116:0/421898013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b7c100d80 0x7f6b7c074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b7aa00700 1 -- 172.21.15.116:0/421898013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c102910 msgr2=0x7f6b7c0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b7aa00700 1 --2- 172.21.15.116:0/421898013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c102910 0x7f6b7c0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b7aa00700 1 -- 172.21.15.116:0/421898013 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b74009c40 con 0x7f6b7c101f70 2024-03-30T01:00:08.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b7b400700 1 --2- 172.21.15.116:0/421898013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b7c100d80 0x7f6b7c074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:00:08.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.676+0000 7f6b7be00700 1 --2- 172.21.15.116:0/421898013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c102910 0x7f6b7c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:08.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.677+0000 7f6b7aa00700 1 --2- 172.21.15.116:0/421898013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b7c101f70 0x7f6b7c0731a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800e9f0 tx=0x7f6b6800edb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:08.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.677+0000 7f6b73e00700 1 -- 172.21.15.116:0/421898013 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6800cbd0 con 0x7f6b7c101f70 2024-03-30T01:00:08.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.677+0000 7f6b73e00700 1 -- 172.21.15.116:0/421898013 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b6800cd30 con 0x7f6b7c101f70 2024-03-30T01:00:08.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.677+0000 7f6b82400700 1 -- 172.21.15.116:0/421898013 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7c1a10b0 con 0x7f6b7c101f70 2024-03-30T01:00:08.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.677+0000 7f6b73e00700 1 -- 172.21.15.116:0/421898013 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68018680 con 0x7f6b7c101f70 2024-03-30T01:00:08.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.677+0000 7f6b82400700 1 -- 172.21.15.116:0/421898013 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7c1a1560 con 0x7f6b7c101f70 2024-03-30T01:00:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.682+0000 7f6b73e00700 1 -- 172.21.15.116:0/421898013 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f6b68010430 con 0x7f6b7c101f70 2024-03-30T01:00:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.683+0000 7f6b95f78b80 1 Processor -- start 2024-03-30T01:00:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.683+0000 7f6b95f78b80 1 -- start start 2024-03-30T01:00:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.683+0000 7f6b95f78b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05c97d0 0x555ea05c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.683+0000 7f6b95f78b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05ca130 0x555ea05ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.683+0000 7f6b95f78b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 0x555ea05cb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.683+0000 7f6b95f78b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea0521990 con 0x555ea05c97d0 2024-03-30T01:00:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.683+0000 7f6b95f78b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea051f350 con 0x555ea05ca130 2024-03-30T01:00:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.683+0000 7f6b95f78b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea051ac40 con 0x555ea05cafd0 2024-03-30T01:00:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.684+0000 7f6b7be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05c97d0 0x555ea05c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.684+0000 7f6b7aa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 0x555ea05cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.684+0000 7f6b7be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05c97d0 0x555ea05c9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60294/0 (socket says 172.21.15.116:60294) 2024-03-30T01:00:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.684+0000 7f6b7be00700 1 -- 172.21.15.116:0/893132263 learned_addr learned my addr 172.21.15.116:0/893132263 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.684+0000 7f6b7b400700 1 --2- 172.21.15.116:0/893132263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05ca130 0x555ea05ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.684+0000 7f6b73e00700 1 --2- 172.21.15.116:0/421898013 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6b600a1120 0x7f6b600a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.684+0000 7f6b73e00700 1 -- 172.21.15.116:0/421898013 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6b68014070 con 0x7f6b7c101f70 2024-03-30T01:00:08.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.684+0000 7f6b7be00700 1 --2- 172.21.15.116:0/421898013 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6b600a1120 0x7f6b600a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.685+0000 7f6b7b400700 1 -- 172.21.15.116:0/893132263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 msgr2=0x555ea05cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.685+0000 7f6b7b400700 1 --2- 172.21.15.116:0/893132263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 0x555ea05cb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.685+0000 7f6b7b400700 1 -- 172.21.15.116:0/893132263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05c97d0 msgr2=0x555ea05c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.685+0000 7f6b7b400700 1 --2- 172.21.15.116:0/893132263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05c97d0 0x555ea05c9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.685+0000 7f6b7b400700 1 -- 172.21.15.116:0/893132263 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ea039c3b0 con 0x555ea05ca130 2024-03-30T01:00:08.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.685+0000 7f6b7be00700 1 --2- 172.21.15.116:0/421898013 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6b600a1120 0x7f6b600a35e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6b74007860 tx=0x7f6b7400b330 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:08.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.685+0000 7f6b7b400700 1 --2- 172.21.15.116:0/893132263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05ca130 0x555ea05ca9f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009d50 tx=0x7f6b6c0116c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81a91a0c0337fc03 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:08.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.686+0000 7f6b70c00700 1 -- 172.21.15.116:0/893132263 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c009ea0 con 0x555ea05ca130 2024-03-30T01:00:08.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.686+0000 7f6b70c00700 1 -- 172.21.15.116:0/893132263 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b6c011e20 con 0x555ea05ca130 2024-03-30T01:00:08.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.686+0000 7f6b70c00700 1 -- 172.21.15.116:0/893132263 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c012450 con 0x555ea05ca130 2024-03-30T01:00:08.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.687+0000 7f6b95f78b80 1 -- 172.21.15.116:0/893132263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05ca130 msgr2=0x555ea05ca9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.687+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/893132263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05ca130 0x555ea05ca9f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009d50 tx=0x7f6b6c0116c0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.687+0000 7f6b95f78b80 1 -- 172.21.15.116:0/893132263 shutdown_connections 2024-03-30T01:00:08.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.687+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/893132263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 0x555ea05cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.687+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/893132263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05c97d0 0x555ea05c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.687+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/893132263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05ca130 0x555ea05ca9f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.687+0000 7f6b95f78b80 1 -- 172.21.15.116:0/893132263 >> 172.21.15.116:0/893132263 conn(0x555ea05286a0 msgr2=0x555ea05c49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:08.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.687+0000 7f6b95f78b80 1 -- 172.21.15.116:0/893132263 shutdown_connections 2024-03-30T01:00:08.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.687+0000 7f6b95f78b80 1 -- 172.21.15.116:0/893132263 wait complete. 2024-03-30T01:00:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.687+0000 7f6b95f78b80 1 -- start start 2024-03-30T01:00:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.688+0000 7f6b95f78b80 1 Processor -- start 2024-03-30T01:00:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.688+0000 7f6b95f78b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05d3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.688+0000 7f6b95f78b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05ca130 0x555ea05d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.688+0000 7f6b95f78b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 0x555ea05dabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.688+0000 7f6b95f78b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea04b6d80 con 0x555ea05ca130 2024-03-30T01:00:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.688+0000 7f6b95f78b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea04b6a20 con 0x555ea05c97d0 2024-03-30T01:00:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.688+0000 7f6b95f78b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea039e6a0 con 0x555ea05cafd0 2024-03-30T01:00:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.688+0000 7f6b95f78b80 10 client.0 _ll_register_callbacks cb 0x555ea04c0470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.688+0000 7f6b7b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05d3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.689+0000 7f6b7b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05d3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38590/0 (socket says 172.21.15.116:38590) 2024-03-30T01:00:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.689+0000 7f6b7b400700 1 -- 172.21.15.116:0/1746467324 learned_addr learned my addr 172.21.15.116:0/1746467324 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.689+0000 7f6b7be00700 1 --2- 172.21.15.116:0/1746467324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 0x555ea05dabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.689+0000 7f6b7aa00700 1 --2- 172.21.15.116:0/1746467324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05ca130 0x555ea05d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.689+0000 7f6b7be00700 1 -- 172.21.15.116:0/1746467324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 msgr2=0x555ea05d3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.689+0000 7f6b7be00700 1 --2- 172.21.15.116:0/1746467324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05d3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.689+0000 7f6b7be00700 1 -- 172.21.15.116:0/1746467324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05ca130 msgr2=0x555ea05d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.689+0000 7f6b7be00700 1 --2- 172.21.15.116:0/1746467324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05ca130 0x555ea05d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.689+0000 7f6b7be00700 1 -- 172.21.15.116:0/1746467324 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6c009980 con 0x555ea05cafd0 2024-03-30T01:00:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.689+0000 7f6b7b400700 1 --2- 172.21.15.116:0/1746467324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05d3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:00:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.690+0000 7f6b7be00700 1 --2- 172.21.15.116:0/1746467324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 0x555ea05dabb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6b74009f90 tx=0x7f6b74009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffbc0290a1acc95f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.690+0000 7f6b65600700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:00:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.690+0000 7f6b65600700 1 -- 172.21.15.116:0/1746467324 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7400e040 con 0x555ea05cafd0 2024-03-30T01:00:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.690+0000 7f6b65600700 1 -- 172.21.15.116:0/1746467324 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6b7400bbf0 con 0x555ea05cafd0 2024-03-30T01:00:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.690+0000 7f6b95f78b80 1 -- 172.21.15.116:0/1746467324 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555ea04db380 con 0x555ea05cafd0 2024-03-30T01:00:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.690+0000 7f6b65600700 1 -- 172.21.15.116:0/1746467324 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b74004e90 con 0x555ea05cafd0 2024-03-30T01:00:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.690+0000 7f6b95f78b80 10 client.25942 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.690+0000 7f6b95f78b80 1 -- 172.21.15.116:0/1746467324 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x555ea0348b60 con 0x555ea05cafd0 2024-03-30T01:00:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.691+0000 7f6b65600700 1 -- 172.21.15.116:0/1746467324 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b74038080 con 0x555ea05cafd0 2024-03-30T01:00:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.691+0000 7f6b65600700 1 -- 172.21.15.116:0/1746467324 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f6b74005040 con 0x555ea05cafd0 2024-03-30T01:00:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.691+0000 7f6b95f78b80 10 client.25942 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.691+0000 7f6b95f78b80 10 client.25942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:00:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.691+0000 7f6b95f78b80 20 client.25942 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.691+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/1746467324 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x555ea05e7d20 0x555ea05ea140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.691+0000 7f6b95f78b80 4 client.25942 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:00:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.691+0000 7f6b95f78b80 1 -- 172.21.15.116:0/1746467324 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ea0513700 con 0x555ea05e7d20 2024-03-30T01:00:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.691+0000 7f6b7b400700 1 --2- 172.21.15.116:0/1746467324 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x555ea05e7d20 0x555ea05ea140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.692+0000 7f6b7b400700 1 --2- 172.21.15.116:0/1746467324 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x555ea05e7d20 0x555ea05ea140 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8e6e383592d7d7d server_cookie=ae9bdc29bab9a0ca in_seq=0 out_seq=0 2024-03-30T01:00:08.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.692+0000 7f6b65600700 10 client.25942 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:08.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.693+0000 7f6b65600700 1 -- 172.21.15.116:0/1746467324 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555ea0513700 con 0x555ea05e7d20 2024-03-30T01:00:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.693+0000 7f6b65600700 10 client.25942 handle_command_reply: tid=0 2024-03-30T01:00:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.693+0000 7f6b95f78b80 1 client.25942 shutdown 2024-03-30T01:00:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.693+0000 7f6b95f78b80 1 -- 172.21.15.116:0/1746467324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 msgr2=0x555ea05dabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.693+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/1746467324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 0x555ea05dabb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6b74009f90 tx=0x7f6b74009920 comp rx=0 tx=0).stop 2024-03-30T01:00:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.693+0000 7f6b95f78b80 1 -- 172.21.15.116:0/1746467324 shutdown_connections 2024-03-30T01:00:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.693+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/1746467324 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x555ea05e7d20 0x555ea05ea140 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.693+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/1746467324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 0x555ea05dabb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.693+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/1746467324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05d3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.693+0000 7f6b65600700 0 client.25942 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.693+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/1746467324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05ca130 0x555ea05d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.693+0000 7f6b95f78b80 1 -- 172.21.15.116:0/1746467324 >> 172.21.15.116:0/1746467324 conn(0x555ea05286a0 msgr2=0x555ea05c8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.693+0000 7f6b95f78b80 1 -- 172.21.15.116:0/1746467324 shutdown_connections 2024-03-30T01:00:08.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.693+0000 7f6b95f78b80 1 -- 172.21.15.116:0/1746467324 wait complete. 2024-03-30T01:00:08.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.694+0000 7f6b95f78b80 20 client.25942 trim_cache size 0 max 16384 2024-03-30T01:00:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.726+0000 7f6b95f78b80 1 Processor -- start 2024-03-30T01:00:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.726+0000 7f6b95f78b80 1 -- start start 2024-03-30T01:00:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.727+0000 7f6b95f78b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.727+0000 7f6b95f78b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05ca130 0x555ea05dab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.727+0000 7f6b95f78b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05cafd0 0x555ea05d14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.727+0000 7f6b95f78b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea05c0480 con 0x555ea05cafd0 2024-03-30T01:00:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.727+0000 7f6b95f78b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea05c58b0 con 0x555ea05c97d0 2024-03-30T01:00:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.727+0000 7f6b95f78b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea05d4820 con 0x555ea05ca130 2024-03-30T01:00:08.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.728+0000 7f6b7aa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05ca130 0x555ea05dab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.728+0000 7f6b7aa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05ca130 0x555ea05dab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:53256/0 (socket says 172.21.15.116:53256) 2024-03-30T01:00:08.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.728+0000 7f6b7aa00700 1 -- 172.21.15.116:0/592900647 learned_addr learned my addr 172.21.15.116:0/592900647 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:08.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.728+0000 7f6b7b400700 1 --2- 172.21.15.116:0/592900647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.728+0000 7f6b7be00700 1 --2- 172.21.15.116:0/592900647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05cafd0 0x555ea05d14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.728+0000 7f6b7aa00700 1 -- 172.21.15.116:0/592900647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 msgr2=0x555ea05c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.728+0000 7f6b7aa00700 1 --2- 172.21.15.116:0/592900647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05c3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.728+0000 7f6b7aa00700 1 -- 172.21.15.116:0/592900647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05cafd0 msgr2=0x555ea05d14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.728+0000 7f6b7aa00700 1 --2- 172.21.15.116:0/592900647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05cafd0 0x555ea05d14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.728+0000 7f6b7aa00700 1 -- 172.21.15.116:0/592900647 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ea04e0ea0 con 0x555ea05ca130 2024-03-30T01:00:08.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.728+0000 7f6b7be00700 1 --2- 172.21.15.116:0/592900647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05cafd0 0x555ea05d14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:08.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.729+0000 7f6b7aa00700 1 --2- 172.21.15.116:0/592900647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05ca130 0x555ea05dab70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800c5b0 tx=0x7f6b6800ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ff88f970f5581bf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:08.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.729+0000 7f6b71600700 1 -- 172.21.15.116:0/592900647 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68099c20 con 0x555ea05ca130 2024-03-30T01:00:08.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.729+0000 7f6b71600700 1 -- 172.21.15.116:0/592900647 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b68099a40 con 0x555ea05ca130 2024-03-30T01:00:08.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.729+0000 7f6b71600700 1 -- 172.21.15.116:0/592900647 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b680995d0 con 0x555ea05ca130 2024-03-30T01:00:08.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.730+0000 7f6b95f78b80 1 -- 172.21.15.116:0/592900647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05ca130 msgr2=0x555ea05dab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.730+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/592900647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05ca130 0x555ea05dab70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800c5b0 tx=0x7f6b6800ef90 comp rx=0 tx=0).stop 2024-03-30T01:00:08.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.731+0000 7f6b95f78b80 1 -- 172.21.15.116:0/592900647 shutdown_connections 2024-03-30T01:00:08.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.731+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/592900647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05cafd0 0x555ea05d14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.731+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/592900647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.731+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/592900647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05ca130 0x555ea05dab70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.731+0000 7f6b95f78b80 1 -- 172.21.15.116:0/592900647 >> 172.21.15.116:0/592900647 conn(0x555ea05286a0 msgr2=0x555ea05bd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:08.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.731+0000 7f6b95f78b80 1 -- 172.21.15.116:0/592900647 shutdown_connections 2024-03-30T01:00:08.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.731+0000 7f6b95f78b80 1 -- 172.21.15.116:0/592900647 wait complete. 2024-03-30T01:00:08.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.731+0000 7f6b95f78b80 1 -- start start 2024-03-30T01:00:08.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.732+0000 7f6b95f78b80 1 Processor -- start 2024-03-30T01:00:08.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.733+0000 7f6b95f78b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05da970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.733+0000 7f6b95f78b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 0x555ea05d9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.733+0000 7f6b95f78b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05d97d0 0x555ea05d9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.733+0000 7f6b95f78b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea063bb00 con 0x555ea05d97d0 2024-03-30T01:00:08.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.733+0000 7f6b95f78b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea063bc40 con 0x555ea05c97d0 2024-03-30T01:00:08.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.733+0000 7f6b95f78b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea063bd80 con 0x555ea05cafd0 2024-03-30T01:00:08.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.733+0000 7f6b95f78b80 10 client.0 _ll_register_callbacks cb 0x555ea04c0470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:08.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.733+0000 7f6b7b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05da970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.733+0000 7f6b7b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05da970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38612/0 (socket says 172.21.15.116:38612) 2024-03-30T01:00:08.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.733+0000 7f6b7b400700 1 -- 172.21.15.116:0/2398398665 learned_addr learned my addr 172.21.15.116:0/2398398665 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:08.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.734+0000 7f6b7be00700 1 --2- 172.21.15.116:0/2398398665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05d97d0 0x555ea05d9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.734+0000 7f6b7aa00700 1 --2- 172.21.15.116:0/2398398665 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 0x555ea05d9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.734+0000 7f6b7be00700 1 -- 172.21.15.116:0/2398398665 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 msgr2=0x555ea05d9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.734+0000 7f6b7be00700 1 --2- 172.21.15.116:0/2398398665 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 0x555ea05d9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.734+0000 7f6b7be00700 1 -- 172.21.15.116:0/2398398665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 msgr2=0x555ea05da970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:08.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.734+0000 7f6b7be00700 1 --2- 172.21.15.116:0/2398398665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05da970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.734+0000 7f6b7be00700 1 -- 172.21.15.116:0/2398398665 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7c1a1560 con 0x555ea05d97d0 2024-03-30T01:00:08.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.734+0000 7f6b7be00700 1 --2- 172.21.15.116:0/2398398665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05d97d0 0x555ea05d9c50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6b74005370 tx=0x7f6b74006060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229c011055a4256c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:08.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.735+0000 7f6b65600700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:00:08.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.735+0000 7f6b65600700 1 -- 172.21.15.116:0/2398398665 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b74020650 con 0x555ea05d97d0 2024-03-30T01:00:08.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.735+0000 7f6b65600700 1 -- 172.21.15.116:0/2398398665 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6b7400b680 con 0x555ea05d97d0 2024-03-30T01:00:08.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.735+0000 7f6b95f78b80 1 -- 172.21.15.116:0/2398398665 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b6c011e20 con 0x555ea05d97d0 2024-03-30T01:00:08.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.735+0000 7f6b65600700 1 -- 172.21.15.116:0/2398398665 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b74009ac0 con 0x555ea05d97d0 2024-03-30T01:00:08.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.735+0000 7f6b95f78b80 10 client.15897 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:08.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.735+0000 7f6b95f78b80 1 -- 172.21.15.116:0/2398398665 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x555ea03a0cd0 con 0x555ea05d97d0 2024-03-30T01:00:08.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.736+0000 7f6b65600700 1 -- 172.21.15.116:0/2398398665 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b6c011e20 con 0x555ea05d97d0 2024-03-30T01:00:08.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.736+0000 7f6b65600700 1 -- 172.21.15.116:0/2398398665 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f6b7401b5e0 con 0x555ea05d97d0 2024-03-30T01:00:08.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.736+0000 7f6b95f78b80 10 client.15897 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:08.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.736+0000 7f6b95f78b80 10 client.15897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:00:08.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.736+0000 7f6b95f78b80 20 client.15897 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:08.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.736+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/2398398665 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x555ea063d490 0x555ea063f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:08.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.736+0000 7f6b95f78b80 4 client.15897 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T01:00:08.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.736+0000 7f6b95f78b80 1 -- 172.21.15.116:0/2398398665 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555ea0369cf0 con 0x555ea063d490 2024-03-30T01:00:08.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.737+0000 7f6b7b400700 1 --2- 172.21.15.116:0/2398398665 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x555ea063d490 0x555ea063f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:08.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.737+0000 7f6b7b400700 1 --2- 172.21.15.116:0/2398398665 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x555ea063d490 0x555ea063f8b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a199d264ed22f180 server_cookie=f282284bb4421336 in_seq=0 out_seq=0 2024-03-30T01:00:08.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.737+0000 7f6b65600700 10 client.15897 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:08.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:08 smithi181 ceph-mon[47622]: pgmap v359: 129 pgs: 129 active+clean; 178 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 9.0 MiB/s wr, 579 op/s 2024-03-30T01:00:08.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.749+0000 7f6b65600700 1 -- 172.21.15.116:0/2398398665 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x555ea0369cf0 con 0x555ea063d490 2024-03-30T01:00:08.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.749+0000 7f6b65600700 10 client.15897 handle_command_reply: tid=0 2024-03-30T01:00:08.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.749+0000 7f6b95f78b80 1 client.15897 shutdown 2024-03-30T01:00:08.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.749+0000 7f6b95f78b80 1 -- 172.21.15.116:0/2398398665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05d97d0 msgr2=0x555ea05d9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.749+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/2398398665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05d97d0 0x555ea05d9c50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6b74005370 tx=0x7f6b74006060 comp rx=0 tx=0).stop 2024-03-30T01:00:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.749+0000 7f6b95f78b80 1 -- 172.21.15.116:0/2398398665 shutdown_connections 2024-03-30T01:00:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.749+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/2398398665 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x555ea063d490 0x555ea063f8b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.749+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/2398398665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ea05d97d0 0x555ea05d9c50 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.749+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/2398398665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea05c97d0 0x555ea05da970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.749+0000 7f6b65600700 0 client.15897 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.749+0000 7f6b95f78b80 1 --2- 172.21.15.116:0/2398398665 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555ea05cafd0 0x555ea05d9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.749+0000 7f6b95f78b80 1 -- 172.21.15.116:0/2398398665 >> 172.21.15.116:0/2398398665 conn(0x555ea05286a0 msgr2=0x555ea05d2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.750+0000 7f6b95f78b80 1 -- 172.21.15.116:0/2398398665 shutdown_connections 2024-03-30T01:00:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.750+0000 7f6b95f78b80 1 -- 172.21.15.116:0/2398398665 wait complete. 2024-03-30T01:00:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.750+0000 7f6b95f78b80 20 client.15897 trim_cache size 0 max 16384 2024-03-30T01:00:08.756 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:00:08.757 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (3738 inodes in the stack)", 2024-03-30T01:00:08.757 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-30T01:00:08.757 INFO:teuthology.orchestra.run.smithi116.stdout: "b51b928f-0c53-48f1-8b7e-9d812744668a": { 2024-03-30T01:00:08.757 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-30T01:00:08.757 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "b51b928f-0c53-48f1-8b7e-9d812744668a", 2024-03-30T01:00:08.757 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-30T01:00:08.757 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:00:08.757 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:00:08.757 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:00:08.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.750+0000 7f6b82400700 1 -- 172.21.15.116:0/421898013 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6b600a1120 msgr2=0x7f6b600a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.750+0000 7f6b82400700 1 --2- 172.21.15.116:0/421898013 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6b600a1120 0x7f6b600a35e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6b74007860 tx=0x7f6b7400b330 comp rx=0 tx=0).stop 2024-03-30T01:00:08.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.750+0000 7f6b82400700 1 -- 172.21.15.116:0/421898013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b7c101f70 msgr2=0x7f6b7c0731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:08.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.750+0000 7f6b82400700 1 --2- 172.21.15.116:0/421898013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b7c101f70 0x7f6b7c0731a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800e9f0 tx=0x7f6b6800edb0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.750+0000 7f6b82400700 1 -- 172.21.15.116:0/421898013 shutdown_connections 2024-03-30T01:00:08.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.750+0000 7f6b82400700 1 --2- 172.21.15.116:0/421898013 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f6b600a1120 0x7f6b600a35e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.750+0000 7f6b82400700 1 --2- 172.21.15.116:0/421898013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c102910 0x7f6b7c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.750+0000 7f6b82400700 1 --2- 172.21.15.116:0/421898013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b7c100d80 0x7f6b7c074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.751+0000 7f6b82400700 1 --2- 172.21.15.116:0/421898013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b7c101f70 0x7f6b7c0731a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:08.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.751+0000 7f6b82400700 1 -- 172.21.15.116:0/421898013 >> 172.21.15.116:0/421898013 conn(0x7f6b7c0fc2f0 msgr2=0x7f6b7c105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:08.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.751+0000 7f6b82400700 1 -- 172.21.15.116:0/421898013 shutdown_connections 2024-03-30T01:00:08.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:08.751+0000 7f6b82400700 1 -- 172.21.15.116:0/421898013 wait complete. 2024-03-30T01:00:08.777 INFO:tasks.cephfs.filesystem:scrub status for tag:b51b928f-0c53-48f1-8b7e-9d812744668a - {'path': '/', 'tag': 'b51b928f-0c53-48f1-8b7e-9d812744668a', 'options': 'recursive,force'} 2024-03-30T01:00:08.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:08 smithi130 ceph-mon[47626]: pgmap v359: 129 pgs: 129 active+clean; 178 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 9.0 MiB/s wr, 579 op/s 2024-03-30T01:00:08.959 INFO:tasks.workunit.client.0.smithi116.stdout: 52 321152 462 244578 372 142785 739 2024-03-30T01:00:09.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:09 smithi116 ceph-mon[38086]: scrub queued for path: / 2024-03-30T01:00:09.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:09 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:09.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:09 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:09.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:09 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:09.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:09 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:09.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:09 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:09.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:09 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:09.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:09 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:09.679 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:09 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:09.679 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:09 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:09.679 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:09 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:09.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:09 smithi181 ceph-mon[47622]: scrub queued for path: / 2024-03-30T01:00:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:09 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:09 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:09 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:09 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:09 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:09 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:09 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:09 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:09 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:09.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:09 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:09.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:09 smithi130 ceph-mon[47626]: scrub queued for path: / 2024-03-30T01:00:09.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:09 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:09.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:09 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:09.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:09 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:09.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:09 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:09.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:09 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:09.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:09 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:09.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:09 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:09.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:09 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:09.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:09 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:09.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:09 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:10.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:10 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:10 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:10.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:10 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:10 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:10.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:10 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:10 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:10.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:10 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:10 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:10 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:10.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:10 smithi116 ceph-mon[38086]: pgmap v360: 129 pgs: 129 active+clean; 198 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 11 MiB/s wr, 706 op/s 2024-03-30T01:00:10.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:10 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:10 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:10 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:10 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:10 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:10 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:10 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:10 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:10 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:10 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:10 smithi181 ceph-mon[47622]: pgmap v360: 129 pgs: 129 active+clean; 198 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 11 MiB/s wr, 706 op/s 2024-03-30T01:00:10.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:10 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:10 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:10 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:10.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:10 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:10 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:10.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:10 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:10 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:10.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:10 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:10 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:10.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:10 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:10.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:10 smithi130 ceph-mon[47626]: pgmap v360: 129 pgs: 129 active+clean; 198 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 11 MiB/s wr, 706 op/s 2024-03-30T01:00:10.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:10 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:11.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:11 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:11.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:11 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.971 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.971 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.971 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.971 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.971 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:11.971 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:11.971 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:11 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: pgmap v361: 129 pgs: 129 active+clean; 231 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 13 MiB/s wr, 847 op/s 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.787 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.787 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.787 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.787 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.787 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.787 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.787 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.787 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.787 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.787 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.787 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.787 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.788 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.789 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.790 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:12.791 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:12 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: pgmap v361: 129 pgs: 129 active+clean; 231 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 13 MiB/s wr, 847 op/s 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.879 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.880 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.880 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.880 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.880 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.880 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.880 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.880 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.880 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.880 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.880 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.881 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.881 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.881 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.881 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.881 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.881 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.881 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.881 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.881 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.881 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.882 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.883 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.884 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.885 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.885 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.885 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.885 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.885 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.885 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.885 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.885 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:12.885 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:12 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: pgmap v361: 129 pgs: 129 active+clean; 231 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 13 MiB/s wr, 847 op/s 2024-03-30T01:00:12.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.964 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:12.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:12.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:12 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.757 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.758 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.759 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.760 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.761 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.763 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.764 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.766 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.767 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.768 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.769 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.770 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.771 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.772 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.775 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.776 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.777 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.778 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.780 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.781 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.782 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.783 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.784 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:00:13.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.786 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:13 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.864 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.866 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.867 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.868 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.872 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.873 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.876 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.877 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:00:13.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.878 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:13 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.980 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.980 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.980 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.980 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.980 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.984 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.984 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.984 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.984 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.984 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.984 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.984 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.984 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.984 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.984 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.984 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.984 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.987 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.987 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.987 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.987 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.987 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.987 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.987 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.987 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.987 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.987 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.987 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.987 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.988 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.988 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.988 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.988 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.988 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.988 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.988 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.988 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.988 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.988 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.988 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.990 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.990 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.990 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.990 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.990 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.990 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.990 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.990 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.990 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.990 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.996 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.996 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.996 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.996 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.996 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.996 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.996 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.996 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.996 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.996 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:13.996 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:13.996 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.018 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.018 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.018 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.018 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.018 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.029 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:00:14.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:13 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:00:14.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:14 smithi181 ceph-mon[47622]: pgmap v362: 129 pgs: 129 active+clean; 242 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 13 MiB/s wr, 860 op/s 2024-03-30T01:00:14.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:14 smithi130 ceph-mon[47626]: pgmap v362: 129 pgs: 129 active+clean; 242 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 13 MiB/s wr, 860 op/s 2024-03-30T01:00:14.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:14 smithi116 ceph-mon[38086]: pgmap v362: 129 pgs: 129 active+clean; 242 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 13 MiB/s wr, 860 op/s 2024-03-30T01:00:16.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:16 smithi181 ceph-mon[47622]: pgmap v363: 129 pgs: 129 active+clean; 241 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 11 MiB/s wr, 787 op/s 2024-03-30T01:00:16.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:16 smithi130 ceph-mon[47626]: pgmap v363: 129 pgs: 129 active+clean; 241 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 11 MiB/s wr, 787 op/s 2024-03-30T01:00:16.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:16 smithi116 ceph-mon[38086]: pgmap v363: 129 pgs: 129 active+clean; 241 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 11 MiB/s wr, 787 op/s 2024-03-30T01:00:18.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:18 smithi181 ceph-mon[47622]: pgmap v364: 129 pgs: 129 active+clean; 244 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 9.4 MiB/s wr, 665 op/s 2024-03-30T01:00:18.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:18 smithi130 ceph-mon[47626]: pgmap v364: 129 pgs: 129 active+clean; 244 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 9.4 MiB/s wr, 665 op/s 2024-03-30T01:00:18.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:18 smithi116 ceph-mon[38086]: pgmap v364: 129 pgs: 129 active+clean; 244 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 9.4 MiB/s wr, 665 op/s 2024-03-30T01:00:18.959 INFO:tasks.workunit.client.0.smithi116.stdout: 68 443282 829 332271 776 180409 452 2024-03-30T01:00:20.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:20 smithi181 ceph-mon[47622]: pgmap v365: 129 pgs: 129 active+clean; 247 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.6 MiB/s wr, 498 op/s 2024-03-30T01:00:20.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:20 smithi130 ceph-mon[47626]: pgmap v365: 129 pgs: 129 active+clean; 247 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.6 MiB/s wr, 498 op/s 2024-03-30T01:00:21.198 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:20 smithi116 ceph-mon[38086]: pgmap v365: 129 pgs: 129 active+clean; 247 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.6 MiB/s wr, 498 op/s 2024-03-30T01:00:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:22 smithi181 ceph-mon[47622]: pgmap v366: 129 pgs: 129 active+clean; 253 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.2 MiB/s wr, 393 op/s 2024-03-30T01:00:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:22 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:00:22.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:22 smithi130 ceph-mon[47626]: pgmap v366: 129 pgs: 129 active+clean; 253 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.2 MiB/s wr, 393 op/s 2024-03-30T01:00:22.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:22 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:00:22.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:22 smithi116 ceph-mon[38086]: pgmap v366: 129 pgs: 129 active+clean; 253 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.2 MiB/s wr, 393 op/s 2024-03-30T01:00:22.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:22 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:00:23.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:23 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:00:23.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:23 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:00:23.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:23 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:00:23.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:23 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:00:23.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:23 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:00:23.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:23 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:00:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:24 smithi181 ceph-mon[47622]: pgmap v367: 129 pgs: 129 active+clean; 253 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 250 op/s 2024-03-30T01:00:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:24 smithi130 ceph-mon[47626]: pgmap v367: 129 pgs: 129 active+clean; 253 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 250 op/s 2024-03-30T01:00:24.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:24 smithi116 ceph-mon[38086]: pgmap v367: 129 pgs: 129 active+clean; 253 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 250 op/s 2024-03-30T01:00:26.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:26 smithi181 ceph-mon[47622]: pgmap v368: 129 pgs: 129 active+clean; 256 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-03-30T01:00:26.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:26 smithi130 ceph-mon[47626]: pgmap v368: 129 pgs: 129 active+clean; 256 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-03-30T01:00:26.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:26 smithi116 ceph-mon[38086]: pgmap v368: 129 pgs: 129 active+clean; 256 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-03-30T01:00:28.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:28 smithi181 ceph-mon[47622]: pgmap v369: 129 pgs: 129 active+clean; 261 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 210 op/s 2024-03-30T01:00:28.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:28 smithi130 ceph-mon[47626]: pgmap v369: 129 pgs: 129 active+clean; 261 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 210 op/s 2024-03-30T01:00:28.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:28 smithi116 ceph-mon[38086]: pgmap v369: 129 pgs: 129 active+clean; 261 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.1 MiB/s wr, 210 op/s 2024-03-30T01:00:28.959 INFO:tasks.workunit.client.0.smithi116.stdout: 87 565705 1097 384773 802 187566 1072 2024-03-30T01:00:30.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:30 smithi181 ceph-mon[47622]: pgmap v370: 129 pgs: 129 active+clean; 277 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 260 op/s 2024-03-30T01:00:30.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:30 smithi130 ceph-mon[47626]: pgmap v370: 129 pgs: 129 active+clean; 277 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 260 op/s 2024-03-30T01:00:30.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:30 smithi116 ceph-mon[38086]: pgmap v370: 129 pgs: 129 active+clean; 277 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 260 op/s 2024-03-30T01:00:32.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:32 smithi130 ceph-mon[47626]: pgmap v371: 129 pgs: 129 active+clean; 308 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.6 MiB/s wr, 352 op/s 2024-03-30T01:00:32.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:32 smithi116 ceph-mon[38086]: pgmap v371: 129 pgs: 129 active+clean; 308 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.6 MiB/s wr, 352 op/s 2024-03-30T01:00:33.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:32 smithi181 ceph-mon[47622]: pgmap v371: 129 pgs: 129 active+clean; 308 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.6 MiB/s wr, 352 op/s 2024-03-30T01:00:34.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:34 smithi130 ceph-mon[47626]: pgmap v372: 129 pgs: 129 active+clean; 329 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.0 MiB/s wr, 426 op/s 2024-03-30T01:00:34.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:34 smithi116 ceph-mon[38086]: pgmap v372: 129 pgs: 129 active+clean; 329 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.0 MiB/s wr, 426 op/s 2024-03-30T01:00:35.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:34 smithi181 ceph-mon[47622]: pgmap v372: 129 pgs: 129 active+clean; 329 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.0 MiB/s wr, 426 op/s 2024-03-30T01:00:36.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:36 smithi130 ceph-mon[47626]: pgmap v373: 129 pgs: 129 active+clean; 354 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.2 MiB/s wr, 528 op/s 2024-03-30T01:00:36.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:36 smithi116 ceph-mon[38086]: pgmap v373: 129 pgs: 129 active+clean; 354 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.2 MiB/s wr, 528 op/s 2024-03-30T01:00:37.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:36 smithi181 ceph-mon[47622]: pgmap v373: 129 pgs: 129 active+clean; 354 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 9.2 MiB/s wr, 528 op/s 2024-03-30T01:00:38.777 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T01:00:38.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:38 smithi130 ceph-mon[47626]: pgmap v374: 129 pgs: 129 active+clean; 384 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 12 MiB/s wr, 653 op/s 2024-03-30T01:00:38.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:38 smithi116 ceph-mon[38086]: pgmap v374: 129 pgs: 129 active+clean; 384 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 12 MiB/s wr, 653 op/s 2024-03-30T01:00:38.966 INFO:tasks.workunit.client.0.smithi116.stdout: 102 391381 1112 264732 898 131750 936 2024-03-30T01:00:39.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:38 smithi181 ceph-mon[47622]: pgmap v374: 129 pgs: 129 active+clean; 384 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 12 MiB/s wr, 653 op/s 2024-03-30T01:00:40.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:40 smithi130 ceph-mon[47626]: pgmap v375: 129 pgs: 129 active+clean; 385 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 11 MiB/s wr, 653 op/s 2024-03-30T01:00:40.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:40 smithi116 ceph-mon[38086]: pgmap v375: 129 pgs: 129 active+clean; 385 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 11 MiB/s wr, 653 op/s 2024-03-30T01:00:41.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:40 smithi181 ceph-mon[47622]: pgmap v375: 129 pgs: 129 active+clean; 385 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 11 MiB/s wr, 653 op/s 2024-03-30T01:00:42.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.086+0000 7f944fc00700 1 -- 172.21.15.116:0/1718440934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9448100d80 msgr2=0x7f94481011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:42.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.086+0000 7f944fc00700 1 --2- 172.21.15.116:0/1718440934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9448100d80 0x7f94481011a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f943c0099c0 tx=0x7f943c009cd0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.086+0000 7f944fc00700 1 -- 172.21.15.116:0/1718440934 shutdown_connections 2024-03-30T01:00:42.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.086+0000 7f944fc00700 1 --2- 172.21.15.116:0/1718440934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9448102910 0x7f94481091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.086+0000 7f944fc00700 1 --2- 172.21.15.116:0/1718440934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9448100d80 0x7f94481011a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.086+0000 7f944fc00700 1 --2- 172.21.15.116:0/1718440934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9448101f70 0x7f94481023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.086+0000 7f944fc00700 1 -- 172.21.15.116:0/1718440934 >> 172.21.15.116:0/1718440934 conn(0x7f94480fc2f0 msgr2=0x7f94480fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:42.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.086+0000 7f944fc00700 1 -- 172.21.15.116:0/1718440934 shutdown_connections 2024-03-30T01:00:42.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.086+0000 7f944fc00700 1 -- 172.21.15.116:0/1718440934 wait complete. 2024-03-30T01:00:42.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.087+0000 7f944fc00700 1 Processor -- start 2024-03-30T01:00:42.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.088+0000 7f944fc00700 1 -- start start 2024-03-30T01:00:42.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.088+0000 7f944fc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9448100d80 0x7f9448196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.088+0000 7f944fc00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9448101f70 0x7f94481972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.088+0000 7f944fc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9448102910 0x7f944819e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.088+0000 7f944fc00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944819e870 con 0x7f9448102910 2024-03-30T01:00:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.088+0000 7f944fc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944819e9e0 con 0x7f9448100d80 2024-03-30T01:00:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.088+0000 7f944fc00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944819eba0 con 0x7f9448101f70 2024-03-30T01:00:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.088+0000 7f944dc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9448102910 0x7f944819e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.088+0000 7f9447e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9448101f70 0x7f94481972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.088+0000 7f944d200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9448100d80 0x7f9448196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.088+0000 7f944dc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9448102910 0x7f944819e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46290/0 (socket says 172.21.15.116:46290) 2024-03-30T01:00:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.088+0000 7f9447e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9448101f70 0x7f94481972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:48726/0 (socket says 172.21.15.116:48726) 2024-03-30T01:00:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.088+0000 7f944dc00700 1 -- 172.21.15.116:0/3882126312 learned_addr learned my addr 172.21.15.116:0/3882126312 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.089+0000 7f944dc00700 1 -- 172.21.15.116:0/3882126312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9448101f70 msgr2=0x7f94481972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.089+0000 7f944dc00700 1 --2- 172.21.15.116:0/3882126312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9448101f70 0x7f94481972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.089+0000 7f944dc00700 1 -- 172.21.15.116:0/3882126312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9448100d80 msgr2=0x7f9448196d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.089+0000 7f944dc00700 1 --2- 172.21.15.116:0/3882126312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9448100d80 0x7f9448196d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.089+0000 7f944dc00700 1 -- 172.21.15.116:0/3882126312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c0095f0 con 0x7f9448102910 2024-03-30T01:00:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.089+0000 7f9447e00700 1 --2- 172.21.15.116:0/3882126312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9448101f70 0x7f94481972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:00:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.089+0000 7f944dc00700 1 --2- 172.21.15.116:0/3882126312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9448102910 0x7f944819e2a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f944000ef30 tx=0x7f944000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.089+0000 7f9445600700 1 -- 172.21.15.116:0/3882126312 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9440018070 con 0x7f9448102910 2024-03-30T01:00:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.089+0000 7f944fc00700 1 -- 172.21.15.116:0/3882126312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9448072fe0 con 0x7f9448102910 2024-03-30T01:00:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.089+0000 7f9445600700 1 -- 172.21.15.116:0/3882126312 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f944000f040 con 0x7f9448102910 2024-03-30T01:00:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.089+0000 7f9445600700 1 -- 172.21.15.116:0/3882126312 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944000cd00 con 0x7f9448102910 2024-03-30T01:00:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.089+0000 7f944fc00700 1 -- 172.21.15.116:0/3882126312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9448073510 con 0x7f9448102910 2024-03-30T01:00:42.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.092+0000 7f9445600700 1 -- 172.21.15.116:0/3882126312 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f9440003a00 con 0x7f9448102910 2024-03-30T01:00:42.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f9463798b80 1 Processor -- start 2024-03-30T01:00:42.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f9445600700 1 --2- 172.21.15.116:0/3882126312 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f94300a1120 0x7f94300a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f9463798b80 1 -- start start 2024-03-30T01:00:42.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f9445600700 1 -- 172.21.15.116:0/3882126312 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f94400cc770 con 0x7f9448102910 2024-03-30T01:00:42.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f9447e00700 1 --2- 172.21.15.116:0/3882126312 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f94300a1120 0x7f94300a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f9463798b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb37d0 0x55ec3ceb3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f9463798b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 0x55ec3ceb49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f9463798b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb4fd0 0x55ec3ceb5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f9463798b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec3ce0b990 con 0x55ec3ceb37d0 2024-03-30T01:00:42.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f9463798b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec3ce09350 con 0x55ec3ceb4130 2024-03-30T01:00:42.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f9463798b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec3ce04c40 con 0x55ec3ceb4fd0 2024-03-30T01:00:42.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f944d200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb37d0 0x55ec3ceb3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f944dc00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb4fd0 0x55ec3ceb5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f944dc00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb4fd0 0x55ec3ceb5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:48740/0 (socket says 172.21.15.116:48740) 2024-03-30T01:00:42.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.094+0000 7f944dc00700 1 -- 172.21.15.116:0/1008507283 learned_addr learned my addr 172.21.15.116:0/1008507283 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:42.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.095+0000 7f9447e00700 1 --2- 172.21.15.116:0/1008507283 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 0x55ec3ceb49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.095+0000 7f944dc00700 1 -- 172.21.15.116:0/1008507283 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 msgr2=0x55ec3ceb49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:42.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.095+0000 7f944dc00700 1 --2- 172.21.15.116:0/1008507283 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 0x55ec3ceb49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.095+0000 7f944dc00700 1 -- 172.21.15.116:0/1008507283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb37d0 msgr2=0x55ec3ceb3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:42.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.095+0000 7f944dc00700 1 --2- 172.21.15.116:0/1008507283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb37d0 0x55ec3ceb3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.095+0000 7f944dc00700 1 -- 172.21.15.116:0/1008507283 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec3cc863b0 con 0x55ec3ceb4fd0 2024-03-30T01:00:42.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.095+0000 7f9447e00700 1 --2- 172.21.15.116:0/1008507283 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 0x55ec3ceb49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:42.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.095+0000 7f944d200700 1 --2- 172.21.15.116:0/1008507283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb37d0 0x55ec3ceb3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:00:42.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.095+0000 7f944dc00700 1 --2- 172.21.15.116:0/1008507283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb4fd0 0x55ec3ceb5450 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9440009c10 tx=0x7f94400994a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4314aa90f742cc52 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:42.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.095+0000 7f9447e00700 1 --2- 172.21.15.116:0/3882126312 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f94300a1120 0x7f94300a35e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f94340098b0 tx=0x7f9434009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:42.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.095+0000 7f943a000700 1 -- 172.21.15.116:0/1008507283 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9440084040 con 0x55ec3ceb4fd0 2024-03-30T01:00:42.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.095+0000 7f943a000700 1 -- 172.21.15.116:0/1008507283 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f944001b320 con 0x55ec3ceb4fd0 2024-03-30T01:00:42.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.095+0000 7f943a000700 1 -- 172.21.15.116:0/1008507283 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94400a1cd0 con 0x55ec3ceb4fd0 2024-03-30T01:00:42.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.096+0000 7f9463798b80 1 -- 172.21.15.116:0/1008507283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb4fd0 msgr2=0x55ec3ceb5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:42.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.096+0000 7f9463798b80 1 --2- 172.21.15.116:0/1008507283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb4fd0 0x55ec3ceb5450 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9440009c10 tx=0x7f94400994a0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.096+0000 7f9463798b80 1 -- 172.21.15.116:0/1008507283 shutdown_connections 2024-03-30T01:00:42.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.096+0000 7f9463798b80 1 --2- 172.21.15.116:0/1008507283 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb4fd0 0x55ec3ceb5450 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.096+0000 7f9463798b80 1 --2- 172.21.15.116:0/1008507283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb37d0 0x55ec3ceb3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.096+0000 7f9463798b80 1 --2- 172.21.15.116:0/1008507283 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 0x55ec3ceb49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.096+0000 7f9463798b80 1 -- 172.21.15.116:0/1008507283 >> 172.21.15.116:0/1008507283 conn(0x55ec3ce126a0 msgr2=0x55ec3ceae9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:42.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.096+0000 7f9463798b80 1 -- 172.21.15.116:0/1008507283 shutdown_connections 2024-03-30T01:00:42.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.096+0000 7f9463798b80 1 -- 172.21.15.116:0/1008507283 wait complete. 2024-03-30T01:00:42.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.097+0000 7f9463798b80 1 -- start start 2024-03-30T01:00:42.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.097+0000 7f9463798b80 1 Processor -- start 2024-03-30T01:00:42.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.097+0000 7f9463798b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb37d0 0x55ec3cebd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.097+0000 7f9463798b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 0x55ec3cebdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.097+0000 7f9463798b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 0x55ec3cec4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.097+0000 7f9463798b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec3cda0d80 con 0x55ec3ceb4fd0 2024-03-30T01:00:42.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.097+0000 7f9463798b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec3cda0a20 con 0x55ec3ceb4130 2024-03-30T01:00:42.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.097+0000 7f9463798b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec3cc886a0 con 0x55ec3ceb37d0 2024-03-30T01:00:42.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.097+0000 7f9463798b80 10 client.0 _ll_register_callbacks cb 0x55ec3cdaa470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:42.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.098+0000 7f9447e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 0x55ec3cebdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.098+0000 7f944d200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb37d0 0x55ec3cebd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.098+0000 7f9447e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 0x55ec3cebdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:37666/0 (socket says 172.21.15.116:37666) 2024-03-30T01:00:42.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.098+0000 7f944d200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb37d0 0x55ec3cebd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:48744/0 (socket says 172.21.15.116:48744) 2024-03-30T01:00:42.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.098+0000 7f9447e00700 1 -- 172.21.15.116:0/2465286858 learned_addr learned my addr 172.21.15.116:0/2465286858 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:42.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.098+0000 7f944dc00700 1 --2- 172.21.15.116:0/2465286858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 0x55ec3cec4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.098+0000 7f944d200700 1 -- 172.21.15.116:0/2465286858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 msgr2=0x55ec3cebdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:42.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.098+0000 7f944d200700 1 --2- 172.21.15.116:0/2465286858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 0x55ec3cebdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.098+0000 7f944d200700 1 -- 172.21.15.116:0/2465286858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 msgr2=0x55ec3cec4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:42.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.098+0000 7f944d200700 1 --2- 172.21.15.116:0/2465286858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 0x55ec3cec4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.098+0000 7f944d200700 1 -- 172.21.15.116:0/2465286858 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9448073510 con 0x55ec3ceb37d0 2024-03-30T01:00:42.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.098+0000 7f944d200700 1 --2- 172.21.15.116:0/2465286858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb37d0 0x55ec3cebd6e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f943c0098e0 tx=0x7f943c00bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc69b0aa8bf825d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:42.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.098+0000 7f942ea00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:00:42.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.098+0000 7f942ea00700 1 -- 172.21.15.116:0/2465286858 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943c003be0 con 0x55ec3ceb37d0 2024-03-30T01:00:42.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.099+0000 7f9463798b80 1 -- 172.21.15.116:0/2465286858 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec3cdc5380 con 0x55ec3ceb37d0 2024-03-30T01:00:42.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.099+0000 7f942ea00700 1 -- 172.21.15.116:0/2465286858 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f943c003d40 con 0x55ec3ceb37d0 2024-03-30T01:00:42.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.099+0000 7f942ea00700 1 -- 172.21.15.116:0/2465286858 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f943c005040 con 0x55ec3ceb37d0 2024-03-30T01:00:42.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.099+0000 7f9463798b80 10 client.25960 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:42.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.099+0000 7f9463798b80 1 -- 172.21.15.116:0/2465286858 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ec3cc32b60 con 0x55ec3ceb37d0 2024-03-30T01:00:42.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.099+0000 7f942ea00700 1 -- 172.21.15.116:0/2465286858 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ec3cdc5380 con 0x55ec3ceb37d0 2024-03-30T01:00:42.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.099+0000 7f942ea00700 1 -- 172.21.15.116:0/2465286858 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f943c00f980 con 0x55ec3ceb37d0 2024-03-30T01:00:42.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.099+0000 7f9463798b80 10 client.25960 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:42.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.099+0000 7f9463798b80 10 client.25960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:00:42.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.099+0000 7f9463798b80 20 client.25960 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:42.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.099+0000 7f9463798b80 1 --2- 172.21.15.116:0/2465286858 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ec3cec9380 0x55ec3cecb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.099+0000 7f9463798b80 4 client.25960 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:00:42.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.099+0000 7f9463798b80 1 -- 172.21.15.116:0/2465286858 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec3cdfd700 con 0x55ec3cec9380 2024-03-30T01:00:42.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.100+0000 7f9447e00700 1 --2- 172.21.15.116:0/2465286858 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ec3cec9380 0x55ec3cecb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.100+0000 7f9447e00700 1 --2- 172.21.15.116:0/2465286858 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ec3cec9380 0x55ec3cecb7a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=585ee6c21b87689 server_cookie=ec8f346fa93dfb84 in_seq=0 out_seq=0 2024-03-30T01:00:42.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.100+0000 7f942ea00700 10 client.25960 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:42.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.104+0000 7f942ea00700 1 -- 172.21.15.116:0/2465286858 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ec3cdfd700 con 0x55ec3cec9380 2024-03-30T01:00:42.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.104+0000 7f942ea00700 10 client.25960 handle_command_reply: tid=0 2024-03-30T01:00:42.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.104+0000 7f9463798b80 1 client.25960 shutdown 2024-03-30T01:00:42.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.104+0000 7f9463798b80 1 -- 172.21.15.116:0/2465286858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb37d0 msgr2=0x55ec3cebd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:42.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.104+0000 7f9463798b80 1 --2- 172.21.15.116:0/2465286858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb37d0 0x55ec3cebd6e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f943c0098e0 tx=0x7f943c00bec0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.104+0000 7f9463798b80 1 -- 172.21.15.116:0/2465286858 shutdown_connections 2024-03-30T01:00:42.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.104+0000 7f9463798b80 1 --2- 172.21.15.116:0/2465286858 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ec3cec9380 0x55ec3cecb7a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.104+0000 7f9463798b80 1 --2- 172.21.15.116:0/2465286858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb37d0 0x55ec3cebd6e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.104+0000 7f9463798b80 1 --2- 172.21.15.116:0/2465286858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 0x55ec3cec4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.104+0000 7f942ea00700 0 client.25960 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:42.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.104+0000 7f9463798b80 1 --2- 172.21.15.116:0/2465286858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 0x55ec3cebdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.104+0000 7f9463798b80 1 -- 172.21.15.116:0/2465286858 >> 172.21.15.116:0/2465286858 conn(0x55ec3ce126a0 msgr2=0x55ec3ceb2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:42.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.104+0000 7f9463798b80 1 -- 172.21.15.116:0/2465286858 shutdown_connections 2024-03-30T01:00:42.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.105+0000 7f9463798b80 1 -- 172.21.15.116:0/2465286858 wait complete. 2024-03-30T01:00:42.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.105+0000 7f9463798b80 20 client.25960 trim_cache size 0 max 16384 2024-03-30T01:00:42.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.137+0000 7f9463798b80 1 Processor -- start 2024-03-30T01:00:42.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.137+0000 7f9463798b80 1 -- start start 2024-03-30T01:00:42.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.138+0000 7f9463798b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb37d0 0x55ec3cebcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.138+0000 7f9463798b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb4130 0x55ec3cec4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.138+0000 7f9463798b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 0x55ec3cebd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.138+0000 7f9463798b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec3cec5dc0 con 0x55ec3ceb4fd0 2024-03-30T01:00:42.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.138+0000 7f9463798b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec3ceaf660 con 0x55ec3ceb37d0 2024-03-30T01:00:42.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.138+0000 7f9463798b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec3ceaf8a0 con 0x55ec3ceb4130 2024-03-30T01:00:42.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.138+0000 7f944d200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb37d0 0x55ec3cebcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.138+0000 7f9447e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb4130 0x55ec3cec4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.138+0000 7f9447e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb4130 0x55ec3cec4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:48758/0 (socket says 172.21.15.116:48758) 2024-03-30T01:00:42.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.138+0000 7f9447e00700 1 -- 172.21.15.116:0/661835769 learned_addr learned my addr 172.21.15.116:0/661835769 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:42.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.138+0000 7f944dc00700 1 --2- 172.21.15.116:0/661835769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 0x55ec3cebd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.138+0000 7f9447e00700 1 -- 172.21.15.116:0/661835769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb37d0 msgr2=0x55ec3cebcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:42.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.138+0000 7f944d200700 1 --2- 172.21.15.116:0/661835769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb37d0 0x55ec3cebcbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:37676/0 (socket says 172.21.15.116:37676) 2024-03-30T01:00:42.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.139+0000 7f9447e00700 1 --2- 172.21.15.116:0/661835769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb37d0 0x55ec3cebcbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.139+0000 7f9447e00700 1 -- 172.21.15.116:0/661835769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 msgr2=0x55ec3cebd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:42.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.139+0000 7f9447e00700 1 --2- 172.21.15.116:0/661835769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 0x55ec3cebd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.139+0000 7f9447e00700 1 -- 172.21.15.116:0/661835769 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec3cdcaea0 con 0x55ec3ceb4130 2024-03-30T01:00:42.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.139+0000 7f944dc00700 1 --2- 172.21.15.116:0/661835769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 0x55ec3cebd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:00:42.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.139+0000 7f944d200700 1 --2- 172.21.15.116:0/661835769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb37d0 0x55ec3cebcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:42.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.139+0000 7f9447e00700 1 --2- 172.21.15.116:0/661835769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb4130 0x55ec3cec4bd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f943402dde0 tx=0x7f9434018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=136b41a2ff97b7b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:42.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.139+0000 7f943aa00700 1 -- 172.21.15.116:0/661835769 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9434003c90 con 0x55ec3ceb4130 2024-03-30T01:00:42.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.139+0000 7f943aa00700 1 -- 172.21.15.116:0/661835769 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9434023040 con 0x55ec3ceb4130 2024-03-30T01:00:42.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.139+0000 7f943aa00700 1 -- 172.21.15.116:0/661835769 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943402a800 con 0x55ec3ceb4130 2024-03-30T01:00:42.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.140+0000 7f9463798b80 1 -- 172.21.15.116:0/661835769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb4130 msgr2=0x55ec3cec4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:42.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.140+0000 7f9463798b80 1 --2- 172.21.15.116:0/661835769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb4130 0x55ec3cec4bd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f943402dde0 tx=0x7f9434018000 comp rx=0 tx=0).stop 2024-03-30T01:00:42.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.140+0000 7f9463798b80 1 -- 172.21.15.116:0/661835769 shutdown_connections 2024-03-30T01:00:42.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.140+0000 7f9463798b80 1 --2- 172.21.15.116:0/661835769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 0x55ec3cebd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.140+0000 7f9463798b80 1 --2- 172.21.15.116:0/661835769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb37d0 0x55ec3cebcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.140+0000 7f9463798b80 1 --2- 172.21.15.116:0/661835769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb4130 0x55ec3cec4bd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.140+0000 7f9463798b80 1 -- 172.21.15.116:0/661835769 >> 172.21.15.116:0/661835769 conn(0x55ec3ce126a0 msgr2=0x55ec3cea75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:42.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.140+0000 7f9463798b80 1 -- 172.21.15.116:0/661835769 shutdown_connections 2024-03-30T01:00:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.140+0000 7f9463798b80 1 -- 172.21.15.116:0/661835769 wait complete. 2024-03-30T01:00:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.141+0000 7f9463798b80 1 -- start start 2024-03-30T01:00:42.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.141+0000 7f9463798b80 1 Processor -- start 2024-03-30T01:00:42.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.142+0000 7f9463798b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb37d0 0x55ec3cec4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.142+0000 7f9463798b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 0x55ec3cec2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.142+0000 7f9463798b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 0x55ec3cec34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.142+0000 7f9463798b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec3cec3a70 con 0x55ec3ceb4fd0 2024-03-30T01:00:42.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.142+0000 7f9463798b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec3cec3be0 con 0x55ec3ceb4130 2024-03-30T01:00:42.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.142+0000 7f9463798b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec3cec3da0 con 0x55ec3ceb37d0 2024-03-30T01:00:42.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.142+0000 7f9463798b80 10 client.0 _ll_register_callbacks cb 0x55ec3cdaa470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:42.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.142+0000 7f944d200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb37d0 0x55ec3cec4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.142+0000 7f9447e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 0x55ec3cec2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.142+0000 7f944d200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb37d0 0x55ec3cec4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:48772/0 (socket says 172.21.15.116:48772) 2024-03-30T01:00:42.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.142+0000 7f944d200700 1 -- 172.21.15.116:0/2595505910 learned_addr learned my addr 172.21.15.116:0/2595505910 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:42.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.143+0000 7f944dc00700 1 --2- 172.21.15.116:0/2595505910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 0x55ec3cec34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.143+0000 7f944d200700 1 -- 172.21.15.116:0/2595505910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 msgr2=0x55ec3cec2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:42.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.143+0000 7f944d200700 1 --2- 172.21.15.116:0/2595505910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 0x55ec3cec2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.143+0000 7f944d200700 1 -- 172.21.15.116:0/2595505910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 msgr2=0x55ec3cec34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:42.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.143+0000 7f944d200700 1 --2- 172.21.15.116:0/2595505910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 0x55ec3cec34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.143+0000 7f944d200700 1 -- 172.21.15.116:0/2595505910 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec3cc863b0 con 0x55ec3ceb37d0 2024-03-30T01:00:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.143+0000 7f944dc00700 1 --2- 172.21.15.116:0/2595505910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 0x55ec3cec34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.143+0000 7f944d200700 1 --2- 172.21.15.116:0/2595505910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb37d0 0x55ec3cec4970 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f943c00a010 tx=0x7f943c003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e6c8c87374b0b10 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.143+0000 7f942ea00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:00:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.143+0000 7f942ea00700 1 -- 172.21.15.116:0/2595505910 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943c00f670 con 0x55ec3ceb37d0 2024-03-30T01:00:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.143+0000 7f942ea00700 1 -- 172.21.15.116:0/2595505910 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f943c00f7d0 con 0x55ec3ceb37d0 2024-03-30T01:00:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.145+0000 7f9463798b80 1 -- 172.21.15.116:0/2595505910 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f944001b320 con 0x55ec3ceb37d0 2024-03-30T01:00:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.146+0000 7f942ea00700 1 -- 172.21.15.116:0/2595505910 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f943c00f990 con 0x55ec3ceb37d0 2024-03-30T01:00:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.146+0000 7f9463798b80 10 client.25966 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.146+0000 7f9463798b80 1 -- 172.21.15.116:0/2595505910 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f943402dae0 con 0x55ec3ceb37d0 2024-03-30T01:00:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.146+0000 7f942ea00700 1 -- 172.21.15.116:0/2595505910 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f944001b320 con 0x55ec3ceb37d0 2024-03-30T01:00:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.146+0000 7f942ea00700 1 -- 172.21.15.116:0/2595505910 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f943c023900 con 0x55ec3ceb37d0 2024-03-30T01:00:42.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.146+0000 7f9463798b80 10 client.25966 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:42.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.146+0000 7f9463798b80 10 client.25966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:00:42.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.146+0000 7f9463798b80 20 client.25966 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:42.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.146+0000 7f9463798b80 1 --2- 172.21.15.116:0/2595505910 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ec3cf26dd0 0x55ec3cf291f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.146+0000 7f9463798b80 4 client.25966 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T01:00:42.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.146+0000 7f9463798b80 1 -- 172.21.15.116:0/2595505910 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ec3cc53cf0 con 0x55ec3cf26dd0 2024-03-30T01:00:42.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.147+0000 7f9447e00700 1 --2- 172.21.15.116:0/2595505910 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ec3cf26dd0 0x55ec3cf291f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.147+0000 7f9447e00700 1 --2- 172.21.15.116:0/2595505910 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ec3cf26dd0 0x55ec3cf291f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71c6034cb6eed68f server_cookie=cf03eeadda496d22 in_seq=0 out_seq=0 2024-03-30T01:00:42.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.147+0000 7f942ea00700 10 client.25966 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:42.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.150+0000 7f942ea00700 1 -- 172.21.15.116:0/2595505910 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ec3cc53cf0 con 0x55ec3cf26dd0 2024-03-30T01:00:42.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.150+0000 7f942ea00700 10 client.25966 handle_command_reply: tid=0 2024-03-30T01:00:42.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.150+0000 7f9463798b80 1 client.25966 shutdown 2024-03-30T01:00:42.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.151+0000 7f9463798b80 1 -- 172.21.15.116:0/2595505910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb37d0 msgr2=0x55ec3cec4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:42.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.151+0000 7f9463798b80 1 --2- 172.21.15.116:0/2595505910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb37d0 0x55ec3cec4970 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f943c00a010 tx=0x7f943c003eb0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.151+0000 7f9463798b80 1 -- 172.21.15.116:0/2595505910 shutdown_connections 2024-03-30T01:00:42.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.151+0000 7f9463798b80 1 --2- 172.21.15.116:0/2595505910 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55ec3cf26dd0 0x55ec3cf291f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.151+0000 7f9463798b80 1 --2- 172.21.15.116:0/2595505910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ec3ceb37d0 0x55ec3cec4970 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.151+0000 7f942ea00700 0 client.25966 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:42.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.151+0000 7f9463798b80 1 --2- 172.21.15.116:0/2595505910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3ceb4fd0 0x55ec3cec34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.151+0000 7f9463798b80 1 --2- 172.21.15.116:0/2595505910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec3ceb4130 0x55ec3cec2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.151+0000 7f9463798b80 1 -- 172.21.15.116:0/2595505910 >> 172.21.15.116:0/2595505910 conn(0x55ec3ce126a0 msgr2=0x55ec3cebcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:42.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.151+0000 7f9463798b80 1 -- 172.21.15.116:0/2595505910 shutdown_connections 2024-03-30T01:00:42.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.151+0000 7f9463798b80 1 -- 172.21.15.116:0/2595505910 wait complete. 2024-03-30T01:00:42.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.151+0000 7f9463798b80 20 client.25966 trim_cache size 0 max 16384 2024-03-30T01:00:42.162 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:00:42.162 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-30T01:00:42.162 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-30T01:00:42.162 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:00:42.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.152+0000 7f944fc00700 1 -- 172.21.15.116:0/3882126312 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f94300a1120 msgr2=0x7f94300a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:42.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.152+0000 7f944fc00700 1 --2- 172.21.15.116:0/3882126312 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f94300a1120 0x7f94300a35e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f94340098b0 tx=0x7f9434009260 comp rx=0 tx=0).stop 2024-03-30T01:00:42.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.152+0000 7f944fc00700 1 -- 172.21.15.116:0/3882126312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9448102910 msgr2=0x7f944819e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:42.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.152+0000 7f944fc00700 1 --2- 172.21.15.116:0/3882126312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9448102910 0x7f944819e2a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f944000ef30 tx=0x7f944000c3a0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.152+0000 7f944fc00700 1 -- 172.21.15.116:0/3882126312 shutdown_connections 2024-03-30T01:00:42.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.152+0000 7f944fc00700 1 --2- 172.21.15.116:0/3882126312 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f94300a1120 0x7f94300a35e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.152+0000 7f944fc00700 1 --2- 172.21.15.116:0/3882126312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9448102910 0x7f944819e2a0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.152+0000 7f944fc00700 1 --2- 172.21.15.116:0/3882126312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9448100d80 0x7f9448196d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.152+0000 7f944fc00700 1 --2- 172.21.15.116:0/3882126312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9448101f70 0x7f94481972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.152+0000 7f944fc00700 1 -- 172.21.15.116:0/3882126312 >> 172.21.15.116:0/3882126312 conn(0x7f94480fc2f0 msgr2=0x7f9448105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:42.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.152+0000 7f944fc00700 1 -- 172.21.15.116:0/3882126312 shutdown_connections 2024-03-30T01:00:42.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.152+0000 7f944fc00700 1 -- 172.21.15.116:0/3882126312 wait complete. 2024-03-30T01:00:42.184 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T01:00:42.184 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T01:00:42.528 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:42 smithi116 ceph-mon[38086]: pgmap v376: 129 pgs: 129 active+clean; 395 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 11 MiB/s wr, 606 op/s 2024-03-30T01:00:42.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.550+0000 7f7581e00700 1 -- 172.21.15.116:0/2917395556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f757c100c50 msgr2=0x7f757c101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:42.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.550+0000 7f7581e00700 1 --2- 172.21.15.116:0/2917395556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f757c100c50 0x7f757c101070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f75700099c0 tx=0x7f7570009cd0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.550+0000 7f7581e00700 1 -- 172.21.15.116:0/2917395556 shutdown_connections 2024-03-30T01:00:42.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.550+0000 7f7581e00700 1 --2- 172.21.15.116:0/2917395556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f757c1027e0 0x7f757c109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.550+0000 7f7581e00700 1 --2- 172.21.15.116:0/2917395556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f757c100c50 0x7f757c101070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.550+0000 7f7581e00700 1 --2- 172.21.15.116:0/2917395556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f757c101e40 0x7f757c1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.550+0000 7f7581e00700 1 -- 172.21.15.116:0/2917395556 >> 172.21.15.116:0/2917395556 conn(0x7f757c0fc1c0 msgr2=0x7f757c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:42.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.550+0000 7f7581e00700 1 -- 172.21.15.116:0/2917395556 shutdown_connections 2024-03-30T01:00:42.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.550+0000 7f7581e00700 1 -- 172.21.15.116:0/2917395556 wait complete. 2024-03-30T01:00:42.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.551+0000 7f7581e00700 1 Processor -- start 2024-03-30T01:00:42.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.551+0000 7f7581e00700 1 -- start start 2024-03-30T01:00:42.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.551+0000 7f7581e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f757c100c50 0x7f757c194990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.551+0000 7f7581e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f757c101e40 0x7f757c194ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.551+0000 7f7581e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f757c1027e0 0x7f757c19bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.551+0000 7f7581e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c19c4c0 con 0x7f757c101e40 2024-03-30T01:00:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.551+0000 7f7581e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c19c600 con 0x7f757c1027e0 2024-03-30T01:00:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.551+0000 7f7581e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c19c790 con 0x7f757c100c50 2024-03-30T01:00:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.552+0000 7f757be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f757c101e40 0x7f757c194ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.552+0000 7f7581400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f757c1027e0 0x7f757c19bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.552+0000 7f757be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f757c101e40 0x7f757c194ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46356/0 (socket says 172.21.15.116:46356) 2024-03-30T01:00:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.552+0000 7f757be00700 1 -- 172.21.15.116:0/3329732375 learned_addr learned my addr 172.21.15.116:0/3329732375 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.552+0000 7f7580a00700 1 --2- 172.21.15.116:0/3329732375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f757c100c50 0x7f757c194990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.552+0000 7f7581400700 1 -- 172.21.15.116:0/3329732375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f757c100c50 msgr2=0x7f757c194990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.552+0000 7f7581400700 1 --2- 172.21.15.116:0/3329732375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f757c100c50 0x7f757c194990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.552+0000 7f7581400700 1 -- 172.21.15.116:0/3329732375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f757c101e40 msgr2=0x7f757c194ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.552+0000 7f7581400700 1 --2- 172.21.15.116:0/3329732375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f757c101e40 0x7f757c194ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.552+0000 7f7581400700 1 -- 172.21.15.116:0/3329732375 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7568009520 con 0x7f757c1027e0 2024-03-30T01:00:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.552+0000 7f757be00700 1 --2- 172.21.15.116:0/3329732375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f757c101e40 0x7f757c194ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:00:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.552+0000 7f7581400700 1 --2- 172.21.15.116:0/3329732375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f757c1027e0 0x7f757c19bef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7574004960 tx=0x7f757400f740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.553+0000 7f7579600700 1 -- 172.21.15.116:0/3329732375 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7574014040 con 0x7f757c1027e0 2024-03-30T01:00:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.553+0000 7f7581e00700 1 -- 172.21.15.116:0/3329732375 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75700095f0 con 0x7f757c1027e0 2024-03-30T01:00:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.553+0000 7f7579600700 1 -- 172.21.15.116:0/3329732375 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7574004db0 con 0x7f757c1027e0 2024-03-30T01:00:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.553+0000 7f7579600700 1 -- 172.21.15.116:0/3329732375 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7574007e20 con 0x7f757c1027e0 2024-03-30T01:00:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.553+0000 7f7581e00700 1 -- 172.21.15.116:0/3329732375 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f757c19cd90 con 0x7f757c1027e0 2024-03-30T01:00:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.554+0000 7f7581e00700 1 -- 172.21.15.116:0/3329732375 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757c108070 con 0x7f757c1027e0 2024-03-30T01:00:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.556+0000 7f7579600700 1 -- 172.21.15.116:0/3329732375 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f75740090d0 con 0x7f757c1027e0 2024-03-30T01:00:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.557+0000 7f7579600700 1 --2- 172.21.15.116:0/3329732375 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f75640a1120 0x7f75640a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.558+0000 7f7579600700 1 -- 172.21.15.116:0/3329732375 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f75740d0d10 con 0x7f757c1027e0 2024-03-30T01:00:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.558+0000 7f7580a00700 1 --2- 172.21.15.116:0/3329732375 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f75640a1120 0x7f75640a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.558+0000 7f7580a00700 1 --2- 172.21.15.116:0/3329732375 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f75640a1120 0x7f75640a35e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7570003730 tx=0x7f7570000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:42.560+0000 7f7579600700 1 -- 172.21.15.116:0/3329732375 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f757409a190 con 0x7f757c1027e0 2024-03-30T01:00:42.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:42 smithi130 ceph-mon[47626]: pgmap v376: 129 pgs: 129 active+clean; 395 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 11 MiB/s wr, 606 op/s 2024-03-30T01:00:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:42 smithi181 ceph-mon[47622]: pgmap v376: 129 pgs: 129 active+clean; 395 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 11 MiB/s wr, 606 op/s 2024-03-30T01:00:44.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:44 smithi130 ceph-mon[47626]: pgmap v377: 129 pgs: 129 active+clean; 394 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 8.1 MiB/s wr, 509 op/s 2024-03-30T01:00:44.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:44 smithi116 ceph-mon[38086]: pgmap v377: 129 pgs: 129 active+clean; 394 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 8.1 MiB/s wr, 509 op/s 2024-03-30T01:00:45.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:44 smithi181 ceph-mon[47622]: pgmap v377: 129 pgs: 129 active+clean; 394 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 8.1 MiB/s wr, 509 op/s 2024-03-30T01:00:45.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:45.915+0000 7f7581e00700 1 -- 172.21.15.116:0/3329732375 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7558005180 con 0x7f757c1027e0 2024-03-30T01:00:45.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:45.948+0000 7f7579600700 1 -- 172.21.15.116:0/3329732375 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10889 (secure 0 0 0) 0x7f7574027070 con 0x7f757c1027e0 2024-03-30T01:00:45.984 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T01:00:45.984 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T00:59:42.300374+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T01:00:46.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:46.072+0000 7f7581e00700 1 -- 172.21.15.116:0/3329732375 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f75640a1120 msgr2=0x7f75640a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:46.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:46.072+0000 7f7581e00700 1 --2- 172.21.15.116:0/3329732375 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f75640a1120 0x7f75640a35e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7570003730 tx=0x7f7570000bc0 comp rx=0 tx=0).stop 2024-03-30T01:00:46.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:46.072+0000 7f7581e00700 1 -- 172.21.15.116:0/3329732375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f757c1027e0 msgr2=0x7f757c19bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:46.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:46.072+0000 7f7581e00700 1 --2- 172.21.15.116:0/3329732375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f757c1027e0 0x7f757c19bef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7574004960 tx=0x7f757400f740 comp rx=0 tx=0).stop 2024-03-30T01:00:46.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:46.077+0000 7f7581e00700 1 -- 172.21.15.116:0/3329732375 shutdown_connections 2024-03-30T01:00:46.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:46.077+0000 7f7581e00700 1 --2- 172.21.15.116:0/3329732375 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f75640a1120 0x7f75640a35e0 secure :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7570003730 tx=0x7f7570000bc0 comp rx=0 tx=0).stop 2024-03-30T01:00:46.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:46.077+0000 7f7581e00700 1 --2- 172.21.15.116:0/3329732375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f757c100c50 0x7f757c194990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:46.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:46.077+0000 7f7581e00700 1 --2- 172.21.15.116:0/3329732375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f757c1027e0 0x7f757c19bef0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:46.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:46.077+0000 7f7581e00700 1 --2- 172.21.15.116:0/3329732375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f757c101e40 0x7f757c194ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:46.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:46.077+0000 7f7581e00700 1 -- 172.21.15.116:0/3329732375 >> 172.21.15.116:0/3329732375 conn(0x7f757c0fc1c0 msgr2=0x7f757c1051b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:46.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:46.140+0000 7f7581e00700 1 -- 172.21.15.116:0/3329732375 shutdown_connections 2024-03-30T01:00:46.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:46.140+0000 7f7581e00700 1 -- 172.21.15.116:0/3329732375 wait complete. 2024-03-30T01:00:46.194 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 25 2024-03-30T01:00:46.478 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T01:00:46.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:46 smithi116 ceph-mon[38086]: pgmap v378: 129 pgs: 129 active+clean; 400 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 434 op/s 2024-03-30T01:00:46.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:46 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3329732375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:00:46.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:46 smithi130 ceph-mon[47626]: pgmap v378: 129 pgs: 129 active+clean; 400 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 434 op/s 2024-03-30T01:00:46.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:46 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3329732375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:00:47.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:46 smithi181 ceph-mon[47622]: pgmap v378: 129 pgs: 129 active+clean; 400 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 434 op/s 2024-03-30T01:00:47.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:46 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3329732375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:00:48.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:48 smithi130 ceph-mon[47626]: pgmap v379: 129 pgs: 129 active+clean; 406 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.0 MiB/s wr, 351 op/s 2024-03-30T01:00:48.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:48 smithi116 ceph-mon[38086]: pgmap v379: 129 pgs: 129 active+clean; 406 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.0 MiB/s wr, 351 op/s 2024-03-30T01:00:48.963 INFO:tasks.workunit.client.0.smithi116.stdout: 123 658982 1141 449158 1027 223596 1818 2024-03-30T01:00:49.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:48 smithi181 ceph-mon[47622]: pgmap v379: 129 pgs: 129 active+clean; 406 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.0 MiB/s wr, 351 op/s 2024-03-30T01:00:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.947+0000 7f33d2600700 1 -- 172.21.15.116:0/1436817542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc10d0f0 msgr2=0x7f33cc10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.947+0000 7f33d2600700 1 --2- 172.21.15.116:0/1436817542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc10d0f0 0x7f33cc10d570 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f33c4009f90 tx=0x7f33c400b330 comp rx=0 tx=0).stop 2024-03-30T01:00:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.950+0000 7f33d2600700 1 -- 172.21.15.116:0/1436817542 shutdown_connections 2024-03-30T01:00:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.950+0000 7f33d2600700 1 --2- 172.21.15.116:0/1436817542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33cc107db0 0x7f33cc10a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.950+0000 7f33d2600700 1 --2- 172.21.15.116:0/1436817542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc10d0f0 0x7f33cc10d570 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.950+0000 7f33d2600700 1 --2- 172.21.15.116:0/1436817542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f33cc10a720 0x7f33cc10cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.950+0000 7f33d2600700 1 -- 172.21.15.116:0/1436817542 >> 172.21.15.116:0/1436817542 conn(0x7f33cc06dd50 msgr2=0x7f33cc070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:49.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.957+0000 7f33d2600700 1 -- 172.21.15.116:0/1436817542 shutdown_connections 2024-03-30T01:00:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.957+0000 7f33d2600700 1 -- 172.21.15.116:0/1436817542 wait complete. 2024-03-30T01:00:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.959+0000 7f33d2600700 1 Processor -- start 2024-03-30T01:00:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.960+0000 7f33d2600700 1 -- start start 2024-03-30T01:00:49.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.967+0000 7f33d2600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f33cc107db0 0x7f33cc118370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:49.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.967+0000 7f33d2600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33cc10a720 0x7f33cc1188b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:49.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.967+0000 7f33d2600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc10d0f0 0x7f33cc1b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:49.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.967+0000 7f33d2600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc119130 con 0x7f33cc10d0f0 2024-03-30T01:00:49.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.967+0000 7f33d2600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc1b5f90 con 0x7f33cc107db0 2024-03-30T01:00:49.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.967+0000 7f33d2600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc1b60d0 con 0x7f33cc10a720 2024-03-30T01:00:49.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.967+0000 7f33caa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33cc10a720 0x7f33cc1188b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:49.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.967+0000 7f33caa00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33cc10a720 0x7f33cc1188b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:33862/0 (socket says 172.21.15.116:33862) 2024-03-30T01:00:49.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.967+0000 7f33caa00700 1 -- 172.21.15.116:0/482655024 learned_addr learned my addr 172.21.15.116:0/482655024 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:49.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.967+0000 7f33caa00700 1 -- 172.21.15.116:0/482655024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f33cc107db0 msgr2=0x7f33cc118370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T01:00:49.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.967+0000 7f33caa00700 1 --2- 172.21.15.116:0/482655024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f33cc107db0 0x7f33cc118370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:49.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.967+0000 7f33caa00700 1 -- 172.21.15.116:0/482655024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc10d0f0 msgr2=0x7f33cc1b5a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T01:00:49.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.968+0000 7f33cbe00700 1 --2- 172.21.15.116:0/482655024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc10d0f0 0x7f33cc1b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:49.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.968+0000 7f33caa00700 1 --2- 172.21.15.116:0/482655024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc10d0f0 0x7f33cc1b5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:49.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.968+0000 7f33caa00700 1 -- 172.21.15.116:0/482655024 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c4009c40 con 0x7f33cc10a720 2024-03-30T01:00:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.971+0000 7f33cbe00700 1 --2- 172.21.15.116:0/482655024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc10d0f0 0x7f33cc1b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.971+0000 7f33caa00700 1 --2- 172.21.15.116:0/482655024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33cc10a720 0x7f33cc1188b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f33b800d950 tx=0x7f33b800dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.971+0000 7f33c3e00700 1 -- 172.21.15.116:0/482655024 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b8009b40 con 0x7f33cc10a720 2024-03-30T01:00:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.971+0000 7f33d2600700 1 -- 172.21.15.116:0/482655024 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33cc1b6300 con 0x7f33cc10a720 2024-03-30T01:00:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.972+0000 7f33c3e00700 1 -- 172.21.15.116:0/482655024 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33b8009ca0 con 0x7f33cc10a720 2024-03-30T01:00:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.972+0000 7f33c3e00700 1 -- 172.21.15.116:0/482655024 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b800f620 con 0x7f33cc10a720 2024-03-30T01:00:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.972+0000 7f33d2600700 1 -- 172.21.15.116:0/482655024 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc1b6800 con 0x7f33cc10a720 2024-03-30T01:00:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.979+0000 7f33c3e00700 1 -- 172.21.15.116:0/482655024 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f33b800f780 con 0x7f33cc10a720 2024-03-30T01:00:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.981+0000 7f33c3e00700 1 --2- 172.21.15.116:0/482655024 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f33b00a11f0 0x7f33b00a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.982+0000 7f33c3e00700 1 -- 172.21.15.116:0/482655024 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f33b80cdac0 con 0x7f33cc10a720 2024-03-30T01:00:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.983+0000 7f33cb400700 1 --2- 172.21.15.116:0/482655024 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f33b00a11f0 0x7f33b00a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:49.993+0000 7f33cb400700 1 --2- 172.21.15.116:0/482655024 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f33b00a11f0 0x7f33b00a36b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f33bc005fd0 tx=0x7f33bc009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:50.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.000+0000 7f33e6078b80 1 Processor -- start 2024-03-30T01:00:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.008+0000 7f33e6078b80 1 -- start start 2024-03-30T01:00:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.008+0000 7f33e6078b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd27d0 0x557cc0cd2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.008+0000 7f33e6078b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3130 0x557cc0cd39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.008+0000 7f33e6078b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3fd0 0x557cc0cd4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.008+0000 7f33e6078b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc0c2a990 con 0x557cc0cd3130 2024-03-30T01:00:50.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.008+0000 7f33e6078b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc0c28350 con 0x557cc0cd3fd0 2024-03-30T01:00:50.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.008+0000 7f33e6078b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc0c23c40 con 0x557cc0cd27d0 2024-03-30T01:00:50.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.015+0000 7f33cbe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3fd0 0x557cc0cd4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:50.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.015+0000 7f33cbe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3fd0 0x557cc0cd4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:39690/0 (socket says 172.21.15.116:39690) 2024-03-30T01:00:50.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.015+0000 7f33cbe00700 1 -- 172.21.15.116:0/111499869 learned_addr learned my addr 172.21.15.116:0/111499869 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:50.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.015+0000 7f33cb400700 1 --2- 172.21.15.116:0/111499869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd27d0 0x557cc0cd2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:50.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.015+0000 7f33cbe00700 1 -- 172.21.15.116:0/111499869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd27d0 msgr2=0x557cc0cd2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:50.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.015+0000 7f33cbe00700 1 --2- 172.21.15.116:0/111499869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd27d0 0x557cc0cd2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.015+0000 7f33cbe00700 1 -- 172.21.15.116:0/111499869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3130 msgr2=0x557cc0cd39f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:00:50.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.015+0000 7f33cbe00700 1 --2- 172.21.15.116:0/111499869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3130 0x557cc0cd39f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.015+0000 7f33cbe00700 1 -- 172.21.15.116:0/111499869 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cc0aa53b0 con 0x557cc0cd3fd0 2024-03-30T01:00:50.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.028+0000 7f33cbe00700 1 --2- 172.21.15.116:0/111499869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3fd0 0x557cc0cd4450 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f33c4009480 tx=0x7f33c40094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6e3d30a32dccd73 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:50.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.032+0000 7f33c0c00700 1 -- 172.21.15.116:0/111499869 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4021070 con 0x557cc0cd3fd0 2024-03-30T01:00:50.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.032+0000 7f33c0c00700 1 -- 172.21.15.116:0/111499869 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33c4004b50 con 0x557cc0cd3fd0 2024-03-30T01:00:50.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.032+0000 7f33c0c00700 1 -- 172.21.15.116:0/111499869 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c400e040 con 0x557cc0cd3fd0 2024-03-30T01:00:50.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.043+0000 7f33e6078b80 1 -- 172.21.15.116:0/111499869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3fd0 msgr2=0x557cc0cd4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:50.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.043+0000 7f33e6078b80 1 --2- 172.21.15.116:0/111499869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3fd0 0x557cc0cd4450 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f33c4009480 tx=0x7f33c40094b0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.052+0000 7f33e6078b80 1 -- 172.21.15.116:0/111499869 shutdown_connections 2024-03-30T01:00:50.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.052+0000 7f33e6078b80 1 --2- 172.21.15.116:0/111499869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd27d0 0x557cc0cd2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.052+0000 7f33e6078b80 1 --2- 172.21.15.116:0/111499869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3fd0 0x557cc0cd4450 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.052+0000 7f33e6078b80 1 --2- 172.21.15.116:0/111499869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3130 0x557cc0cd39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.052+0000 7f33e6078b80 1 -- 172.21.15.116:0/111499869 >> 172.21.15.116:0/111499869 conn(0x557cc0c316a0 msgr2=0x557cc0ccd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:50.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.061+0000 7f33e6078b80 1 -- 172.21.15.116:0/111499869 shutdown_connections 2024-03-30T01:00:50.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.061+0000 7f33e6078b80 1 -- 172.21.15.116:0/111499869 wait complete. 2024-03-30T01:00:50.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.062+0000 7f33e6078b80 1 -- start start 2024-03-30T01:00:50.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.063+0000 7f33e6078b80 1 Processor -- start 2024-03-30T01:00:50.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.065+0000 7f33e6078b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd27d0 0x557cc0cdc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.066+0000 7f33e6078b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3130 0x557cc0cdcc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.066+0000 7f33e6078b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3fd0 0x557cc0ce3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.066+0000 7f33cb400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd27d0 0x557cc0cdc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:50.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.066+0000 7f33cb400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd27d0 0x557cc0cdc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:33872/0 (socket says 172.21.15.116:33872) 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.066+0000 7f33cbe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3fd0 0x557cc0ce3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.067+0000 7f33cbe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3fd0 0x557cc0ce3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52272/0 (socket says 172.21.15.116:52272) 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.067+0000 7f33caa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3130 0x557cc0cdcc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.067+0000 7f33caa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3130 0x557cc0cdcc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:39694/0 (socket says 172.21.15.116:39694) 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.067+0000 7f33cb400700 1 -- 172.21.15.116:0/160254505 learned_addr learned my addr 172.21.15.116:0/160254505 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.067+0000 7f33e6078b80 1 -- 172.21.15.116:0/160254505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc0bbfd80 con 0x557cc0cd3fd0 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.067+0000 7f33e6078b80 1 -- 172.21.15.116:0/160254505 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc0bbfa20 con 0x557cc0cd3130 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.067+0000 7f33e6078b80 1 -- 172.21.15.116:0/160254505 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc0aa76a0 con 0x557cc0cd27d0 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.067+0000 7f33cbe00700 1 -- 172.21.15.116:0/160254505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd27d0 msgr2=0x557cc0cdc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.067+0000 7f33cbe00700 1 --2- 172.21.15.116:0/160254505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd27d0 0x557cc0cdc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.067+0000 7f33cbe00700 1 -- 172.21.15.116:0/160254505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3130 msgr2=0x557cc0cdcc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.067+0000 7f33cbe00700 1 --2- 172.21.15.116:0/160254505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3130 0x557cc0cdcc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.067+0000 7f33cbe00700 1 -- 172.21.15.116:0/160254505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc10e0a0 con 0x557cc0cd3fd0 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.068+0000 7f33cbe00700 1 --2- 172.21.15.116:0/160254505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3fd0 0x557cc0ce3c60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f33c4009450 tx=0x7f33c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7226f3a0ee2a466 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:50.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.069+0000 7f33e6078b80 10 client.0 _ll_register_callbacks cb 0x557cc0bc9470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:50.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.069+0000 7f33e6078b80 1 -- 172.21.15.116:0/160254505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557cc0be4380 con 0x557cc0cd3fd0 2024-03-30T01:00:50.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.069+0000 7f33b5600700 10 client.15924 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:00:50.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.069+0000 7f33b5600700 1 -- 172.21.15.116:0/160254505 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4021070 con 0x557cc0cd3fd0 2024-03-30T01:00:50.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.077+0000 7f33b5600700 1 -- 172.21.15.116:0/160254505 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f33c401b5a0 con 0x557cc0cd3fd0 2024-03-30T01:00:50.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.077+0000 7f33b5600700 1 -- 172.21.15.116:0/160254505 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33c400e040 con 0x557cc0cd3fd0 2024-03-30T01:00:50.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.077+0000 7f33e6078b80 10 client.15924 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:50.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.077+0000 7f33e6078b80 1 -- 172.21.15.116:0/160254505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x557cc0a51b60 con 0x557cc0cd3fd0 2024-03-30T01:00:50.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.078+0000 7f33b5600700 1 -- 172.21.15.116:0/160254505 <== mon.0 v2:172.21.15.116:3300/0 4 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f33c4019040 con 0x557cc0cd3fd0 2024-03-30T01:00:50.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.086+0000 7f33e6078b80 10 client.15924 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:50.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.086+0000 7f33e6078b80 10 client.15924 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T01:00:50.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.086+0000 7f33e6078b80 20 client.15924 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:50.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.086+0000 7f33e6078b80 1 --2- 172.21.15.116:0/160254505 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x557cc0ce7e00 0x557cc0cea2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.086+0000 7f33e6078b80 4 client.15924 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:00:50.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.086+0000 7f33e6078b80 1 -- 172.21.15.116:0/160254505 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557cc0be4380 con 0x557cc0ce7e00 2024-03-30T01:00:50.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.090+0000 7f33cb400700 1 --2- 172.21.15.116:0/160254505 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x557cc0ce7e00 0x557cc0cea2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:50.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.095+0000 7f33cb400700 1 --2- 172.21.15.116:0/160254505 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x557cc0ce7e00 0x557cc0cea2c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=931098fa011688a9 server_cookie=6eac776abef7be0d in_seq=0 out_seq=0 2024-03-30T01:00:50.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.101+0000 7f33b5600700 10 client.15924 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T01:00:50.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.101+0000 7f33b5600700 1 -- 172.21.15.116:0/160254505 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557cc0be4380 con 0x557cc0ce7e00 2024-03-30T01:00:50.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.101+0000 7f33b5600700 10 client.15924 handle_command_reply: tid=0 2024-03-30T01:00:50.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.101+0000 7f33e6078b80 1 client.15924 shutdown 2024-03-30T01:00:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.103+0000 7f33e6078b80 1 -- 172.21.15.116:0/160254505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3fd0 msgr2=0x557cc0ce3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.103+0000 7f33e6078b80 1 --2- 172.21.15.116:0/160254505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3fd0 0x557cc0ce3c60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f33c4009450 tx=0x7f33c400b330 comp rx=0 tx=0).stop 2024-03-30T01:00:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.103+0000 7f33e6078b80 1 -- 172.21.15.116:0/160254505 shutdown_connections 2024-03-30T01:00:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.103+0000 7f33e6078b80 1 --2- 172.21.15.116:0/160254505 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x557cc0ce7e00 0x557cc0cea2c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.103+0000 7f33e6078b80 1 --2- 172.21.15.116:0/160254505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd27d0 0x557cc0cdc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.103+0000 7f33e6078b80 1 --2- 172.21.15.116:0/160254505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3fd0 0x557cc0ce3c60 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.103+0000 7f33e6078b80 1 --2- 172.21.15.116:0/160254505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3130 0x557cc0cdcc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.103+0000 7f33e6078b80 1 -- 172.21.15.116:0/160254505 >> 172.21.15.116:0/160254505 conn(0x557cc0c316a0 msgr2=0x557cc0cd1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:50.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.103+0000 7f33b5600700 0 client.15924 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T01:00:50.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.104+0000 7f33e6078b80 1 -- 172.21.15.116:0/160254505 shutdown_connections 2024-03-30T01:00:50.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.104+0000 7f33e6078b80 1 -- 172.21.15.116:0/160254505 wait complete. 2024-03-30T01:00:50.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.104+0000 7f33e6078b80 20 client.15924 trim_cache size 0 max 16384 2024-03-30T01:00:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.324+0000 7f33e6078b80 1 Processor -- start 2024-03-30T01:00:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.335+0000 7f33e6078b80 1 -- start start 2024-03-30T01:00:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.335+0000 7f33e6078b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd27d0 0x557cc0cd43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.335+0000 7f33e6078b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3130 0x557cc0ce3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.335+0000 7f33e6078b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd3fd0 0x557cc0cdc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.335+0000 7f33e6078b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc0be4380 con 0x557cc0cd3130 2024-03-30T01:00:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.335+0000 7f33e6078b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc0cce4b0 con 0x557cc0cd27d0 2024-03-30T01:00:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.335+0000 7f33e6078b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc0cdc250 con 0x557cc0cd3fd0 2024-03-30T01:00:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.336+0000 7f33caa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3130 0x557cc0ce3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.337+0000 7f33caa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3130 0x557cc0ce3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52276/0 (socket says 172.21.15.116:52276) 2024-03-30T01:00:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.337+0000 7f33caa00700 1 -- 172.21.15.116:0/964530197 learned_addr learned my addr 172.21.15.116:0/964530197 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.338+0000 7f33caa00700 1 -- 172.21.15.116:0/964530197 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd3fd0 msgr2=0x557cc0cdc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.338+0000 7f33caa00700 1 --2- 172.21.15.116:0/964530197 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd3fd0 0x557cc0cdc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.338+0000 7f33caa00700 1 -- 172.21.15.116:0/964530197 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd27d0 msgr2=0x557cc0cd43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.338+0000 7f33caa00700 1 --2- 172.21.15.116:0/964530197 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd27d0 0x557cc0cd43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.338+0000 7f33caa00700 1 -- 172.21.15.116:0/964530197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cc0be9ea0 con 0x557cc0cd3130 2024-03-30T01:00:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.338+0000 7f33caa00700 1 --2- 172.21.15.116:0/964530197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3130 0x557cc0ce3c20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f33b8010dc0 tx=0x7f33b800dfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8362acdf79293b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:50.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.342+0000 7f33c1600700 1 -- 172.21.15.116:0/964530197 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b809ea30 con 0x557cc0cd3130 2024-03-30T01:00:50.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.342+0000 7f33c1600700 1 -- 172.21.15.116:0/964530197 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33b80a3080 con 0x557cc0cd3130 2024-03-30T01:00:50.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.342+0000 7f33c1600700 1 -- 172.21.15.116:0/964530197 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b80a2e50 con 0x557cc0cd3130 2024-03-30T01:00:50.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.355+0000 7f33e6078b80 1 -- 172.21.15.116:0/964530197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3130 msgr2=0x557cc0ce3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:50.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.355+0000 7f33e6078b80 1 --2- 172.21.15.116:0/964530197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3130 0x557cc0ce3c20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f33b8010dc0 tx=0x7f33b800dfd0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.359+0000 7f33e6078b80 1 -- 172.21.15.116:0/964530197 shutdown_connections 2024-03-30T01:00:50.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.359+0000 7f33e6078b80 1 --2- 172.21.15.116:0/964530197 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd3fd0 0x557cc0cdc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.359+0000 7f33e6078b80 1 --2- 172.21.15.116:0/964530197 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd27d0 0x557cc0cd43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.359+0000 7f33e6078b80 1 --2- 172.21.15.116:0/964530197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd3130 0x557cc0ce3c20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.359+0000 7f33e6078b80 1 -- 172.21.15.116:0/964530197 >> 172.21.15.116:0/964530197 conn(0x557cc0c316a0 msgr2=0x557cc0cda5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:50.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.368+0000 7f33e6078b80 1 -- 172.21.15.116:0/964530197 shutdown_connections 2024-03-30T01:00:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.370+0000 7f33e6078b80 1 -- 172.21.15.116:0/964530197 wait complete. 2024-03-30T01:00:50.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.373+0000 7f33e6078b80 1 -- start start 2024-03-30T01:00:50.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.374+0000 7f33e6078b80 1 Processor -- start 2024-03-30T01:00:50.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.379+0000 7f33e6078b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd27d0 0x557cc0ce3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.379+0000 7f33e6078b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3130 0x557cc0ce1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.379+0000 7f33e6078b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd3fd0 0x557cc0ce2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.379+0000 7f33e6078b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc0ce3f90 con 0x557cc0cd27d0 2024-03-30T01:00:50.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.379+0000 7f33e6078b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc0ce2850 con 0x557cc0cd3130 2024-03-30T01:00:50.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.379+0000 7f33e6078b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc0ce2a10 con 0x557cc0cd3fd0 2024-03-30T01:00:50.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.379+0000 7f33cb400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd27d0 0x557cc0ce3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:50.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.380+0000 7f33cbe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd3fd0 0x557cc0ce2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:50.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.380+0000 7f33cb400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd27d0 0x557cc0ce3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52282/0 (socket says 172.21.15.116:52282) 2024-03-30T01:00:50.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.380+0000 7f33cb400700 1 -- 172.21.15.116:0/1487238432 learned_addr learned my addr 172.21.15.116:0/1487238432 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:50.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.380+0000 7f33e6078b80 10 client.0 _ll_register_callbacks cb 0x557cc0bc9470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:50.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.380+0000 7f33cb400700 1 -- 172.21.15.116:0/1487238432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd3fd0 msgr2=0x557cc0ce2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:50.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.380+0000 7f33cb400700 1 --2- 172.21.15.116:0/1487238432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd3fd0 0x557cc0ce2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.380+0000 7f33cb400700 1 -- 172.21.15.116:0/1487238432 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3130 msgr2=0x557cc0ce1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:50.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.380+0000 7f33cb400700 1 --2- 172.21.15.116:0/1487238432 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3130 0x557cc0ce1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.380+0000 7f33cb400700 1 -- 172.21.15.116:0/1487238432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc1b6800 con 0x557cc0cd27d0 2024-03-30T01:00:50.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.381+0000 7f33cb400700 1 --2- 172.21.15.116:0/1487238432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd27d0 0x557cc0ce3770 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f33bc005b40 tx=0x7f33bc00ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60bddaae1dc9ebc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:50.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.383+0000 7f33b5600700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:00:50.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.384+0000 7f33b5600700 1 -- 172.21.15.116:0/1487238432 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc022040 con 0x557cc0cd27d0 2024-03-30T01:00:50.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.384+0000 7f33b5600700 1 -- 172.21.15.116:0/1487238432 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f33bc004830 con 0x557cc0cd27d0 2024-03-30T01:00:50.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.384+0000 7f33e6078b80 1 -- 172.21.15.116:0/1487238432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33c4004b50 con 0x557cc0cd27d0 2024-03-30T01:00:50.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.393+0000 7f33b5600700 1 -- 172.21.15.116:0/1487238432 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33bc0275c0 con 0x557cc0cd27d0 2024-03-30T01:00:50.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.394+0000 7f33e6078b80 10 client.15930 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:50.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.394+0000 7f33e6078b80 1 -- 172.21.15.116:0/1487238432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557cc0aa9cd0 con 0x557cc0cd27d0 2024-03-30T01:00:50.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.394+0000 7f33b5600700 1 -- 172.21.15.116:0/1487238432 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33c4004b50 con 0x557cc0cd27d0 2024-03-30T01:00:50.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.395+0000 7f33b5600700 1 -- 172.21.15.116:0/1487238432 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f33bc010430 con 0x557cc0cd27d0 2024-03-30T01:00:50.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.396+0000 7f33e6078b80 10 client.15930 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:50.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.396+0000 7f33e6078b80 10 client.15930 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T01:00:50.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.396+0000 7f33e6078b80 20 client.15930 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:50.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.396+0000 7f33e6078b80 1 --2- 172.21.15.116:0/1487238432 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x557cc0d3d350 0x557cc0d3f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:50.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.396+0000 7f33e6078b80 4 client.15930 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:00:50.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.396+0000 7f33e6078b80 1 -- 172.21.15.116:0/1487238432 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557cc0c1c700 con 0x557cc0d3d350 2024-03-30T01:00:50.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.400+0000 7f33caa00700 1 --2- 172.21.15.116:0/1487238432 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x557cc0d3d350 0x557cc0d3f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:50.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.414+0000 7f33b5600700 10 client.15930 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T01:00:50.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.414+0000 7f33caa00700 1 --2- 172.21.15.116:0/1487238432 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x557cc0d3d350 0x557cc0d3f770 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e074303256433cd2 server_cookie=47c72c7aab56db22 in_seq=0 out_seq=0 2024-03-30T01:00:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.417+0000 7f33b5600700 1 -- 172.21.15.116:0/1487238432 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557cc0c1c700 con 0x557cc0d3d350 2024-03-30T01:00:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.417+0000 7f33b5600700 10 client.15930 handle_command_reply: tid=0 2024-03-30T01:00:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.417+0000 7f33e6078b80 1 client.15930 shutdown 2024-03-30T01:00:50.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.429+0000 7f33e6078b80 1 -- 172.21.15.116:0/1487238432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd27d0 msgr2=0x557cc0ce3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.429+0000 7f33e6078b80 1 --2- 172.21.15.116:0/1487238432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd27d0 0x557cc0ce3770 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f33bc005b40 tx=0x7f33bc00ed20 comp rx=0 tx=0).stop 2024-03-30T01:00:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.433+0000 7f33e6078b80 1 -- 172.21.15.116:0/1487238432 shutdown_connections 2024-03-30T01:00:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.433+0000 7f33e6078b80 1 --2- 172.21.15.116:0/1487238432 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x557cc0d3d350 0x557cc0d3f770 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.433+0000 7f33e6078b80 1 --2- 172.21.15.116:0/1487238432 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557cc0cd3fd0 0x557cc0ce2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.433+0000 7f33e6078b80 1 --2- 172.21.15.116:0/1487238432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cc0cd27d0 0x557cc0ce3770 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.433+0000 7f33e6078b80 1 --2- 172.21.15.116:0/1487238432 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc0cd3130 0x557cc0ce1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.433+0000 7f33e6078b80 1 -- 172.21.15.116:0/1487238432 >> 172.21.15.116:0/1487238432 conn(0x557cc0c316a0 msgr2=0x557cc0cdc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:50.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.433+0000 7f33b5600700 0 client.15930 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T01:00:50.449 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:00:50.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.438+0000 7f33e6078b80 1 -- 172.21.15.116:0/1487238432 shutdown_connections 2024-03-30T01:00:50.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.438+0000 7f33e6078b80 1 -- 172.21.15.116:0/1487238432 wait complete. 2024-03-30T01:00:50.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.442+0000 7f33e6078b80 20 client.15930 trim_cache size 0 max 16384 2024-03-30T01:00:50.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.444+0000 7f33d2600700 1 -- 172.21.15.116:0/482655024 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f33b00a11f0 msgr2=0x7f33b00a36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:50.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.444+0000 7f33d2600700 1 --2- 172.21.15.116:0/482655024 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f33b00a11f0 0x7f33b00a36b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f33bc005fd0 tx=0x7f33bc009310 comp rx=0 tx=0).stop 2024-03-30T01:00:50.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.444+0000 7f33d2600700 1 -- 172.21.15.116:0/482655024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33cc10a720 msgr2=0x7f33cc1188b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:50.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.444+0000 7f33d2600700 1 --2- 172.21.15.116:0/482655024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33cc10a720 0x7f33cc1188b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f33b800d950 tx=0x7f33b800dd10 comp rx=0 tx=0).stop 2024-03-30T01:00:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.449+0000 7f33d2600700 1 -- 172.21.15.116:0/482655024 shutdown_connections 2024-03-30T01:00:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.449+0000 7f33d2600700 1 --2- 172.21.15.116:0/482655024 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f33b00a11f0 0x7f33b00a36b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.449+0000 7f33d2600700 1 --2- 172.21.15.116:0/482655024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33cc10d0f0 0x7f33cc1b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.449+0000 7f33d2600700 1 --2- 172.21.15.116:0/482655024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f33cc107db0 0x7f33cc118370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.449+0000 7f33d2600700 1 --2- 172.21.15.116:0/482655024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f33cc10a720 0x7f33cc1188b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.449+0000 7f33d2600700 1 -- 172.21.15.116:0/482655024 >> 172.21.15.116:0/482655024 conn(0x7f33cc06dd50 msgr2=0x7f33cc10ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.450+0000 7f33d2600700 1 -- 172.21.15.116:0/482655024 shutdown_connections 2024-03-30T01:00:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:50.450+0000 7f33d2600700 1 -- 172.21.15.116:0/482655024 wait complete. 2024-03-30T01:00:50.561 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T01:00:50.798 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:50 smithi116 ceph-mon[38086]: pgmap v380: 129 pgs: 129 active+clean; 407 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2024-03-30T01:00:50.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:50 smithi130 ceph-mon[47626]: pgmap v380: 129 pgs: 129 active+clean; 407 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2024-03-30T01:00:51.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:50 smithi181 ceph-mon[47622]: pgmap v380: 129 pgs: 129 active+clean; 407 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2024-03-30T01:00:51.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.423+0000 7f0a83c00700 1 -- 172.21.15.116:0/703922140 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7c10d0f0 msgr2=0x7f0a7c10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:51.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.423+0000 7f0a83c00700 1 --2- 172.21.15.116:0/703922140 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7c10d0f0 0x7f0a7c10d570 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0098f0 tx=0x7f0a6c009c00 comp rx=0 tx=0).stop 2024-03-30T01:00:51.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.423+0000 7f0a83c00700 1 -- 172.21.15.116:0/703922140 shutdown_connections 2024-03-30T01:00:51.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.423+0000 7f0a83c00700 1 --2- 172.21.15.116:0/703922140 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7c10d0f0 0x7f0a7c10d570 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.423+0000 7f0a83c00700 1 --2- 172.21.15.116:0/703922140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7c107db0 0x7f0a7c10a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.423+0000 7f0a83c00700 1 --2- 172.21.15.116:0/703922140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a7c10a720 0x7f0a7c10cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.423+0000 7f0a83c00700 1 -- 172.21.15.116:0/703922140 >> 172.21.15.116:0/703922140 conn(0x7f0a7c06dd50 msgr2=0x7f0a7c070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:51.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.430+0000 7f0a83c00700 1 -- 172.21.15.116:0/703922140 shutdown_connections 2024-03-30T01:00:51.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.430+0000 7f0a83c00700 1 -- 172.21.15.116:0/703922140 wait complete. 2024-03-30T01:00:51.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.432+0000 7f0a83c00700 1 Processor -- start 2024-03-30T01:00:51.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.432+0000 7f0a83c00700 1 -- start start 2024-03-30T01:00:51.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.432+0000 7f0a83c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7c107db0 0x7f0a7c1a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.432+0000 7f0a83c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7c10a720 0x7f0a7c1a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.433+0000 7f0a83c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a7c10d0f0 0x7f0a7c1ad100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.433+0000 7f0a81200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7c107db0 0x7f0a7c1a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.433+0000 7f0a81200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7c107db0 0x7f0a7c1a5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:33894/0 (socket says 172.21.15.116:33894) 2024-03-30T01:00:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.435+0000 7f0a7be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7c10a720 0x7f0a7c1a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.435+0000 7f0a7be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7c10a720 0x7f0a7c1a60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:39738/0 (socket says 172.21.15.116:39738) 2024-03-30T01:00:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.437+0000 7f0a81200700 1 -- 172.21.15.116:0/1228463951 learned_addr learned my addr 172.21.15.116:0/1228463951 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.437+0000 7f0a83c00700 1 -- 172.21.15.116:0/1228463951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c1ad6d0 con 0x7f0a7c10d0f0 2024-03-30T01:00:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.437+0000 7f0a83c00700 1 -- 172.21.15.116:0/1228463951 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c1ad810 con 0x7f0a7c10a720 2024-03-30T01:00:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.437+0000 7f0a83c00700 1 -- 172.21.15.116:0/1228463951 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c1ad9a0 con 0x7f0a7c107db0 2024-03-30T01:00:51.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.437+0000 7f0a81c00700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a7c10d0f0 0x7f0a7c1ad100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:51.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.438+0000 7f0a81c00700 1 -- 172.21.15.116:0/1228463951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7c107db0 msgr2=0x7f0a7c1a5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:51.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.438+0000 7f0a81c00700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7c107db0 0x7f0a7c1a5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.438+0000 7f0a81c00700 1 -- 172.21.15.116:0/1228463951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7c10a720 msgr2=0x7f0a7c1a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:51.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.438+0000 7f0a81c00700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7c10a720 0x7f0a7c1a60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.438+0000 7f0a81c00700 1 -- 172.21.15.116:0/1228463951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c009520 con 0x7f0a7c10d0f0 2024-03-30T01:00:51.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.439+0000 7f0a7be00700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7c10a720 0x7f0a7c1a60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-30T01:00:51.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.439+0000 7f0a81c00700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a7c10d0f0 0x7f0a7c1ad100 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009810 tx=0x7f0a6c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:51.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.444+0000 7f0a79600700 1 -- 172.21.15.116:0/1228463951 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c01c070 con 0x7f0a7c10d0f0 2024-03-30T01:00:51.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.444+0000 7f0a79600700 1 -- 172.21.15.116:0/1228463951 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a6c00bcc0 con 0x7f0a7c10d0f0 2024-03-30T01:00:51.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.444+0000 7f0a79600700 1 -- 172.21.15.116:0/1228463951 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c003e00 con 0x7f0a7c10d0f0 2024-03-30T01:00:51.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.444+0000 7f0a83c00700 1 -- 172.21.15.116:0/1228463951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7c1adc20 con 0x7f0a7c10d0f0 2024-03-30T01:00:51.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.445+0000 7f0a81200700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7c107db0 0x7f0a7c1a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:51.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.450+0000 7f0a83c00700 1 -- 172.21.15.116:0/1228463951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7c1adff0 con 0x7f0a7c10d0f0 2024-03-30T01:00:51.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.460+0000 7f0a79600700 1 -- 172.21.15.116:0/1228463951 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f0a6c00e3e0 con 0x7f0a7c10d0f0 2024-03-30T01:00:51.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.462+0000 7f0a79600700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0a600a12a0 0x7f0a600a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.468+0000 7f0a81200700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0a600a12a0 0x7f0a600a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:51.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.469+0000 7f0a81200700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0a600a12a0 0x7f0a600a3760 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009d20 tx=0x7f0a680092a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:51.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.470+0000 7f0a79600700 1 -- 172.21.15.116:0/1228463951 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0a6c0cf220 con 0x7f0a7c10d0f0 2024-03-30T01:00:51.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.474+0000 7f0a97645b80 1 Processor -- start 2024-03-30T01:00:51.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.483+0000 7f0a97645b80 1 -- start start 2024-03-30T01:00:51.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.483+0000 7f0a97645b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 0x55f3b424bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.483+0000 7f0a97645b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424c130 0x55f3b424c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.483+0000 7f0a97645b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424cfd0 0x55f3b424d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.483+0000 7f0a97645b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b41a3990 con 0x55f3b424cfd0 2024-03-30T01:00:51.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.483+0000 7f0a97645b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b41a1350 con 0x55f3b424b7d0 2024-03-30T01:00:51.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.483+0000 7f0a97645b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b419cc40 con 0x55f3b424c130 2024-03-30T01:00:51.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.485+0000 7f0a81c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424cfd0 0x55f3b424d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:51.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.485+0000 7f0a81c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424cfd0 0x55f3b424d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52300/0 (socket says 172.21.15.116:52300) 2024-03-30T01:00:51.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.485+0000 7f0a81c00700 1 -- 172.21.15.116:0/1516982316 learned_addr learned my addr 172.21.15.116:0/1516982316 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:51.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.485+0000 7f0a81200700 1 --2- 172.21.15.116:0/1516982316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 0x55f3b424bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:51.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.486+0000 7f0a81c00700 1 -- 172.21.15.116:0/1516982316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424c130 msgr2=0x55f3b424c9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:00:51.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.486+0000 7f0a81c00700 1 --2- 172.21.15.116:0/1516982316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424c130 0x55f3b424c9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.486+0000 7f0a81c00700 1 -- 172.21.15.116:0/1516982316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 msgr2=0x55f3b424bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:51.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.486+0000 7f0a81c00700 1 --2- 172.21.15.116:0/1516982316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 0x55f3b424bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.486+0000 7f0a81c00700 1 -- 172.21.15.116:0/1516982316 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3b401e3b0 con 0x55f3b424cfd0 2024-03-30T01:00:51.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.486+0000 7f0a81c00700 1 --2- 172.21.15.116:0/1516982316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424cfd0 0x55f3b424d450 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0967e0 tx=0x7f0a6c0038d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c41819eb05fa40 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:51.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.487+0000 7f0a72000700 1 -- 172.21.15.116:0/1516982316 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c005730 con 0x55f3b424cfd0 2024-03-30T01:00:51.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.487+0000 7f0a72000700 1 -- 172.21.15.116:0/1516982316 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a6c09a9e0 con 0x55f3b424cfd0 2024-03-30T01:00:51.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.489+0000 7f0a72000700 1 -- 172.21.15.116:0/1516982316 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c098420 con 0x55f3b424cfd0 2024-03-30T01:00:51.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.495+0000 7f0a97645b80 1 -- 172.21.15.116:0/1516982316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424cfd0 msgr2=0x55f3b424d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:51.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.495+0000 7f0a97645b80 1 --2- 172.21.15.116:0/1516982316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424cfd0 0x55f3b424d450 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0967e0 tx=0x7f0a6c0038d0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.501+0000 7f0a97645b80 1 -- 172.21.15.116:0/1516982316 shutdown_connections 2024-03-30T01:00:51.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.501+0000 7f0a97645b80 1 --2- 172.21.15.116:0/1516982316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424cfd0 0x55f3b424d450 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.501+0000 7f0a97645b80 1 --2- 172.21.15.116:0/1516982316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 0x55f3b424bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.501+0000 7f0a97645b80 1 --2- 172.21.15.116:0/1516982316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424c130 0x55f3b424c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.501+0000 7f0a97645b80 1 -- 172.21.15.116:0/1516982316 >> 172.21.15.116:0/1516982316 conn(0x55f3b41aa6a0 msgr2=0x55f3b42469a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:51.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.505+0000 7f0a97645b80 1 -- 172.21.15.116:0/1516982316 shutdown_connections 2024-03-30T01:00:51.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.509+0000 7f0a97645b80 1 -- 172.21.15.116:0/1516982316 wait complete. 2024-03-30T01:00:51.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.509+0000 7f0a97645b80 1 -- start start 2024-03-30T01:00:51.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.510+0000 7f0a97645b80 1 Processor -- start 2024-03-30T01:00:51.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.514+0000 7f0a97645b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 0x55f3b4255350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.514+0000 7f0a97645b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424c130 0x55f3b4255890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.514+0000 7f0a97645b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424cfd0 0x55f3b425c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.514+0000 7f0a97645b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b4138d80 con 0x55f3b424cfd0 2024-03-30T01:00:51.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.514+0000 7f0a97645b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b4138a20 con 0x55f3b424b7d0 2024-03-30T01:00:51.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.514+0000 7f0a97645b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b40206a0 con 0x55f3b424c130 2024-03-30T01:00:51.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.514+0000 7f0a97645b80 10 client.0 _ll_register_callbacks cb 0x55f3b4142470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:51.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.515+0000 7f0a7be00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424c130 0x55f3b4255890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:51.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.515+0000 7f0a7be00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424c130 0x55f3b4255890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:33906/0 (socket says 172.21.15.116:33906) 2024-03-30T01:00:51.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.515+0000 7f0a7be00700 1 -- 172.21.15.116:0/3495594462 learned_addr learned my addr 172.21.15.116:0/3495594462 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:51.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.515+0000 7f0a7be00700 1 -- 172.21.15.116:0/3495594462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 msgr2=0x55f3b4255350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:51.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.515+0000 7f0a7be00700 1 --2- 172.21.15.116:0/3495594462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 0x55f3b4255350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.515+0000 7f0a7be00700 1 -- 172.21.15.116:0/3495594462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424cfd0 msgr2=0x55f3b425c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:51.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.515+0000 7f0a7be00700 1 --2- 172.21.15.116:0/3495594462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424cfd0 0x55f3b425c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.515+0000 7f0a7be00700 1 -- 172.21.15.116:0/3495594462 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7c1adff0 con 0x55f3b424c130 2024-03-30T01:00:51.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.515+0000 7f0a7be00700 1 --2- 172.21.15.116:0/3495594462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424c130 0x55f3b4255890 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400c8c0 tx=0x7f0a7400cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bb5d6610f44ffb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:51.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.516+0000 7f0a66a00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:00:51.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.516+0000 7f0a66a00700 1 -- 172.21.15.116:0/3495594462 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7400ab60 con 0x55f3b424c130 2024-03-30T01:00:51.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.516+0000 7f0a66a00700 1 -- 172.21.15.116:0/3495594462 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0a74008040 con 0x55f3b424c130 2024-03-30T01:00:51.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.517+0000 7f0a97645b80 1 -- 172.21.15.116:0/3495594462 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f3b415d380 con 0x55f3b424c130 2024-03-30T01:00:51.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.523+0000 7f0a66a00700 1 -- 172.21.15.116:0/3495594462 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a7400acc0 con 0x55f3b424c130 2024-03-30T01:00:51.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.531+0000 7f0a97645b80 10 client.25993 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:51.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.531+0000 7f0a97645b80 1 -- 172.21.15.116:0/3495594462 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f3b3fcab60 con 0x55f3b424c130 2024-03-30T01:00:51.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.532+0000 7f0a66a00700 1 -- 172.21.15.116:0/3495594462 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f3b415d380 con 0x55f3b424c130 2024-03-30T01:00:51.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.533+0000 7f0a66a00700 1 -- 172.21.15.116:0/3495594462 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f0a74016020 con 0x55f3b424c130 2024-03-30T01:00:51.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.533+0000 7f0a97645b80 10 client.25993 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:51.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.533+0000 7f0a97645b80 10 client.25993 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T01:00:51.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.533+0000 7f0a97645b80 20 client.25993 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:51.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.533+0000 7f0a97645b80 1 --2- 172.21.15.116:0/3495594462 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55f3b4261160 0x55f3b4263580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.533+0000 7f0a97645b80 4 client.25993 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:00:51.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.533+0000 7f0a97645b80 1 -- 172.21.15.116:0/3495594462 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f3b4195700 con 0x55f3b4261160 2024-03-30T01:00:51.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.533+0000 7f0a81200700 1 --2- 172.21.15.116:0/3495594462 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55f3b4261160 0x55f3b4263580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:51.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.535+0000 7f0a81200700 1 --2- 172.21.15.116:0/3495594462 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55f3b4261160 0x55f3b4263580 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=24db46e1dca88ac9 server_cookie=4dc5b04a245c29c3 in_seq=0 out_seq=0 2024-03-30T01:00:51.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.535+0000 7f0a66a00700 10 client.25993 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T01:00:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.536+0000 7f0a66a00700 1 -- 172.21.15.116:0/3495594462 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f3b4195700 con 0x55f3b4261160 2024-03-30T01:00:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.536+0000 7f0a66a00700 10 client.25993 handle_command_reply: tid=0 2024-03-30T01:00:51.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.537+0000 7f0a97645b80 1 client.25993 shutdown 2024-03-30T01:00:51.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.551+0000 7f0a97645b80 1 -- 172.21.15.116:0/3495594462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424c130 msgr2=0x55f3b4255890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:51.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.551+0000 7f0a97645b80 1 --2- 172.21.15.116:0/3495594462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424c130 0x55f3b4255890 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400c8c0 tx=0x7f0a7400cc80 comp rx=0 tx=0).stop 2024-03-30T01:00:51.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.551+0000 7f0a97645b80 1 -- 172.21.15.116:0/3495594462 shutdown_connections 2024-03-30T01:00:51.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.551+0000 7f0a97645b80 1 --2- 172.21.15.116:0/3495594462 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55f3b4261160 0x55f3b4263580 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.551+0000 7f0a97645b80 1 --2- 172.21.15.116:0/3495594462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424cfd0 0x55f3b425c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.551+0000 7f0a97645b80 1 --2- 172.21.15.116:0/3495594462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 0x55f3b4255350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.551+0000 7f0a97645b80 1 --2- 172.21.15.116:0/3495594462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424c130 0x55f3b4255890 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.551+0000 7f0a66a00700 0 client.25993 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T01:00:51.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.551+0000 7f0a97645b80 1 -- 172.21.15.116:0/3495594462 >> 172.21.15.116:0/3495594462 conn(0x55f3b41aa6a0 msgr2=0x55f3b424a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:51.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.551+0000 7f0a97645b80 1 -- 172.21.15.116:0/3495594462 shutdown_connections 2024-03-30T01:00:51.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.555+0000 7f0a97645b80 1 -- 172.21.15.116:0/3495594462 wait complete. 2024-03-30T01:00:51.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.560+0000 7f0a97645b80 20 client.25993 trim_cache size 0 max 16384 2024-03-30T01:00:51.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.656+0000 7f0a97645b80 1 Processor -- start 2024-03-30T01:00:51.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.667+0000 7f0a97645b80 1 -- start start 2024-03-30T01:00:51.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.668+0000 7f0a97645b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424b7d0 0x55f3b424bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.668+0000 7f0a97645b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424c130 0x55f3b425c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.668+0000 7f0a97645b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424cfd0 0x55f3b425cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.668+0000 7f0a97645b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b424c510 con 0x55f3b424b7d0 2024-03-30T01:00:51.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.668+0000 7f0a97645b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b425da50 con 0x55f3b424c130 2024-03-30T01:00:51.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.668+0000 7f0a97645b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b4260c50 con 0x55f3b424cfd0 2024-03-30T01:00:51.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.668+0000 7f0a7be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424c130 0x55f3b425c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:51.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.668+0000 7f0a7be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424c130 0x55f3b425c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:39774/0 (socket says 172.21.15.116:39774) 2024-03-30T01:00:51.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.668+0000 7f0a7be00700 1 -- 172.21.15.116:0/3554987322 learned_addr learned my addr 172.21.15.116:0/3554987322 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:51.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.668+0000 7f0a81c00700 1 --2- 172.21.15.116:0/3554987322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424cfd0 0x55f3b425cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:51.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.668+0000 7f0a7be00700 1 -- 172.21.15.116:0/3554987322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424cfd0 msgr2=0x55f3b425cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:51.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.668+0000 7f0a7be00700 1 --2- 172.21.15.116:0/3554987322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424cfd0 0x55f3b425cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.668+0000 7f0a7be00700 1 -- 172.21.15.116:0/3554987322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424b7d0 msgr2=0x55f3b424bbb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:00:51.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.668+0000 7f0a81200700 1 --2- 172.21.15.116:0/3554987322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424b7d0 0x55f3b424bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:51.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.669+0000 7f0a7be00700 1 --2- 172.21.15.116:0/3554987322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424b7d0 0x55f3b424bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.669+0000 7f0a7be00700 1 -- 172.21.15.116:0/3554987322 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3b4162ea0 con 0x55f3b424c130 2024-03-30T01:00:51.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.669+0000 7f0a81c00700 1 --2- 172.21.15.116:0/3554987322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424cfd0 0x55f3b425cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:51.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.669+0000 7f0a81200700 1 --2- 172.21.15.116:0/3554987322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424b7d0 0x55f3b424bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:00:51.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.669+0000 7f0a7be00700 1 --2- 172.21.15.116:0/3554987322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424c130 0x55f3b425c890 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0a740022f0 tx=0x7f0a7400cf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f2f99225f215a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:51.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.671+0000 7f0a72a00700 1 -- 172.21.15.116:0/3554987322 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7400f6e0 con 0x55f3b424c130 2024-03-30T01:00:51.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.671+0000 7f0a72a00700 1 -- 172.21.15.116:0/3554987322 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a7400a960 con 0x55f3b424c130 2024-03-30T01:00:51.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.672+0000 7f0a97645b80 1 -- 172.21.15.116:0/3554987322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424c130 msgr2=0x55f3b425c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:51.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.672+0000 7f0a97645b80 1 --2- 172.21.15.116:0/3554987322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424c130 0x55f3b425c890 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0a740022f0 tx=0x7f0a7400cf40 comp rx=0 tx=0).stop 2024-03-30T01:00:51.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.675+0000 7f0a97645b80 1 -- 172.21.15.116:0/3554987322 shutdown_connections 2024-03-30T01:00:51.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.675+0000 7f0a97645b80 1 --2- 172.21.15.116:0/3554987322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424cfd0 0x55f3b425cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.675+0000 7f0a97645b80 1 --2- 172.21.15.116:0/3554987322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424b7d0 0x55f3b424bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.675+0000 7f0a97645b80 1 --2- 172.21.15.116:0/3554987322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424c130 0x55f3b425c890 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.675+0000 7f0a97645b80 1 -- 172.21.15.116:0/3554987322 >> 172.21.15.116:0/3554987322 conn(0x55f3b41aa6a0 msgr2=0x55f3b423f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:51.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.680+0000 7f0a97645b80 1 -- 172.21.15.116:0/3554987322 shutdown_connections 2024-03-30T01:00:51.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.680+0000 7f0a97645b80 1 -- 172.21.15.116:0/3554987322 wait complete. 2024-03-30T01:00:51.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.680+0000 7f0a97645b80 1 -- start start 2024-03-30T01:00:51.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.681+0000 7f0a97645b80 1 Processor -- start 2024-03-30T01:00:51.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.683+0000 7f0a97645b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 0x55f3b425a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.685+0000 7f0a97645b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424c130 0x55f3b425a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.686+0000 7f0a81200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 0x55f3b425a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.686+0000 7f0a81200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 0x55f3b425a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:39778/0 (socket says 172.21.15.116:39778) 2024-03-30T01:00:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.686+0000 7f0a7be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424c130 0x55f3b425a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.686+0000 7f0a7be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424c130 0x55f3b425a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52330/0 (socket says 172.21.15.116:52330) 2024-03-30T01:00:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.686+0000 7f0a81200700 1 -- 172.21.15.116:0/1084454563 learned_addr learned my addr 172.21.15.116:0/1084454563 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.686+0000 7f0a97645b80 1 --2- 172.21.15.116:0/1084454563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424cfd0 0x55f3b425b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.686+0000 7f0a97645b80 1 -- 172.21.15.116:0/1084454563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b425be90 con 0x55f3b424c130 2024-03-30T01:00:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.686+0000 7f0a97645b80 1 -- 172.21.15.116:0/1084454563 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b425c000 con 0x55f3b424b7d0 2024-03-30T01:00:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.686+0000 7f0a97645b80 1 -- 172.21.15.116:0/1084454563 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b425c1c0 con 0x55f3b424cfd0 2024-03-30T01:00:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.687+0000 7f0a7be00700 1 -- 172.21.15.116:0/1084454563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424cfd0 msgr2=0x55f3b425b950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:00:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.687+0000 7f0a7be00700 1 --2- 172.21.15.116:0/1084454563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424cfd0 0x55f3b425b950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.687+0000 7f0a7be00700 1 -- 172.21.15.116:0/1084454563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 msgr2=0x55f3b425a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.687+0000 7f0a7be00700 1 --2- 172.21.15.116:0/1084454563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 0x55f3b425a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.687+0000 7f0a7be00700 1 -- 172.21.15.116:0/1084454563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3b3fcab60 con 0x55f3b424c130 2024-03-30T01:00:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.688+0000 7f0a7be00700 1 --2- 172.21.15.116:0/1084454563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424c130 0x55f3b425a940 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400ccc0 tx=0x7f0a7400af20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8758f17d234b1b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.689+0000 7f0a66a00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:00:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.689+0000 7f0a66a00700 1 -- 172.21.15.116:0/1084454563 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a74022810 con 0x55f3b424c130 2024-03-30T01:00:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.689+0000 7f0a97645b80 10 client.0 _ll_register_callbacks cb 0x55f3b4142470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.689+0000 7f0a66a00700 1 -- 172.21.15.116:0/1084454563 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0a74026070 con 0x55f3b424c130 2024-03-30T01:00:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.689+0000 7f0a97645b80 1 -- 172.21.15.116:0/1084454563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0a6c09a9e0 con 0x55f3b424c130 2024-03-30T01:00:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.690+0000 7f0a66a00700 1 -- 172.21.15.116:0/1084454563 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a74022970 con 0x55f3b424c130 2024-03-30T01:00:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.690+0000 7f0a97645b80 10 client.15939 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.690+0000 7f0a97645b80 1 -- 172.21.15.116:0/1084454563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x55f3b4022cd0 con 0x55f3b424c130 2024-03-30T01:00:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.690+0000 7f0a66a00700 1 -- 172.21.15.116:0/1084454563 <== mon.0 v2:172.21.15.116:3300/0 4 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f0a7400c9a0 con 0x55f3b424c130 2024-03-30T01:00:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.690+0000 7f0a97645b80 10 client.15939 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.690+0000 7f0a97645b80 10 client.15939 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T01:00:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.690+0000 7f0a97645b80 20 client.15939 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.690+0000 7f0a97645b80 1 --2- 172.21.15.116:0/1084454563 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55f3b42b5390 0x55f3b42b7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.690+0000 7f0a97645b80 4 client.15939 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:00:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.690+0000 7f0a97645b80 1 -- 172.21.15.116:0/1084454563 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f3b3febcf0 con 0x55f3b42b5390 2024-03-30T01:00:51.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.694+0000 7f0a81200700 1 --2- 172.21.15.116:0/1084454563 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55f3b42b5390 0x55f3b42b7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:51.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.700+0000 7f0a81200700 1 --2- 172.21.15.116:0/1084454563 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55f3b42b5390 0x55f3b42b7830 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=49421c6a81a80807 server_cookie=c01d5e468fbea96e in_seq=0 out_seq=0 2024-03-30T01:00:51.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.706+0000 7f0a66a00700 10 client.15939 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T01:00:51.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.706+0000 7f0a66a00700 1 -- 172.21.15.116:0/1084454563 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f3b3febcf0 con 0x55f3b42b5390 2024-03-30T01:00:51.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.706+0000 7f0a66a00700 10 client.15939 handle_command_reply: tid=0 2024-03-30T01:00:51.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.707+0000 7f0a97645b80 1 client.15939 shutdown 2024-03-30T01:00:51.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.707+0000 7f0a97645b80 1 -- 172.21.15.116:0/1084454563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424c130 msgr2=0x55f3b425a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:51.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.707+0000 7f0a97645b80 1 --2- 172.21.15.116:0/1084454563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424c130 0x55f3b425a940 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400ccc0 tx=0x7f0a7400af20 comp rx=0 tx=0).stop 2024-03-30T01:00:51.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.708+0000 7f0a97645b80 1 -- 172.21.15.116:0/1084454563 shutdown_connections 2024-03-30T01:00:51.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.708+0000 7f0a97645b80 1 --2- 172.21.15.116:0/1084454563 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55f3b42b5390 0x55f3b42b7830 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.708+0000 7f0a97645b80 1 --2- 172.21.15.116:0/1084454563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f3b424cfd0 0x55f3b425b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.708+0000 7f0a97645b80 1 --2- 172.21.15.116:0/1084454563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f3b424b7d0 0x55f3b425a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.708+0000 7f0a66a00700 0 client.15939 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T01:00:51.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.708+0000 7f0a97645b80 1 --2- 172.21.15.116:0/1084454563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b424c130 0x55f3b425a940 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.708+0000 7f0a97645b80 1 -- 172.21.15.116:0/1084454563 >> 172.21.15.116:0/1084454563 conn(0x55f3b41aa6a0 msgr2=0x55f3b4254860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:51.716 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:00:51.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.708+0000 7f0a97645b80 1 -- 172.21.15.116:0/1084454563 shutdown_connections 2024-03-30T01:00:51.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.708+0000 7f0a97645b80 1 -- 172.21.15.116:0/1084454563 wait complete. 2024-03-30T01:00:51.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.709+0000 7f0a97645b80 20 client.15939 trim_cache size 0 max 16384 2024-03-30T01:00:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.710+0000 7f0a83c00700 1 -- 172.21.15.116:0/1228463951 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0a600a12a0 msgr2=0x7f0a600a3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.710+0000 7f0a83c00700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0a600a12a0 0x7f0a600a3760 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009d20 tx=0x7f0a680092a0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.710+0000 7f0a83c00700 1 -- 172.21.15.116:0/1228463951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a7c10d0f0 msgr2=0x7f0a7c1ad100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.710+0000 7f0a83c00700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a7c10d0f0 0x7f0a7c1ad100 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009810 tx=0x7f0a6c00b8d0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.711+0000 7f0a83c00700 1 -- 172.21.15.116:0/1228463951 shutdown_connections 2024-03-30T01:00:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.711+0000 7f0a83c00700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f0a600a12a0 0x7f0a600a3760 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.711+0000 7f0a83c00700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a7c107db0 0x7f0a7c1a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.711+0000 7f0a83c00700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a7c10d0f0 0x7f0a7c1ad100 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.711+0000 7f0a83c00700 1 --2- 172.21.15.116:0/1228463951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7c10a720 0x7f0a7c1a60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.711+0000 7f0a83c00700 1 -- 172.21.15.116:0/1228463951 >> 172.21.15.116:0/1228463951 conn(0x7f0a7c06dd50 msgr2=0x7f0a7c109370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:51.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.714+0000 7f0a83c00700 1 -- 172.21.15.116:0/1228463951 shutdown_connections 2024-03-30T01:00:51.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:51.714+0000 7f0a83c00700 1 -- 172.21.15.116:0/1228463951 wait complete. 2024-03-30T01:00:51.899 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T01:00:52.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:52 smithi130 ceph-mon[47626]: pgmap v381: 129 pgs: 129 active+clean; 416 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 242 op/s 2024-03-30T01:00:52.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:52 smithi116 ceph-mon[38086]: pgmap v381: 129 pgs: 129 active+clean; 416 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 242 op/s 2024-03-30T01:00:53.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:52 smithi181 ceph-mon[47622]: pgmap v381: 129 pgs: 129 active+clean; 416 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 242 op/s 2024-03-30T01:00:54.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:54 smithi130 ceph-mon[47626]: pgmap v382: 129 pgs: 129 active+clean; 416 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 219 op/s 2024-03-30T01:00:54.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:54 smithi116 ceph-mon[38086]: pgmap v382: 129 pgs: 129 active+clean; 416 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 219 op/s 2024-03-30T01:00:55.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:54 smithi181 ceph-mon[47622]: pgmap v382: 129 pgs: 129 active+clean; 416 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 219 op/s 2024-03-30T01:00:56.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.710+0000 7eff0b600700 1 -- 172.21.15.116:0/678050344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0410a720 msgr2=0x7eff0410cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:56.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.710+0000 7eff0b600700 1 --2- 172.21.15.116:0/678050344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0410a720 0x7eff0410cbb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7efef0009940 tx=0x7efef0009c50 comp rx=0 tx=0).stop 2024-03-30T01:00:56.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.747+0000 7eff0b600700 1 -- 172.21.15.116:0/678050344 shutdown_connections 2024-03-30T01:00:56.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.747+0000 7eff0b600700 1 --2- 172.21.15.116:0/678050344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff04107db0 0x7eff0410a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:56.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.747+0000 7eff0b600700 1 --2- 172.21.15.116:0/678050344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff0410d0f0 0x7eff0410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:56.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.747+0000 7eff0b600700 1 --2- 172.21.15.116:0/678050344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0410a720 0x7eff0410cbb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:56.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.747+0000 7eff0b600700 1 -- 172.21.15.116:0/678050344 >> 172.21.15.116:0/678050344 conn(0x7eff0406dd50 msgr2=0x7eff04070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:56.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.761+0000 7eff0b600700 1 -- 172.21.15.116:0/678050344 shutdown_connections 2024-03-30T01:00:56.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.761+0000 7eff0b600700 1 -- 172.21.15.116:0/678050344 wait complete. 2024-03-30T01:00:56.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.762+0000 7eff0b600700 1 Processor -- start 2024-03-30T01:00:56.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.784+0000 7eff0b600700 1 -- start start 2024-03-30T01:00:56.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.784+0000 7eff0b600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff04107db0 0x7eff0419d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:56.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.784+0000 7eff0b600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0410a720 0x7eff0419d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:56.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.784+0000 7eff0b600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff0410d0f0 0x7eff041a4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:56.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.784+0000 7eff0b600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff041a4f50 con 0x7eff0410a720 2024-03-30T01:00:56.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.784+0000 7eff0b600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff041a5090 con 0x7eff04107db0 2024-03-30T01:00:56.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.784+0000 7eff0b600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff041a5220 con 0x7eff0410d0f0 2024-03-30T01:00:56.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.785+0000 7eff09600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff0410d0f0 0x7eff041a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:56.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.785+0000 7eff09600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff0410d0f0 0x7eff041a4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59064/0 (socket says 172.21.15.116:59064) 2024-03-30T01:00:56.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.785+0000 7eff09600700 1 -- 172.21.15.116:0/137650709 learned_addr learned my addr 172.21.15.116:0/137650709 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:56.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.785+0000 7eff03e00700 1 --2- 172.21.15.116:0/137650709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0410a720 0x7eff0419d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.786+0000 7eff09600700 1 -- 172.21.15.116:0/137650709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff04107db0 msgr2=0x7eff0419d420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T01:00:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.786+0000 7eff09600700 1 --2- 172.21.15.116:0/137650709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff04107db0 0x7eff0419d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.786+0000 7eff09600700 1 -- 172.21.15.116:0/137650709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0410a720 msgr2=0x7eff0419d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.786+0000 7eff09600700 1 --2- 172.21.15.116:0/137650709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0410a720 0x7eff0419d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.786+0000 7eff09600700 1 -- 172.21.15.116:0/137650709 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef0009520 con 0x7eff0410d0f0 2024-03-30T01:00:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.786+0000 7eff03e00700 1 --2- 172.21.15.116:0/137650709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0410a720 0x7eff0419d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:00:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.790+0000 7eff09600700 1 --2- 172.21.15.116:0/137650709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff0410d0f0 0x7eff041a4980 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efefc01a530 tx=0x7efefc01e3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.790+0000 7eff01600700 1 -- 172.21.15.116:0/137650709 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc004db0 con 0x7eff0410d0f0 2024-03-30T01:00:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.790+0000 7eff01600700 1 -- 172.21.15.116:0/137650709 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efefc00b070 con 0x7eff0410d0f0 2024-03-30T01:00:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.790+0000 7eff01600700 1 -- 172.21.15.116:0/137650709 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc007cc0 con 0x7eff0410d0f0 2024-03-30T01:00:56.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.794+0000 7eff0b600700 1 -- 172.21.15.116:0/137650709 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff041a5500 con 0x7eff0410d0f0 2024-03-30T01:00:56.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.794+0000 7eff0b600700 1 -- 172.21.15.116:0/137650709 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff041a5990 con 0x7eff0410d0f0 2024-03-30T01:00:56.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:56 smithi130 ceph-mon[47626]: pgmap v383: 129 pgs: 129 active+clean; 422 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 206 op/s 2024-03-30T01:00:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.848+0000 7eff01600700 1 -- 172.21.15.116:0/137650709 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7efefc00d030 con 0x7eff0410d0f0 2024-03-30T01:00:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.855+0000 7eff01600700 1 --2- 172.21.15.116:0/137650709 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efee80a1320 0x7efee80a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.855+0000 7eff01600700 1 -- 172.21.15.116:0/137650709 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efefc0edaf0 con 0x7eff0410d0f0 2024-03-30T01:00:56.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.856+0000 7eff1f194b80 1 Processor -- start 2024-03-30T01:00:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.869+0000 7eff08c00700 1 --2- 172.21.15.116:0/137650709 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efee80a1320 0x7efee80a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.870+0000 7eff08c00700 1 --2- 172.21.15.116:0/137650709 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efee80a1320 0x7efee80a37e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7efef4009710 tx=0x7efef4006c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:56.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.880+0000 7eff1f194b80 1 -- start start 2024-03-30T01:00:56.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.880+0000 7eff1f194b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22b7d0 0x55c95a22bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:56.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.880+0000 7eff1f194b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22c130 0x55c95a22c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:56.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.880+0000 7eff1f194b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a22d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:56.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.881+0000 7eff1f194b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95a183990 con 0x55c95a22c130 2024-03-30T01:00:56.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.881+0000 7eff1f194b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95a181350 con 0x55c95a22cfd0 2024-03-30T01:00:56.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.881+0000 7eff1f194b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95a17cc40 con 0x55c95a22b7d0 2024-03-30T01:00:56.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.881+0000 7eff08c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22b7d0 0x55c95a22bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:56.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.881+0000 7eff08c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22b7d0 0x55c95a22bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59072/0 (socket says 172.21.15.116:59072) 2024-03-30T01:00:56.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.881+0000 7eff08c00700 1 -- 172.21.15.116:0/2299265954 learned_addr learned my addr 172.21.15.116:0/2299265954 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:56.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.881+0000 7eff09600700 1 --2- 172.21.15.116:0/2299265954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a22d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:56.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.882+0000 7eff08c00700 1 -- 172.21.15.116:0/2299265954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 msgr2=0x55c95a22d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:56.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.882+0000 7eff08c00700 1 --2- 172.21.15.116:0/2299265954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a22d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:56.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.882+0000 7eff08c00700 1 -- 172.21.15.116:0/2299265954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22c130 msgr2=0x55c95a22c9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:00:56.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.882+0000 7eff08c00700 1 --2- 172.21.15.116:0/2299265954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22c130 0x55c95a22c9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:56.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.882+0000 7eff08c00700 1 -- 172.21.15.116:0/2299265954 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c959ffe3b0 con 0x55c95a22b7d0 2024-03-30T01:00:56.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.884+0000 7eff08c00700 1 --2- 172.21.15.116:0/2299265954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22b7d0 0x55c95a22bbf0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efef4015920 tx=0x7efef4009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89e3c6f76aad139 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:56.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.886+0000 7efef9c00700 1 -- 172.21.15.116:0/2299265954 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef401a070 con 0x55c95a22b7d0 2024-03-30T01:00:56.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.886+0000 7efef9c00700 1 -- 172.21.15.116:0/2299265954 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efef4012a00 con 0x55c95a22b7d0 2024-03-30T01:00:56.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.886+0000 7efef9c00700 1 -- 172.21.15.116:0/2299265954 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef40138e0 con 0x55c95a22b7d0 2024-03-30T01:00:56.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.888+0000 7eff1f194b80 1 -- 172.21.15.116:0/2299265954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22b7d0 msgr2=0x55c95a22bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:56.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.888+0000 7eff1f194b80 1 --2- 172.21.15.116:0/2299265954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22b7d0 0x55c95a22bbf0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efef4015920 tx=0x7efef4009f90 comp rx=0 tx=0).stop 2024-03-30T01:00:56.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.909+0000 7eff1f194b80 1 -- 172.21.15.116:0/2299265954 shutdown_connections 2024-03-30T01:00:56.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.909+0000 7eff1f194b80 1 --2- 172.21.15.116:0/2299265954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22b7d0 0x55c95a22bbf0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:56.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.909+0000 7eff1f194b80 1 --2- 172.21.15.116:0/2299265954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a22d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:56.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.909+0000 7eff1f194b80 1 --2- 172.21.15.116:0/2299265954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22c130 0x55c95a22c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:56.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.909+0000 7eff1f194b80 1 -- 172.21.15.116:0/2299265954 >> 172.21.15.116:0/2299265954 conn(0x55c95a18a6a0 msgr2=0x55c95a2269a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:56.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:56 smithi116 ceph-mon[38086]: pgmap v383: 129 pgs: 129 active+clean; 422 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 206 op/s 2024-03-30T01:00:56.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.926+0000 7eff1f194b80 1 -- 172.21.15.116:0/2299265954 shutdown_connections 2024-03-30T01:00:56.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.926+0000 7eff1f194b80 1 -- 172.21.15.116:0/2299265954 wait complete. 2024-03-30T01:00:56.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.928+0000 7eff1f194b80 1 -- start start 2024-03-30T01:00:56.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.928+0000 7eff1f194b80 1 Processor -- start 2024-03-30T01:00:56.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.938+0000 7eff1f194b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a235760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:56.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.940+0000 7eff1f194b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 0x55c95a235ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:56.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.940+0000 7eff1f194b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a23cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:56.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.941+0000 7eff03e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 0x55c95a235ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:56.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.941+0000 7eff03e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 0x55c95a235ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59088/0 (socket says 172.21.15.116:59088) 2024-03-30T01:00:56.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.941+0000 7eff03e00700 1 -- 172.21.15.116:0/1415331514 learned_addr learned my addr 172.21.15.116:0/1415331514 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.941+0000 7eff1f194b80 1 -- 172.21.15.116:0/1415331514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95a118d80 con 0x55c95a22b7d0 2024-03-30T01:00:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.941+0000 7eff1f194b80 1 -- 172.21.15.116:0/1415331514 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95a118a20 con 0x55c95a22cfd0 2024-03-30T01:00:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.941+0000 7eff1f194b80 1 -- 172.21.15.116:0/1415331514 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95a0006a0 con 0x55c95a22c130 2024-03-30T01:00:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.941+0000 7eff1f194b80 10 client.0 _ll_register_callbacks cb 0x55c95a122470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.942+0000 7eff09600700 1 --2- 172.21.15.116:0/1415331514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a23cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.946+0000 7eff03e00700 1 -- 172.21.15.116:0/1415331514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 msgr2=0x55c95a23cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.946+0000 7eff03e00700 1 --2- 172.21.15.116:0/1415331514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a23cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.946+0000 7eff03e00700 1 -- 172.21.15.116:0/1415331514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 msgr2=0x55c95a235760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.946+0000 7eff03e00700 1 --2- 172.21.15.116:0/1415331514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a235760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.946+0000 7eff03e00700 1 -- 172.21.15.116:0/1415331514 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef4015660 con 0x55c95a22c130 2024-03-30T01:00:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.956+0000 7eff03e00700 1 --2- 172.21.15.116:0/1415331514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 0x55c95a235ca0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efef0009860 tx=0x7efef0005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92dae89e7852bb7f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.968+0000 7efeee000700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:00:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.968+0000 7efeee000700 1 -- 172.21.15.116:0/1415331514 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef000b9d0 con 0x55c95a22c130 2024-03-30T01:00:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.968+0000 7efeee000700 1 -- 172.21.15.116:0/1415331514 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efef00056a0 con 0x55c95a22c130 2024-03-30T01:00:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.992+0000 7eff1f194b80 1 -- 172.21.15.116:0/1415331514 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c95a13d380 con 0x55c95a22c130 2024-03-30T01:00:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.994+0000 7efeee000700 1 -- 172.21.15.116:0/1415331514 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7efef000bb30 con 0x55c95a22c130 2024-03-30T01:00:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.996+0000 7eff1f194b80 10 client.26002 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.996+0000 7eff1f194b80 1 -- 172.21.15.116:0/1415331514 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c959faab60 con 0x55c95a22c130 2024-03-30T01:00:57.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:56 smithi181 ceph-mon[47622]: pgmap v383: 129 pgs: 129 active+clean; 422 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 206 op/s 2024-03-30T01:00:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.997+0000 7efeee000700 1 -- 172.21.15.116:0/1415331514 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c95a13d380 con 0x55c95a22c130 2024-03-30T01:00:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.997+0000 7efeee000700 1 -- 172.21.15.116:0/1415331514 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7efef002b070 con 0x55c95a22c130 2024-03-30T01:00:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.997+0000 7eff1f194b80 10 client.26002 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.997+0000 7eff1f194b80 10 client.26002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:00:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.997+0000 7eff1f194b80 20 client.26002 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.997+0000 7eff1f194b80 1 --2- 172.21.15.116:0/1415331514 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c95a241480 0x55c95a2438a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.997+0000 7eff1f194b80 4 client.26002 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:00:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.997+0000 7eff1f194b80 1 -- 172.21.15.116:0/1415331514 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c95a175700 con 0x55c95a241480 2024-03-30T01:00:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:56.998+0000 7eff08c00700 1 --2- 172.21.15.116:0/1415331514 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c95a241480 0x55c95a2438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:57.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.000+0000 7eff08c00700 1 --2- 172.21.15.116:0/1415331514 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c95a241480 0x55c95a2438a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9a0e91abfcfa287 server_cookie=bd169be83a53d9db in_seq=0 out_seq=0 2024-03-30T01:00:57.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.001+0000 7efeee000700 10 client.26002 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.002+0000 7efeee000700 1 -- 172.21.15.116:0/1415331514 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c95a175700 con 0x55c95a241480 2024-03-30T01:00:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.002+0000 7efeee000700 10 client.26002 handle_command_reply: tid=0 2024-03-30T01:00:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.002+0000 7eff1f194b80 1 client.26002 shutdown 2024-03-30T01:00:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.007+0000 7eff1f194b80 1 -- 172.21.15.116:0/1415331514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 msgr2=0x55c95a235ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.007+0000 7eff1f194b80 1 --2- 172.21.15.116:0/1415331514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 0x55c95a235ca0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efef0009860 tx=0x7efef0005270 comp rx=0 tx=0).stop 2024-03-30T01:00:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.007+0000 7eff1f194b80 1 -- 172.21.15.116:0/1415331514 shutdown_connections 2024-03-30T01:00:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.007+0000 7eff1f194b80 1 --2- 172.21.15.116:0/1415331514 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c95a241480 0x55c95a2438a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.007+0000 7eff1f194b80 1 --2- 172.21.15.116:0/1415331514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a23cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.007+0000 7eff1f194b80 1 --2- 172.21.15.116:0/1415331514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a235760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.007+0000 7eff1f194b80 1 --2- 172.21.15.116:0/1415331514 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 0x55c95a235ca0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.007+0000 7eff1f194b80 1 -- 172.21.15.116:0/1415331514 >> 172.21.15.116:0/1415331514 conn(0x55c95a18a6a0 msgr2=0x55c95a22a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.007+0000 7efeee000700 0 client.26002 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.023+0000 7eff1f194b80 1 -- 172.21.15.116:0/1415331514 shutdown_connections 2024-03-30T01:00:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.024+0000 7eff1f194b80 1 -- 172.21.15.116:0/1415331514 wait complete. 2024-03-30T01:00:57.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.032+0000 7eff1f194b80 20 client.26002 trim_cache size 0 max 16384 2024-03-30T01:00:57.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.340+0000 7eff1f194b80 1 Processor -- start 2024-03-30T01:00:57.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.340+0000 7eff1f194b80 1 -- start start 2024-03-30T01:00:57.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.341+0000 7eff1f194b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a22bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:57.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.341+0000 7eff1f194b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 0x55c95a23cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:57.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.341+0000 7eff1f194b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a21f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:57.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.341+0000 7eff1f194b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95a222810 con 0x55c95a22b7d0 2024-03-30T01:00:57.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.341+0000 7eff1f194b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95a2278b0 con 0x55c95a22cfd0 2024-03-30T01:00:57.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.341+0000 7eff1f194b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95a226e90 con 0x55c95a22c130 2024-03-30T01:00:57.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.342+0000 7eff08c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a22bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:57.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.342+0000 7eff08c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a22bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34612/0 (socket says 172.21.15.116:34612) 2024-03-30T01:00:57.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.342+0000 7eff08c00700 1 -- 172.21.15.116:0/2394706806 learned_addr learned my addr 172.21.15.116:0/2394706806 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:57.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.342+0000 7eff08c00700 1 -- 172.21.15.116:0/2394706806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 msgr2=0x55c95a23cc00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:00:57.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.342+0000 7eff08c00700 1 --2- 172.21.15.116:0/2394706806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 0x55c95a23cc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.342+0000 7eff08c00700 1 -- 172.21.15.116:0/2394706806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 msgr2=0x55c95a21f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:57.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.342+0000 7eff08c00700 1 --2- 172.21.15.116:0/2394706806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a21f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.342+0000 7eff08c00700 1 -- 172.21.15.116:0/2394706806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c95a142ea0 con 0x55c95a22b7d0 2024-03-30T01:00:57.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.343+0000 7eff08c00700 1 --2- 172.21.15.116:0/2394706806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a22bbb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7efef4015410 tx=0x7efef4013490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2a23b7f2759f422 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.354+0000 7efefa600700 1 -- 172.21.15.116:0/2394706806 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef401a040 con 0x55c95a22b7d0 2024-03-30T01:00:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.354+0000 7efefa600700 1 -- 172.21.15.116:0/2394706806 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efef4050030 con 0x55c95a22b7d0 2024-03-30T01:00:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.354+0000 7efefa600700 1 -- 172.21.15.116:0/2394706806 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef402a330 con 0x55c95a22b7d0 2024-03-30T01:00:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.358+0000 7eff1f194b80 1 -- 172.21.15.116:0/2394706806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 msgr2=0x55c95a22bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.358+0000 7eff1f194b80 1 --2- 172.21.15.116:0/2394706806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a22bbb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7efef4015410 tx=0x7efef4013490 comp rx=0 tx=0).stop 2024-03-30T01:00:57.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.372+0000 7eff1f194b80 1 -- 172.21.15.116:0/2394706806 shutdown_connections 2024-03-30T01:00:57.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.372+0000 7eff1f194b80 1 --2- 172.21.15.116:0/2394706806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a21f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.372+0000 7eff1f194b80 1 --2- 172.21.15.116:0/2394706806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a22bbb0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.372+0000 7eff1f194b80 1 --2- 172.21.15.116:0/2394706806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 0x55c95a23cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.372+0000 7eff1f194b80 1 -- 172.21.15.116:0/2394706806 >> 172.21.15.116:0/2394706806 conn(0x55c95a18a6a0 msgr2=0x55c95a233600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:57.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.374+0000 7eff1f194b80 1 -- 172.21.15.116:0/2394706806 shutdown_connections 2024-03-30T01:00:57.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.378+0000 7eff1f194b80 1 -- 172.21.15.116:0/2394706806 wait complete. 2024-03-30T01:00:57.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.378+0000 7eff1f194b80 1 -- start start 2024-03-30T01:00:57.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.379+0000 7eff1f194b80 1 Processor -- start 2024-03-30T01:00:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.381+0000 7eff1f194b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a23aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:57.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.386+0000 7eff08c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a23aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:57.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.387+0000 7eff08c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a23aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34622/0 (socket says 172.21.15.116:34622) 2024-03-30T01:00:57.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.391+0000 7eff1f194b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 0x55c95a23af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:57.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.391+0000 7eff1f194b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a23bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:57.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.391+0000 7eff1f194b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95a23c4e0 con 0x55c95a22b7d0 2024-03-30T01:00:57.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.391+0000 7eff1f194b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95a23c650 con 0x55c95a22cfd0 2024-03-30T01:00:57.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.391+0000 7eff1f194b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c95a23c810 con 0x55c95a22c130 2024-03-30T01:00:57.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.391+0000 7eff1f194b80 10 client.0 _ll_register_callbacks cb 0x55c95a122470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:00:57.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.392+0000 7eff03e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 0x55c95a23af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:57.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.392+0000 7eff03e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 0x55c95a23af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59104/0 (socket says 172.21.15.116:59104) 2024-03-30T01:00:57.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.392+0000 7eff03e00700 1 -- 172.21.15.116:0/973305946 learned_addr learned my addr 172.21.15.116:0/973305946 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:00:57.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.392+0000 7eff09600700 1 --2- 172.21.15.116:0/973305946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a23bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.398+0000 7eff09600700 1 -- 172.21.15.116:0/973305946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 msgr2=0x55c95a23af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.398+0000 7eff09600700 1 --2- 172.21.15.116:0/973305946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 0x55c95a23af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.398+0000 7eff09600700 1 -- 172.21.15.116:0/973305946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 msgr2=0x55c95a23aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:00:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.398+0000 7eff09600700 1 --2- 172.21.15.116:0/973305946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a23aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.398+0000 7eff09600700 1 -- 172.21.15.116:0/973305946 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c959ffe3b0 con 0x55c95a22cfd0 2024-03-30T01:00:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.398+0000 7eff08c00700 1 --2- 172.21.15.116:0/973305946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a23aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-30T01:00:57.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.404+0000 7eff09600700 1 --2- 172.21.15.116:0/973305946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a23bfa0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efefc0b70d0 tx=0x7efefc004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7680ea77b940f07d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:00:57.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.404+0000 7efeee000700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T01:00:57.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.404+0000 7efeee000700 1 -- 172.21.15.116:0/973305946 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc0bb030 con 0x55c95a22cfd0 2024-03-30T01:00:57.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.404+0000 7efeee000700 1 -- 172.21.15.116:0/973305946 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efefc0babd0 con 0x55c95a22cfd0 2024-03-30T01:00:57.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.411+0000 7eff1f194b80 1 -- 172.21.15.116:0/973305946 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efef4012a00 con 0x55c95a22cfd0 2024-03-30T01:00:57.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.411+0000 7efeee000700 1 -- 172.21.15.116:0/973305946 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7efefc0bea00 con 0x55c95a22cfd0 2024-03-30T01:00:57.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.411+0000 7eff1f194b80 10 client.26000 fetch_fsmap learned FSMap version 25 2024-03-30T01:00:57.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.411+0000 7eff1f194b80 1 -- 172.21.15.116:0/973305946 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c95a002cd0 con 0x55c95a22cfd0 2024-03-30T01:00:57.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.415+0000 7efeee000700 1 -- 172.21.15.116:0/973305946 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efef4012a00 con 0x55c95a22cfd0 2024-03-30T01:00:57.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.415+0000 7efeee000700 1 -- 172.21.15.116:0/973305946 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7efefc0c2bc0 con 0x55c95a22cfd0 2024-03-30T01:00:57.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.421+0000 7eff1f194b80 10 client.26000 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:00:57.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.421+0000 7eff1f194b80 10 client.26000 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:00:57.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.421+0000 7eff1f194b80 20 client.26000 populate_metadata read hostname 'smithi116' 2024-03-30T01:00:57.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.421+0000 7eff1f194b80 1 --2- 172.21.15.116:0/973305946 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c95a296380 0x55c95a2987a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:00:57.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.421+0000 7eff1f194b80 4 client.26000 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:00:57.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.421+0000 7eff1f194b80 1 -- 172.21.15.116:0/973305946 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c959fcbcf0 con 0x55c95a296380 2024-03-30T01:00:57.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.425+0000 7eff08c00700 1 --2- 172.21.15.116:0/973305946 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c95a296380 0x55c95a2987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:00:57.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.435+0000 7eff08c00700 1 --2- 172.21.15.116:0/973305946 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c95a296380 0x55c95a2987a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79db3cfe6997a24e server_cookie=addcdc0fc6764a82 in_seq=0 out_seq=0 2024-03-30T01:00:57.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.436+0000 7efeee000700 10 client.26000 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:57.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.436+0000 7efeee000700 1 -- 172.21.15.116:0/973305946 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c959fcbcf0 con 0x55c95a296380 2024-03-30T01:00:57.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.436+0000 7efeee000700 10 client.26000 handle_command_reply: tid=0 2024-03-30T01:00:57.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.436+0000 7eff1f194b80 1 client.26000 shutdown 2024-03-30T01:00:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.440+0000 7eff1f194b80 1 -- 172.21.15.116:0/973305946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 msgr2=0x55c95a23bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.440+0000 7eff1f194b80 1 --2- 172.21.15.116:0/973305946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a23bfa0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efefc0b70d0 tx=0x7efefc004ad0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.441+0000 7eff1f194b80 1 -- 172.21.15.116:0/973305946 shutdown_connections 2024-03-30T01:00:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.441+0000 7eff1f194b80 1 --2- 172.21.15.116:0/973305946 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55c95a296380 0x55c95a2987a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.441+0000 7eff1f194b80 1 --2- 172.21.15.116:0/973305946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c95a22cfd0 0x55c95a23bfa0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.441+0000 7eff1f194b80 1 --2- 172.21.15.116:0/973305946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c95a22b7d0 0x55c95a23aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.441+0000 7eff1f194b80 1 --2- 172.21.15.116:0/973305946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c95a22c130 0x55c95a23af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.441+0000 7eff1f194b80 1 -- 172.21.15.116:0/973305946 >> 172.21.15.116:0/973305946 conn(0x55c95a18a6a0 msgr2=0x55c95a234bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.441+0000 7efeee000700 0 client.26000 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:00:57.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.451+0000 7eff1f194b80 1 -- 172.21.15.116:0/973305946 shutdown_connections 2024-03-30T01:00:57.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.451+0000 7eff1f194b80 1 -- 172.21.15.116:0/973305946 wait complete. 2024-03-30T01:00:57.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.470+0000 7eff1f194b80 20 client.26000 trim_cache size 0 max 16384 2024-03-30T01:00:57.477 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:00:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.497+0000 7eff0b600700 1 -- 172.21.15.116:0/137650709 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efee80a1320 msgr2=0x7efee80a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.497+0000 7eff0b600700 1 --2- 172.21.15.116:0/137650709 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efee80a1320 0x7efee80a37e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7efef4009710 tx=0x7efef4006c60 comp rx=0 tx=0).stop 2024-03-30T01:00:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.497+0000 7eff0b600700 1 -- 172.21.15.116:0/137650709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff0410d0f0 msgr2=0x7eff041a4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:00:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.497+0000 7eff0b600700 1 --2- 172.21.15.116:0/137650709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff0410d0f0 0x7eff041a4980 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efefc01a530 tx=0x7efefc01e3b0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.501+0000 7eff0b600700 1 -- 172.21.15.116:0/137650709 shutdown_connections 2024-03-30T01:00:57.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.501+0000 7eff0b600700 1 --2- 172.21.15.116:0/137650709 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efee80a1320 0x7efee80a37e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.501+0000 7eff0b600700 1 --2- 172.21.15.116:0/137650709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff0410d0f0 0x7eff041a4980 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.501+0000 7eff0b600700 1 --2- 172.21.15.116:0/137650709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff04107db0 0x7eff0419d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.501+0000 7eff0b600700 1 --2- 172.21.15.116:0/137650709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0410a720 0x7eff0419d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:00:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.501+0000 7eff0b600700 1 -- 172.21.15.116:0/137650709 >> 172.21.15.116:0/137650709 conn(0x7eff0406dd50 msgr2=0x7eff0410afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:00:57.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.507+0000 7eff0b600700 1 -- 172.21.15.116:0/137650709 shutdown_connections 2024-03-30T01:00:57.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:00:57.509+0000 7eff0b600700 1 -- 172.21.15.116:0/137650709 wait complete. 2024-03-30T01:00:58.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:00:58 smithi130 ceph-mon[47626]: pgmap v384: 129 pgs: 129 active+clean; 424 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 210 op/s 2024-03-30T01:00:58.890 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T01:00:58.890 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T01:00:58.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:00:58 smithi116 ceph-mon[38086]: pgmap v384: 129 pgs: 129 active+clean; 424 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 210 op/s 2024-03-30T01:00:58.963 INFO:tasks.workunit.client.0.smithi116.stdout: 145 658656 870 457893 1203 234076 1556 2024-03-30T01:00:59.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:00:58 smithi181 ceph-mon[47622]: pgmap v384: 129 pgs: 129 active+clean; 424 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 210 op/s 2024-03-30T01:01:00.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.430+0000 7f687d400700 1 --2- 172.21.15.116:0/4099791149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687810ce40 0x7f6878117cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:01:00.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.432+0000 7f687de00700 1 -- 172.21.15.116:0/4099791149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f687810a390 msgr2=0x7f687810c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:00.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.432+0000 7f687de00700 1 --2- 172.21.15.116:0/4099791149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f687810a390 0x7f687810c820 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f68640098f0 tx=0x7f6864009c00 comp rx=0 tx=0).stop 2024-03-30T01:01:00.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.433+0000 7f687de00700 1 -- 172.21.15.116:0/4099791149 shutdown_connections 2024-03-30T01:01:00.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.433+0000 7f687de00700 1 --2- 172.21.15.116:0/4099791149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687810ce40 0x7f6878117cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.433+0000 7f687de00700 1 --2- 172.21.15.116:0/4099791149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6878072b80 0x7f6878109dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.433+0000 7f687de00700 1 --2- 172.21.15.116:0/4099791149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f687810a390 0x7f687810c820 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.433+0000 7f687de00700 1 -- 172.21.15.116:0/4099791149 >> 172.21.15.116:0/4099791149 conn(0x7f687806da50 msgr2=0x7f687806fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:00.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.438+0000 7f687de00700 1 -- 172.21.15.116:0/4099791149 shutdown_connections 2024-03-30T01:01:00.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.438+0000 7f687de00700 1 -- 172.21.15.116:0/4099791149 wait complete. 2024-03-30T01:01:00.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.440+0000 7f687de00700 1 Processor -- start 2024-03-30T01:01:00.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.441+0000 7f687de00700 1 -- start start 2024-03-30T01:01:00.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.441+0000 7f687de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6878072b80 0x7f68781176c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.442+0000 7f687de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687810a390 0x7f6878114180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.442+0000 7f687de00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f687810ce40 0x7f6878114720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.442+0000 7f687de00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68781b5b60 con 0x7f687810a390 2024-03-30T01:01:00.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.442+0000 7f687de00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68781b5ca0 con 0x7f6878072b80 2024-03-30T01:01:00.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.442+0000 7f687de00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68781b5de0 con 0x7f687810ce40 2024-03-30T01:01:00.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.442+0000 7f687ca00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6878072b80 0x7f68781176c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:00.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.443+0000 7f687ca00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6878072b80 0x7f68781176c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:39436/0 (socket says 172.21.15.116:39436) 2024-03-30T01:01:00.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.443+0000 7f687ca00700 1 -- 172.21.15.116:0/2206857064 learned_addr learned my addr 172.21.15.116:0/2206857064 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:00.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.443+0000 7f687d400700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f687810ce40 0x7f6878114720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:00.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.444+0000 7f687ca00700 1 -- 172.21.15.116:0/2206857064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f687810ce40 msgr2=0x7f6878114720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:00.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.444+0000 7f687ca00700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f687810ce40 0x7f6878114720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.444+0000 7f687ca00700 1 -- 172.21.15.116:0/2206857064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687810a390 msgr2=0x7f6878114180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:00.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.444+0000 7f6877e00700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687810a390 0x7f6878114180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:00.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.444+0000 7f687ca00700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687810a390 0x7f6878114180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.444+0000 7f687ca00700 1 -- 172.21.15.116:0/2206857064 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6864009520 con 0x7f6878072b80 2024-03-30T01:01:00.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.444+0000 7f6877e00700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687810a390 0x7f6878114180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:01:00.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.444+0000 7f687ca00700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6878072b80 0x7f68781176c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f686800d950 tx=0x7f686800dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:00.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.445+0000 7f6875600700 1 -- 172.21.15.116:0/2206857064 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6868009b40 con 0x7f6878072b80 2024-03-30T01:01:00.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.445+0000 7f6875600700 1 -- 172.21.15.116:0/2206857064 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6868009ca0 con 0x7f6878072b80 2024-03-30T01:01:00.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.445+0000 7f6875600700 1 -- 172.21.15.116:0/2206857064 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686800f620 con 0x7f6878072b80 2024-03-30T01:01:00.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.445+0000 7f687de00700 1 -- 172.21.15.116:0/2206857064 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68781b60c0 con 0x7f6878072b80 2024-03-30T01:01:00.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.446+0000 7f687de00700 1 -- 172.21.15.116:0/2206857064 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68781b6580 con 0x7f6878072b80 2024-03-30T01:01:00.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.462+0000 7f6875600700 1 -- 172.21.15.116:0/2206857064 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f6868010460 con 0x7f6878072b80 2024-03-30T01:01:00.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.464+0000 7f6875600700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f685c0a1270 0x7f685c0a3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.465+0000 7f6892e23b80 1 Processor -- start 2024-03-30T01:01:00.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.469+0000 7f6877e00700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f685c0a1270 0x7f685c0a3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.470+0000 7f6892e23b80 1 -- start start 2024-03-30T01:01:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.471+0000 7f6892e23b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f6157ffa60 0x55f6157ffe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.471+0000 7f6892e23b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f6158003c0 0x55f615800c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.471+0000 7f6892e23b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f615801210 0x55f615801690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.471+0000 7f6877e00700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f685c0a1270 0x7f685c0a3730 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6864009230 tx=0x7f6864010d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.472+0000 7f687d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f615801210 0x55f615801690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.472+0000 7f687d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f615801210 0x55f615801690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34636/0 (socket says 172.21.15.116:34636) 2024-03-30T01:01:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.472+0000 7f6892e23b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f615757a10 con 0x55f615801210 2024-03-30T01:01:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.472+0000 7f6892e23b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6157553d0 con 0x55f6158003c0 2024-03-30T01:01:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.472+0000 7f6892e23b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f615750cc0 con 0x55f6157ffa60 2024-03-30T01:01:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.473+0000 7f6875600700 1 -- 172.21.15.116:0/2206857064 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6868099ef0 con 0x7f6878072b80 2024-03-30T01:01:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.473+0000 7f6877e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f6158003c0 0x55f615800c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.473+0000 7f6877e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f6158003c0 0x55f615800c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:39438/0 (socket says 172.21.15.116:39438) 2024-03-30T01:01:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.473+0000 7f6877e00700 1 -- 172.21.15.116:0/3174418150 learned_addr learned my addr 172.21.15.116:0/3174418150 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.474+0000 7f687d400700 1 -- 172.21.15.116:0/3174418150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f6157ffa60 msgr2=0x55f6157ffe80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:01:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.474+0000 7f687d400700 1 --2- 172.21.15.116:0/3174418150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f6157ffa60 0x55f6157ffe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.474+0000 7f687d400700 1 -- 172.21.15.116:0/3174418150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f6158003c0 msgr2=0x55f615800c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:01:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.474+0000 7f687d400700 1 --2- 172.21.15.116:0/3174418150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f6158003c0 0x55f615800c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.474+0000 7f687d400700 1 -- 172.21.15.116:0/3174418150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f6155d2430 con 0x55f615801210 2024-03-30T01:01:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.474+0000 7f687d400700 1 --2- 172.21.15.116:0/3174418150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f615801210 0x55f615801690 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6870009d10 tx=0x7f68700078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae8474b63a30b3c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.487+0000 7f686dc00700 1 -- 172.21.15.116:0/3174418150 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6870004bb0 con 0x55f615801210 2024-03-30T01:01:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.488+0000 7f686dc00700 1 -- 172.21.15.116:0/3174418150 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6870004d10 con 0x55f615801210 2024-03-30T01:01:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.488+0000 7f686dc00700 1 -- 172.21.15.116:0/3174418150 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6870014660 con 0x55f615801210 2024-03-30T01:01:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.491+0000 7f6892e23b80 1 -- 172.21.15.116:0/3174418150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f615801210 msgr2=0x55f615801690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.491+0000 7f6892e23b80 1 --2- 172.21.15.116:0/3174418150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f615801210 0x55f615801690 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6870009d10 tx=0x7f68700078a0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.510+0000 7f6892e23b80 1 -- 172.21.15.116:0/3174418150 shutdown_connections 2024-03-30T01:01:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.510+0000 7f6892e23b80 1 --2- 172.21.15.116:0/3174418150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f6157ffa60 0x55f6157ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.510+0000 7f6892e23b80 1 --2- 172.21.15.116:0/3174418150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f615801210 0x55f615801690 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.510+0000 7f6892e23b80 1 --2- 172.21.15.116:0/3174418150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f6158003c0 0x55f615800c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.510+0000 7f6892e23b80 1 -- 172.21.15.116:0/3174418150 >> 172.21.15.116:0/3174418150 conn(0x55f61575e720 msgr2=0x55f6157fab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:00.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.528+0000 7f6892e23b80 1 -- 172.21.15.116:0/3174418150 shutdown_connections 2024-03-30T01:01:00.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.532+0000 7f6892e23b80 1 -- 172.21.15.116:0/3174418150 wait complete. 2024-03-30T01:01:00.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.532+0000 7f6892e23b80 1 -- start start 2024-03-30T01:01:00.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.533+0000 7f6892e23b80 1 Processor -- start 2024-03-30T01:01:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.538+0000 7f6892e23b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6157ffa60 0x55f615809980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.545+0000 7f6892e23b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f6158003c0 0x55f615809ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.545+0000 7f6892e23b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f615801210 0x55f615810f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.545+0000 7f6892e23b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6156ece00 con 0x55f6157ffa60 2024-03-30T01:01:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.545+0000 7f6892e23b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6156ecaa0 con 0x55f6158003c0 2024-03-30T01:01:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.545+0000 7f6892e23b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6155d4720 con 0x55f615801210 2024-03-30T01:01:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.546+0000 7f687ca00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6157ffa60 0x55f615809980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.546+0000 7f687ca00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6157ffa60 0x55f615809980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34646/0 (socket says 172.21.15.116:34646) 2024-03-30T01:01:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.546+0000 7f687ca00700 1 -- 172.21.15.116:0/3171747062 learned_addr learned my addr 172.21.15.116:0/3171747062 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.546+0000 7f6892e23b80 10 client.0 _ll_register_callbacks cb 0x55f6156f64f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:01:00.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.547+0000 7f687ca00700 1 -- 172.21.15.116:0/3171747062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f615801210 msgr2=0x55f615810f00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:01:00.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.547+0000 7f687ca00700 1 --2- 172.21.15.116:0/3171747062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f615801210 0x55f615810f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.547+0000 7f687ca00700 1 -- 172.21.15.116:0/3171747062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f6158003c0 msgr2=0x55f615809ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:01:00.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.547+0000 7f687ca00700 1 --2- 172.21.15.116:0/3171747062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f6158003c0 0x55f615809ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.547+0000 7f687ca00700 1 -- 172.21.15.116:0/3171747062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6870009430 con 0x55f6157ffa60 2024-03-30T01:01:00.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.548+0000 7f687ca00700 1 --2- 172.21.15.116:0/3171747062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6157ffa60 0x55f615809980 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f68680a1740 tx=0x7f6868094390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eeb6252835de75 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:00.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.548+0000 7f6862000700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:01:00.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.548+0000 7f6862000700 1 -- 172.21.15.116:0/3171747062 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6868095b70 con 0x55f6157ffa60 2024-03-30T01:01:00.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.548+0000 7f6862000700 1 -- 172.21.15.116:0/3171747062 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f68680a1c70 con 0x55f6157ffa60 2024-03-30T01:01:00.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.554+0000 7f6892e23b80 1 -- 172.21.15.116:0/3171747062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f615711400 con 0x55f6157ffa60 2024-03-30T01:01:00.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.561+0000 7f6862000700 1 -- 172.21.15.116:0/3171747062 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6868099150 con 0x55f6157ffa60 2024-03-30T01:01:00.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.561+0000 7f6892e23b80 10 client.15957 fetch_fsmap learned FSMap version 25 2024-03-30T01:01:00.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.561+0000 7f6892e23b80 1 -- 172.21.15.116:0/3171747062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f61557ebe0 con 0x55f6157ffa60 2024-03-30T01:01:00.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.579+0000 7f6862000700 1 -- 172.21.15.116:0/3171747062 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f615711400 con 0x55f6157ffa60 2024-03-30T01:01:00.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.579+0000 7f6862000700 1 -- 172.21.15.116:0/3171747062 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f6868098db0 con 0x55f6157ffa60 2024-03-30T01:01:00.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.579+0000 7f6892e23b80 10 client.15957 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:01:00.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.579+0000 7f6892e23b80 10 client.15957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:01:00.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.579+0000 7f6892e23b80 20 client.15957 populate_metadata read hostname 'smithi116' 2024-03-30T01:01:00.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.579+0000 7f6892e23b80 1 --2- 172.21.15.116:0/3171747062 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f6158156f0 0x55f615817b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.579+0000 7f6892e23b80 4 client.15957 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:01:00.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.579+0000 7f6892e23b80 1 -- 172.21.15.116:0/3171747062 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f615749780 con 0x55f6158156f0 2024-03-30T01:01:00.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.588+0000 7f6877e00700 1 --2- 172.21.15.116:0/3171747062 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f6158156f0 0x55f615817b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:00.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.606+0000 7f6877e00700 1 --2- 172.21.15.116:0/3171747062 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f6158156f0 0x55f615817b10 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccdb68537dd24542 server_cookie=7dfbe86852c0712e in_seq=0 out_seq=0 2024-03-30T01:01:00.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.606+0000 7f6862000700 10 client.15957 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:01:00.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.609+0000 7f6862000700 1 -- 172.21.15.116:0/3171747062 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f615749780 con 0x55f6158156f0 2024-03-30T01:01:00.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.609+0000 7f6862000700 10 client.15957 handle_command_reply: tid=0 2024-03-30T01:01:00.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.609+0000 7f6892e23b80 1 client.15957 shutdown 2024-03-30T01:01:00.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.610+0000 7f6892e23b80 1 -- 172.21.15.116:0/3171747062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6157ffa60 msgr2=0x55f615809980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:00.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.610+0000 7f6892e23b80 1 --2- 172.21.15.116:0/3171747062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6157ffa60 0x55f615809980 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f68680a1740 tx=0x7f6868094390 comp rx=0 tx=0).stop 2024-03-30T01:01:00.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.610+0000 7f6892e23b80 1 -- 172.21.15.116:0/3171747062 shutdown_connections 2024-03-30T01:01:00.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.610+0000 7f6892e23b80 1 --2- 172.21.15.116:0/3171747062 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f6158156f0 0x55f615817b10 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.610+0000 7f6892e23b80 1 --2- 172.21.15.116:0/3171747062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f615801210 0x55f615810f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.610+0000 7f6892e23b80 1 --2- 172.21.15.116:0/3171747062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6157ffa60 0x55f615809980 secure :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f68680a1740 tx=0x7f6868094390 comp rx=0 tx=0).stop 2024-03-30T01:01:00.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.610+0000 7f6892e23b80 1 --2- 172.21.15.116:0/3171747062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f6158003c0 0x55f615809ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.610+0000 7f6892e23b80 1 -- 172.21.15.116:0/3171747062 >> 172.21.15.116:0/3171747062 conn(0x55f61575e720 msgr2=0x55f6157fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:00.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.610+0000 7f6862000700 0 client.15957 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:01:00.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.620+0000 7f6892e23b80 1 -- 172.21.15.116:0/3171747062 shutdown_connections 2024-03-30T01:01:00.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.620+0000 7f6892e23b80 1 -- 172.21.15.116:0/3171747062 wait complete. 2024-03-30T01:01:00.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.621+0000 7f6892e23b80 20 client.15957 trim_cache size 0 max 16384 2024-03-30T01:01:00.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.805+0000 7f6892e23b80 1 Processor -- start 2024-03-30T01:01:00.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.805+0000 7f6892e23b80 1 -- start start 2024-03-30T01:01:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.814+0000 7f6892e23b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6158003c0 0x55f615801610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.816+0000 7f687ca00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6158003c0 0x55f615801610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.817+0000 7f687ca00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6158003c0 0x55f615801610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34656/0 (socket says 172.21.15.116:34656) 2024-03-30T01:01:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.817+0000 7f687ca00700 1 -- 172.21.15.116:0/1462949070 learned_addr learned my addr 172.21.15.116:0/1462949070 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.823+0000 7f6892e23b80 1 --2- 172.21.15.116:0/1462949070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f615801210 0x55f615810e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.823+0000 7f6892e23b80 1 --2- 172.21.15.116:0/1462949070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f615807770 0x55f615804420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.823+0000 7f6892e23b80 1 -- 172.21.15.116:0/1462949070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f615749780 con 0x55f6158003c0 2024-03-30T01:01:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.823+0000 7f6892e23b80 1 -- 172.21.15.116:0/1462949070 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f615711400 con 0x55f615807770 2024-03-30T01:01:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.823+0000 7f6892e23b80 1 -- 172.21.15.116:0/1462949070 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6157fb8b0 con 0x55f615801210 2024-03-30T01:01:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.823+0000 7f6877e00700 1 --2- 172.21.15.116:0/1462949070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f615801210 0x55f615810e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.823+0000 7f6877e00700 1 -- 172.21.15.116:0/1462949070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f615807770 msgr2=0x55f615804420 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:01:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.823+0000 7f6877e00700 1 --2- 172.21.15.116:0/1462949070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f615807770 0x55f615804420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.823+0000 7f6877e00700 1 -- 172.21.15.116:0/1462949070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6158003c0 msgr2=0x55f615801610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:01:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.824+0000 7f6877e00700 1 --2- 172.21.15.116:0/1462949070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6158003c0 0x55f615801610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.824+0000 7f6877e00700 1 -- 172.21.15.116:0/1462949070 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f61581c740 con 0x55f615801210 2024-03-30T01:01:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.824+0000 7f6877e00700 1 --2- 172.21.15.116:0/1462949070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f615801210 0x55f615810e90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x55f6158180f0 tx=0x7f686401d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f32d96529f239e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.824+0000 7f686e600700 1 -- 172.21.15.116:0/1462949070 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686402b040 con 0x55f615801210 2024-03-30T01:01:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.824+0000 7f686e600700 1 -- 172.21.15.116:0/1462949070 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6864030dc0 con 0x55f615801210 2024-03-30T01:01:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.824+0000 7f686e600700 1 -- 172.21.15.116:0/1462949070 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686401d070 con 0x55f615801210 2024-03-30T01:01:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.825+0000 7f6892e23b80 1 -- 172.21.15.116:0/1462949070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f615801210 msgr2=0x55f615810e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.825+0000 7f6892e23b80 1 --2- 172.21.15.116:0/1462949070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f615801210 0x55f615810e90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x55f6158180f0 tx=0x7f686401d000 comp rx=0 tx=0).stop 2024-03-30T01:01:00.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:00 smithi130 ceph-mon[47626]: pgmap v385: 129 pgs: 129 active+clean; 428 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 204 op/s 2024-03-30T01:01:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.854+0000 7f687ca00700 1 --2- 172.21.15.116:0/1462949070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6158003c0 0x55f615801610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:01:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.854+0000 7f6892e23b80 1 -- 172.21.15.116:0/1462949070 shutdown_connections 2024-03-30T01:01:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.854+0000 7f6892e23b80 1 --2- 172.21.15.116:0/1462949070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f615807770 0x55f615804420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.854+0000 7f6892e23b80 1 --2- 172.21.15.116:0/1462949070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6158003c0 0x55f615801610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.854+0000 7f6892e23b80 1 --2- 172.21.15.116:0/1462949070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f615801210 0x55f615810e90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.854+0000 7f6892e23b80 1 -- 172.21.15.116:0/1462949070 >> 172.21.15.116:0/1462949070 conn(0x55f61575e720 msgr2=0x55f615808ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.855+0000 7f6892e23b80 1 -- 172.21.15.116:0/1462949070 shutdown_connections 2024-03-30T01:01:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.855+0000 7f6892e23b80 1 -- 172.21.15.116:0/1462949070 wait complete. 2024-03-30T01:01:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.856+0000 7f6892e23b80 1 -- start start 2024-03-30T01:01:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.856+0000 7f6892e23b80 1 Processor -- start 2024-03-30T01:01:00.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.877+0000 7f6892e23b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f615801210 0x55f615810340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.877+0000 7f6892e23b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f615807770 0x55f615810880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.877+0000 7f6892e23b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f615867880 0x55f615869c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.877+0000 7f6892e23b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61586a1b0 con 0x55f615867880 2024-03-30T01:01:00.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.877+0000 7f6892e23b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61586a2f0 con 0x55f615801210 2024-03-30T01:01:00.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.877+0000 7f6892e23b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61586a4b0 con 0x55f615807770 2024-03-30T01:01:00.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.877+0000 7f6892e23b80 10 client.0 _ll_register_callbacks cb 0x55f6156f64f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:01:00.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.878+0000 7f687d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f615867880 0x55f615869c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:00.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.878+0000 7f687d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f615867880 0x55f615869c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34670/0 (socket says 172.21.15.116:34670) 2024-03-30T01:01:00.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.878+0000 7f687d400700 1 -- 172.21.15.116:0/124971504 learned_addr learned my addr 172.21.15.116:0/124971504 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:00.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.878+0000 7f687d400700 1 -- 172.21.15.116:0/124971504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f615807770 msgr2=0x55f615810880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:01:00.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.879+0000 7f687d400700 1 --2- 172.21.15.116:0/124971504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f615807770 0x55f615810880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.879+0000 7f687d400700 1 -- 172.21.15.116:0/124971504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f615801210 msgr2=0x55f615810340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:01:00.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.879+0000 7f687d400700 1 --2- 172.21.15.116:0/124971504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f615801210 0x55f615810340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.879+0000 7f687d400700 1 -- 172.21.15.116:0/124971504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6878118720 con 0x55f615867880 2024-03-30T01:01:00.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.880+0000 7f687d400700 1 --2- 172.21.15.116:0/124971504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f615867880 0x55f615869c70 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6870009ce0 tx=0x7f6870005a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98fb3b0bc7d89def server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:00.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.880+0000 7f6862000700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:01:00.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.880+0000 7f6862000700 1 -- 172.21.15.116:0/124971504 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6870004e80 con 0x55f615867880 2024-03-30T01:01:00.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.881+0000 7f6862000700 1 -- 172.21.15.116:0/124971504 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6870004a30 con 0x55f615867880 2024-03-30T01:01:00.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.881+0000 7f6892e23b80 1 -- 172.21.15.116:0/124971504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6870004d10 con 0x55f615867880 2024-03-30T01:01:00.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.882+0000 7f6862000700 1 -- 172.21.15.116:0/124971504 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6870004e80 con 0x55f615867880 2024-03-30T01:01:00.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:00 smithi116 ceph-mon[38086]: pgmap v385: 129 pgs: 129 active+clean; 428 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 204 op/s 2024-03-30T01:01:00.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.915+0000 7f6892e23b80 10 client.15960 fetch_fsmap learned FSMap version 25 2024-03-30T01:01:00.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.915+0000 7f6892e23b80 1 -- 172.21.15.116:0/124971504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f615716f20 con 0x55f615867880 2024-03-30T01:01:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.956+0000 7f6862000700 1 -- 172.21.15.116:0/124971504 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6870004d10 con 0x55f615867880 2024-03-30T01:01:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.961+0000 7f6862000700 1 -- 172.21.15.116:0/124971504 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f68700144f0 con 0x55f615867880 2024-03-30T01:01:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.966+0000 7f6892e23b80 10 client.15960 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:01:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.966+0000 7f6892e23b80 10 client.15960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:01:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.966+0000 7f6892e23b80 20 client.15960 populate_metadata read hostname 'smithi116' 2024-03-30T01:01:00.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.966+0000 7f6892e23b80 1 --2- 172.21.15.116:0/124971504 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f61586be30 0x55f61586e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:00.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.966+0000 7f6892e23b80 4 client.15960 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T01:01:00.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.966+0000 7f6892e23b80 1 -- 172.21.15.116:0/124971504 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f61559fd70 con 0x55f61586be30 2024-03-30T01:01:00.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.970+0000 7f687ca00700 1 --2- 172.21.15.116:0/124971504 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f61586be30 0x55f61586e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.972+0000 7f687ca00700 1 --2- 172.21.15.116:0/124971504 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f61586be30 0x55f61586e250 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=804c4f9ca2ce67e2 server_cookie=38aa45db02f7ffba in_seq=0 out_seq=0 2024-03-30T01:01:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.972+0000 7f6862000700 10 client.15960 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:01:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.976+0000 7f6862000700 1 -- 172.21.15.116:0/124971504 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f61559fd70 con 0x55f61586be30 2024-03-30T01:01:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.976+0000 7f6862000700 10 client.15960 handle_command_reply: tid=0 2024-03-30T01:01:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.976+0000 7f6892e23b80 1 client.15960 shutdown 2024-03-30T01:01:00.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.985+0000 7f6892e23b80 1 -- 172.21.15.116:0/124971504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f615867880 msgr2=0x55f615869c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:00.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.985+0000 7f6892e23b80 1 --2- 172.21.15.116:0/124971504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f615867880 0x55f615869c70 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6870009ce0 tx=0x7f6870005a10 comp rx=0 tx=0).stop 2024-03-30T01:01:00.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.985+0000 7f6892e23b80 1 -- 172.21.15.116:0/124971504 shutdown_connections 2024-03-30T01:01:00.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.985+0000 7f6892e23b80 1 --2- 172.21.15.116:0/124971504 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55f61586be30 0x55f61586e250 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.985+0000 7f6892e23b80 1 --2- 172.21.15.116:0/124971504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f615867880 0x55f615869c70 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.985+0000 7f6892e23b80 1 --2- 172.21.15.116:0/124971504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f615801210 0x55f615810340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.985+0000 7f6892e23b80 1 --2- 172.21.15.116:0/124971504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f615807770 0x55f615810880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:00.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.985+0000 7f6892e23b80 1 -- 172.21.15.116:0/124971504 >> 172.21.15.116:0/124971504 conn(0x55f61575e720 msgr2=0x55f615804420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:00.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.985+0000 7f6862000700 0 client.15960 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:01:00.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.986+0000 7f6892e23b80 1 -- 172.21.15.116:0/124971504 shutdown_connections 2024-03-30T01:01:00.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:00.986+0000 7f6892e23b80 1 -- 172.21.15.116:0/124971504 wait complete. 2024-03-30T01:01:01.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:00 smithi181 ceph-mon[47622]: pgmap v385: 129 pgs: 129 active+clean; 428 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 204 op/s 2024-03-30T01:01:01.014 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:01:01.014 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-30T01:01:01.014 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "667811a6-53e8-4022-a5a1-930373772864", 2024-03-30T01:01:01.015 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-30T01:01:01.015 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:01:01.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:01.007+0000 7f6892e23b80 20 client.15960 trim_cache size 0 max 16384 2024-03-30T01:01:01.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:01.047+0000 7f687de00700 1 -- 172.21.15.116:0/2206857064 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f685c0a1270 msgr2=0x7f685c0a3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:01.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:01.047+0000 7f687de00700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f685c0a1270 0x7f685c0a3730 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6864009230 tx=0x7f6864010d20 comp rx=0 tx=0).stop 2024-03-30T01:01:01.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:01.047+0000 7f687de00700 1 -- 172.21.15.116:0/2206857064 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6878072b80 msgr2=0x7f68781176c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:01.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:01.047+0000 7f687de00700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6878072b80 0x7f68781176c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f686800d950 tx=0x7f686800dd10 comp rx=0 tx=0).stop 2024-03-30T01:01:01.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:01.052+0000 7f687de00700 1 -- 172.21.15.116:0/2206857064 shutdown_connections 2024-03-30T01:01:01.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:01.052+0000 7f687de00700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f685c0a1270 0x7f685c0a3730 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:01.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:01.052+0000 7f687de00700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f687810ce40 0x7f6878114720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:01.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:01.052+0000 7f687de00700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6878072b80 0x7f68781176c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:01.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:01.052+0000 7f687de00700 1 --2- 172.21.15.116:0/2206857064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687810a390 0x7f6878114180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:01.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:01.052+0000 7f687de00700 1 -- 172.21.15.116:0/2206857064 >> 172.21.15.116:0/2206857064 conn(0x7f687806da50 msgr2=0x7f6878108690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:01.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:01.061+0000 7f687de00700 1 -- 172.21.15.116:0/2206857064 shutdown_connections 2024-03-30T01:01:01.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:01.062+0000 7f687de00700 1 -- 172.21.15.116:0/2206857064 wait complete. 2024-03-30T01:01:01.502 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T01:01:01.787 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:01 smithi130 ceph-mon[47626]: scrub queued for path: / 2024-03-30T01:01:01.787 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:01 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:01.787 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:01 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:01:01.787 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:01 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:01:01.787 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:01 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:01.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:01 smithi116 ceph-mon[38086]: scrub queued for path: / 2024-03-30T01:01:01.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:01 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:01.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:01 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:01:01.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:01 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:01:01.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:01 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:02.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:01 smithi181 ceph-mon[47622]: scrub queued for path: / 2024-03-30T01:01:02.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:01 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:02.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:01 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:01:02.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:01 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:01:02.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:01 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:02.594 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:02 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:01:02.594 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:02 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:01:02.594 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:02 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:02.594 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:02 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:01:02.595 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:02 smithi116 ceph-mon[38086]: pgmap v386: 129 pgs: 129 active+clean; 439 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 255 op/s 2024-03-30T01:01:02.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:02 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:01:02.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:02 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:01:02.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:02 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:02.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:02 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:01:02.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:02 smithi130 ceph-mon[47626]: pgmap v386: 129 pgs: 129 active+clean; 439 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 255 op/s 2024-03-30T01:01:03.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:02 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:01:03.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:02 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:01:03.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:02 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:03.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:02 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:01:03.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:02 smithi181 ceph-mon[47622]: pgmap v386: 129 pgs: 129 active+clean; 439 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.5 MiB/s wr, 255 op/s 2024-03-30T01:01:03.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:03 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:03.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:03 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:04.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:03 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:04.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:04 smithi116 ceph-mon[38086]: pgmap v387: 129 pgs: 129 active+clean; 452 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 268 op/s 2024-03-30T01:01:04.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:04 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:05.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:04 smithi181 ceph-mon[47622]: pgmap v387: 129 pgs: 129 active+clean; 452 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 268 op/s 2024-03-30T01:01:05.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:04 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:05.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:04 smithi130 ceph-mon[47626]: pgmap v387: 129 pgs: 129 active+clean; 452 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 268 op/s 2024-03-30T01:01:05.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:04 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:06.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:06 smithi116 ceph-mon[38086]: pgmap v388: 129 pgs: 129 active+clean; 470 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.3 MiB/s wr, 330 op/s 2024-03-30T01:01:07.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:06 smithi181 ceph-mon[47622]: pgmap v388: 129 pgs: 129 active+clean; 470 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.3 MiB/s wr, 330 op/s 2024-03-30T01:01:07.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:06 smithi130 ceph-mon[47626]: pgmap v388: 129 pgs: 129 active+clean; 470 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.3 MiB/s wr, 330 op/s 2024-03-30T01:01:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.810+0000 7fcc1dc00700 1 -- 172.21.15.116:0/2842515800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc18107e00 msgr2=0x7fcc18108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.810+0000 7fcc1dc00700 1 --2- 172.21.15.116:0/2842515800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc18107e00 0x7fcc18108220 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009f90 tx=0x7fcc1000b330 comp rx=0 tx=0).stop 2024-03-30T01:01:07.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.829+0000 7fcc1dc00700 1 -- 172.21.15.116:0/2842515800 shutdown_connections 2024-03-30T01:01:07.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.829+0000 7fcc1dc00700 1 --2- 172.21.15.116:0/2842515800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc18109270 0x7fcc1810fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:07.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.829+0000 7fcc1dc00700 1 --2- 172.21.15.116:0/2842515800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc18107e00 0x7fcc18108220 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:07.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.829+0000 7fcc1dc00700 1 --2- 172.21.15.116:0/2842515800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc181087f0 0x7fcc18108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:07.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.829+0000 7fcc1dc00700 1 -- 172.21.15.116:0/2842515800 >> 172.21.15.116:0/2842515800 conn(0x7fcc1806dbc0 msgr2=0x7fcc18070000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:07.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.861+0000 7fcc1dc00700 1 -- 172.21.15.116:0/2842515800 shutdown_connections 2024-03-30T01:01:07.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.869+0000 7fcc1dc00700 1 -- 172.21.15.116:0/2842515800 wait complete. 2024-03-30T01:01:07.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.871+0000 7fcc1dc00700 1 Processor -- start 2024-03-30T01:01:07.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.890+0000 7fcc1dc00700 1 -- start start 2024-03-30T01:01:07.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.891+0000 7fcc1dc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc18107e00 0x7fcc1819d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:07.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.891+0000 7fcc1dc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc181087f0 0x7fcc1819d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:07.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.891+0000 7fcc1dc00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc18109270 0x7fcc181b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:07.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.892+0000 7fcc16000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc181087f0 0x7fcc1819d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:07.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.892+0000 7fcc16000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc181087f0 0x7fcc1819d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41648/0 (socket says 172.21.15.116:41648) 2024-03-30T01:01:07.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.892+0000 7fcc1dc00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc181a0930 con 0x7fcc181087f0 2024-03-30T01:01:07.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.892+0000 7fcc1dc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc181a0aa0 con 0x7fcc18107e00 2024-03-30T01:01:07.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.892+0000 7fcc1dc00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc181b61d0 con 0x7fcc18109270 2024-03-30T01:01:07.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.892+0000 7fcc16000700 1 -- 172.21.15.116:0/2893561146 learned_addr learned my addr 172.21.15.116:0/2893561146 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:07.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.892+0000 7fcc17400700 1 --2- 172.21.15.116:0/2893561146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc18109270 0x7fcc181b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:07.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.895+0000 7fcc16000700 1 -- 172.21.15.116:0/2893561146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc18109270 msgr2=0x7fcc181b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:07.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.895+0000 7fcc16000700 1 --2- 172.21.15.116:0/2893561146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc18109270 0x7fcc181b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:07.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.895+0000 7fcc16000700 1 -- 172.21.15.116:0/2893561146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc18107e00 msgr2=0x7fcc1819d3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T01:01:07.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.895+0000 7fcc16000700 1 --2- 172.21.15.116:0/2893561146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc18107e00 0x7fcc1819d3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:07.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.895+0000 7fcc16000700 1 -- 172.21.15.116:0/2893561146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc10009c40 con 0x7fcc181087f0 2024-03-30T01:01:07.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.896+0000 7fcc16000700 1 --2- 172.21.15.116:0/2893561146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc181087f0 0x7fcc1819d8f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcc04007920 tx=0x7fcc04007c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:07.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:01:07.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:07.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:07.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:07.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:07 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.915+0000 7fcc0f400700 1 -- 172.21.15.116:0/2893561146 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04017070 con 0x7fcc181087f0 2024-03-30T01:01:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.915+0000 7fcc0f400700 1 -- 172.21.15.116:0/2893561146 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc040044e0 con 0x7fcc181087f0 2024-03-30T01:01:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.915+0000 7fcc0f400700 1 -- 172.21.15.116:0/2893561146 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0400f040 con 0x7fcc181087f0 2024-03-30T01:01:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.932+0000 7fcc1dc00700 1 -- 172.21.15.116:0/2893561146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc181b6460 con 0x7fcc181087f0 2024-03-30T01:01:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.932+0000 7fcc1dc00700 1 -- 172.21.15.116:0/2893561146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc181b68f0 con 0x7fcc181087f0 2024-03-30T01:01:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:07.992+0000 7fcc31793b80 1 Processor -- start 2024-03-30T01:01:08.005 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:01:08.005 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:08.005 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:08.005 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:08.005 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:07 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:08.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.004+0000 7fcc0f400700 1 -- 172.21.15.116:0/2893561146 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fcc04004750 con 0x7fcc181087f0 2024-03-30T01:01:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.023+0000 7fcc0f400700 1 --2- 172.21.15.116:0/2893561146 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcbfc0a12a0 0x7fcbfc0a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.025+0000 7fcc0f400700 1 -- 172.21.15.116:0/2893561146 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcc040cba50 con 0x7fcc181087f0 2024-03-30T01:01:08.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.031+0000 7fcc17400700 1 --2- 172.21.15.116:0/2893561146 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcbfc0a12a0 0x7fcbfc0a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:08.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.045+0000 7fcc31793b80 1 -- start start 2024-03-30T01:01:08.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.045+0000 7fcc31793b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b147d0 0x55d8c7b14bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:08.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.045+0000 7fcc31793b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b15130 0x55d8c7b159f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:08.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.045+0000 7fcc31793b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b16450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:08.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.045+0000 7fcc31793b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8c7a6c990 con 0x55d8c7b15130 2024-03-30T01:01:08.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.045+0000 7fcc31793b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8c7a6a350 con 0x55d8c7b147d0 2024-03-30T01:01:08.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.045+0000 7fcc31793b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8c7a65c40 con 0x55d8c7b15fd0 2024-03-30T01:01:08.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.052+0000 7fcc17400700 1 --2- 172.21.15.116:0/2893561146 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcbfc0a12a0 0x7fcbfc0a3760 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcc1806aa50 tx=0x7fcc08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:08.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.053+0000 7fcc17400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b16450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:08.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.053+0000 7fcc17400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b16450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:54584/0 (socket says 172.21.15.116:54584) 2024-03-30T01:01:08.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.053+0000 7fcc17400700 1 -- 172.21.15.116:0/2350544395 learned_addr learned my addr 172.21.15.116:0/2350544395 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:08.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.053+0000 7fcc16000700 1 --2- 172.21.15.116:0/2350544395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b15130 0x55d8c7b159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:08.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.056+0000 7fcc16000700 1 -- 172.21.15.116:0/2350544395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 msgr2=0x55d8c7b16450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:01:08.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.056+0000 7fcc16000700 1 --2- 172.21.15.116:0/2350544395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b16450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.056+0000 7fcc16000700 1 -- 172.21.15.116:0/2350544395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b147d0 msgr2=0x55d8c7b14bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:01:08.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.056+0000 7fcc16000700 1 --2- 172.21.15.116:0/2350544395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b147d0 0x55d8c7b14bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.056+0000 7fcc16000700 1 -- 172.21.15.116:0/2350544395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8c78e73b0 con 0x55d8c7b15130 2024-03-30T01:01:08.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.056+0000 7fcc16000700 1 --2- 172.21.15.116:0/2350544395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b15130 0x55d8c7b159f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcc040078f0 tx=0x7fcc0400af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a425b364bdb4fe1d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:08.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.065+0000 7fcc03e00700 1 -- 172.21.15.116:0/2350544395 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040a0fb0 con 0x55d8c7b15130 2024-03-30T01:01:08.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.073+0000 7fcc03e00700 1 -- 172.21.15.116:0/2350544395 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc04096ed0 con 0x55d8c7b15130 2024-03-30T01:01:08.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.073+0000 7fcc03e00700 1 -- 172.21.15.116:0/2350544395 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04098260 con 0x55d8c7b15130 2024-03-30T01:01:08.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:01:08.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:08.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:08.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:08.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:07 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.085+0000 7fcc31793b80 1 -- 172.21.15.116:0/2350544395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b15130 msgr2=0x55d8c7b159f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.085+0000 7fcc31793b80 1 --2- 172.21.15.116:0/2350544395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b15130 0x55d8c7b159f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcc040078f0 tx=0x7fcc0400af90 comp rx=0 tx=0).stop 2024-03-30T01:01:08.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.110+0000 7fcc31793b80 1 -- 172.21.15.116:0/2350544395 shutdown_connections 2024-03-30T01:01:08.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.110+0000 7fcc31793b80 1 --2- 172.21.15.116:0/2350544395 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b16450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.110+0000 7fcc31793b80 1 --2- 172.21.15.116:0/2350544395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b147d0 0x55d8c7b14bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.110+0000 7fcc31793b80 1 --2- 172.21.15.116:0/2350544395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b15130 0x55d8c7b159f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.110+0000 7fcc31793b80 1 -- 172.21.15.116:0/2350544395 >> 172.21.15.116:0/2350544395 conn(0x55d8c7a736a0 msgr2=0x55d8c7b0f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:08.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.120+0000 7fcc31793b80 1 -- 172.21.15.116:0/2350544395 shutdown_connections 2024-03-30T01:01:08.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.124+0000 7fcc31793b80 1 -- 172.21.15.116:0/2350544395 wait complete. 2024-03-30T01:01:08.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.124+0000 7fcc31793b80 1 -- start start 2024-03-30T01:01:08.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.125+0000 7fcc31793b80 1 Processor -- start 2024-03-30T01:01:08.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.151+0000 7fcc31793b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b147d0 0x55d8c7b1e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:08.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.151+0000 7fcc31793b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b15130 0x55d8c7b1ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:08.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.151+0000 7fcc31793b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b25c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:08.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.151+0000 7fcc31793b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8c7a01d80 con 0x55d8c7b15130 2024-03-30T01:01:08.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.151+0000 7fcc31793b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8c7a01a20 con 0x55d8c7b15fd0 2024-03-30T01:01:08.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.151+0000 7fcc31793b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8c78e96a0 con 0x55d8c7b147d0 2024-03-30T01:01:08.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.151+0000 7fcc31793b80 10 client.0 _ll_register_callbacks cb 0x55d8c7a0b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:01:08.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.152+0000 7fcc16000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b15130 0x55d8c7b1ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:08.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.156+0000 7fcc16000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b15130 0x55d8c7b1ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41674/0 (socket says 172.21.15.116:41674) 2024-03-30T01:01:08.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.156+0000 7fcc16000700 1 -- 172.21.15.116:0/3020422461 learned_addr learned my addr 172.21.15.116:0/3020422461 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:08.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.156+0000 7fcc17400700 1 --2- 172.21.15.116:0/3020422461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b25c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:08.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.156+0000 7fcc16000700 1 -- 172.21.15.116:0/3020422461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b147d0 msgr2=0x55d8c7b1e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:01:08.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.156+0000 7fcc16000700 1 --2- 172.21.15.116:0/3020422461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b147d0 0x55d8c7b1e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.156+0000 7fcc16000700 1 -- 172.21.15.116:0/3020422461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15fd0 msgr2=0x55d8c7b25c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:01:08.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.156+0000 7fcc16000700 1 --2- 172.21.15.116:0/3020422461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b25c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.156+0000 7fcc16000700 1 -- 172.21.15.116:0/3020422461 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc181b68f0 con 0x55d8c7b15130 2024-03-30T01:01:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.160+0000 7fcc16000700 1 --2- 172.21.15.116:0/3020422461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b15130 0x55d8c7b1ec00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcc040955a0 tx=0x7fcc04098d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7a03af81280bd3b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:08.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.174+0000 7fcc00c00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:01:08.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.189+0000 7fcc00c00700 1 -- 172.21.15.116:0/3020422461 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040a0fb0 con 0x55d8c7b15130 2024-03-30T01:01:08.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.190+0000 7fcc00c00700 1 -- 172.21.15.116:0/3020422461 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcc040b1270 con 0x55d8c7b15130 2024-03-30T01:01:08.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.211+0000 7fcc31793b80 1 -- 172.21.15.116:0/3020422461 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d8c7a26380 con 0x55d8c7b15130 2024-03-30T01:01:08.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.211+0000 7fcc00c00700 1 -- 172.21.15.116:0/3020422461 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc040b1430 con 0x55d8c7b15130 2024-03-30T01:01:08.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.211+0000 7fcc31793b80 10 client.15972 fetch_fsmap learned FSMap version 25 2024-03-30T01:01:08.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.211+0000 7fcc31793b80 1 -- 172.21.15.116:0/3020422461 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d8c7893b60 con 0x55d8c7b15130 2024-03-30T01:01:08.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.216+0000 7fcc00c00700 1 -- 172.21.15.116:0/3020422461 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d8c7a26380 con 0x55d8c7b15130 2024-03-30T01:01:08.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.216+0000 7fcc00c00700 1 -- 172.21.15.116:0/3020422461 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fcc040a4070 con 0x55d8c7b15130 2024-03-30T01:01:08.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.216+0000 7fcc31793b80 10 client.15972 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:01:08.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.216+0000 7fcc31793b80 10 client.15972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:01:08.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.216+0000 7fcc31793b80 20 client.15972 populate_metadata read hostname 'smithi116' 2024-03-30T01:01:08.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.216+0000 7fcc31793b80 1 --2- 172.21.15.116:0/3020422461 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d8c7b2a180 0x55d8c7b2c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:08.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.216+0000 7fcc31793b80 4 client.15972 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:01:08.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.216+0000 7fcc31793b80 1 -- 172.21.15.116:0/3020422461 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d8c7a5e700 con 0x55d8c7b2a180 2024-03-30T01:01:08.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.222+0000 7fcc16a00700 1 --2- 172.21.15.116:0/3020422461 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d8c7b2a180 0x55d8c7b2c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:08.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.244+0000 7fcc16a00700 1 --2- 172.21.15.116:0/3020422461 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d8c7b2a180 0x55d8c7b2c5a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ea8c742bac350a8 server_cookie=4a5a0bd39215f880 in_seq=0 out_seq=0 2024-03-30T01:01:08.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.260+0000 7fcc00c00700 10 client.15972 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:01:08.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.260+0000 7fcc00c00700 1 -- 172.21.15.116:0/3020422461 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d8c7a5e700 con 0x55d8c7b2a180 2024-03-30T01:01:08.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.260+0000 7fcc00c00700 10 client.15972 handle_command_reply: tid=0 2024-03-30T01:01:08.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.264+0000 7fcc31793b80 1 client.15972 shutdown 2024-03-30T01:01:08.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.264+0000 7fcc31793b80 1 -- 172.21.15.116:0/3020422461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b15130 msgr2=0x55d8c7b1ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:08.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.264+0000 7fcc31793b80 1 --2- 172.21.15.116:0/3020422461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b15130 0x55d8c7b1ec00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcc040955a0 tx=0x7fcc04098d90 comp rx=0 tx=0).stop 2024-03-30T01:01:08.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.268+0000 7fcc31793b80 1 -- 172.21.15.116:0/3020422461 shutdown_connections 2024-03-30T01:01:08.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.268+0000 7fcc31793b80 1 --2- 172.21.15.116:0/3020422461 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d8c7b2a180 0x55d8c7b2c5a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.268+0000 7fcc31793b80 1 --2- 172.21.15.116:0/3020422461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b147d0 0x55d8c7b1e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.268+0000 7fcc31793b80 1 --2- 172.21.15.116:0/3020422461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b25c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.268+0000 7fcc31793b80 1 --2- 172.21.15.116:0/3020422461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b15130 0x55d8c7b1ec00 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.268+0000 7fcc31793b80 1 -- 172.21.15.116:0/3020422461 >> 172.21.15.116:0/3020422461 conn(0x55d8c7a736a0 msgr2=0x55d8c7b13170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:08.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.268+0000 7fcc00c00700 0 client.15972 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:01:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.273+0000 7fcc31793b80 1 -- 172.21.15.116:0/3020422461 shutdown_connections 2024-03-30T01:01:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.273+0000 7fcc31793b80 1 -- 172.21.15.116:0/3020422461 wait complete. 2024-03-30T01:01:08.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.283+0000 7fcc31793b80 20 client.15972 trim_cache size 0 max 16384 2024-03-30T01:01:08.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.873+0000 7fcc31793b80 1 Processor -- start 2024-03-30T01:01:08.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.894+0000 7fcc31793b80 1 -- start start 2024-03-30T01:01:08.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.895+0000 7fcc31793b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b147d0 0x55d8c7b262f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:08.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.895+0000 7fcc31793b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15130 0x55d8c7b25c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:08.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.895+0000 7fcc31793b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b163b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.895+0000 7fcc31793b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8c7b104b0 con 0x55d8c7b147d0 2024-03-30T01:01:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.895+0000 7fcc31793b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8c7b15e50 con 0x55d8c7b15130 2024-03-30T01:01:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.895+0000 7fcc31793b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8c7b1f610 con 0x55d8c7b15fd0 2024-03-30T01:01:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.896+0000 7fcc16a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b147d0 0x55d8c7b262f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.896+0000 7fcc16a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b147d0 0x55d8c7b262f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41690/0 (socket says 172.21.15.116:41690) 2024-03-30T01:01:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.896+0000 7fcc16a00700 1 -- 172.21.15.116:0/3692243858 learned_addr learned my addr 172.21.15.116:0/3692243858 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.896+0000 7fcc16000700 1 --2- 172.21.15.116:0/3692243858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15130 0x55d8c7b25c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.896+0000 7fcc16a00700 1 -- 172.21.15.116:0/3692243858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 msgr2=0x55d8c7b163b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:01:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.896+0000 7fcc17400700 1 --2- 172.21.15.116:0/3692243858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b163b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.900+0000 7fcc16a00700 1 --2- 172.21.15.116:0/3692243858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b163b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.900+0000 7fcc16a00700 1 -- 172.21.15.116:0/3692243858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15130 msgr2=0x55d8c7b25c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:01:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.900+0000 7fcc16a00700 1 --2- 172.21.15.116:0/3692243858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15130 0x55d8c7b25c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.900+0000 7fcc16a00700 1 -- 172.21.15.116:0/3692243858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8c7a2bea0 con 0x55d8c7b147d0 2024-03-30T01:01:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.900+0000 7fcc17400700 1 --2- 172.21.15.116:0/3692243858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b163b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:01:08.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.900+0000 7fcc16000700 1 --2- 172.21.15.116:0/3692243858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15130 0x55d8c7b25c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:01:08.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.900+0000 7fcc16a00700 1 --2- 172.21.15.116:0/3692243858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b147d0 0x55d8c7b262f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009f90 tx=0x7fcc10006040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f222bcd7c88bf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.918+0000 7fcc0cc00700 1 -- 172.21.15.116:0/3692243858 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10020070 con 0x55d8c7b147d0 2024-03-30T01:01:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.918+0000 7fcc0cc00700 1 -- 172.21.15.116:0/3692243858 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc10024070 con 0x55d8c7b147d0 2024-03-30T01:01:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.918+0000 7fcc0cc00700 1 -- 172.21.15.116:0/3692243858 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1000e040 con 0x55d8c7b147d0 2024-03-30T01:01:08.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:08 smithi116 ceph-mon[38086]: pgmap v389: 129 pgs: 129 active+clean; 502 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.5 MiB/s wr, 462 op/s 2024-03-30T01:01:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.930+0000 7fcc31793b80 1 -- 172.21.15.116:0/3692243858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b147d0 msgr2=0x55d8c7b262f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.930+0000 7fcc31793b80 1 --2- 172.21.15.116:0/3692243858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b147d0 0x55d8c7b262f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009f90 tx=0x7fcc10006040 comp rx=0 tx=0).stop 2024-03-30T01:01:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.936+0000 7fcc31793b80 1 -- 172.21.15.116:0/3692243858 shutdown_connections 2024-03-30T01:01:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.936+0000 7fcc31793b80 1 --2- 172.21.15.116:0/3692243858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b163b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.937+0000 7fcc31793b80 1 --2- 172.21.15.116:0/3692243858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b147d0 0x55d8c7b262f0 secure :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009f90 tx=0x7fcc10006040 comp rx=0 tx=0).stop 2024-03-30T01:01:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.938+0000 7fcc31793b80 1 --2- 172.21.15.116:0/3692243858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15130 0x55d8c7b25c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.941+0000 7fcc31793b80 1 -- 172.21.15.116:0/3692243858 >> 172.21.15.116:0/3692243858 conn(0x55d8c7a736a0 msgr2=0x55d8c7b1c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:08.985 INFO:tasks.workunit.client.0.smithi116.stdout: 159 512691 750 358455 794 178768 41 2024-03-30T01:01:08.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.975+0000 7fcc31793b80 1 -- 172.21.15.116:0/3692243858 shutdown_connections 2024-03-30T01:01:08.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.981+0000 7fcc31793b80 1 -- 172.21.15.116:0/3692243858 wait complete. 2024-03-30T01:01:08.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.981+0000 7fcc31793b80 1 -- start start 2024-03-30T01:01:08.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.982+0000 7fcc31793b80 1 Processor -- start 2024-03-30T01:01:08.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.992+0000 7fcc31793b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b147d0 0x55d8c7b25790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:09.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.992+0000 7fcc31793b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15130 0x55d8c7b23ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:09.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.992+0000 7fcc31793b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b24420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:09.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.992+0000 7fcc31793b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8c7b25fb0 con 0x55d8c7b147d0 2024-03-30T01:01:09.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.992+0000 7fcc31793b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8c7b24960 con 0x55d8c7b15130 2024-03-30T01:01:09.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.992+0000 7fcc31793b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8c7b24b20 con 0x55d8c7b15fd0 2024-03-30T01:01:09.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.992+0000 7fcc31793b80 10 client.0 _ll_register_callbacks cb 0x55d8c7a0b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:01:09.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.993+0000 7fcc16000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15130 0x55d8c7b23ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:09.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.993+0000 7fcc16000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15130 0x55d8c7b23ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38908/0 (socket says 172.21.15.116:38908) 2024-03-30T01:01:09.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.993+0000 7fcc16000700 1 -- 172.21.15.116:0/2035042852 learned_addr learned my addr 172.21.15.116:0/2035042852 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:09.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.993+0000 7fcc16a00700 1 --2- 172.21.15.116:0/2035042852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b147d0 0x55d8c7b25790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:09.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.994+0000 7fcc16000700 1 -- 172.21.15.116:0/2035042852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 msgr2=0x55d8c7b24420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:01:09.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.994+0000 7fcc16000700 1 --2- 172.21.15.116:0/2035042852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b24420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:09.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.994+0000 7fcc16000700 1 -- 172.21.15.116:0/2035042852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b147d0 msgr2=0x55d8c7b25790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:01:09.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.994+0000 7fcc16000700 1 --2- 172.21.15.116:0/2035042852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b147d0 0x55d8c7b25790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:09.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.994+0000 7fcc16000700 1 -- 172.21.15.116:0/2035042852 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc18110520 con 0x55d8c7b15130 2024-03-30T01:01:09.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.994+0000 7fcc16a00700 1 --2- 172.21.15.116:0/2035042852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b147d0 0x55d8c7b25790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:01:09.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:08.995+0000 7fcc16000700 1 --2- 172.21.15.116:0/2035042852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15130 0x55d8c7b23ee0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcc0409c260 tx=0x7fcc04004e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81858f21c8dbd495 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:09.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:08 smithi181 ceph-mon[47622]: pgmap v389: 129 pgs: 129 active+clean; 502 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.5 MiB/s wr, 462 op/s 2024-03-30T01:01:09.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.016+0000 7fcc00c00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T01:01:09.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.016+0000 7fcc00c00700 1 -- 172.21.15.116:0/2035042852 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04004020 con 0x55d8c7b15130 2024-03-30T01:01:09.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.016+0000 7fcc00c00700 1 -- 172.21.15.116:0/2035042852 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcc04094920 con 0x55d8c7b15130 2024-03-30T01:01:09.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.037+0000 7fcc31793b80 1 -- 172.21.15.116:0/2035042852 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc04096ed0 con 0x55d8c7b15130 2024-03-30T01:01:09.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.044+0000 7fcc00c00700 1 -- 172.21.15.116:0/2035042852 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc040a0fb0 con 0x55d8c7b15130 2024-03-30T01:01:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.051+0000 7fcc31793b80 10 client.26018 fetch_fsmap learned FSMap version 25 2024-03-30T01:01:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.051+0000 7fcc31793b80 1 -- 172.21.15.116:0/2035042852 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d8c78ebcd0 con 0x55d8c7b15130 2024-03-30T01:01:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.057+0000 7fcc00c00700 1 -- 172.21.15.116:0/2035042852 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc04096ed0 con 0x55d8c7b15130 2024-03-30T01:01:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.057+0000 7fcc00c00700 1 -- 172.21.15.116:0/2035042852 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fcc04095240 con 0x55d8c7b15130 2024-03-30T01:01:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.057+0000 7fcc31793b80 10 client.26018 fetch_fsmap finished waiting for FSMap version 25 2024-03-30T01:01:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.058+0000 7fcc31793b80 10 client.26018 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:01:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.058+0000 7fcc31793b80 20 client.26018 populate_metadata read hostname 'smithi116' 2024-03-30T01:01:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.058+0000 7fcc31793b80 1 --2- 172.21.15.116:0/2035042852 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d8c7b7f000 0x55d8c7b813f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.058+0000 7fcc31793b80 4 client.26018 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T01:01:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.058+0000 7fcc31793b80 1 -- 172.21.15.116:0/2035042852 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d8c78b4cf0 con 0x55d8c7b7f000 2024-03-30T01:01:09.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.064+0000 7fcc16a00700 1 --2- 172.21.15.116:0/2035042852 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d8c7b7f000 0x55d8c7b813f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.073+0000 7fcc16a00700 1 --2- 172.21.15.116:0/2035042852 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d8c7b7f000 0x55d8c7b813f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e9ea12f865688f0 server_cookie=c41433f7847205a9 in_seq=0 out_seq=0 2024-03-30T01:01:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.078+0000 7fcc00c00700 10 client.26018 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:01:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.078+0000 7fcc00c00700 1 -- 172.21.15.116:0/2035042852 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55d8c78b4cf0 con 0x55d8c7b7f000 2024-03-30T01:01:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.078+0000 7fcc00c00700 10 client.26018 handle_command_reply: tid=0 2024-03-30T01:01:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.078+0000 7fcc31793b80 1 client.26018 shutdown 2024-03-30T01:01:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.084+0000 7fcc31793b80 1 -- 172.21.15.116:0/2035042852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15130 msgr2=0x55d8c7b23ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.084+0000 7fcc31793b80 1 --2- 172.21.15.116:0/2035042852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15130 0x55d8c7b23ee0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcc0409c260 tx=0x7fcc04004e30 comp rx=0 tx=0).stop 2024-03-30T01:01:09.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:08 smithi130 ceph-mon[47626]: pgmap v389: 129 pgs: 129 active+clean; 502 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.5 MiB/s wr, 462 op/s 2024-03-30T01:01:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.105+0000 7fcc31793b80 1 -- 172.21.15.116:0/2035042852 shutdown_connections 2024-03-30T01:01:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.105+0000 7fcc31793b80 1 --2- 172.21.15.116:0/2035042852 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d8c7b7f000 0x55d8c7b813f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.105+0000 7fcc31793b80 1 --2- 172.21.15.116:0/2035042852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8c7b15fd0 0x55d8c7b24420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.105+0000 7fcc31793b80 1 --2- 172.21.15.116:0/2035042852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8c7b147d0 0x55d8c7b25790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.105+0000 7fcc31793b80 1 --2- 172.21.15.116:0/2035042852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8c7b15130 0x55d8c7b23ee0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.105+0000 7fcc31793b80 1 -- 172.21.15.116:0/2035042852 >> 172.21.15.116:0/2035042852 conn(0x55d8c7a736a0 msgr2=0x55d8c7b1c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.105+0000 7fcc00c00700 0 client.26018 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:01:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.110+0000 7fcc31793b80 1 -- 172.21.15.116:0/2035042852 shutdown_connections 2024-03-30T01:01:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.110+0000 7fcc31793b80 1 -- 172.21.15.116:0/2035042852 wait complete. 2024-03-30T01:01:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.111+0000 7fcc31793b80 20 client.26018 trim_cache size 0 max 16384 2024-03-30T01:01:09.137 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:01:09.138 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (5016 inodes in the stack)", 2024-03-30T01:01:09.138 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-30T01:01:09.138 INFO:teuthology.orchestra.run.smithi116.stdout: "667811a6-53e8-4022-a5a1-930373772864": { 2024-03-30T01:01:09.138 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-30T01:01:09.138 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "667811a6-53e8-4022-a5a1-930373772864", 2024-03-30T01:01:09.138 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-30T01:01:09.138 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:01:09.138 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:01:09.138 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:01:09.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.136+0000 7fcc1dc00700 1 -- 172.21.15.116:0/2893561146 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcbfc0a12a0 msgr2=0x7fcbfc0a3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:09.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.137+0000 7fcc1dc00700 1 --2- 172.21.15.116:0/2893561146 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcbfc0a12a0 0x7fcbfc0a3760 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcc1806aa50 tx=0x7fcc08009260 comp rx=0 tx=0).stop 2024-03-30T01:01:09.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.137+0000 7fcc1dc00700 1 -- 172.21.15.116:0/2893561146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc181087f0 msgr2=0x7fcc1819d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:09.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.137+0000 7fcc1dc00700 1 --2- 172.21.15.116:0/2893561146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc181087f0 0x7fcc1819d8f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcc04007920 tx=0x7fcc04007c30 comp rx=0 tx=0).stop 2024-03-30T01:01:09.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.139+0000 7fcc1dc00700 1 -- 172.21.15.116:0/2893561146 shutdown_connections 2024-03-30T01:01:09.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.139+0000 7fcc1dc00700 1 --2- 172.21.15.116:0/2893561146 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fcbfc0a12a0 0x7fcbfc0a3760 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:09.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.139+0000 7fcc1dc00700 1 --2- 172.21.15.116:0/2893561146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc18109270 0x7fcc181b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:09.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.139+0000 7fcc1dc00700 1 --2- 172.21.15.116:0/2893561146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc18107e00 0x7fcc1819d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:09.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.139+0000 7fcc1dc00700 1 --2- 172.21.15.116:0/2893561146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc181087f0 0x7fcc1819d8f0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:09.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.139+0000 7fcc1dc00700 1 -- 172.21.15.116:0/2893561146 >> 172.21.15.116:0/2893561146 conn(0x7fcc1806dbc0 msgr2=0x7fcc1810bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:09.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.159+0000 7fcc1dc00700 1 -- 172.21.15.116:0/2893561146 shutdown_connections 2024-03-30T01:01:09.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:09.162+0000 7fcc1dc00700 1 -- 172.21.15.116:0/2893561146 wait complete. 2024-03-30T01:01:09.658 INFO:tasks.cephfs.filesystem:scrub status for tag:667811a6-53e8-4022-a5a1-930373772864 - {'path': '/', 'tag': '667811a6-53e8-4022-a5a1-930373772864', 'options': 'recursive,force'} 2024-03-30T01:01:09.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:09 smithi116 ceph-mon[38086]: pgmap v390: 129 pgs: 129 active+clean; 564 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 12 MiB/s wr, 673 op/s 2024-03-30T01:01:10.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:09 smithi181 ceph-mon[47622]: pgmap v390: 129 pgs: 129 active+clean; 564 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 12 MiB/s wr, 673 op/s 2024-03-30T01:01:10.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:09 smithi130 ceph-mon[47626]: pgmap v390: 129 pgs: 129 active+clean; 564 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 12 MiB/s wr, 673 op/s 2024-03-30T01:01:10.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:10 smithi116 ceph-mon[38086]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:01:11.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:10 smithi181 ceph-mon[47622]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:01:11.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:10 smithi130 ceph-mon[47626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:01:12.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:11 smithi181 ceph-mon[47622]: pgmap v391: 129 pgs: 129 active+clean; 600 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 15 MiB/s wr, 788 op/s 2024-03-30T01:01:12.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:11 smithi130 ceph-mon[47626]: pgmap v391: 129 pgs: 129 active+clean; 600 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 15 MiB/s wr, 788 op/s 2024-03-30T01:01:12.186 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:11 smithi116 ceph-mon[38086]: pgmap v391: 129 pgs: 129 active+clean; 600 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 15 MiB/s wr, 788 op/s 2024-03-30T01:01:14.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:14 smithi130 ceph-mon[47626]: pgmap v392: 129 pgs: 129 active+clean; 629 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 871 op/s 2024-03-30T01:01:14.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:14 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:14.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:14 smithi116 ceph-mon[38086]: pgmap v392: 129 pgs: 129 active+clean; 629 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 871 op/s 2024-03-30T01:01:14.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:14 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:14.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:14 smithi181 ceph-mon[47622]: pgmap v392: 129 pgs: 129 active+clean; 629 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 871 op/s 2024-03-30T01:01:14.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:14 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:01:15.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:15 smithi130 ceph-mon[47626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:01:15.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:15 smithi116 ceph-mon[38086]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:01:15.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:15 smithi181 ceph-mon[47622]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:01:16.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:16 smithi130 ceph-mon[47626]: pgmap v393: 129 pgs: 129 active+clean; 658 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 18 MiB/s wr, 960 op/s 2024-03-30T01:01:16.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:16 smithi116 ceph-mon[38086]: pgmap v393: 129 pgs: 129 active+clean; 658 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 18 MiB/s wr, 960 op/s 2024-03-30T01:01:16.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:16 smithi181 ceph-mon[47622]: pgmap v393: 129 pgs: 129 active+clean; 658 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 18 MiB/s wr, 960 op/s 2024-03-30T01:01:17.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:17 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:17.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:17 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:17.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:17 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:01:17.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:17 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:17.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:17 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:17.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:17 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:01:18.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:17 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:18.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:17 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:18.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:17 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:01:18.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:18 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:01:18.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:18 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:01:18.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:18 smithi130 ceph-mon[47626]: pgmap v394: 129 pgs: 129 active+clean; 656 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 17 MiB/s wr, 902 op/s 2024-03-30T01:01:18.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:18 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:01:18.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:18 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:01:18.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:18 smithi116 ceph-mon[38086]: pgmap v394: 129 pgs: 129 active+clean; 656 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 17 MiB/s wr, 902 op/s 2024-03-30T01:01:18.974 INFO:tasks.workunit.client.0.smithi116.stdout: 188 597214 1494 413359 1472 184683 167 2024-03-30T01:01:19.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:18 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:01:19.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:18 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:01:19.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:18 smithi181 ceph-mon[47622]: pgmap v394: 129 pgs: 129 active+clean; 656 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 17 MiB/s wr, 902 op/s 2024-03-30T01:01:20.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:20 smithi130 ceph-mon[47626]: pgmap v395: 129 pgs: 129 active+clean; 666 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 15 MiB/s wr, 765 op/s 2024-03-30T01:01:20.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:20 smithi116 ceph-mon[38086]: pgmap v395: 129 pgs: 129 active+clean; 666 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 15 MiB/s wr, 765 op/s 2024-03-30T01:01:21.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:20 smithi181 ceph-mon[47622]: pgmap v395: 129 pgs: 129 active+clean; 666 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 15 MiB/s wr, 765 op/s 2024-03-30T01:01:22.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:22 smithi130 ceph-mon[47626]: pgmap v396: 129 pgs: 129 active+clean; 666 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 9.5 MiB/s wr, 528 op/s 2024-03-30T01:01:22.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:22 smithi116 ceph-mon[38086]: pgmap v396: 129 pgs: 129 active+clean; 666 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 9.5 MiB/s wr, 528 op/s 2024-03-30T01:01:23.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:22 smithi181 ceph-mon[47622]: pgmap v396: 129 pgs: 129 active+clean; 666 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 9.5 MiB/s wr, 528 op/s 2024-03-30T01:01:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:24 smithi130 ceph-mon[47626]: pgmap v397: 129 pgs: 129 active+clean; 670 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.8 MiB/s wr, 411 op/s 2024-03-30T01:01:24.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:24 smithi116 ceph-mon[38086]: pgmap v397: 129 pgs: 129 active+clean; 670 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.8 MiB/s wr, 411 op/s 2024-03-30T01:01:25.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:24 smithi181 ceph-mon[47622]: pgmap v397: 129 pgs: 129 active+clean; 670 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.8 MiB/s wr, 411 op/s 2024-03-30T01:01:26.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:26 smithi130 ceph-mon[47626]: pgmap v398: 129 pgs: 129 active+clean; 675 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.8 MiB/s wr, 287 op/s 2024-03-30T01:01:26.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:26 smithi116 ceph-mon[38086]: pgmap v398: 129 pgs: 129 active+clean; 675 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.8 MiB/s wr, 287 op/s 2024-03-30T01:01:27.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:26 smithi181 ceph-mon[47622]: pgmap v398: 129 pgs: 129 active+clean; 675 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.8 MiB/s wr, 287 op/s 2024-03-30T01:01:28.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:28 smithi130 ceph-mon[47626]: pgmap v399: 129 pgs: 129 active+clean; 688 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 199 op/s 2024-03-30T01:01:28.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:28 smithi116 ceph-mon[38086]: pgmap v399: 129 pgs: 129 active+clean; 688 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 199 op/s 2024-03-30T01:01:28.979 INFO:tasks.workunit.client.0.smithi116.stdout: 218 626167 1455 436630 1487 187377 2291 2024-03-30T01:01:29.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:28 smithi181 ceph-mon[47622]: pgmap v399: 129 pgs: 129 active+clean; 688 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 199 op/s 2024-03-30T01:01:30.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:30 smithi116 ceph-mon[38086]: pgmap v400: 129 pgs: 129 active+clean; 703 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.8 MiB/s wr, 230 op/s 2024-03-30T01:01:31.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:30 smithi181 ceph-mon[47622]: pgmap v400: 129 pgs: 129 active+clean; 703 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.8 MiB/s wr, 230 op/s 2024-03-30T01:01:31.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:30 smithi130 ceph-mon[47626]: pgmap v400: 129 pgs: 129 active+clean; 703 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.8 MiB/s wr, 230 op/s 2024-03-30T01:01:31.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:01:31.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:01:31.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:32.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:01:32.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:01:32.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:32.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:01:32.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:01:32.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:01:32.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:32 smithi116 ceph-mon[38086]: pgmap v401: 129 pgs: 129 active+clean; 715 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.2 MiB/s wr, 278 op/s 2024-03-30T01:01:33.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:32 smithi181 ceph-mon[47622]: pgmap v401: 129 pgs: 129 active+clean; 715 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.2 MiB/s wr, 278 op/s 2024-03-30T01:01:33.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:32 smithi130 ceph-mon[47626]: pgmap v401: 129 pgs: 129 active+clean; 715 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.2 MiB/s wr, 278 op/s 2024-03-30T01:01:33.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:33 smithi116 ceph-mon[38086]: pgmap v402: 129 pgs: 129 active+clean; 738 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.1 MiB/s wr, 366 op/s 2024-03-30T01:01:34.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:33 smithi181 ceph-mon[47622]: pgmap v402: 129 pgs: 129 active+clean; 738 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.1 MiB/s wr, 366 op/s 2024-03-30T01:01:34.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:33 smithi130 ceph-mon[47626]: pgmap v402: 129 pgs: 129 active+clean; 738 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.1 MiB/s wr, 366 op/s 2024-03-30T01:01:36.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:36 smithi130 ceph-mon[47626]: pgmap v403: 129 pgs: 129 active+clean; 802 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 12 MiB/s wr, 546 op/s 2024-03-30T01:01:36.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:36 smithi116 ceph-mon[38086]: pgmap v403: 129 pgs: 129 active+clean; 802 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 12 MiB/s wr, 546 op/s 2024-03-30T01:01:36.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:36 smithi181 ceph-mon[47622]: pgmap v403: 129 pgs: 129 active+clean; 802 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 12 MiB/s wr, 546 op/s 2024-03-30T01:01:38.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:38 smithi130 ceph-mon[47626]: pgmap v404: 129 pgs: 129 active+clean; 852 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 16 MiB/s wr, 710 op/s 2024-03-30T01:01:38.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:38 smithi116 ceph-mon[38086]: pgmap v404: 129 pgs: 129 active+clean; 852 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 16 MiB/s wr, 710 op/s 2024-03-30T01:01:38.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:38 smithi181 ceph-mon[47622]: pgmap v404: 129 pgs: 129 active+clean; 852 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 16 MiB/s wr, 710 op/s 2024-03-30T01:01:38.998 INFO:tasks.workunit.client.0.smithi116.stdout: 231 458501 665 318136 615 147006 1085 2024-03-30T01:01:39.659 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T01:01:40.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:40 smithi130 ceph-mon[47626]: pgmap v405: 129 pgs: 129 active+clean; 888 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 18 MiB/s wr, 835 op/s 2024-03-30T01:01:40.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:40 smithi116 ceph-mon[38086]: pgmap v405: 129 pgs: 129 active+clean; 888 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 18 MiB/s wr, 835 op/s 2024-03-30T01:01:40.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:40 smithi181 ceph-mon[47622]: pgmap v405: 129 pgs: 129 active+clean; 888 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 18 MiB/s wr, 835 op/s 2024-03-30T01:01:42.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:42 smithi130 ceph-mon[47626]: pgmap v406: 129 pgs: 129 active+clean; 904 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 18 MiB/s wr, 870 op/s 2024-03-30T01:01:42.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:42 smithi116 ceph-mon[38086]: pgmap v406: 129 pgs: 129 active+clean; 904 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 18 MiB/s wr, 870 op/s 2024-03-30T01:01:42.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:42 smithi181 ceph-mon[47622]: pgmap v406: 129 pgs: 129 active+clean; 904 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 18 MiB/s wr, 870 op/s 2024-03-30T01:01:43.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.651+0000 7f2e3b000700 1 -- 172.21.15.116:0/1475481556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e34108de0 msgr2=0x7f2e34109270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:43.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.651+0000 7f2e3b000700 1 --2- 172.21.15.116:0/1475481556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e34108de0 0x7f2e34109270 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00b3e0 tx=0x7f2e2c00b6f0 comp rx=0 tx=0).stop 2024-03-30T01:01:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.674+0000 7f2e3b000700 1 -- 172.21.15.116:0/1475481556 shutdown_connections 2024-03-30T01:01:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.674+0000 7f2e3b000700 1 --2- 172.21.15.116:0/1475481556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e34108de0 0x7f2e34109270 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.674+0000 7f2e3b000700 1 --2- 172.21.15.116:0/1475481556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e340722b0 0x7f2e34107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.674+0000 7f2e3b000700 1 --2- 172.21.15.116:0/1475481556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e34072740 0x7f2e34108540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.674+0000 7f2e3b000700 1 -- 172.21.15.116:0/1475481556 >> 172.21.15.116:0/1475481556 conn(0x7f2e3406dbc0 msgr2=0x7f2e34070000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.680+0000 7f2e3b000700 1 -- 172.21.15.116:0/1475481556 shutdown_connections 2024-03-30T01:01:43.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.680+0000 7f2e3b000700 1 -- 172.21.15.116:0/1475481556 wait complete. 2024-03-30T01:01:43.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.684+0000 7f2e3b000700 1 Processor -- start 2024-03-30T01:01:43.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.686+0000 7f2e3b000700 1 -- start start 2024-03-30T01:01:43.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.693+0000 7f2e3b000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e340722b0 0x7f2e3419d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:43.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.693+0000 7f2e3b000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e34072740 0x7f2e3419d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:43.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.693+0000 7f2e3b000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e34108de0 0x7f2e341a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:43.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.693+0000 7f2e3b000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e341a4f00 con 0x7f2e340722b0 2024-03-30T01:01:43.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.693+0000 7f2e33e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e340722b0 0x7f2e3419d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:43.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.693+0000 7f2e39000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e34108de0 0x7f2e341a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:43.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.693+0000 7f2e39000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e34108de0 0x7f2e341a4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:45796/0 (socket says 172.21.15.116:45796) 2024-03-30T01:01:43.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.693+0000 7f2e33e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e340722b0 0x7f2e3419d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43704/0 (socket says 172.21.15.116:43704) 2024-03-30T01:01:43.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.693+0000 7f2e39000700 1 -- 172.21.15.116:0/3728423210 learned_addr learned my addr 172.21.15.116:0/3728423210 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:43.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.694+0000 7f2e33400700 1 --2- 172.21.15.116:0/3728423210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e34072740 0x7f2e3419d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:43.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.718+0000 7f2e3b000700 1 -- 172.21.15.116:0/3728423210 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e341b6160 con 0x7f2e34108de0 2024-03-30T01:01:43.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.718+0000 7f2e3b000700 1 -- 172.21.15.116:0/3728423210 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e341b62a0 con 0x7f2e34072740 2024-03-30T01:01:43.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.719+0000 7f2e33e00700 1 -- 172.21.15.116:0/3728423210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e34072740 msgr2=0x7f2e3419d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:43.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.719+0000 7f2e33e00700 1 --2- 172.21.15.116:0/3728423210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e34072740 0x7f2e3419d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:43.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.719+0000 7f2e33e00700 1 -- 172.21.15.116:0/3728423210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e34108de0 msgr2=0x7f2e341a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:43.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.719+0000 7f2e33e00700 1 --2- 172.21.15.116:0/3728423210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e34108de0 0x7f2e341a4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:43.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.719+0000 7f2e33e00700 1 -- 172.21.15.116:0/3728423210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c009d00 con 0x7f2e340722b0 2024-03-30T01:01:43.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.720+0000 7f2e33e00700 1 --2- 172.21.15.116:0/3728423210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e340722b0 0x7f2e3419d3d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400b6a0 tx=0x7f2e2400ba60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:43.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.726+0000 7f2e33400700 1 --2- 172.21.15.116:0/3728423210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e34072740 0x7f2e3419d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:01:43.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.729+0000 7f2e30c00700 1 -- 172.21.15.116:0/3728423210 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24010680 con 0x7f2e340722b0 2024-03-30T01:01:43.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.729+0000 7f2e30c00700 1 -- 172.21.15.116:0/3728423210 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e24010ca0 con 0x7f2e340722b0 2024-03-30T01:01:43.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.729+0000 7f2e30c00700 1 -- 172.21.15.116:0/3728423210 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2400fe60 con 0x7f2e340722b0 2024-03-30T01:01:43.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.733+0000 7f2e3b000700 1 -- 172.21.15.116:0/3728423210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e341b6500 con 0x7f2e340722b0 2024-03-30T01:01:43.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.733+0000 7f2e3b000700 1 -- 172.21.15.116:0/3728423210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e341b6990 con 0x7f2e340722b0 2024-03-30T01:01:43.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.805+0000 7f2e4eacab80 1 Processor -- start 2024-03-30T01:01:43.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.810+0000 7f2e30c00700 1 -- 172.21.15.116:0/3728423210 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f2e2400d380 con 0x7f2e340722b0 2024-03-30T01:01:43.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.816+0000 7f2e4eacab80 1 -- start start 2024-03-30T01:01:43.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.817+0000 7f2e4eacab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c78f7d0 0x55703c78fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:43.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.817+0000 7f2e4eacab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 0x55703c7909f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:43.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.817+0000 7f2e4eacab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c790fd0 0x55703c791450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:43.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.817+0000 7f2e4eacab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55703c6e7990 con 0x55703c790130 2024-03-30T01:01:43.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.817+0000 7f2e4eacab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55703c6e5350 con 0x55703c790fd0 2024-03-30T01:01:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.817+0000 7f2e4eacab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55703c6e0c40 con 0x55703c78f7d0 2024-03-30T01:01:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.817+0000 7f2e39000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c78f7d0 0x55703c78fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.817+0000 7f2e39000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c78f7d0 0x55703c78fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:60508/0 (socket says 172.21.15.116:60508) 2024-03-30T01:01:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.817+0000 7f2e39000700 1 -- 172.21.15.116:0/3869485422 learned_addr learned my addr 172.21.15.116:0/3869485422 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.817+0000 7f2e33400700 1 --2- 172.21.15.116:0/3869485422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c790fd0 0x55703c791450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.818+0000 7f2e39000700 1 -- 172.21.15.116:0/3869485422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c790fd0 msgr2=0x55703c791450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:01:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.818+0000 7f2e39000700 1 --2- 172.21.15.116:0/3869485422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c790fd0 0x55703c791450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.818+0000 7f2e39000700 1 -- 172.21.15.116:0/3869485422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 msgr2=0x55703c7909f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:01:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.818+0000 7f2e39000700 1 --2- 172.21.15.116:0/3869485422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 0x55703c7909f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.818+0000 7f2e39000700 1 -- 172.21.15.116:0/3869485422 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55703c5623b0 con 0x55703c78f7d0 2024-03-30T01:01:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.820+0000 7f2e39000700 1 --2- 172.21.15.116:0/3869485422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c78f7d0 0x55703c78fbf0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c0092c0 tx=0x7f2e2c0092f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba290bf8d10238 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:43.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.839+0000 7f2e30c00700 1 --2- 172.21.15.116:0/3728423210 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2e180a1320 0x7f2e180a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:43.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.843+0000 7f2e29600700 1 -- 172.21.15.116:0/3869485422 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c00e040 con 0x55703c78f7d0 2024-03-30T01:01:43.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.843+0000 7f2e29600700 1 -- 172.21.15.116:0/3869485422 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e2c0074e0 con 0x55703c78f7d0 2024-03-30T01:01:43.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.843+0000 7f2e29600700 1 -- 172.21.15.116:0/3869485422 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c005380 con 0x55703c78f7d0 2024-03-30T01:01:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.844+0000 7f2e33e00700 1 --2- 172.21.15.116:0/3728423210 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2e180a1320 0x7f2e180a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.845+0000 7f2e33e00700 1 --2- 172.21.15.116:0/3728423210 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2e180a1320 0x7f2e180a37e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2e240ca020 tx=0x7f2e24010a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.847+0000 7f2e30c00700 1 -- 172.21.15.116:0/3728423210 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2e240cc8c0 con 0x7f2e340722b0 2024-03-30T01:01:43.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.848+0000 7f2e4eacab80 1 -- 172.21.15.116:0/3869485422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c78f7d0 msgr2=0x55703c78fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:43.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.848+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/3869485422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c78f7d0 0x55703c78fbf0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c0092c0 tx=0x7f2e2c0092f0 comp rx=0 tx=0).stop 2024-03-30T01:01:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.876+0000 7f2e4eacab80 1 -- 172.21.15.116:0/3869485422 shutdown_connections 2024-03-30T01:01:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.876+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/3869485422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c78f7d0 0x55703c78fbf0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.876+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/3869485422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c790fd0 0x55703c791450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.876+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/3869485422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 0x55703c7909f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.876+0000 7f2e4eacab80 1 -- 172.21.15.116:0/3869485422 >> 172.21.15.116:0/3869485422 conn(0x55703c6ee6a0 msgr2=0x55703c78a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.882+0000 7f2e4eacab80 1 -- 172.21.15.116:0/3869485422 shutdown_connections 2024-03-30T01:01:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.883+0000 7f2e4eacab80 1 -- 172.21.15.116:0/3869485422 wait complete. 2024-03-30T01:01:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.885+0000 7f2e4eacab80 1 -- start start 2024-03-30T01:01:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.886+0000 7f2e4eacab80 1 Processor -- start 2024-03-30T01:01:43.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.934+0000 7f2e4eacab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c78f7d0 0x55703c799630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.934+0000 7f2e4eacab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 0x55703c799b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.934+0000 7f2e4eacab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c790fd0 0x55703c7a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.934+0000 7f2e4eacab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55703c67cd80 con 0x55703c790130 2024-03-30T01:01:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.934+0000 7f2e4eacab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55703c67ca20 con 0x55703c790fd0 2024-03-30T01:01:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.934+0000 7f2e4eacab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55703c5646a0 con 0x55703c78f7d0 2024-03-30T01:01:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.934+0000 7f2e4eacab80 10 client.0 _ll_register_callbacks cb 0x55703c686470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:01:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.935+0000 7f2e39000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c78f7d0 0x55703c799630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.935+0000 7f2e39000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c78f7d0 0x55703c799630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:60520/0 (socket says 172.21.15.116:60520) 2024-03-30T01:01:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.935+0000 7f2e39000700 1 -- 172.21.15.116:0/4191953042 learned_addr learned my addr 172.21.15.116:0/4191953042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:43.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.936+0000 7f2e39000700 1 --2- 172.21.15.116:0/4191953042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c790fd0 0x55703c7a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.949+0000 7f2e33400700 1 --2- 172.21.15.116:0/4191953042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 0x55703c799b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.950+0000 7f2e33400700 1 -- 172.21.15.116:0/4191953042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c78f7d0 msgr2=0x55703c799630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:01:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.950+0000 7f2e33400700 1 --2- 172.21.15.116:0/4191953042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c78f7d0 0x55703c799630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.950+0000 7f2e33400700 1 -- 172.21.15.116:0/4191953042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c790fd0 msgr2=0x55703c7a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:01:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.950+0000 7f2e33400700 1 --2- 172.21.15.116:0/4191953042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c790fd0 0x55703c7a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.950+0000 7f2e33400700 1 -- 172.21.15.116:0/4191953042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3410a170 con 0x55703c790130 2024-03-30T01:01:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.950+0000 7f2e33400700 1 --2- 172.21.15.116:0/4191953042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 0x55703c799b70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000deb0 tx=0x7f2e2000df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d64d7fac1abf5807 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.967+0000 7f2e1e000700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:01:43.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.971+0000 7f2e1e000700 1 -- 172.21.15.116:0/4191953042 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e20014480 con 0x55703c790130 2024-03-30T01:01:43.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.971+0000 7f2e1e000700 1 -- 172.21.15.116:0/4191953042 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2e200145e0 con 0x55703c790130 2024-03-30T01:01:44.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:43.998+0000 7f2e4eacab80 1 -- 172.21.15.116:0/4191953042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55703c6a1380 con 0x55703c790130 2024-03-30T01:01:44.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.007+0000 7f2e1e000700 1 -- 172.21.15.116:0/4191953042 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e200147a0 con 0x55703c790130 2024-03-30T01:01:44.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.007+0000 7f2e4eacab80 10 client.15987 fetch_fsmap learned FSMap version 27 2024-03-30T01:01:44.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.007+0000 7f2e4eacab80 1 -- 172.21.15.116:0/4191953042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55703c50eb60 con 0x55703c790130 2024-03-30T01:01:44.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.012+0000 7f2e1e000700 1 -- 172.21.15.116:0/4191953042 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55703c6a1380 con 0x55703c790130 2024-03-30T01:01:44.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.012+0000 7f2e1e000700 1 -- 172.21.15.116:0/4191953042 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f2e2001dc30 con 0x55703c790130 2024-03-30T01:01:44.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.012+0000 7f2e4eacab80 10 client.15987 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T01:01:44.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.012+0000 7f2e4eacab80 10 client.15987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:01:44.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.012+0000 7f2e4eacab80 20 client.15987 populate_metadata read hostname 'smithi116' 2024-03-30T01:01:44.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.012+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/4191953042 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55703c7a54a0 0x55703c7a78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:44.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.012+0000 7f2e4eacab80 4 client.15987 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:01:44.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.012+0000 7f2e4eacab80 1 -- 172.21.15.116:0/4191953042 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55703c6d9700 con 0x55703c7a54a0 2024-03-30T01:01:44.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.013+0000 7f2e39000700 1 --2- 172.21.15.116:0/4191953042 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55703c7a54a0 0x55703c7a78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:44.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.014+0000 7f2e39000700 1 --2- 172.21.15.116:0/4191953042 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55703c7a54a0 0x55703c7a78c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9818510868b4774 server_cookie=35b5bbaf8593c99f in_seq=0 out_seq=0 2024-03-30T01:01:44.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.017+0000 7f2e1e000700 10 client.15987 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:01:44.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.017+0000 7f2e1e000700 1 -- 172.21.15.116:0/4191953042 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55703c6d9700 con 0x55703c7a54a0 2024-03-30T01:01:44.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.017+0000 7f2e1e000700 10 client.15987 handle_command_reply: tid=0 2024-03-30T01:01:44.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.017+0000 7f2e4eacab80 1 client.15987 shutdown 2024-03-30T01:01:44.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.029+0000 7f2e4eacab80 1 -- 172.21.15.116:0/4191953042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 msgr2=0x55703c799b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:44.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.029+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/4191953042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 0x55703c799b70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2e2000deb0 tx=0x7f2e2000df90 comp rx=0 tx=0).stop 2024-03-30T01:01:44.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.043+0000 7f2e4eacab80 1 -- 172.21.15.116:0/4191953042 shutdown_connections 2024-03-30T01:01:44.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.043+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/4191953042 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55703c7a54a0 0x55703c7a78c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.043+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/4191953042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c78f7d0 0x55703c799630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.043+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/4191953042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c790fd0 0x55703c7a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.043+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/4191953042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 0x55703c799b70 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.043+0000 7f2e4eacab80 1 -- 172.21.15.116:0/4191953042 >> 172.21.15.116:0/4191953042 conn(0x55703c6ee6a0 msgr2=0x55703c78e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:44.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.043+0000 7f2e1e000700 0 client.15987 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:01:44.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.043+0000 7f2e4eacab80 1 -- 172.21.15.116:0/4191953042 shutdown_connections 2024-03-30T01:01:44.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.044+0000 7f2e4eacab80 1 -- 172.21.15.116:0/4191953042 wait complete. 2024-03-30T01:01:44.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.049+0000 7f2e4eacab80 20 client.15987 trim_cache size 0 max 16384 2024-03-30T01:01:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.540+0000 7f2e4eacab80 1 Processor -- start 2024-03-30T01:01:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.579+0000 7f2e4eacab80 1 -- start start 2024-03-30T01:01:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.579+0000 7f2e4eacab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c78f7d0 0x55703c797600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.579+0000 7f2e4eacab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c790130 0x55703c7a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.579+0000 7f2e4eacab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c790fd0 0x55703c7941e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.579+0000 7f2e4eacab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55703c790880 con 0x55703c78f7d0 2024-03-30T01:01:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.579+0000 7f2e4eacab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55703c78f370 con 0x55703c790fd0 2024-03-30T01:01:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.579+0000 7f2e4eacab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55703c7a28a0 con 0x55703c790130 2024-03-30T01:01:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.580+0000 7f2e39000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c78f7d0 0x55703c797600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.580+0000 7f2e39000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c78f7d0 0x55703c797600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43724/0 (socket says 172.21.15.116:43724) 2024-03-30T01:01:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.581+0000 7f2e39000700 1 -- 172.21.15.116:0/1894117263 learned_addr learned my addr 172.21.15.116:0/1894117263 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.581+0000 7f2e39000700 1 -- 172.21.15.116:0/1894117263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c790130 msgr2=0x55703c7a0a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:01:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.581+0000 7f2e39000700 1 --2- 172.21.15.116:0/1894117263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c790130 0x55703c7a0a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.581+0000 7f2e39000700 1 -- 172.21.15.116:0/1894117263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c790fd0 msgr2=0x55703c7941e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:01:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.581+0000 7f2e39000700 1 --2- 172.21.15.116:0/1894117263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c790fd0 0x55703c7941e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.581+0000 7f2e39000700 1 -- 172.21.15.116:0/1894117263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55703c6a6ea0 con 0x55703c78f7d0 2024-03-30T01:01:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.582+0000 7f2e39000700 1 --2- 172.21.15.116:0/1894117263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c78f7d0 0x55703c797600 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c002230 tx=0x7f2e2c0092f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea7fbe5d277f0c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.585+0000 7f2e2a000700 1 -- 172.21.15.116:0/1894117263 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c003d80 con 0x55703c78f7d0 2024-03-30T01:01:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.585+0000 7f2e2a000700 1 -- 172.21.15.116:0/1894117263 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e2c0050e0 con 0x55703c78f7d0 2024-03-30T01:01:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.585+0000 7f2e2a000700 1 -- 172.21.15.116:0/1894117263 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c029130 con 0x55703c78f7d0 2024-03-30T01:01:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.586+0000 7f2e4eacab80 1 -- 172.21.15.116:0/1894117263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c78f7d0 msgr2=0x55703c797600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.586+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/1894117263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c78f7d0 0x55703c797600 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c002230 tx=0x7f2e2c0092f0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:44 smithi130 ceph-mon[47626]: pgmap v407: 129 pgs: 129 active+clean; 909 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 17 MiB/s wr, 844 op/s 2024-03-30T01:01:44.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.614+0000 7f2e4eacab80 1 -- 172.21.15.116:0/1894117263 shutdown_connections 2024-03-30T01:01:44.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.614+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/1894117263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c790fd0 0x55703c7941e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.614+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/1894117263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c78f7d0 0x55703c797600 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.614+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/1894117263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c790130 0x55703c7a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.614+0000 7f2e4eacab80 1 -- 172.21.15.116:0/1894117263 >> 172.21.15.116:0/1894117263 conn(0x55703c6ee6a0 msgr2=0x55703c783f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.614+0000 7f2e4eacab80 1 -- 172.21.15.116:0/1894117263 shutdown_connections 2024-03-30T01:01:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.614+0000 7f2e4eacab80 1 -- 172.21.15.116:0/1894117263 wait complete. 2024-03-30T01:01:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.615+0000 7f2e4eacab80 1 -- start start 2024-03-30T01:01:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.616+0000 7f2e4eacab80 1 Processor -- start 2024-03-30T01:01:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.616+0000 7f2e4eacab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c78f7d0 0x55703c7a0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.617+0000 7f2e4eacab80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 0x55703c7a0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.617+0000 7f2e4eacab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c790fd0 0x55703c79ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.617+0000 7f2e4eacab80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55703c79ef60 con 0x55703c790130 2024-03-30T01:01:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.617+0000 7f2e4eacab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55703c79f0d0 con 0x55703c78f7d0 2024-03-30T01:01:44.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.617+0000 7f2e4eacab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55703c79f290 con 0x55703c790fd0 2024-03-30T01:01:44.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.617+0000 7f2e4eacab80 10 client.0 _ll_register_callbacks cb 0x55703c686470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:01:44.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.618+0000 7f2e33400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 0x55703c7a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:44.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.618+0000 7f2e33400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 0x55703c7a0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43740/0 (socket says 172.21.15.116:43740) 2024-03-30T01:01:44.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.618+0000 7f2e33400700 1 -- 172.21.15.116:0/3114301530 learned_addr learned my addr 172.21.15.116:0/3114301530 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:44.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.618+0000 7f2e33400700 1 -- 172.21.15.116:0/3114301530 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c790fd0 msgr2=0x55703c79ea20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:01:44.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.618+0000 7f2e33400700 1 --2- 172.21.15.116:0/3114301530 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c790fd0 0x55703c79ea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.618+0000 7f2e33400700 1 -- 172.21.15.116:0/3114301530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c78f7d0 msgr2=0x55703c7a0410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:01:44.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.618+0000 7f2e33400700 1 --2- 172.21.15.116:0/3114301530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c78f7d0 0x55703c7a0410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.618+0000 7f2e33400700 1 -- 172.21.15.116:0/3114301530 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55703c5623b0 con 0x55703c790130 2024-03-30T01:01:44.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.619+0000 7f2e33400700 1 --2- 172.21.15.116:0/3114301530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 0x55703c7a0950 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2e2001efd0 tx=0x7f2e20004800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4800d371bc7183c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:44.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.631+0000 7f2e1e000700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:01:44.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.631+0000 7f2e1e000700 1 -- 172.21.15.116:0/3114301530 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e20013ba0 con 0x55703c790130 2024-03-30T01:01:44.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.631+0000 7f2e1e000700 1 -- 172.21.15.116:0/3114301530 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2e2002d030 con 0x55703c790130 2024-03-30T01:01:44.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.655+0000 7f2e4eacab80 1 -- 172.21.15.116:0/3114301530 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2e2c0074e0 con 0x55703c790130 2024-03-30T01:01:44.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.657+0000 7f2e1e000700 1 -- 172.21.15.116:0/3114301530 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e20013d00 con 0x55703c790130 2024-03-30T01:01:44.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.658+0000 7f2e4eacab80 10 client.15993 fetch_fsmap learned FSMap version 27 2024-03-30T01:01:44.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.658+0000 7f2e4eacab80 1 -- 172.21.15.116:0/3114301530 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55703c566cd0 con 0x55703c790130 2024-03-30T01:01:44.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.659+0000 7f2e1e000700 1 -- 172.21.15.116:0/3114301530 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e2c0074e0 con 0x55703c790130 2024-03-30T01:01:44.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.659+0000 7f2e1e000700 1 -- 172.21.15.116:0/3114301530 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f2e20012360 con 0x55703c790130 2024-03-30T01:01:44.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.659+0000 7f2e4eacab80 10 client.15993 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T01:01:44.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.659+0000 7f2e4eacab80 10 client.15993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:01:44.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.659+0000 7f2e4eacab80 20 client.15993 populate_metadata read hostname 'smithi116' 2024-03-30T01:01:44.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.659+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/3114301530 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55703c81c550 0x55703c81e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:44.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.659+0000 7f2e4eacab80 4 client.15993 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T01:01:44.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.659+0000 7f2e4eacab80 1 -- 172.21.15.116:0/3114301530 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55703c52fcf0 con 0x55703c81c550 2024-03-30T01:01:44.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:44 smithi116 ceph-mon[38086]: pgmap v407: 129 pgs: 129 active+clean; 909 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 17 MiB/s wr, 844 op/s 2024-03-30T01:01:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.665+0000 7f2e39000700 1 --2- 172.21.15.116:0/3114301530 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55703c81c550 0x55703c81e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.681+0000 7f2e39000700 1 --2- 172.21.15.116:0/3114301530 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55703c81c550 0x55703c81e970 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f571923f10ae511c server_cookie=e4e78db3a8758d52 in_seq=0 out_seq=0 2024-03-30T01:01:44.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.687+0000 7f2e1e000700 10 client.15993 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:01:44.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.687+0000 7f2e1e000700 1 -- 172.21.15.116:0/3114301530 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55703c52fcf0 con 0x55703c81c550 2024-03-30T01:01:44.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.687+0000 7f2e1e000700 10 client.15993 handle_command_reply: tid=0 2024-03-30T01:01:44.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.687+0000 7f2e4eacab80 1 client.15993 shutdown 2024-03-30T01:01:44.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.694+0000 7f2e4eacab80 1 -- 172.21.15.116:0/3114301530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 msgr2=0x55703c7a0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:44.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.694+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/3114301530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 0x55703c7a0950 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2e2001efd0 tx=0x7f2e20004800 comp rx=0 tx=0).stop 2024-03-30T01:01:44.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.695+0000 7f2e4eacab80 1 -- 172.21.15.116:0/3114301530 shutdown_connections 2024-03-30T01:01:44.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.695+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/3114301530 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55703c81c550 0x55703c81e970 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.695+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/3114301530 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55703c790fd0 0x55703c79ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.695+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/3114301530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55703c78f7d0 0x55703c7a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.695+0000 7f2e4eacab80 1 --2- 172.21.15.116:0/3114301530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703c790130 0x55703c7a0950 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.695+0000 7f2e4eacab80 1 -- 172.21.15.116:0/3114301530 >> 172.21.15.116:0/3114301530 conn(0x55703c6ee6a0 msgr2=0x55703c7975a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:44.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.695+0000 7f2e1e000700 0 client.15993 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:01:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.706+0000 7f2e4eacab80 1 -- 172.21.15.116:0/3114301530 shutdown_connections 2024-03-30T01:01:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.708+0000 7f2e4eacab80 1 -- 172.21.15.116:0/3114301530 wait complete. 2024-03-30T01:01:44.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:44 smithi181 ceph-mon[47622]: pgmap v407: 129 pgs: 129 active+clean; 909 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 17 MiB/s wr, 844 op/s 2024-03-30T01:01:44.768 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:01:44.769 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-30T01:01:44.769 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-30T01:01:44.769 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:01:44.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.757+0000 7f2e4eacab80 20 client.15993 trim_cache size 0 max 16384 2024-03-30T01:01:44.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.833+0000 7f2e3b000700 1 -- 172.21.15.116:0/3728423210 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2e180a1320 msgr2=0x7f2e180a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:44.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.833+0000 7f2e3b000700 1 --2- 172.21.15.116:0/3728423210 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2e180a1320 0x7f2e180a37e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2e240ca020 tx=0x7f2e24010a90 comp rx=0 tx=0).stop 2024-03-30T01:01:44.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.833+0000 7f2e3b000700 1 -- 172.21.15.116:0/3728423210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e340722b0 msgr2=0x7f2e3419d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:44.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.833+0000 7f2e3b000700 1 --2- 172.21.15.116:0/3728423210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e340722b0 0x7f2e3419d3d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400b6a0 tx=0x7f2e2400ba60 comp rx=0 tx=0).stop 2024-03-30T01:01:44.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.834+0000 7f2e3b000700 1 -- 172.21.15.116:0/3728423210 shutdown_connections 2024-03-30T01:01:44.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.834+0000 7f2e3b000700 1 --2- 172.21.15.116:0/3728423210 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2e180a1320 0x7f2e180a37e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.834+0000 7f2e3b000700 1 --2- 172.21.15.116:0/3728423210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e34108de0 0x7f2e341a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.834+0000 7f2e3b000700 1 --2- 172.21.15.116:0/3728423210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e340722b0 0x7f2e3419d3d0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.834+0000 7f2e3b000700 1 --2- 172.21.15.116:0/3728423210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e34072740 0x7f2e3419d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:44.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.834+0000 7f2e3b000700 1 -- 172.21.15.116:0/3728423210 >> 172.21.15.116:0/3728423210 conn(0x7f2e3406dbc0 msgr2=0x7f2e3410b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.892+0000 7f2e3b000700 1 -- 172.21.15.116:0/3728423210 shutdown_connections 2024-03-30T01:01:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:44.892+0000 7f2e3b000700 1 -- 172.21.15.116:0/3728423210 wait complete. 2024-03-30T01:01:45.435 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T01:01:45.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T01:01:46.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:46 smithi130 ceph-mon[47626]: pgmap v408: 129 pgs: 129 active+clean; 911 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 16 MiB/s wr, 785 op/s 2024-03-30T01:01:46.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:46 smithi116 ceph-mon[38086]: pgmap v408: 129 pgs: 129 active+clean; 911 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 16 MiB/s wr, 785 op/s 2024-03-30T01:01:46.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:46 smithi181 ceph-mon[47622]: pgmap v408: 129 pgs: 129 active+clean; 911 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 16 MiB/s wr, 785 op/s 2024-03-30T01:01:48.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:48 smithi130 ceph-mon[47626]: pgmap v409: 129 pgs: 129 active+clean; 914 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 11 MiB/s wr, 606 op/s 2024-03-30T01:01:48.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:48 smithi116 ceph-mon[38086]: pgmap v409: 129 pgs: 129 active+clean; 914 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 11 MiB/s wr, 606 op/s 2024-03-30T01:01:48.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:48 smithi181 ceph-mon[47622]: pgmap v409: 129 pgs: 129 active+clean; 914 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 11 MiB/s wr, 606 op/s 2024-03-30T01:01:49.021 INFO:tasks.workunit.client.0.smithi116.stdout: 264 643525 1729 448620 1720 206886 1601 2024-03-30T01:01:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.160+0000 7f5749200700 1 -- 172.21.15.116:0/1836221900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f574410a500 msgr2=0x7f574410c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.160+0000 7f5749200700 1 --2- 172.21.15.116:0/1836221900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f574410a500 0x7f574410c990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f573c00b3e0 tx=0x7f573c00b6f0 comp rx=0 tx=0).stop 2024-03-30T01:01:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.161+0000 7f5749200700 1 -- 172.21.15.116:0/1836221900 shutdown_connections 2024-03-30T01:01:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.161+0000 7f5749200700 1 --2- 172.21.15.116:0/1836221900 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f574410cf20 0x7f574410d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.161+0000 7f5749200700 1 --2- 172.21.15.116:0/1836221900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5744107b90 0x7f5744109fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.161+0000 7f5749200700 1 --2- 172.21.15.116:0/1836221900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f574410a500 0x7f574410c990 secure :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f573c00b3e0 tx=0x7f573c00b6f0 comp rx=0 tx=0).stop 2024-03-30T01:01:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.161+0000 7f5749200700 1 -- 172.21.15.116:0/1836221900 >> 172.21.15.116:0/1836221900 conn(0x7f574406dc30 msgr2=0x7f5744070070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:01:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.163+0000 7f5749200700 1 -- 172.21.15.116:0/1836221900 shutdown_connections 2024-03-30T01:01:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.165+0000 7f5749200700 1 -- 172.21.15.116:0/1836221900 wait complete. 2024-03-30T01:01:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.166+0000 7f5749200700 1 Processor -- start 2024-03-30T01:01:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.173+0000 7f5749200700 1 -- start start 2024-03-30T01:01:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.175+0000 7f5749200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5744107b90 0x7f574419d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:50.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.185+0000 7f5749200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f574410cf20 0x7f574419d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:50.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.185+0000 7f5749200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57441a2970 0x7f57441a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:50.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.185+0000 7f5749200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57441a5330 con 0x7f5744107b90 2024-03-30T01:01:50.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.185+0000 7f5749200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57441a5470 con 0x7f57441a2970 2024-03-30T01:01:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.185+0000 7f5749200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57441a5600 con 0x7f574410cf20 2024-03-30T01:01:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.190+0000 7f5742800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57441a2970 0x7f57441a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.190+0000 7f5742800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57441a2970 0x7f57441a4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:42608/0 (socket says 172.21.15.116:42608) 2024-03-30T01:01:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.190+0000 7f5742800700 1 -- 172.21.15.116:0/2597067489 learned_addr learned my addr 172.21.15.116:0/2597067489 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:01:50.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.191+0000 7f5741e00700 1 --2- 172.21.15.116:0/2597067489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5744107b90 0x7f574419d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:50.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.192+0000 7f5741e00700 1 -- 172.21.15.116:0/2597067489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f574410cf20 msgr2=0x7f574419d960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T01:01:50.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.192+0000 7f5741e00700 1 --2- 172.21.15.116:0/2597067489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f574410cf20 0x7f574419d960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:50.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.192+0000 7f5741e00700 1 -- 172.21.15.116:0/2597067489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57441a2970 msgr2=0x7f57441a4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:01:50.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.192+0000 7f5741e00700 1 --2- 172.21.15.116:0/2597067489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57441a2970 0x7f57441a4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:01:50.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.192+0000 7f5741e00700 1 -- 172.21.15.116:0/2597067489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c009c40 con 0x7f5744107b90 2024-03-30T01:01:50.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.192+0000 7f5741e00700 1 --2- 172.21.15.116:0/2597067489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5744107b90 0x7f574419d420 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f573000e9f0 tx=0x7f573000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:50.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.231+0000 7f573aa00700 1 -- 172.21.15.116:0/2597067489 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573000eee0 con 0x7f5744107b90 2024-03-30T01:01:50.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.231+0000 7f573aa00700 1 -- 172.21.15.116:0/2597067489 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f573000cbd0 con 0x7f5744107b90 2024-03-30T01:01:50.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.231+0000 7f573aa00700 1 -- 172.21.15.116:0/2597067489 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5730018680 con 0x7f5744107b90 2024-03-30T01:01:50.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.256+0000 7f5749200700 1 -- 172.21.15.116:0/2597067489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57441a58e0 con 0x7f5744107b90 2024-03-30T01:01:50.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.256+0000 7f5749200700 1 -- 172.21.15.116:0/2597067489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57441a5d70 con 0x7f5744107b90 2024-03-30T01:01:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.309+0000 7f573aa00700 1 -- 172.21.15.116:0/2597067489 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f573000cd40 con 0x7f5744107b90 2024-03-30T01:01:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.312+0000 7f573aa00700 1 --2- 172.21.15.116:0/2597067489 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f57280a1320 0x7f57280a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:01:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.316+0000 7f573aa00700 1 -- 172.21.15.116:0/2597067489 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5730014070 con 0x7f5744107b90 2024-03-30T01:01:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.286+0000 7f5749200700 1 -- 172.21.15.116:0/2597067489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5744067970 con 0x7f5744107b90 2024-03-30T01:01:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.321+0000 7f5741400700 1 --2- 172.21.15.116:0/2597067489 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f57280a1320 0x7f57280a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:01:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.326+0000 7f5741400700 1 --2- 172.21.15.116:0/2597067489 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f57280a1320 0x7f57280a37e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f573c000f80 tx=0x7f573c007b00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:01:50.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:01:50.371+0000 7f573aa00700 1 -- 172.21.15.116:0/2597067489 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5730096070 con 0x7f5744107b90 2024-03-30T01:01:50.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:50 smithi130 ceph-mon[47626]: pgmap v410: 129 pgs: 129 active+clean; 913 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.2 MiB/s wr, 442 op/s 2024-03-30T01:01:50.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:50 smithi116 ceph-mon[38086]: pgmap v410: 129 pgs: 129 active+clean; 913 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.2 MiB/s wr, 442 op/s 2024-03-30T01:01:50.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:50 smithi181 ceph-mon[47622]: pgmap v410: 129 pgs: 129 active+clean; 913 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.2 MiB/s wr, 442 op/s 2024-03-30T01:01:52.598 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:52 smithi130 ceph-mon[47626]: pgmap v411: 129 pgs: 129 active+clean; 928 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 321 op/s 2024-03-30T01:01:52.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:52 smithi116 ceph-mon[38086]: pgmap v411: 129 pgs: 129 active+clean; 928 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 321 op/s 2024-03-30T01:01:52.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:52 smithi181 ceph-mon[47622]: pgmap v411: 129 pgs: 129 active+clean; 928 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 321 op/s 2024-03-30T01:01:54.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:54 smithi130 ceph-mon[47626]: pgmap v412: 129 pgs: 129 active+clean; 926 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 253 op/s 2024-03-30T01:01:54.678 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:54 smithi116 ceph-mon[38086]: pgmap v412: 129 pgs: 129 active+clean; 926 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 253 op/s 2024-03-30T01:01:54.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:54 smithi181 ceph-mon[47622]: pgmap v412: 129 pgs: 129 active+clean; 926 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 253 op/s 2024-03-30T01:01:56.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:56 smithi116 ceph-mon[38086]: pgmap v413: 129 pgs: 129 active+clean; 933 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 275 op/s 2024-03-30T01:01:56.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:56 smithi181 ceph-mon[47622]: pgmap v413: 129 pgs: 129 active+clean; 933 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 275 op/s 2024-03-30T01:01:56.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:56 smithi130 ceph-mon[47626]: pgmap v413: 129 pgs: 129 active+clean; 933 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 275 op/s 2024-03-30T01:01:58.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:01:58 smithi116 ceph-mon[38086]: pgmap v414: 129 pgs: 129 active+clean; 939 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 267 op/s 2024-03-30T01:01:58.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:01:58 smithi181 ceph-mon[47622]: pgmap v414: 129 pgs: 129 active+clean; 939 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 267 op/s 2024-03-30T01:01:58.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:01:58 smithi130 ceph-mon[47626]: pgmap v414: 129 pgs: 129 active+clean; 939 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 267 op/s 2024-03-30T01:01:59.021 INFO:tasks.workunit.client.0.smithi116.stdout: 285 620939 1381 436530 1122 203639 2291 2024-03-30T01:02:00.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:00.213+0000 7f5749200700 1 -- 172.21.15.116:0/2597067489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f571c005180 con 0x7f5744107b90 2024-03-30T01:02:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:00.214+0000 7f573aa00700 1 -- 172.21.15.116:0/2597067489 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10892 (secure 0 0 0) 0x7f5730099690 con 0x7f5744107b90 2024-03-30T01:02:00.245 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T01:02:00.245 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T01:01:14.261079+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T01:02:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:00.319+0000 7f5749200700 1 -- 172.21.15.116:0/2597067489 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f57280a1320 msgr2=0x7f57280a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:00.319+0000 7f5749200700 1 --2- 172.21.15.116:0/2597067489 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f57280a1320 0x7f57280a37e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f573c000f80 tx=0x7f573c007b00 comp rx=0 tx=0).stop 2024-03-30T01:02:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:00.319+0000 7f5749200700 1 -- 172.21.15.116:0/2597067489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5744107b90 msgr2=0x7f574419d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:00.319+0000 7f5749200700 1 --2- 172.21.15.116:0/2597067489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5744107b90 0x7f574419d420 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f573000e9f0 tx=0x7f573000ed00 comp rx=0 tx=0).stop 2024-03-30T01:02:00.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:00.323+0000 7f5749200700 1 -- 172.21.15.116:0/2597067489 shutdown_connections 2024-03-30T01:02:00.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:00.323+0000 7f5749200700 1 --2- 172.21.15.116:0/2597067489 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f57280a1320 0x7f57280a37e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:00.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:00.323+0000 7f5749200700 1 --2- 172.21.15.116:0/2597067489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57441a2970 0x7f57441a4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:00.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:00.323+0000 7f5749200700 1 --2- 172.21.15.116:0/2597067489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5744107b90 0x7f574419d420 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:00.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:00.323+0000 7f5749200700 1 --2- 172.21.15.116:0/2597067489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f574410cf20 0x7f574419d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:00.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:00.323+0000 7f5749200700 1 -- 172.21.15.116:0/2597067489 >> 172.21.15.116:0/2597067489 conn(0x7f574406dc30 msgr2=0x7f574410ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:00.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:00.326+0000 7f5749200700 1 -- 172.21.15.116:0/2597067489 shutdown_connections 2024-03-30T01:02:00.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:00.331+0000 7f5749200700 1 -- 172.21.15.116:0/2597067489 wait complete. 2024-03-30T01:02:00.412 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 27 2024-03-30T01:02:00.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:00 smithi116 ceph-mon[38086]: pgmap v415: 129 pgs: 129 active+clean; 943 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 298 op/s 2024-03-30T01:02:00.674 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:00 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2597067489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:02:00.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:00 smithi181 ceph-mon[47622]: pgmap v415: 129 pgs: 129 active+clean; 943 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 298 op/s 2024-03-30T01:02:00.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:00 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2597067489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:02:00.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:00 smithi130 ceph-mon[47626]: pgmap v415: 129 pgs: 129 active+clean; 943 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 298 op/s 2024-03-30T01:02:00.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:00 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2597067489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:02:01.033 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T01:02:02.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:02 smithi116 ceph-mon[38086]: pgmap v416: 129 pgs: 129 active+clean; 955 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.7 MiB/s wr, 354 op/s 2024-03-30T01:02:02.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:02 smithi181 ceph-mon[47622]: pgmap v416: 129 pgs: 129 active+clean; 955 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.7 MiB/s wr, 354 op/s 2024-03-30T01:02:02.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:02 smithi130 ceph-mon[47626]: pgmap v416: 129 pgs: 129 active+clean; 955 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.7 MiB/s wr, 354 op/s 2024-03-30T01:02:04.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:04 smithi116 ceph-mon[38086]: pgmap v417: 129 pgs: 129 active+clean; 963 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 358 op/s 2024-03-30T01:02:04.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:04 smithi181 ceph-mon[47622]: pgmap v417: 129 pgs: 129 active+clean; 963 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 358 op/s 2024-03-30T01:02:04.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:04 smithi130 ceph-mon[47626]: pgmap v417: 129 pgs: 129 active+clean; 963 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 358 op/s 2024-03-30T01:02:06.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.200+0000 7fd37b200700 1 -- 172.21.15.116:0/3232642550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd37410d0f0 msgr2=0x7fd37410d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:06.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.200+0000 7fd37b200700 1 --2- 172.21.15.116:0/3232642550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd37410d0f0 0x7fd37410d570 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd36c00b3e0 tx=0x7fd36c00b6f0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.208+0000 7fd37b200700 1 -- 172.21.15.116:0/3232642550 shutdown_connections 2024-03-30T01:02:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.208+0000 7fd37b200700 1 --2- 172.21.15.116:0/3232642550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd37410d0f0 0x7fd37410d570 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.208+0000 7fd37b200700 1 --2- 172.21.15.116:0/3232642550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd374107db0 0x7fd37410a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.208+0000 7fd37b200700 1 --2- 172.21.15.116:0/3232642550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd37410a720 0x7fd37410cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.208+0000 7fd37b200700 1 -- 172.21.15.116:0/3232642550 >> 172.21.15.116:0/3232642550 conn(0x7fd37406dd50 msgr2=0x7fd374070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:06.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.210+0000 7fd37b200700 1 -- 172.21.15.116:0/3232642550 shutdown_connections 2024-03-30T01:02:06.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.210+0000 7fd37b200700 1 -- 172.21.15.116:0/3232642550 wait complete. 2024-03-30T01:02:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.215+0000 7fd37b200700 1 Processor -- start 2024-03-30T01:02:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.218+0000 7fd37b200700 1 -- start start 2024-03-30T01:02:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.218+0000 7fd37b200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd374107db0 0x7fd3741a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.219+0000 7fd37b200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd37410a720 0x7fd3741a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.220+0000 7fd373400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd37410a720 0x7fd3741a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.220+0000 7fd373400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd37410a720 0x7fd3741a6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36058/0 (socket says 172.21.15.116:36058) 2024-03-30T01:02:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.220+0000 7fd373e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd374107db0 0x7fd3741a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.220+0000 7fd373e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd374107db0 0x7fd3741a5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:34120/0 (socket says 172.21.15.116:34120) 2024-03-30T01:02:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.220+0000 7fd37b200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd37410d0f0 0x7fd3741ad050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.221+0000 7fd373e00700 1 -- 172.21.15.116:0/1056308 learned_addr learned my addr 172.21.15.116:0/1056308 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.221+0000 7fd37b200700 1 -- 172.21.15.116:0/1056308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3741ad620 con 0x7fd37410a720 2024-03-30T01:02:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.221+0000 7fd37b200700 1 -- 172.21.15.116:0/1056308 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3741ad760 con 0x7fd37410d0f0 2024-03-30T01:02:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.221+0000 7fd37b200700 1 -- 172.21.15.116:0/1056308 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3741ad8f0 con 0x7fd374107db0 2024-03-30T01:02:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.221+0000 7fd379200700 1 --2- 172.21.15.116:0/1056308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd37410d0f0 0x7fd3741ad050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.222+0000 7fd373400700 1 -- 172.21.15.116:0/1056308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd374107db0 msgr2=0x7fd3741a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.222+0000 7fd373400700 1 --2- 172.21.15.116:0/1056308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd374107db0 0x7fd3741a5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.222+0000 7fd373400700 1 -- 172.21.15.116:0/1056308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd37410d0f0 msgr2=0x7fd3741ad050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.222+0000 7fd373400700 1 --2- 172.21.15.116:0/1056308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd37410d0f0 0x7fd3741ad050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.222+0000 7fd373400700 1 -- 172.21.15.116:0/1056308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36c009c40 con 0x7fd37410a720 2024-03-30T01:02:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.222+0000 7fd373400700 1 --2- 172.21.15.116:0/1056308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd37410a720 0x7fd3741a6080 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd36000d9c0 tx=0x7fd36000dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.223+0000 7fd370c00700 1 -- 172.21.15.116:0/1056308 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360009930 con 0x7fd37410a720 2024-03-30T01:02:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.223+0000 7fd370c00700 1 -- 172.21.15.116:0/1056308 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd360010460 con 0x7fd37410a720 2024-03-30T01:02:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.223+0000 7fd370c00700 1 -- 172.21.15.116:0/1056308 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36000b5d0 con 0x7fd37410a720 2024-03-30T01:02:06.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.225+0000 7fd37b200700 1 -- 172.21.15.116:0/1056308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3741adbd0 con 0x7fd37410a720 2024-03-30T01:02:06.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.225+0000 7fd37b200700 1 -- 172.21.15.116:0/1056308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3741ae090 con 0x7fd37410a720 2024-03-30T01:02:06.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.249+0000 7fd38ed51b80 1 Processor -- start 2024-03-30T01:02:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.250+0000 7fd370c00700 1 -- 172.21.15.116:0/1056308 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fd3600105d0 con 0x7fd37410a720 2024-03-30T01:02:06.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.254+0000 7fd38ed51b80 1 -- start start 2024-03-30T01:02:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.262+0000 7fd370c00700 1 --2- 172.21.15.116:0/1056308 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd3580a1320 0x7fd3580a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.262+0000 7fd370c00700 1 -- 172.21.15.116:0/1056308 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd3600cc440 con 0x7fd37410a720 2024-03-30T01:02:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.263+0000 7fd38ed51b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb7fabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.263+0000 7fd38ed51b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fb130 0x559fbb7fb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.263+0000 7fd38ed51b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb7fc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.263+0000 7fd38ed51b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fbb752990 con 0x559fbb7fbfd0 2024-03-30T01:02:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.263+0000 7fd38ed51b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fbb750350 con 0x559fbb7fb130 2024-03-30T01:02:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.263+0000 7fd38ed51b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fbb74bc40 con 0x559fbb7fa7d0 2024-03-30T01:02:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.264+0000 7fd379200700 1 --2- 172.21.15.116:0/1056308 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd3580a1320 0x7fd3580a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.265+0000 7fd379200700 1 --2- 172.21.15.116:0/1056308 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd3580a1320 0x7fd3580a37e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd36c00d010 tx=0x7fd36c00ba40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.266+0000 7fd373e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb7fabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.266+0000 7fd373e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb7fabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:34132/0 (socket says 172.21.15.116:34132) 2024-03-30T01:02:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.266+0000 7fd373e00700 1 -- 172.21.15.116:0/2574342237 learned_addr learned my addr 172.21.15.116:0/2574342237 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.268+0000 7fd373e00700 1 -- 172.21.15.116:0/2574342237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fb130 msgr2=0x559fbb7fb9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:02:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.268+0000 7fd373e00700 1 --2- 172.21.15.116:0/2574342237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fb130 0x559fbb7fb9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.268+0000 7fd373e00700 1 -- 172.21.15.116:0/2574342237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 msgr2=0x559fbb7fc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.268+0000 7fd373e00700 1 --2- 172.21.15.116:0/2574342237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb7fc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.268+0000 7fd373e00700 1 -- 172.21.15.116:0/2574342237 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fbb5cd3b0 con 0x559fbb7fa7d0 2024-03-30T01:02:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.268+0000 7fd373e00700 1 --2- 172.21.15.116:0/2574342237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb7fabf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd364009dc0 tx=0x7fd3640116c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=738c3b38a8d8c699 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:06.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.282+0000 7fd369600700 1 -- 172.21.15.116:0/2574342237 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd364009890 con 0x559fbb7fa7d0 2024-03-30T01:02:06.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.282+0000 7fd369600700 1 -- 172.21.15.116:0/2574342237 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd364011ed0 con 0x559fbb7fa7d0 2024-03-30T01:02:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.283+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2574342237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 msgr2=0x559fbb7fabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.283+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2574342237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb7fabf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd364009dc0 tx=0x7fd3640116c0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.283+0000 7fd369600700 1 -- 172.21.15.116:0/2574342237 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36400f8e0 con 0x559fbb7fa7d0 2024-03-30T01:02:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.283+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2574342237 shutdown_connections 2024-03-30T01:02:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.283+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2574342237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb7fabf0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.283+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2574342237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb7fc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.283+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2574342237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fb130 0x559fbb7fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.283+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2574342237 >> 172.21.15.116:0/2574342237 conn(0x559fbb7596a0 msgr2=0x559fbb7f59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.339+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2574342237 shutdown_connections 2024-03-30T01:02:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.339+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2574342237 wait complete. 2024-03-30T01:02:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.342+0000 7fd38ed51b80 1 -- start start 2024-03-30T01:02:06.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.355+0000 7fd38ed51b80 1 Processor -- start 2024-03-30T01:02:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.372+0000 7fd38ed51b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb804470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.377+0000 7fd373e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb804470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.377+0000 7fd373e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb804470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:34144/0 (socket says 172.21.15.116:34144) 2024-03-30T01:02:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.378+0000 7fd373e00700 1 -- 172.21.15.116:0/2446908021 learned_addr learned my addr 172.21.15.116:0/2446908021 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.378+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2446908021 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fb130 0x559fbb8049b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.378+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2446908021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb80b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:06.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.383+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2446908021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fbb6e7d80 con 0x559fbb7fbfd0 2024-03-30T01:02:06.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.383+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2446908021 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fbb6e7a20 con 0x559fbb7fb130 2024-03-30T01:02:06.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.383+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2446908021 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fbb5cf6a0 con 0x559fbb7fa7d0 2024-03-30T01:02:06.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.384+0000 7fd379200700 1 --2- 172.21.15.116:0/2446908021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb80b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:06.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.386+0000 7fd38ed51b80 10 client.0 _ll_register_callbacks cb 0x559fbb6f1470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:02:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.390+0000 7fd379200700 1 -- 172.21.15.116:0/2446908021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 msgr2=0x559fbb804470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.390+0000 7fd379200700 1 --2- 172.21.15.116:0/2446908021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb804470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.390+0000 7fd379200700 1 -- 172.21.15.116:0/2446908021 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fb130 msgr2=0x559fbb8049b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:02:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.390+0000 7fd379200700 1 --2- 172.21.15.116:0/2446908021 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fb130 0x559fbb8049b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.390+0000 7fd379200700 1 -- 172.21.15.116:0/2446908021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3640099f0 con 0x559fbb7fbfd0 2024-03-30T01:02:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.393+0000 7fd35e000700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:02:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.393+0000 7fd379200700 1 --2- 172.21.15.116:0/2446908021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb80b9f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd36c007a30 tx=0x7fd36c0078f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afefe0431b1bde06 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.395+0000 7fd35e000700 1 -- 172.21.15.116:0/2446908021 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c00e040 con 0x559fbb7fbfd0 2024-03-30T01:02:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.396+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2446908021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559fbb70c380 con 0x559fbb7fbfd0 2024-03-30T01:02:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.396+0000 7fd35e000700 1 -- 172.21.15.116:0/2446908021 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd36c00be40 con 0x559fbb7fbfd0 2024-03-30T01:02:06.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.403+0000 7fd35e000700 1 -- 172.21.15.116:0/2446908021 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd36c00e040 con 0x559fbb7fbfd0 2024-03-30T01:02:06.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.407+0000 7fd38ed51b80 10 client.16008 fetch_fsmap learned FSMap version 27 2024-03-30T01:02:06.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.407+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2446908021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x559fbb579b60 con 0x559fbb7fbfd0 2024-03-30T01:02:06.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.408+0000 7fd35e000700 1 -- 172.21.15.116:0/2446908021 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559fbb70c380 con 0x559fbb7fbfd0 2024-03-30T01:02:06.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.419+0000 7fd35e000700 1 -- 172.21.15.116:0/2446908021 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fd36c01c560 con 0x559fbb7fbfd0 2024-03-30T01:02:06.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.419+0000 7fd38ed51b80 10 client.16008 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T01:02:06.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.419+0000 7fd38ed51b80 10 client.16008 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T01:02:06.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.419+0000 7fd38ed51b80 20 client.16008 populate_metadata read hostname 'smithi116' 2024-03-30T01:02:06.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.419+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2446908021 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x559fbb810060 0x559fbb812480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:06.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.419+0000 7fd38ed51b80 4 client.16008 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:02:06.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.419+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2446908021 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559fbb744700 con 0x559fbb810060 2024-03-30T01:02:06.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.419+0000 7fd373e00700 1 --2- 172.21.15.116:0/2446908021 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x559fbb810060 0x559fbb812480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:06.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.430+0000 7fd373e00700 1 --2- 172.21.15.116:0/2446908021 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x559fbb810060 0x559fbb812480 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e0ad61182ad11189 server_cookie=d003c77b902f420f in_seq=0 out_seq=0 2024-03-30T01:02:06.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.458+0000 7fd35e000700 10 client.16008 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T01:02:06.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.458+0000 7fd35e000700 1 -- 172.21.15.116:0/2446908021 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559fbb744700 con 0x559fbb810060 2024-03-30T01:02:06.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.458+0000 7fd35e000700 10 client.16008 handle_command_reply: tid=0 2024-03-30T01:02:06.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.466+0000 7fd38ed51b80 1 client.16008 shutdown 2024-03-30T01:02:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.473+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2446908021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 msgr2=0x559fbb80b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.473+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2446908021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb80b9f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd36c007a30 tx=0x7fd36c0078f0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.487+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2446908021 shutdown_connections 2024-03-30T01:02:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.487+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2446908021 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x559fbb810060 0x559fbb812480 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.487+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2446908021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb804470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.487+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2446908021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb80b9f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.487+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2446908021 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fb130 0x559fbb8049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.487+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2446908021 >> 172.21.15.116:0/2446908021 conn(0x559fbb7596a0 msgr2=0x559fbb7f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.487+0000 7fd35e000700 0 client.16008 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T01:02:06.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.491+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2446908021 shutdown_connections 2024-03-30T01:02:06.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.501+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2446908021 wait complete. 2024-03-30T01:02:06.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:06.536+0000 7fd38ed51b80 20 client.16008 trim_cache size 0 max 16384 2024-03-30T01:02:06.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:06 smithi181 ceph-mon[47622]: pgmap v418: 129 pgs: 129 active+clean; 1000 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 7.3 MiB/s wr, 493 op/s 2024-03-30T01:02:06.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:06 smithi130 ceph-mon[47626]: pgmap v418: 129 pgs: 129 active+clean; 1000 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 7.3 MiB/s wr, 493 op/s 2024-03-30T01:02:06.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:06 smithi116 ceph-mon[38086]: pgmap v418: 129 pgs: 129 active+clean; 1000 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 7.3 MiB/s wr, 493 op/s 2024-03-30T01:02:07.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.440+0000 7fd38ed51b80 1 Processor -- start 2024-03-30T01:02:07.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.505+0000 7fd38ed51b80 1 -- start start 2024-03-30T01:02:07.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.505+0000 7fd38ed51b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb7fc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:07.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.505+0000 7fd38ed51b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fb130 0x559fbb80b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:07.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.505+0000 7fd38ed51b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb7fb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:07.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.505+0000 7fd38ed51b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fbb7f64f0 con 0x559fbb7fbfd0 2024-03-30T01:02:07.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.505+0000 7fd38ed51b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fbb80cb60 con 0x559fbb7fb130 2024-03-30T01:02:07.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.505+0000 7fd38ed51b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fbb802ff0 con 0x559fbb7fa7d0 2024-03-30T01:02:07.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.506+0000 7fd379200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb7fb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:07.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.507+0000 7fd379200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb7fb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36082/0 (socket says 172.21.15.116:36082) 2024-03-30T01:02:07.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.507+0000 7fd379200700 1 -- 172.21.15.116:0/2343134066 learned_addr learned my addr 172.21.15.116:0/2343134066 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:07.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.507+0000 7fd373400700 1 --2- 172.21.15.116:0/2343134066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fb130 0x559fbb80b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:07.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.507+0000 7fd373e00700 1 --2- 172.21.15.116:0/2343134066 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb7fc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:07.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.507+0000 7fd373400700 1 -- 172.21.15.116:0/2343134066 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 msgr2=0x559fbb7fc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:07.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.507+0000 7fd373400700 1 --2- 172.21.15.116:0/2343134066 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb7fc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.508+0000 7fd373400700 1 -- 172.21.15.116:0/2343134066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 msgr2=0x559fbb7fb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:07.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.508+0000 7fd373400700 1 --2- 172.21.15.116:0/2343134066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb7fb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.508+0000 7fd373400700 1 -- 172.21.15.116:0/2343134066 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fbb711ea0 con 0x559fbb7fb130 2024-03-30T01:02:07.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.508+0000 7fd373400700 1 --2- 172.21.15.116:0/2343134066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fb130 0x559fbb80b9b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd3600992a0 tx=0x7fd3600993f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6d82b2298927bed server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:07.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.508+0000 7fd36a000700 1 -- 172.21.15.116:0/2343134066 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360098f00 con 0x559fbb7fb130 2024-03-30T01:02:07.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.509+0000 7fd36a000700 1 -- 172.21.15.116:0/2343134066 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd360099ca0 con 0x559fbb7fb130 2024-03-30T01:02:07.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.509+0000 7fd36a000700 1 -- 172.21.15.116:0/2343134066 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3600cc270 con 0x559fbb7fb130 2024-03-30T01:02:07.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.513+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2343134066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fb130 msgr2=0x559fbb80b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:07.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.513+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2343134066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fb130 0x559fbb80b9b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd3600992a0 tx=0x7fd3600993f0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.518+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2343134066 shutdown_connections 2024-03-30T01:02:07.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.518+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2343134066 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fa7d0 0x559fbb7fc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.518+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2343134066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb7fb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.518+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/2343134066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fb130 0x559fbb80b9b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.518+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2343134066 >> 172.21.15.116:0/2343134066 conn(0x559fbb7596a0 msgr2=0x559fbb7ee5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:07.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.522+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2343134066 shutdown_connections 2024-03-30T01:02:07.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.523+0000 7fd38ed51b80 1 -- 172.21.15.116:0/2343134066 wait complete. 2024-03-30T01:02:07.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.568+0000 7fd38ed51b80 1 -- start start 2024-03-30T01:02:07.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.568+0000 7fd38ed51b80 1 Processor -- start 2024-03-30T01:02:07.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.600+0000 7fd38ed51b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fa7d0 0x559fbb80b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:07.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.600+0000 7fd38ed51b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fb130 0x559fbb80b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:07.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.600+0000 7fd38ed51b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb807fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:07.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.600+0000 7fd38ed51b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fbb8086f0 con 0x559fbb7fbfd0 2024-03-30T01:02:07.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.600+0000 7fd38ed51b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fbb808860 con 0x559fbb7fa7d0 2024-03-30T01:02:07.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.600+0000 7fd38ed51b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fbb808a20 con 0x559fbb7fb130 2024-03-30T01:02:07.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.601+0000 7fd379200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb807fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:07.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.601+0000 7fd379200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb807fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36094/0 (socket says 172.21.15.116:36094) 2024-03-30T01:02:07.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.601+0000 7fd379200700 1 -- 172.21.15.116:0/805043538 learned_addr learned my addr 172.21.15.116:0/805043538 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:07.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.601+0000 7fd373400700 1 --2- 172.21.15.116:0/805043538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fb130 0x559fbb80b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:07.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.602+0000 7fd373400700 1 -- 172.21.15.116:0/805043538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fa7d0 msgr2=0x559fbb80b450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:02:07.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.602+0000 7fd373400700 1 --2- 172.21.15.116:0/805043538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fa7d0 0x559fbb80b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.602+0000 7fd373400700 1 -- 172.21.15.116:0/805043538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 msgr2=0x559fbb807fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:07.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.602+0000 7fd373400700 1 --2- 172.21.15.116:0/805043538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb807fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.602+0000 7fd38ed51b80 10 client.0 _ll_register_callbacks cb 0x559fbb6f1470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:02:07.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.602+0000 7fd373400700 1 -- 172.21.15.116:0/805043538 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3741ae090 con 0x559fbb7fb130 2024-03-30T01:02:07.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.602+0000 7fd379200700 1 --2- 172.21.15.116:0/805043538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb807fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:02:07.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.602+0000 7fd38ed51b80 1 -- 172.21.15.116:0/805043538 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd364011ed0 con 0x559fbb7fb130 2024-03-30T01:02:07.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.603+0000 7fd373400700 1 --2- 172.21.15.116:0/805043538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fb130 0x559fbb80b990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x559fbb804030 tx=0x7fd360099e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee184946c088083d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:07.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.644+0000 7fd35e000700 10 client.26047 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:02:07.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.644+0000 7fd35e000700 1 -- 172.21.15.116:0/805043538 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3600cc210 con 0x559fbb7fb130 2024-03-30T01:02:07.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.645+0000 7fd35e000700 1 -- 172.21.15.116:0/805043538 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd3600ad0a0 con 0x559fbb7fb130 2024-03-30T01:02:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.645+0000 7fd35e000700 1 -- 172.21.15.116:0/805043538 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3600c9c50 con 0x559fbb7fb130 2024-03-30T01:02:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.659+0000 7fd38ed51b80 10 client.26047 fetch_fsmap learned FSMap version 27 2024-03-30T01:02:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.659+0000 7fd38ed51b80 1 -- 172.21.15.116:0/805043538 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x559fbb5d1cd0 con 0x559fbb7fb130 2024-03-30T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.662+0000 7fd35e000700 1 -- 172.21.15.116:0/805043538 <== mon.1 v2:172.21.15.181:3300/0 4 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fd360098a90 con 0x559fbb7fb130 2024-03-30T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.662+0000 7fd38ed51b80 10 client.26047 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.662+0000 7fd38ed51b80 10 client.26047 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T01:02:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.662+0000 7fd38ed51b80 20 client.26047 populate_metadata read hostname 'smithi116' 2024-03-30T01:02:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.662+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/805043538 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x559fbb875fe0 0x559fbb878400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.662+0000 7fd38ed51b80 4 client.26047 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:02:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.662+0000 7fd38ed51b80 1 -- 172.21.15.116:0/805043538 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559fbb59acf0 con 0x559fbb875fe0 2024-03-30T01:02:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.663+0000 7fd373e00700 1 --2- 172.21.15.116:0/805043538 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x559fbb875fe0 0x559fbb878400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.663+0000 7fd373e00700 1 --2- 172.21.15.116:0/805043538 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x559fbb875fe0 0x559fbb878400 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=de09f36ac673922d server_cookie=474745d8935949cb in_seq=0 out_seq=0 2024-03-30T01:02:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.663+0000 7fd35e000700 10 client.26047 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T01:02:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.668+0000 7fd35e000700 1 -- 172.21.15.116:0/805043538 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559fbb59acf0 con 0x559fbb875fe0 2024-03-30T01:02:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.668+0000 7fd35e000700 10 client.26047 handle_command_reply: tid=0 2024-03-30T01:02:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.668+0000 7fd38ed51b80 1 client.26047 shutdown 2024-03-30T01:02:07.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.689+0000 7fd38ed51b80 1 -- 172.21.15.116:0/805043538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fb130 msgr2=0x559fbb80b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:07.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.689+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/805043538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fb130 0x559fbb80b990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x559fbb804030 tx=0x7fd360099e10 comp rx=0 tx=0).stop 2024-03-30T01:02:07.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.689+0000 7fd38ed51b80 1 -- 172.21.15.116:0/805043538 shutdown_connections 2024-03-30T01:02:07.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.689+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/805043538 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x559fbb875fe0 0x559fbb878400 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.689+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/805043538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fbb7fbfd0 0x559fbb807fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.689+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/805043538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559fbb7fa7d0 0x559fbb80b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.689+0000 7fd38ed51b80 1 --2- 172.21.15.116:0/805043538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559fbb7fb130 0x559fbb80b990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.689+0000 7fd38ed51b80 1 -- 172.21.15.116:0/805043538 >> 172.21.15.116:0/805043538 conn(0x559fbb7596a0 msgr2=0x559fbb7fb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:07.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.689+0000 7fd35e000700 0 client.26047 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T01:02:07.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.690+0000 7fd38ed51b80 1 -- 172.21.15.116:0/805043538 shutdown_connections 2024-03-30T01:02:07.696 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:02:07.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.690+0000 7fd38ed51b80 1 -- 172.21.15.116:0/805043538 wait complete. 2024-03-30T01:02:07.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.691+0000 7fd38ed51b80 20 client.26047 trim_cache size 0 max 16384 2024-03-30T01:02:07.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.695+0000 7fd37b200700 1 -- 172.21.15.116:0/1056308 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd3580a1320 msgr2=0x7fd3580a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:07.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.695+0000 7fd37b200700 1 --2- 172.21.15.116:0/1056308 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd3580a1320 0x7fd3580a37e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd36c00d010 tx=0x7fd36c00ba40 comp rx=0 tx=0).stop 2024-03-30T01:02:07.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.695+0000 7fd37b200700 1 -- 172.21.15.116:0/1056308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd37410a720 msgr2=0x7fd3741a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:07.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.695+0000 7fd37b200700 1 --2- 172.21.15.116:0/1056308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd37410a720 0x7fd3741a6080 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd36000d9c0 tx=0x7fd36000dcd0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.695+0000 7fd37b200700 1 -- 172.21.15.116:0/1056308 shutdown_connections 2024-03-30T01:02:07.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.695+0000 7fd37b200700 1 --2- 172.21.15.116:0/1056308 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd3580a1320 0x7fd3580a37e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.695+0000 7fd37b200700 1 --2- 172.21.15.116:0/1056308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd374107db0 0x7fd3741a5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.695+0000 7fd37b200700 1 --2- 172.21.15.116:0/1056308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd37410d0f0 0x7fd3741ad050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.695+0000 7fd37b200700 1 --2- 172.21.15.116:0/1056308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd37410a720 0x7fd3741a6080 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:07.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.695+0000 7fd37b200700 1 -- 172.21.15.116:0/1056308 >> 172.21.15.116:0/1056308 conn(0x7fd37406dd50 msgr2=0x7fd374109330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.696+0000 7fd37b200700 1 -- 172.21.15.116:0/1056308 shutdown_connections 2024-03-30T01:02:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:07.696+0000 7fd37b200700 1 -- 172.21.15.116:0/1056308 wait complete. 2024-03-30T01:02:08.291 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T01:02:08.467 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:08 smithi116 ceph-mon[38086]: pgmap v419: 129 pgs: 129 active+clean; 1024 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.6 MiB/s wr, 545 op/s 2024-03-30T01:02:08.762 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:08 smithi181 ceph-mon[47622]: pgmap v419: 129 pgs: 129 active+clean; 1024 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.6 MiB/s wr, 545 op/s 2024-03-30T01:02:08.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:08 smithi130 ceph-mon[47626]: pgmap v419: 129 pgs: 129 active+clean; 1024 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.6 MiB/s wr, 545 op/s 2024-03-30T01:02:09.036 INFO:tasks.workunit.client.0.smithi116.stdout: 297 496801 717 351837 868 177293 1532 2024-03-30T01:02:10.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:10 smithi116 ceph-mon[38086]: pgmap v420: 129 pgs: 129 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 10 MiB/s wr, 640 op/s 2024-03-30T01:02:10.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:10 smithi181 ceph-mon[47622]: pgmap v420: 129 pgs: 129 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 10 MiB/s wr, 640 op/s 2024-03-30T01:02:10.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:10 smithi130 ceph-mon[47626]: pgmap v420: 129 pgs: 129 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 10 MiB/s wr, 640 op/s 2024-03-30T01:02:12.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:12 smithi181 ceph-mon[47622]: pgmap v421: 129 pgs: 129 active+clean; 1.1 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 14 MiB/s wr, 823 op/s 2024-03-30T01:02:12.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:12 smithi130 ceph-mon[47626]: pgmap v421: 129 pgs: 129 active+clean; 1.1 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 14 MiB/s wr, 823 op/s 2024-03-30T01:02:12.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:12 smithi116 ceph-mon[38086]: pgmap v421: 129 pgs: 129 active+clean; 1.1 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 14 MiB/s wr, 823 op/s 2024-03-30T01:02:14.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:14 smithi181 ceph-mon[47622]: pgmap v422: 129 pgs: 129 active+clean; 1.1 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 16 MiB/s wr, 862 op/s 2024-03-30T01:02:14.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:14 smithi130 ceph-mon[47626]: pgmap v422: 129 pgs: 129 active+clean; 1.1 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 16 MiB/s wr, 862 op/s 2024-03-30T01:02:14.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:14 smithi116 ceph-mon[38086]: pgmap v422: 129 pgs: 129 active+clean; 1.1 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 16 MiB/s wr, 862 op/s 2024-03-30T01:02:16.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.155+0000 7f27db200700 1 -- 172.21.15.116:0/731800417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27dc10a390 msgr2=0x7f27dc10c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:16.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.155+0000 7f27db200700 1 --2- 172.21.15.116:0/731800417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27dc10a390 0x7f27dc10c820 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f27c80098f0 tx=0x7f27c8009c00 comp rx=0 tx=0).stop 2024-03-30T01:02:16.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.160+0000 7f27db200700 1 -- 172.21.15.116:0/731800417 shutdown_connections 2024-03-30T01:02:16.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.165+0000 7f27db200700 1 --2- 172.21.15.116:0/731800417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27dc072b80 0x7f27dc109dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.165+0000 7f27db200700 1 --2- 172.21.15.116:0/731800417 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27dc10ce40 0x7f27dc117cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.165+0000 7f27db200700 1 --2- 172.21.15.116:0/731800417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27dc10a390 0x7f27dc10c820 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.165+0000 7f27db200700 1 -- 172.21.15.116:0/731800417 >> 172.21.15.116:0/731800417 conn(0x7f27dc06da50 msgr2=0x7f27dc06fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:16.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.202+0000 7f27db200700 1 -- 172.21.15.116:0/731800417 shutdown_connections 2024-03-30T01:02:16.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.215+0000 7f27db200700 1 -- 172.21.15.116:0/731800417 wait complete. 2024-03-30T01:02:16.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.217+0000 7f27db200700 1 Processor -- start 2024-03-30T01:02:16.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.238+0000 7f27db200700 1 -- start start 2024-03-30T01:02:16.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.239+0000 7f27db200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27dc072b80 0x7f27dc19d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:16.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.239+0000 7f27db200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27dc10a390 0x7f27dc19dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:16.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.240+0000 7f27db200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27dc10ce40 0x7f27dc1a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:16.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.240+0000 7f27db200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc1a4d70 con 0x7f27dc072b80 2024-03-30T01:02:16.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.240+0000 7f27db200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc1a4eb0 con 0x7f27dc10a390 2024-03-30T01:02:16.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.240+0000 7f27db200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc1a5040 con 0x7f27dc10ce40 2024-03-30T01:02:16.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.242+0000 7f27da800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27dc10ce40 0x7f27dc1a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:16.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.243+0000 7f27da800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27dc10ce40 0x7f27dc1a47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:56056/0 (socket says 172.21.15.116:56056) 2024-03-30T01:02:16.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.243+0000 7f27da800700 1 -- 172.21.15.116:0/3297815135 learned_addr learned my addr 172.21.15.116:0/3297815135 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:16.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.243+0000 7f27d9400700 1 --2- 172.21.15.116:0/3297815135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27dc10a390 0x7f27dc19dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:16.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.243+0000 7f27d9e00700 1 --2- 172.21.15.116:0/3297815135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27dc072b80 0x7f27dc19d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:16.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.244+0000 7f27d9e00700 1 -- 172.21.15.116:0/3297815135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27dc10ce40 msgr2=0x7f27dc1a47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:16.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.244+0000 7f27d9e00700 1 --2- 172.21.15.116:0/3297815135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27dc10ce40 0x7f27dc1a47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.244+0000 7f27d9e00700 1 -- 172.21.15.116:0/3297815135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27dc10a390 msgr2=0x7f27dc19dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:16.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.244+0000 7f27d9e00700 1 --2- 172.21.15.116:0/3297815135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27dc10a390 0x7f27dc19dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.244+0000 7f27d9e00700 1 -- 172.21.15.116:0/3297815135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c8009520 con 0x7f27dc072b80 2024-03-30T01:02:16.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.245+0000 7f27d9e00700 1 --2- 172.21.15.116:0/3297815135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27dc072b80 0x7f27dc19d8a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f27cc01ecd0 tx=0x7f27cc01a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:16.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.276+0000 7f27d2a00700 1 -- 172.21.15.116:0/3297815135 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc01ee70 con 0x7f27dc072b80 2024-03-30T01:02:16.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.276+0000 7f27d2a00700 1 -- 172.21.15.116:0/3297815135 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27cc021470 con 0x7f27dc072b80 2024-03-30T01:02:16.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.276+0000 7f27d2a00700 1 -- 172.21.15.116:0/3297815135 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc020630 con 0x7f27dc072b80 2024-03-30T01:02:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.309+0000 7f27db200700 1 -- 172.21.15.116:0/3297815135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27dc1a5320 con 0x7f27dc072b80 2024-03-30T01:02:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.309+0000 7f27db200700 1 -- 172.21.15.116:0/3297815135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27dc1a57b0 con 0x7f27dc072b80 2024-03-30T01:02:16.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.407+0000 7f27d2a00700 1 -- 172.21.15.116:0/3297815135 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f27cc0215e0 con 0x7f27dc072b80 2024-03-30T01:02:16.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.415+0000 7f27d2a00700 1 --2- 172.21.15.116:0/3297815135 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f27c00a1320 0x7f27c00a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:16.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.417+0000 7f27f4cf1b80 1 Processor -- start 2024-03-30T01:02:16.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.437+0000 7f27d2a00700 1 -- 172.21.15.116:0/3297815135 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f27cc0dd590 con 0x7f27dc072b80 2024-03-30T01:02:16.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.443+0000 7f27d9400700 1 --2- 172.21.15.116:0/3297815135 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f27c00a1320 0x7f27c00a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:16.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.448+0000 7f27d9400700 1 --2- 172.21.15.116:0/3297815135 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f27c00a1320 0x7f27c00a37e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f27c800fd00 tx=0x7f27c80056a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:16.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.454+0000 7f27f4cf1b80 1 -- start start 2024-03-30T01:02:16.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.455+0000 7f27f4cf1b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd249de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:16.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.455+0000 7f27f4cf1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249e340 0x55afd249ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:16.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.455+0000 7f27f4cf1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249f190 0x55afd249f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:16.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.455+0000 7f27f4cf1b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd23f5990 con 0x55afd249d9e0 2024-03-30T01:02:16.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.455+0000 7f27f4cf1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd23f3350 con 0x55afd249f190 2024-03-30T01:02:16.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.455+0000 7f27f4cf1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd23eec40 con 0x55afd249e340 2024-03-30T01:02:16.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.455+0000 7f27d9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd249de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:16.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.455+0000 7f27d9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd249de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59784/0 (socket says 172.21.15.116:59784) 2024-03-30T01:02:16.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.455+0000 7f27d9e00700 1 -- 172.21.15.116:0/1239029139 learned_addr learned my addr 172.21.15.116:0/1239029139 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:16.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.456+0000 7f27da800700 1 --2- 172.21.15.116:0/1239029139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249f190 0x55afd249f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:16.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.457+0000 7f27d9e00700 1 -- 172.21.15.116:0/1239029139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249e340 msgr2=0x55afd249ebb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:02:16.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.457+0000 7f27d9e00700 1 --2- 172.21.15.116:0/1239029139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249e340 0x55afd249ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.457+0000 7f27d9e00700 1 -- 172.21.15.116:0/1239029139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249f190 msgr2=0x55afd249f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:16.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.457+0000 7f27d9e00700 1 --2- 172.21.15.116:0/1239029139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249f190 0x55afd249f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.457+0000 7f27d9e00700 1 -- 172.21.15.116:0/1239029139 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afd22703b0 con 0x55afd249d9e0 2024-03-30T01:02:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.460+0000 7f27d9e00700 1 --2- 172.21.15.116:0/1239029139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd249de00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f27cc0b2640 tx=0x7f27cc021f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52011336c1260d9d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.490+0000 7f27c7400700 1 -- 172.21.15.116:0/1239029139 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc0ae520 con 0x55afd249d9e0 2024-03-30T01:02:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.490+0000 7f27c7400700 1 -- 172.21.15.116:0/1239029139 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27cc0b2b70 con 0x55afd249d9e0 2024-03-30T01:02:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.490+0000 7f27c7400700 1 -- 172.21.15.116:0/1239029139 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc0218c0 con 0x55afd249d9e0 2024-03-30T01:02:16.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.518+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/1239029139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 msgr2=0x55afd249de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:16.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.518+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/1239029139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd249de00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f27cc0b2640 tx=0x7f27cc021f90 comp rx=0 tx=0).stop 2024-03-30T01:02:16.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.546+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/1239029139 shutdown_connections 2024-03-30T01:02:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.546+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/1239029139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249f190 0x55afd249f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.546+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/1239029139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd249de00 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.546+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/1239029139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249e340 0x55afd249ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.546+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/1239029139 >> 172.21.15.116:0/1239029139 conn(0x55afd23fc6a0 msgr2=0x55afd2498ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:16.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.548+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/1239029139 shutdown_connections 2024-03-30T01:02:16.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.550+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/1239029139 wait complete. 2024-03-30T01:02:16.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.552+0000 7f27f4cf1b80 1 -- start start 2024-03-30T01:02:16.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.553+0000 7f27f4cf1b80 1 Processor -- start 2024-03-30T01:02:16.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.574+0000 7f27f4cf1b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd24a7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:16.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.574+0000 7f27f4cf1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249e340 0x55afd24a7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.574+0000 7f27f4cf1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249f190 0x55afd24aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.576+0000 7f27f4cf1b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd238ad80 con 0x55afd249d9e0 2024-03-30T01:02:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.576+0000 7f27f4cf1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd238aa20 con 0x55afd249f190 2024-03-30T01:02:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.576+0000 7f27f4cf1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd22726a0 con 0x55afd249e340 2024-03-30T01:02:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.576+0000 7f27f4cf1b80 10 client.0 _ll_register_callbacks cb 0x55afd2394470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:02:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.577+0000 7f27d9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd24a7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.578+0000 7f27d9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd24a7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59794/0 (socket says 172.21.15.116:59794) 2024-03-30T01:02:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.578+0000 7f27d9e00700 1 -- 172.21.15.116:0/1527376651 learned_addr learned my addr 172.21.15.116:0/1527376651 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:16.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.584+0000 7f27d9400700 1 --2- 172.21.15.116:0/1527376651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249e340 0x55afd24a7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:16.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.584+0000 7f27d9e00700 1 -- 172.21.15.116:0/1527376651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249e340 msgr2=0x55afd24a7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:16.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.584+0000 7f27d9e00700 1 --2- 172.21.15.116:0/1527376651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249e340 0x55afd24a7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.584+0000 7f27d9e00700 1 -- 172.21.15.116:0/1527376651 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249f190 msgr2=0x55afd24aeea0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:02:16.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.584+0000 7f27d9e00700 1 --2- 172.21.15.116:0/1527376651 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249f190 0x55afd24aeea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.584+0000 7f27d9e00700 1 -- 172.21.15.116:0/1527376651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27dc1a57b0 con 0x55afd249d9e0 2024-03-30T01:02:16.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.584+0000 7f27d9400700 1 --2- 172.21.15.116:0/1527376651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249e340 0x55afd24a7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:02:16.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.596+0000 7f27d9e00700 1 --2- 172.21.15.116:0/1527376651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd24a7920 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f27cc01a930 tx=0x7f27cc0aa7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc802eeb9514295c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.609+0000 7f27bbe00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:02:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.609+0000 7f27bbe00700 1 -- 172.21.15.116:0/1527376651 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc0dd300 con 0x55afd249d9e0 2024-03-30T01:02:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.609+0000 7f27bbe00700 1 -- 172.21.15.116:0/1527376651 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f27cc0ca070 con 0x55afd249d9e0 2024-03-30T01:02:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.610+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/1527376651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55afd23af380 con 0x55afd249d9e0 2024-03-30T01:02:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.611+0000 7f27bbe00700 1 -- 172.21.15.116:0/1527376651 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27cc0a9be0 con 0x55afd249d9e0 2024-03-30T01:02:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.611+0000 7f27f4cf1b80 10 client.16026 fetch_fsmap learned FSMap version 27 2024-03-30T01:02:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.611+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/1527376651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55afd221cb60 con 0x55afd249d9e0 2024-03-30T01:02:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.612+0000 7f27bbe00700 1 -- 172.21.15.116:0/1527376651 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27cc0a9d40 con 0x55afd249d9e0 2024-03-30T01:02:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.612+0000 7f27bbe00700 1 -- 172.21.15.116:0/1527376651 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f27cc0b2770 con 0x55afd249d9e0 2024-03-30T01:02:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.612+0000 7f27f4cf1b80 10 client.16026 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T01:02:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.612+0000 7f27f4cf1b80 10 client.16026 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T01:02:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.612+0000 7f27f4cf1b80 20 client.16026 populate_metadata read hostname 'smithi116' 2024-03-30T01:02:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.623+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/1527376651 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55afd24b3140 0x55afd24b5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.633+0000 7f27d9400700 1 --2- 172.21.15.116:0/1527376651 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55afd24b3140 0x55afd24b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.639+0000 7f27d9400700 1 --2- 172.21.15.116:0/1527376651 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55afd24b3140 0x55afd24b5580 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=93faa50008e51555 server_cookie=dd61dc1c11efa950 in_seq=0 out_seq=0 2024-03-30T01:02:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.650+0000 7f27bbe00700 10 client.16026 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T01:02:16.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.654+0000 7f27f4cf1b80 4 client.16026 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:02:16.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.656+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/1527376651 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55afd23af380 con 0x55afd24b3140 2024-03-30T01:02:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.666+0000 7f27bbe00700 1 -- 172.21.15.116:0/1527376651 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55afd23af380 con 0x55afd24b3140 2024-03-30T01:02:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.666+0000 7f27bbe00700 10 client.16026 handle_command_reply: tid=0 2024-03-30T01:02:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.668+0000 7f27f4cf1b80 1 client.16026 shutdown 2024-03-30T01:02:16.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.732+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/1527376651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 msgr2=0x55afd24a7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:16.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.732+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/1527376651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd24a7920 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f27cc01a930 tx=0x7f27cc0aa7b0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.735+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/1527376651 shutdown_connections 2024-03-30T01:02:16.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.735+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/1527376651 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55afd24b3140 0x55afd24b5580 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.735+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/1527376651 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249f190 0x55afd24aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.735+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/1527376651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd24a7920 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.735+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/1527376651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249e340 0x55afd24a7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:16.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.735+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/1527376651 >> 172.21.15.116:0/1527376651 conn(0x55afd23fc6a0 msgr2=0x55afd249c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:16.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:16 smithi181 ceph-mon[47622]: pgmap v423: 129 pgs: 129 active+clean; 1.1 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 19 MiB/s wr, 977 op/s 2024-03-30T01:02:16.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.758+0000 7f27bbe00700 0 client.16026 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T01:02:16.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.758+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/1527376651 shutdown_connections 2024-03-30T01:02:16.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.758+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/1527376651 wait complete. 2024-03-30T01:02:16.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:16 smithi130 ceph-mon[47626]: pgmap v423: 129 pgs: 129 active+clean; 1.1 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 19 MiB/s wr, 977 op/s 2024-03-30T01:02:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:16.795+0000 7f27f4cf1b80 20 client.16026 trim_cache size 0 max 16384 2024-03-30T01:02:16.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:16 smithi116 ceph-mon[38086]: pgmap v423: 129 pgs: 129 active+clean; 1.1 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 19 MiB/s wr, 977 op/s 2024-03-30T01:02:17.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.388+0000 7f27f4cf1b80 1 Processor -- start 2024-03-30T01:02:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.442+0000 7f27f4cf1b80 1 -- start start 2024-03-30T01:02:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.442+0000 7f27f4cf1b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd249f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.442+0000 7f27f4cf1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249e340 0x55afd24aee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.442+0000 7f27f4cf1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249f190 0x55afd24917f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.442+0000 7f27f4cf1b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd2499ed0 con 0x55afd249d9e0 2024-03-30T01:02:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.442+0000 7f27f4cf1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd2499830 con 0x55afd249e340 2024-03-30T01:02:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.442+0000 7f27f4cf1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd2492780 con 0x55afd249f190 2024-03-30T01:02:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.443+0000 7f27d9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd249f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.443+0000 7f27d9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd249f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59808/0 (socket says 172.21.15.116:59808) 2024-03-30T01:02:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.443+0000 7f27d9e00700 1 -- 172.21.15.116:0/2009985849 learned_addr learned my addr 172.21.15.116:0/2009985849 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.444+0000 7f27d9400700 1 --2- 172.21.15.116:0/2009985849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249e340 0x55afd24aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.451+0000 7f27d9400700 1 -- 172.21.15.116:0/2009985849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249f190 msgr2=0x55afd24917f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:02:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.451+0000 7f27d9400700 1 --2- 172.21.15.116:0/2009985849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249f190 0x55afd24917f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.451+0000 7f27d9400700 1 -- 172.21.15.116:0/2009985849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 msgr2=0x55afd249f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.451+0000 7f27d9400700 1 --2- 172.21.15.116:0/2009985849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd249f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.451+0000 7f27d9400700 1 -- 172.21.15.116:0/2009985849 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afd23b4ea0 con 0x55afd249e340 2024-03-30T01:02:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.455+0000 7f27d9400700 1 --2- 172.21.15.116:0/2009985849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249e340 0x55afd24aee60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f27c800fbd0 tx=0x7f27c801e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acd1a67dbccf173c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:17.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.465+0000 7f27c7e00700 1 -- 172.21.15.116:0/2009985849 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c801c070 con 0x55afd249e340 2024-03-30T01:02:17.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.465+0000 7f27c7e00700 1 -- 172.21.15.116:0/2009985849 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27c8028040 con 0x55afd249e340 2024-03-30T01:02:17.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.465+0000 7f27c7e00700 1 -- 172.21.15.116:0/2009985849 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c800b900 con 0x55afd249e340 2024-03-30T01:02:17.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.476+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/2009985849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249e340 msgr2=0x55afd24aee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:17.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.476+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/2009985849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249e340 0x55afd24aee60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f27c800fbd0 tx=0x7f27c801e000 comp rx=0 tx=0).stop 2024-03-30T01:02:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.528+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/2009985849 shutdown_connections 2024-03-30T01:02:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.528+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/2009985849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249f190 0x55afd24917f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.528+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/2009985849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249d9e0 0x55afd249f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.528+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/2009985849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249e340 0x55afd24aee60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.528+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/2009985849 >> 172.21.15.116:0/2009985849 conn(0x55afd23fc6a0 msgr2=0x55afd24b9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:17.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.546+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/2009985849 shutdown_connections 2024-03-30T01:02:17.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.547+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/2009985849 wait complete. 2024-03-30T01:02:17.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.547+0000 7f27f4cf1b80 1 -- start start 2024-03-30T01:02:17.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.548+0000 7f27f4cf1b80 1 Processor -- start 2024-03-30T01:02:17.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.557+0000 7f27f4cf1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249d9e0 0x55afd24ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:17.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.557+0000 7f27f4cf1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249e340 0x55afd24aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:17.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.557+0000 7f27f4cf1b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249f190 0x55afd24acda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:17.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.557+0000 7f27f4cf1b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd24ad320 con 0x55afd249f190 2024-03-30T01:02:17.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.557+0000 7f27f4cf1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd24ad460 con 0x55afd249e340 2024-03-30T01:02:17.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.557+0000 7f27f4cf1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd24ad620 con 0x55afd249d9e0 2024-03-30T01:02:17.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.557+0000 7f27f4cf1b80 10 client.0 _ll_register_callbacks cb 0x55afd2394470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:02:17.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.558+0000 7f27da800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249f190 0x55afd24acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:17.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.558+0000 7f27da800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249f190 0x55afd24acda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59816/0 (socket says 172.21.15.116:59816) 2024-03-30T01:02:17.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.558+0000 7f27da800700 1 -- 172.21.15.116:0/2026685630 learned_addr learned my addr 172.21.15.116:0/2026685630 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:17.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.559+0000 7f27da800700 1 -- 172.21.15.116:0/2026685630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249d9e0 msgr2=0x55afd24ae650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:02:17.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.559+0000 7f27da800700 1 --2- 172.21.15.116:0/2026685630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249d9e0 0x55afd24ae650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.559+0000 7f27da800700 1 -- 172.21.15.116:0/2026685630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249e340 msgr2=0x55afd24aeb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:02:17.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.559+0000 7f27da800700 1 --2- 172.21.15.116:0/2026685630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249e340 0x55afd24aeb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.559+0000 7f27da800700 1 -- 172.21.15.116:0/2026685630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27dc118720 con 0x55afd249f190 2024-03-30T01:02:17.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.560+0000 7f27da800700 1 --2- 172.21.15.116:0/2026685630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249f190 0x55afd24acda0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f27d40078b0 tx=0x7f27d4007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c131d12e5677ef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:17.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.566+0000 7f27bbe00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:02:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.566+0000 7f27bbe00700 1 -- 172.21.15.116:0/2026685630 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d400f5c0 con 0x55afd249f190 2024-03-30T01:02:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.566+0000 7f27bbe00700 1 -- 172.21.15.116:0/2026685630 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f27d400f720 con 0x55afd249f190 2024-03-30T01:02:17.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.575+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/2026685630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f27cc0b2b70 con 0x55afd249f190 2024-03-30T01:02:17.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.586+0000 7f27bbe00700 1 -- 172.21.15.116:0/2026685630 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27d400f8e0 con 0x55afd249f190 2024-03-30T01:02:17.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.599+0000 7f27f4cf1b80 10 client.16032 fetch_fsmap learned FSMap version 27 2024-03-30T01:02:17.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.599+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/2026685630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55afd2274cd0 con 0x55afd249f190 2024-03-30T01:02:17.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.600+0000 7f27bbe00700 1 -- 172.21.15.116:0/2026685630 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27cc0b2b70 con 0x55afd249f190 2024-03-30T01:02:17.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.600+0000 7f27bbe00700 1 -- 172.21.15.116:0/2026685630 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f27d4016cc0 con 0x55afd249f190 2024-03-30T01:02:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.602+0000 7f27f4cf1b80 10 client.16032 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T01:02:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.602+0000 7f27f4cf1b80 10 client.16032 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T01:02:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.602+0000 7f27f4cf1b80 20 client.16032 populate_metadata read hostname 'smithi116' 2024-03-30T01:02:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.602+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/2026685630 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55afd2508320 0x55afd250a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.602+0000 7f27f4cf1b80 4 client.16032 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:02:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.602+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/2026685630 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55afd23e7700 con 0x55afd2508320 2024-03-30T01:02:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.602+0000 7f27d9e00700 1 --2- 172.21.15.116:0/2026685630 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55afd2508320 0x55afd250a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.605+0000 7f27d9e00700 1 --2- 172.21.15.116:0/2026685630 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55afd2508320 0x55afd250a740 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=134681b226bfe8e0 server_cookie=7b13ac57a4e29fee in_seq=0 out_seq=0 2024-03-30T01:02:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.610+0000 7f27bbe00700 10 client.16032 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T01:02:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.610+0000 7f27bbe00700 1 -- 172.21.15.116:0/2026685630 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55afd23e7700 con 0x55afd2508320 2024-03-30T01:02:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.610+0000 7f27bbe00700 10 client.16032 handle_command_reply: tid=0 2024-03-30T01:02:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.610+0000 7f27f4cf1b80 1 client.16032 shutdown 2024-03-30T01:02:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.653+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/2026685630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249f190 msgr2=0x55afd24acda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.653+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/2026685630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249f190 0x55afd24acda0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f27d40078b0 tx=0x7f27d4007c70 comp rx=0 tx=0).stop 2024-03-30T01:02:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.664+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/2026685630 shutdown_connections 2024-03-30T01:02:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.664+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/2026685630 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55afd2508320 0x55afd250a740 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.664+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/2026685630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afd249d9e0 0x55afd24ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.664+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/2026685630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afd249f190 0x55afd24acda0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.664+0000 7f27f4cf1b80 1 --2- 172.21.15.116:0/2026685630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55afd249e340 0x55afd24aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.664+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/2026685630 >> 172.21.15.116:0/2026685630 conn(0x55afd23fc6a0 msgr2=0x55afd24a6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.664+0000 7f27bbe00700 0 client.16032 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T01:02:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.665+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/2026685630 shutdown_connections 2024-03-30T01:02:17.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.696+0000 7f27f4cf1b80 1 -- 172.21.15.116:0/2026685630 wait complete. 2024-03-30T01:02:17.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.721+0000 7f27f4cf1b80 20 client.16032 trim_cache size 0 max 16384 2024-03-30T01:02:17.727 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:02:17.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.750+0000 7f27db200700 1 -- 172.21.15.116:0/3297815135 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f27c00a1320 msgr2=0x7f27c00a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:17.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.750+0000 7f27db200700 1 --2- 172.21.15.116:0/3297815135 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f27c00a1320 0x7f27c00a37e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f27c800fd00 tx=0x7f27c80056a0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.750+0000 7f27db200700 1 -- 172.21.15.116:0/3297815135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27dc072b80 msgr2=0x7f27dc19d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:17.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.750+0000 7f27db200700 1 --2- 172.21.15.116:0/3297815135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27dc072b80 0x7f27dc19d8a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f27cc01ecd0 tx=0x7f27cc01a530 comp rx=0 tx=0).stop 2024-03-30T01:02:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.755+0000 7f27db200700 1 -- 172.21.15.116:0/3297815135 shutdown_connections 2024-03-30T01:02:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.755+0000 7f27db200700 1 --2- 172.21.15.116:0/3297815135 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f27c00a1320 0x7f27c00a37e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.755+0000 7f27db200700 1 --2- 172.21.15.116:0/3297815135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27dc10ce40 0x7f27dc1a47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.755+0000 7f27db200700 1 --2- 172.21.15.116:0/3297815135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27dc072b80 0x7f27dc19d8a0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.755+0000 7f27db200700 1 --2- 172.21.15.116:0/3297815135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27dc10a390 0x7f27dc19dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.755+0000 7f27db200700 1 -- 172.21.15.116:0/3297815135 >> 172.21.15.116:0/3297815135 conn(0x7f27dc06da50 msgr2=0x7f27dc06de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:17.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.770+0000 7f27db200700 1 -- 172.21.15.116:0/3297815135 shutdown_connections 2024-03-30T01:02:17.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:17.787+0000 7f27db200700 1 -- 172.21.15.116:0/3297815135 wait complete. 2024-03-30T01:02:18.577 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T01:02:18.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:18 smithi130 ceph-mon[47626]: pgmap v424: 129 pgs: 129 active+clean; 1.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 18 MiB/s wr, 966 op/s 2024-03-30T01:02:18.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:18 smithi116 ceph-mon[38086]: pgmap v424: 129 pgs: 129 active+clean; 1.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 18 MiB/s wr, 966 op/s 2024-03-30T01:02:19.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:18 smithi181 ceph-mon[47622]: pgmap v424: 129 pgs: 129 active+clean; 1.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 18 MiB/s wr, 966 op/s 2024-03-30T01:02:19.031 INFO:tasks.workunit.client.0.smithi116.stdout: 308 398903 644 284840 612 142490 1203 2024-03-30T01:02:20.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:20 smithi130 ceph-mon[47626]: pgmap v425: 129 pgs: 129 active+clean; 1.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 18 MiB/s wr, 960 op/s 2024-03-30T01:02:20.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:20 smithi116 ceph-mon[38086]: pgmap v425: 129 pgs: 129 active+clean; 1.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 18 MiB/s wr, 960 op/s 2024-03-30T01:02:21.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:20 smithi181 ceph-mon[47622]: pgmap v425: 129 pgs: 129 active+clean; 1.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 18 MiB/s wr, 960 op/s 2024-03-30T01:02:22.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:22 smithi130 ceph-mon[47626]: pgmap v426: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 18 MiB/s wr, 979 op/s 2024-03-30T01:02:22.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:22 smithi116 ceph-mon[38086]: pgmap v426: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 18 MiB/s wr, 979 op/s 2024-03-30T01:02:23.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:22 smithi181 ceph-mon[47622]: pgmap v426: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 18 MiB/s wr, 979 op/s 2024-03-30T01:02:23.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.787+0000 7ff092600700 1 -- 172.21.15.116:0/374385901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff08c10ce40 msgr2=0x7ff08c117cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:23.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.787+0000 7ff092600700 1 --2- 172.21.15.116:0/374385901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff08c10ce40 0x7ff08c117cf0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff08400b0a0 tx=0x7ff08400b3b0 comp rx=0 tx=0).stop 2024-03-30T01:02:23.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.808+0000 7ff092600700 1 -- 172.21.15.116:0/374385901 shutdown_connections 2024-03-30T01:02:23.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.808+0000 7ff092600700 1 --2- 172.21.15.116:0/374385901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff08c10ce40 0x7ff08c117cf0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:23.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.808+0000 7ff092600700 1 --2- 172.21.15.116:0/374385901 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c072b80 0x7ff08c109dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:23.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.808+0000 7ff092600700 1 --2- 172.21.15.116:0/374385901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff08c10a390 0x7ff08c10c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:23.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.808+0000 7ff092600700 1 -- 172.21.15.116:0/374385901 >> 172.21.15.116:0/374385901 conn(0x7ff08c06da50 msgr2=0x7ff08c06fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:23.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.820+0000 7ff092600700 1 -- 172.21.15.116:0/374385901 shutdown_connections 2024-03-30T01:02:23.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.824+0000 7ff092600700 1 -- 172.21.15.116:0/374385901 wait complete. 2024-03-30T01:02:23.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.826+0000 7ff092600700 1 Processor -- start 2024-03-30T01:02:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.858+0000 7ff092600700 1 -- start start 2024-03-30T01:02:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.860+0000 7ff092600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff08c072b80 0x7ff08c1a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.860+0000 7ff092600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c10a390 0x7ff08c1a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.860+0000 7ff092600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff08c10ce40 0x7ff08c1acfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.860+0000 7ff08be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c10a390 0x7ff08c1a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.860+0000 7ff08be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c10a390 0x7ff08c1a5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:52804/0 (socket says 172.21.15.116:52804) 2024-03-30T01:02:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.860+0000 7ff091200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff08c072b80 0x7ff08c1a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.860+0000 7ff091200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff08c072b80 0x7ff08c1a5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:56092/0 (socket says 172.21.15.116:56092) 2024-03-30T01:02:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.860+0000 7ff092600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c10f570 con 0x7ff08c10ce40 2024-03-30T01:02:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.860+0000 7ff092600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c10f6e0 con 0x7ff08c10a390 2024-03-30T01:02:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.860+0000 7ff092600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c10f8a0 con 0x7ff08c072b80 2024-03-30T01:02:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.861+0000 7ff091c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff08c10ce40 0x7ff08c1acfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.861+0000 7ff091c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff08c10ce40 0x7ff08c1acfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59832/0 (socket says 172.21.15.116:59832) 2024-03-30T01:02:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.861+0000 7ff091c00700 1 -- 172.21.15.116:0/2230643386 learned_addr learned my addr 172.21.15.116:0/2230643386 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.862+0000 7ff091c00700 1 -- 172.21.15.116:0/2230643386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff08c072b80 msgr2=0x7ff08c1a5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.862+0000 7ff091c00700 1 --2- 172.21.15.116:0/2230643386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff08c072b80 0x7ff08c1a5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.862+0000 7ff091c00700 1 -- 172.21.15.116:0/2230643386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c10a390 msgr2=0x7ff08c1a5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.862+0000 7ff091c00700 1 --2- 172.21.15.116:0/2230643386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c10a390 0x7ff08c1a5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.862+0000 7ff091c00700 1 -- 172.21.15.116:0/2230643386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff084009d00 con 0x7ff08c10ce40 2024-03-30T01:02:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.863+0000 7ff091c00700 1 --2- 172.21.15.116:0/2230643386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff08c10ce40 0x7ff08c1acfc0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff084007a60 tx=0x7ff08400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.863+0000 7ff08be00700 1 --2- 172.21.15.116:0/2230643386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c10a390 0x7ff08c1a5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-30T01:02:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.863+0000 7ff091200700 1 --2- 172.21.15.116:0/2230643386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff08c072b80 0x7ff08c1a5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-30T01:02:23.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.886+0000 7ff089600700 1 -- 172.21.15.116:0/2230643386 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08400e040 con 0x7ff08c10ce40 2024-03-30T01:02:23.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.886+0000 7ff089600700 1 -- 172.21.15.116:0/2230643386 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff084004d90 con 0x7ff08c10ce40 2024-03-30T01:02:23.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.886+0000 7ff089600700 1 -- 172.21.15.116:0/2230643386 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084005390 con 0x7ff08c10ce40 2024-03-30T01:02:23.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.903+0000 7ff092600700 1 -- 172.21.15.116:0/2230643386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08c10fac0 con 0x7ff08c10ce40 2024-03-30T01:02:23.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.903+0000 7ff092600700 1 -- 172.21.15.116:0/2230643386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08c10ff50 con 0x7ff08c10ce40 2024-03-30T01:02:23.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.936+0000 7ff089600700 1 -- 172.21.15.116:0/2230643386 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7ff08401da30 con 0x7ff08c10ce40 2024-03-30T01:02:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.943+0000 7ff089600700 1 --2- 172.21.15.116:0/2230643386 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff0700a1320 0x7ff0700a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.943+0000 7ff089600700 1 -- 172.21.15.116:0/2230643386 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff0840cda30 con 0x7ff08c10ce40 2024-03-30T01:02:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.945+0000 7ff091200700 1 --2- 172.21.15.116:0/2230643386 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff0700a1320 0x7ff0700a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.946+0000 7ff091200700 1 --2- 172.21.15.116:0/2230643386 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff0700a1320 0x7ff0700a37e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff07c005fd0 tx=0x7ff07c005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:23.967+0000 7ff0a780bb80 1 Processor -- start 2024-03-30T01:02:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.045+0000 7ff0a780bb80 1 -- start start 2024-03-30T01:02:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.045+0000 7ff0a780bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa879e0 0x556b5fa87e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.045+0000 7ff0a780bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa88340 0x556b5fa88bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.045+0000 7ff0a780bb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa89610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.046+0000 7ff0a780bb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b5f9df990 con 0x556b5fa89190 2024-03-30T01:02:24.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.046+0000 7ff0a780bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b5f9dd350 con 0x556b5fa879e0 2024-03-30T01:02:24.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.046+0000 7ff0a780bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b5f9d8c40 con 0x556b5fa88340 2024-03-30T01:02:24.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.046+0000 7ff091c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa89610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:24.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.046+0000 7ff091c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa89610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59836/0 (socket says 172.21.15.116:59836) 2024-03-30T01:02:24.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.046+0000 7ff091c00700 1 -- 172.21.15.116:0/4288245311 learned_addr learned my addr 172.21.15.116:0/4288245311 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:24.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.047+0000 7ff091c00700 1 -- 172.21.15.116:0/4288245311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa88340 msgr2=0x556b5fa88bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:02:24.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.047+0000 7ff091c00700 1 --2- 172.21.15.116:0/4288245311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa88340 0x556b5fa88bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.047+0000 7ff091c00700 1 -- 172.21.15.116:0/4288245311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa879e0 msgr2=0x556b5fa87e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:02:24.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.047+0000 7ff091c00700 1 --2- 172.21.15.116:0/4288245311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa879e0 0x556b5fa87e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.047+0000 7ff091c00700 1 -- 172.21.15.116:0/4288245311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b5f85a3b0 con 0x556b5fa89190 2024-03-30T01:02:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.047+0000 7ff091c00700 1 --2- 172.21.15.116:0/4288245311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa89610 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff084097610 tx=0x7ff0840092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c181495797ea50c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:24.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.064+0000 7ff082000700 1 -- 172.21.15.116:0/4288245311 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08409b2a0 con 0x556b5fa89190 2024-03-30T01:02:24.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.064+0000 7ff082000700 1 -- 172.21.15.116:0/4288245311 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff08409aba0 con 0x556b5fa89190 2024-03-30T01:02:24.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.064+0000 7ff082000700 1 -- 172.21.15.116:0/4288245311 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08409b040 con 0x556b5fa89190 2024-03-30T01:02:24.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.066+0000 7ff0a780bb80 1 -- 172.21.15.116:0/4288245311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 msgr2=0x556b5fa89610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:24.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.066+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/4288245311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa89610 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff084097610 tx=0x7ff0840092e0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.079+0000 7ff0a780bb80 1 -- 172.21.15.116:0/4288245311 shutdown_connections 2024-03-30T01:02:24.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.079+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/4288245311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa89610 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.079+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/4288245311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa879e0 0x556b5fa87e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.079+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/4288245311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa88340 0x556b5fa88bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.079+0000 7ff0a780bb80 1 -- 172.21.15.116:0/4288245311 >> 172.21.15.116:0/4288245311 conn(0x556b5f9e66a0 msgr2=0x556b5fa82ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:24.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.085+0000 7ff0a780bb80 1 -- 172.21.15.116:0/4288245311 shutdown_connections 2024-03-30T01:02:24.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.085+0000 7ff0a780bb80 1 -- 172.21.15.116:0/4288245311 wait complete. 2024-03-30T01:02:24.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.086+0000 7ff0a780bb80 1 -- start start 2024-03-30T01:02:24.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.091+0000 7ff0a780bb80 1 Processor -- start 2024-03-30T01:02:24.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.102+0000 7ff0a780bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa879e0 0x556b5fa914d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:24.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.102+0000 7ff0a780bb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa88340 0x556b5fa91a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:24.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.102+0000 7ff0a780bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa89190 0x556b5fa98a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:24.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.102+0000 7ff0a780bb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b5f974d80 con 0x556b5fa88340 2024-03-30T01:02:24.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.102+0000 7ff0a780bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b5f974a20 con 0x556b5fa879e0 2024-03-30T01:02:24.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.102+0000 7ff0a780bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b5f85c6a0 con 0x556b5fa89190 2024-03-30T01:02:24.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.102+0000 7ff0a780bb80 10 client.0 _ll_register_callbacks cb 0x556b5f97e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:02:24.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.103+0000 7ff08be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa88340 0x556b5fa91a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:24.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.103+0000 7ff091200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa879e0 0x556b5fa914d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:24.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.103+0000 7ff091200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa879e0 0x556b5fa914d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:52816/0 (socket says 172.21.15.116:52816) 2024-03-30T01:02:24.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.103+0000 7ff091200700 1 -- 172.21.15.116:0/1722779252 learned_addr learned my addr 172.21.15.116:0/1722779252 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:24.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.103+0000 7ff091c00700 1 --2- 172.21.15.116:0/1722779252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa89190 0x556b5fa98a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:24.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.103+0000 7ff08be00700 1 -- 172.21.15.116:0/1722779252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa89190 msgr2=0x556b5fa98a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:24.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.103+0000 7ff08be00700 1 --2- 172.21.15.116:0/1722779252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa89190 0x556b5fa98a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.103+0000 7ff08be00700 1 -- 172.21.15.116:0/1722779252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa879e0 msgr2=0x556b5fa914d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:24.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.103+0000 7ff08be00700 1 --2- 172.21.15.116:0/1722779252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa879e0 0x556b5fa914d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.103+0000 7ff08be00700 1 -- 172.21.15.116:0/1722779252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08c10ff50 con 0x556b5fa88340 2024-03-30T01:02:24.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.104+0000 7ff08be00700 1 --2- 172.21.15.116:0/1722779252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa88340 0x556b5fa91a10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff0780105f0 tx=0x7ff078010900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db21f25336d437e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:24.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.114+0000 7ff076a00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:02:24.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.114+0000 7ff076a00700 1 -- 172.21.15.116:0/1722779252 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078012860 con 0x556b5fa88340 2024-03-30T01:02:24.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.114+0000 7ff076a00700 1 -- 172.21.15.116:0/1722779252 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff078016070 con 0x556b5fa88340 2024-03-30T01:02:24.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.119+0000 7ff0a780bb80 1 -- 172.21.15.116:0/1722779252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556b5f999380 con 0x556b5fa88340 2024-03-30T01:02:24.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.121+0000 7ff076a00700 1 -- 172.21.15.116:0/1722779252 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff078009e20 con 0x556b5fa88340 2024-03-30T01:02:24.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.147+0000 7ff0a780bb80 10 client.16044 fetch_fsmap learned FSMap version 27 2024-03-30T01:02:24.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.147+0000 7ff0a780bb80 1 -- 172.21.15.116:0/1722779252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x556b5f806b60 con 0x556b5fa88340 2024-03-30T01:02:24.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.151+0000 7ff076a00700 1 -- 172.21.15.116:0/1722779252 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556b5f999380 con 0x556b5fa88340 2024-03-30T01:02:24.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.151+0000 7ff076a00700 1 -- 172.21.15.116:0/1722779252 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7ff07801bbd0 con 0x556b5fa88340 2024-03-30T01:02:24.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.152+0000 7ff0a780bb80 10 client.16044 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T01:02:24.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.152+0000 7ff0a780bb80 10 client.16044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:02:24.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.152+0000 7ff0a780bb80 20 client.16044 populate_metadata read hostname 'smithi116' 2024-03-30T01:02:24.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.152+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/1722779252 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556b5fa9d360 0x556b5fa9f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:24.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.152+0000 7ff0a780bb80 4 client.16044 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:02:24.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.152+0000 7ff0a780bb80 1 -- 172.21.15.116:0/1722779252 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b5f9d1700 con 0x556b5fa9d360 2024-03-30T01:02:24.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.152+0000 7ff091200700 1 --2- 172.21.15.116:0/1722779252 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556b5fa9d360 0x556b5fa9f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:24.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.162+0000 7ff091200700 1 --2- 172.21.15.116:0/1722779252 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556b5fa9d360 0x556b5fa9f780 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1aa4dcf310b89439 server_cookie=b7b4903f8004fa7b in_seq=0 out_seq=0 2024-03-30T01:02:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.165+0000 7ff076a00700 10 client.16044 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:02:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.165+0000 7ff076a00700 1 -- 172.21.15.116:0/1722779252 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556b5f9d1700 con 0x556b5fa9d360 2024-03-30T01:02:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.165+0000 7ff076a00700 10 client.16044 handle_command_reply: tid=0 2024-03-30T01:02:24.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.170+0000 7ff0a780bb80 1 client.16044 shutdown 2024-03-30T01:02:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.177+0000 7ff0a780bb80 1 -- 172.21.15.116:0/1722779252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa88340 msgr2=0x556b5fa91a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.177+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/1722779252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa88340 0x556b5fa91a10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff0780105f0 tx=0x7ff078010900 comp rx=0 tx=0).stop 2024-03-30T01:02:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.178+0000 7ff0a780bb80 1 -- 172.21.15.116:0/1722779252 shutdown_connections 2024-03-30T01:02:24.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.178+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/1722779252 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556b5fa9d360 0x556b5fa9f780 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.178+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/1722779252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa89190 0x556b5fa98a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.178+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/1722779252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa879e0 0x556b5fa914d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.178+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/1722779252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa88340 0x556b5fa91a10 secure :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff0780105f0 tx=0x7ff078010900 comp rx=0 tx=0).stop 2024-03-30T01:02:24.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.178+0000 7ff0a780bb80 1 -- 172.21.15.116:0/1722779252 >> 172.21.15.116:0/1722779252 conn(0x556b5f9e66a0 msgr2=0x556b5fa863c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:24.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.178+0000 7ff076a00700 0 client.16044 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:02:24.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.179+0000 7ff0a780bb80 1 -- 172.21.15.116:0/1722779252 shutdown_connections 2024-03-30T01:02:24.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.186+0000 7ff0a780bb80 1 -- 172.21.15.116:0/1722779252 wait complete. 2024-03-30T01:02:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.192+0000 7ff0a780bb80 20 client.16044 trim_cache size 0 max 16384 2024-03-30T01:02:24.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.452+0000 7ff0a780bb80 1 Processor -- start 2024-03-30T01:02:24.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.471+0000 7ff0a780bb80 1 -- start start 2024-03-30T01:02:24.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.471+0000 7ff0a780bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa879e0 0x556b5fa909e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:24.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.471+0000 7ff0a780bb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa98a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:24.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.471+0000 7ff0a780bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa99100 0x556b5fa7b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:24.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.471+0000 7ff0a780bb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b5fa830d0 con 0x556b5fa89190 2024-03-30T01:02:24.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.471+0000 7ff0a780bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b5fa83ed0 con 0x556b5fa99100 2024-03-30T01:02:24.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.471+0000 7ff0a780bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b5fa82550 con 0x556b5fa879e0 2024-03-30T01:02:24.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.475+0000 7ff091c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa99100 0x556b5fa7b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:24.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.475+0000 7ff091c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa99100 0x556b5fa7b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:52818/0 (socket says 172.21.15.116:52818) 2024-03-30T01:02:24.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.475+0000 7ff091c00700 1 -- 172.21.15.116:0/3101607431 learned_addr learned my addr 172.21.15.116:0/3101607431 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:24.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.479+0000 7ff091200700 1 --2- 172.21.15.116:0/3101607431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa879e0 0x556b5fa909e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.480+0000 7ff08be00700 1 --2- 172.21.15.116:0/3101607431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa98a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.480+0000 7ff08be00700 1 -- 172.21.15.116:0/3101607431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa879e0 msgr2=0x556b5fa909e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.481+0000 7ff08be00700 1 --2- 172.21.15.116:0/3101607431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa879e0 0x556b5fa909e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.481+0000 7ff08be00700 1 -- 172.21.15.116:0/3101607431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa99100 msgr2=0x556b5fa7b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.481+0000 7ff08be00700 1 --2- 172.21.15.116:0/3101607431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa99100 0x556b5fa7b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.481+0000 7ff08be00700 1 -- 172.21.15.116:0/3101607431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b5f99eea0 con 0x556b5fa89190 2024-03-30T01:02:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.481+0000 7ff08be00700 1 --2- 172.21.15.116:0/3101607431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa98a10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff07800d000 tx=0x7ff078009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7620faf705f6b1d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:24.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.492+0000 7ff082a00700 1 -- 172.21.15.116:0/3101607431 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078025690 con 0x556b5fa89190 2024-03-30T01:02:24.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.492+0000 7ff082a00700 1 -- 172.21.15.116:0/3101607431 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff078025d90 con 0x556b5fa89190 2024-03-30T01:02:24.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.492+0000 7ff082a00700 1 -- 172.21.15.116:0/3101607431 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078003680 con 0x556b5fa89190 2024-03-30T01:02:24.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.493+0000 7ff0a780bb80 1 -- 172.21.15.116:0/3101607431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 msgr2=0x556b5fa98a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:24.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.493+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/3101607431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa98a10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff07800d000 tx=0x7ff078009480 comp rx=0 tx=0).stop 2024-03-30T01:02:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.510+0000 7ff0a780bb80 1 -- 172.21.15.116:0/3101607431 shutdown_connections 2024-03-30T01:02:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.510+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/3101607431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa879e0 0x556b5fa909e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.510+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/3101607431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa99100 0x556b5fa7b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.510+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/3101607431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa98a10 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.510+0000 7ff0a780bb80 1 -- 172.21.15.116:0/3101607431 >> 172.21.15.116:0/3101607431 conn(0x556b5f9e66a0 msgr2=0x556b5faa40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:24.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.514+0000 7ff0a780bb80 1 -- 172.21.15.116:0/3101607431 shutdown_connections 2024-03-30T01:02:24.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.522+0000 7ff0a780bb80 1 -- 172.21.15.116:0/3101607431 wait complete. 2024-03-30T01:02:24.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.523+0000 7ff0a780bb80 1 -- start start 2024-03-30T01:02:24.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.523+0000 7ff0a780bb80 1 Processor -- start 2024-03-30T01:02:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.539+0000 7ff0a780bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa879e0 0x556b5fa96ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.539+0000 7ff0a780bb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa97410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.539+0000 7ff0a780bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa99100 0x556b5fa97ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.539+0000 7ff0a780bb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b5fa98420 con 0x556b5fa89190 2024-03-30T01:02:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.539+0000 7ff0a780bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b5fa98590 con 0x556b5fa99100 2024-03-30T01:02:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.539+0000 7ff0a780bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b5fa98750 con 0x556b5fa879e0 2024-03-30T01:02:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.539+0000 7ff0a780bb80 10 client.0 _ll_register_callbacks cb 0x556b5f97e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:02:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.540+0000 7ff08be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa97410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.540+0000 7ff08be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa97410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59872/0 (socket says 172.21.15.116:59872) 2024-03-30T01:02:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.540+0000 7ff08be00700 1 -- 172.21.15.116:0/2053957142 learned_addr learned my addr 172.21.15.116:0/2053957142 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.541+0000 7ff08be00700 1 -- 172.21.15.116:0/2053957142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa879e0 msgr2=0x556b5fa96ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:02:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.541+0000 7ff08be00700 1 --2- 172.21.15.116:0/2053957142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa879e0 0x556b5fa96ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.541+0000 7ff08be00700 1 -- 172.21.15.116:0/2053957142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa99100 msgr2=0x556b5fa97ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:02:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.541+0000 7ff08be00700 1 --2- 172.21.15.116:0/2053957142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa99100 0x556b5fa97ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.541+0000 7ff08be00700 1 -- 172.21.15.116:0/2053957142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b5f806b60 con 0x556b5fa89190 2024-03-30T01:02:24.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.541+0000 7ff08be00700 1 --2- 172.21.15.116:0/2053957142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa97410 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff078000c00 tx=0x7ff078009fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5331442642457b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:24.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.549+0000 7ff076a00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:02:24.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.549+0000 7ff076a00700 1 -- 172.21.15.116:0/2053957142 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078025690 con 0x556b5fa89190 2024-03-30T01:02:24.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.549+0000 7ff076a00700 1 -- 172.21.15.116:0/2053957142 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff078000e40 con 0x556b5fa89190 2024-03-30T01:02:24.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.550+0000 7ff0a780bb80 1 -- 172.21.15.116:0/2053957142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff08409aba0 con 0x556b5fa89190 2024-03-30T01:02:24.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.578+0000 7ff076a00700 1 -- 172.21.15.116:0/2053957142 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff078005550 con 0x556b5fa89190 2024-03-30T01:02:24.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.581+0000 7ff0a780bb80 10 client.16050 fetch_fsmap learned FSMap version 27 2024-03-30T01:02:24.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.581+0000 7ff0a780bb80 1 -- 172.21.15.116:0/2053957142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x556b5f85ecd0 con 0x556b5fa89190 2024-03-30T01:02:24.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.581+0000 7ff076a00700 1 -- 172.21.15.116:0/2053957142 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff08409aba0 con 0x556b5fa89190 2024-03-30T01:02:24.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.581+0000 7ff076a00700 1 -- 172.21.15.116:0/2053957142 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7ff078060050 con 0x556b5fa89190 2024-03-30T01:02:24.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.582+0000 7ff0a780bb80 10 client.16050 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T01:02:24.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.582+0000 7ff0a780bb80 10 client.16050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:02:24.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.582+0000 7ff0a780bb80 20 client.16050 populate_metadata read hostname 'smithi116' 2024-03-30T01:02:24.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.582+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/2053957142 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556b5faf1e10 0x556b5faf4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:24.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.582+0000 7ff0a780bb80 4 client.16050 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:02:24.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.582+0000 7ff0a780bb80 1 -- 172.21.15.116:0/2053957142 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556b5f827cf0 con 0x556b5faf1e10 2024-03-30T01:02:24.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.585+0000 7ff091200700 1 --2- 172.21.15.116:0/2053957142 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556b5faf1e10 0x556b5faf4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:24.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.598+0000 7ff091200700 1 --2- 172.21.15.116:0/2053957142 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556b5faf1e10 0x556b5faf4230 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca7e76cfbf5de6ad server_cookie=40b2ba3628f4bea in_seq=0 out_seq=0 2024-03-30T01:02:24.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.600+0000 7ff076a00700 10 client.16050 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:02:24.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.600+0000 7ff076a00700 1 -- 172.21.15.116:0/2053957142 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556b5f827cf0 con 0x556b5faf1e10 2024-03-30T01:02:24.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.600+0000 7ff076a00700 10 client.16050 handle_command_reply: tid=0 2024-03-30T01:02:24.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.611+0000 7ff0a780bb80 1 client.16050 shutdown 2024-03-30T01:02:24.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.622+0000 7ff0a780bb80 1 -- 172.21.15.116:0/2053957142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 msgr2=0x556b5fa97410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:24.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.622+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/2053957142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa97410 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff078000c00 tx=0x7ff078009fc0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.622+0000 7ff0a780bb80 1 -- 172.21.15.116:0/2053957142 shutdown_connections 2024-03-30T01:02:24.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.622+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/2053957142 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x556b5faf1e10 0x556b5faf4230 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.622+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/2053957142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b5fa879e0 0x556b5fa96ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.622+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/2053957142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b5fa99100 0x556b5fa97ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.622+0000 7ff0a780bb80 1 --2- 172.21.15.116:0/2053957142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b5fa89190 0x556b5fa97410 secure :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff078000c00 tx=0x7ff078009fc0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.622+0000 7ff0a780bb80 1 -- 172.21.15.116:0/2053957142 >> 172.21.15.116:0/2053957142 conn(0x556b5f9e66a0 msgr2=0x556b5fa7b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:24.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.622+0000 7ff076a00700 0 client.16050 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:02:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.623+0000 7ff0a780bb80 1 -- 172.21.15.116:0/2053957142 shutdown_connections 2024-03-30T01:02:24.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.630+0000 7ff0a780bb80 1 -- 172.21.15.116:0/2053957142 wait complete. 2024-03-30T01:02:24.639 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:02:24.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.634+0000 7ff0a780bb80 20 client.16050 trim_cache size 0 max 16384 2024-03-30T01:02:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.643+0000 7ff092600700 1 -- 172.21.15.116:0/2230643386 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff0700a1320 msgr2=0x7ff0700a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.643+0000 7ff092600700 1 --2- 172.21.15.116:0/2230643386 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff0700a1320 0x7ff0700a37e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff07c005fd0 tx=0x7ff07c005ee0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.644+0000 7ff092600700 1 -- 172.21.15.116:0/2230643386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff08c10ce40 msgr2=0x7ff08c1acfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:24.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.644+0000 7ff092600700 1 --2- 172.21.15.116:0/2230643386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff08c10ce40 0x7ff08c1acfc0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff084007a60 tx=0x7ff08400be00 comp rx=0 tx=0).stop 2024-03-30T01:02:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.649+0000 7ff092600700 1 -- 172.21.15.116:0/2230643386 shutdown_connections 2024-03-30T01:02:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.649+0000 7ff092600700 1 --2- 172.21.15.116:0/2230643386 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff0700a1320 0x7ff0700a37e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.649+0000 7ff092600700 1 --2- 172.21.15.116:0/2230643386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff08c072b80 0x7ff08c1a5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.649+0000 7ff092600700 1 --2- 172.21.15.116:0/2230643386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff08c10ce40 0x7ff08c1acfc0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.649+0000 7ff092600700 1 --2- 172.21.15.116:0/2230643386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c10a390 0x7ff08c1a5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.649+0000 7ff092600700 1 -- 172.21.15.116:0/2230643386 >> 172.21.15.116:0/2230643386 conn(0x7ff08c06da50 msgr2=0x7ff08c109b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:24.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.661+0000 7ff092600700 1 -- 172.21.15.116:0/2230643386 shutdown_connections 2024-03-30T01:02:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:24.679+0000 7ff092600700 1 -- 172.21.15.116:0/2230643386 wait complete. 2024-03-30T01:02:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:24 smithi130 ceph-mon[47626]: pgmap v427: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 14 MiB/s wr, 767 op/s 2024-03-30T01:02:24.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:24 smithi116 ceph-mon[38086]: pgmap v427: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 14 MiB/s wr, 767 op/s 2024-03-30T01:02:25.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:24 smithi181 ceph-mon[47622]: pgmap v427: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 14 MiB/s wr, 767 op/s 2024-03-30T01:02:26.094 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T01:02:26.094 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T01:02:26.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:26 smithi130 ceph-mon[47626]: pgmap v428: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 12 MiB/s wr, 705 op/s 2024-03-30T01:02:26.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:26 smithi116 ceph-mon[38086]: pgmap v428: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 12 MiB/s wr, 705 op/s 2024-03-30T01:02:27.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:26 smithi181 ceph-mon[47622]: pgmap v428: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 12 MiB/s wr, 705 op/s 2024-03-30T01:02:28.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:28 smithi130 ceph-mon[47626]: pgmap v429: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.3 MiB/s wr, 544 op/s 2024-03-30T01:02:28.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:28 smithi116 ceph-mon[38086]: pgmap v429: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.3 MiB/s wr, 544 op/s 2024-03-30T01:02:29.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:28 smithi181 ceph-mon[47622]: pgmap v429: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.3 MiB/s wr, 544 op/s 2024-03-30T01:02:29.029 INFO:tasks.workunit.client.0.smithi116.stdout: 330 614896 1469 444209 1369 224770 1680 2024-03-30T01:02:30.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:30 smithi130 ceph-mon[47626]: pgmap v430: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.9 MiB/s wr, 409 op/s 2024-03-30T01:02:30.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:30 smithi116 ceph-mon[38086]: pgmap v430: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.9 MiB/s wr, 409 op/s 2024-03-30T01:02:31.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:30 smithi181 ceph-mon[47622]: pgmap v430: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.9 MiB/s wr, 409 op/s 2024-03-30T01:02:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.021+0000 7f07d9c00700 1 -- 172.21.15.116:0/2963406965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07d410a720 msgr2=0x7f07d410cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:31.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.021+0000 7f07d9c00700 1 --2- 172.21.15.116:0/2963406965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07d410a720 0x7f07d410cbb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f07c00098f0 tx=0x7f07c0009c00 comp rx=0 tx=0).stop 2024-03-30T01:02:31.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.026+0000 7f07d9c00700 1 -- 172.21.15.116:0/2963406965 shutdown_connections 2024-03-30T01:02:31.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.026+0000 7f07d9c00700 1 --2- 172.21.15.116:0/2963406965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d410d0f0 0x7f07d410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.026+0000 7f07d9c00700 1 --2- 172.21.15.116:0/2963406965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07d4107db0 0x7f07d410a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.026+0000 7f07d9c00700 1 --2- 172.21.15.116:0/2963406965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07d410a720 0x7f07d410cbb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.026+0000 7f07d9c00700 1 -- 172.21.15.116:0/2963406965 >> 172.21.15.116:0/2963406965 conn(0x7f07d406dd50 msgr2=0x7f07d4070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:31.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.034+0000 7f07d9c00700 1 -- 172.21.15.116:0/2963406965 shutdown_connections 2024-03-30T01:02:31.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.034+0000 7f07d9c00700 1 -- 172.21.15.116:0/2963406965 wait complete. 2024-03-30T01:02:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.054+0000 7f07d9c00700 1 Processor -- start 2024-03-30T01:02:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.065+0000 7f07d9c00700 1 -- start start 2024-03-30T01:02:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.066+0000 7f07d9c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07d4107db0 0x7f07d419d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.066+0000 7f07d9c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07d410a720 0x7f07d419d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.066+0000 7f07d9c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d410d0f0 0x7f07d41a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.066+0000 7f07d9c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d41a4ea0 con 0x7f07d410d0f0 2024-03-30T01:02:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.066+0000 7f07d9c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d41a4fe0 con 0x7f07d410a720 2024-03-30T01:02:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.066+0000 7f07d9c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d41a5170 con 0x7f07d4107db0 2024-03-30T01:02:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.067+0000 7f07d3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d410d0f0 0x7f07d41a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.067+0000 7f07d3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d410d0f0 0x7f07d41a48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49178/0 (socket says 172.21.15.116:49178) 2024-03-30T01:02:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.067+0000 7f07d3400700 1 -- 172.21.15.116:0/2303433730 learned_addr learned my addr 172.21.15.116:0/2303433730 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.068+0000 7f07d2000700 1 --2- 172.21.15.116:0/2303433730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07d410a720 0x7f07d419d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.068+0000 7f07d2a00700 1 --2- 172.21.15.116:0/2303433730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07d4107db0 0x7f07d419d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.069+0000 7f07d3400700 1 -- 172.21.15.116:0/2303433730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07d4107db0 msgr2=0x7f07d419d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.069+0000 7f07d3400700 1 --2- 172.21.15.116:0/2303433730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07d4107db0 0x7f07d419d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.069+0000 7f07d3400700 1 -- 172.21.15.116:0/2303433730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07d410a720 msgr2=0x7f07d419d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.069+0000 7f07d3400700 1 --2- 172.21.15.116:0/2303433730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07d410a720 0x7f07d419d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.069+0000 7f07d3400700 1 -- 172.21.15.116:0/2303433730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c0009520 con 0x7f07d410d0f0 2024-03-30T01:02:31.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.071+0000 7f07d3400700 1 --2- 172.21.15.116:0/2303433730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d410d0f0 0x7f07d41a48d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f07c401e9f0 tx=0x7f07c401edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:31.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.125+0000 7f07cb400700 1 -- 172.21.15.116:0/2303433730 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c401aa10 con 0x7f07d410d0f0 2024-03-30T01:02:31.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.125+0000 7f07cb400700 1 -- 172.21.15.116:0/2303433730 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f07c4004cf0 con 0x7f07d410d0f0 2024-03-30T01:02:31.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.125+0000 7f07cb400700 1 -- 172.21.15.116:0/2303433730 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c4005680 con 0x7f07d410d0f0 2024-03-30T01:02:31.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.146+0000 7f07d9c00700 1 -- 172.21.15.116:0/2303433730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d41a5450 con 0x7f07d410d0f0 2024-03-30T01:02:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.147+0000 7f07d9c00700 1 -- 172.21.15.116:0/2303433730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d41a58e0 con 0x7f07d410d0f0 2024-03-30T01:02:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.203+0000 7f07cb400700 1 -- 172.21.15.116:0/2303433730 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f07c401ab70 con 0x7f07d410d0f0 2024-03-30T01:02:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.206+0000 7f07cb400700 1 --2- 172.21.15.116:0/2303433730 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f07b80a1320 0x7f07b80a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.206+0000 7f07cb400700 1 -- 172.21.15.116:0/2303433730 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f07c40ddbc0 con 0x7f07d410d0f0 2024-03-30T01:02:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.219+0000 7f07d2a00700 1 --2- 172.21.15.116:0/2303433730 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f07b80a1320 0x7f07b80a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.220+0000 7f07d2a00700 1 --2- 172.21.15.116:0/2303433730 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f07b80a1320 0x7f07b80a37e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009d60 tx=0x7f07cc009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:31.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.232+0000 7f07ed7b2b80 1 Processor -- start 2024-03-30T01:02:31.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.262+0000 7f07ed7b2b80 1 -- start start 2024-03-30T01:02:31.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.262+0000 7f07ed7b2b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd17850 0x5620bcd17c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.262+0000 7f07ed7b2b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd181b0 0x5620bcd18a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.263+0000 7f07ed7b2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd19050 0x5620bcd194d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.263+0000 7f07ed7b2b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620bcc6fa10 con 0x5620bcd17850 2024-03-30T01:02:31.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.263+0000 7f07ed7b2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620bcc6d3d0 con 0x5620bcd19050 2024-03-30T01:02:31.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.263+0000 7f07ed7b2b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620bcc68cc0 con 0x5620bcd181b0 2024-03-30T01:02:31.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.267+0000 7f07d3400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd19050 0x5620bcd194d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:31.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.267+0000 7f07d3400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd19050 0x5620bcd194d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:51488/0 (socket says 172.21.15.116:51488) 2024-03-30T01:02:31.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.267+0000 7f07d3400700 1 -- 172.21.15.116:0/625216272 learned_addr learned my addr 172.21.15.116:0/625216272 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:31.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.268+0000 7f07d3400700 1 -- 172.21.15.116:0/625216272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd181b0 msgr2=0x5620bcd18a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:31.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.268+0000 7f07d3400700 1 --2- 172.21.15.116:0/625216272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd181b0 0x5620bcd18a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.268+0000 7f07d3400700 1 -- 172.21.15.116:0/625216272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd17850 msgr2=0x5620bcd17c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:02:31.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.268+0000 7f07d3400700 1 --2- 172.21.15.116:0/625216272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd17850 0x5620bcd17c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.268+0000 7f07d3400700 1 -- 172.21.15.116:0/625216272 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620bcaea430 con 0x5620bcd19050 2024-03-30T01:02:31.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.268+0000 7f07d3400700 1 --2- 172.21.15.116:0/625216272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd19050 0x5620bcd194d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f07c40b2bf0 tx=0x7f07c40aa680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9e0c9c7023e9880 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:31.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.288+0000 7f07bfe00700 1 -- 172.21.15.116:0/625216272 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c40aead0 con 0x5620bcd19050 2024-03-30T01:02:31.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.288+0000 7f07bfe00700 1 -- 172.21.15.116:0/625216272 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f07c40b3120 con 0x5620bcd19050 2024-03-30T01:02:31.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.288+0000 7f07bfe00700 1 -- 172.21.15.116:0/625216272 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c40aa190 con 0x5620bcd19050 2024-03-30T01:02:31.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.298+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/625216272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd19050 msgr2=0x5620bcd194d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:31.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.298+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/625216272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd19050 0x5620bcd194d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f07c40b2bf0 tx=0x7f07c40aa680 comp rx=0 tx=0).stop 2024-03-30T01:02:31.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.312+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/625216272 shutdown_connections 2024-03-30T01:02:31.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.312+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/625216272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd19050 0x5620bcd194d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.312+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/625216272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd17850 0x5620bcd17c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.312+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/625216272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd181b0 0x5620bcd18a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.312+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/625216272 >> 172.21.15.116:0/625216272 conn(0x5620bcc76720 msgr2=0x5620bcd12a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:31.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.312+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/625216272 shutdown_connections 2024-03-30T01:02:31.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.312+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/625216272 wait complete. 2024-03-30T01:02:31.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.333+0000 7f07ed7b2b80 1 -- start start 2024-03-30T01:02:31.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.333+0000 7f07ed7b2b80 1 Processor -- start 2024-03-30T01:02:31.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.349+0000 7f07ed7b2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd17850 0x5620bcd21830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.349+0000 7f07ed7b2b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd181b0 0x5620bcd21d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.349+0000 7f07ed7b2b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd28cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.349+0000 7f07ed7b2b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620bcc04e00 con 0x5620bcd19050 2024-03-30T01:02:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.349+0000 7f07ed7b2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620bcc04aa0 con 0x5620bcd17850 2024-03-30T01:02:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.349+0000 7f07ed7b2b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620bcaec720 con 0x5620bcd181b0 2024-03-30T01:02:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.350+0000 7f07ed7b2b80 10 client.0 _ll_register_callbacks cb 0x5620bcc0e4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:02:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.350+0000 7f07d3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd28cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:31.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.350+0000 7f07d3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd28cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49194/0 (socket says 172.21.15.116:49194) 2024-03-30T01:02:31.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.350+0000 7f07d3400700 1 -- 172.21.15.116:0/4011737667 learned_addr learned my addr 172.21.15.116:0/4011737667 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:31.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.351+0000 7f07d3400700 1 -- 172.21.15.116:0/4011737667 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd181b0 msgr2=0x5620bcd21d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:02:31.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.351+0000 7f07d3400700 1 --2- 172.21.15.116:0/4011737667 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd181b0 0x5620bcd21d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.351+0000 7f07d3400700 1 -- 172.21.15.116:0/4011737667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd17850 msgr2=0x5620bcd21830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:02:31.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.351+0000 7f07d3400700 1 --2- 172.21.15.116:0/4011737667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd17850 0x5620bcd21830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.351+0000 7f07d3400700 1 -- 172.21.15.116:0/4011737667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d41a58e0 con 0x5620bcd19050 2024-03-30T01:02:31.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.351+0000 7f07d3400700 1 --2- 172.21.15.116:0/4011737667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd28cc0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f07c40aac30 tx=0x7f07c40295f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5df2367299e1fd35 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:31.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.375+0000 7f07bcc00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:02:31.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.375+0000 7f07bcc00700 1 -- 172.21.15.116:0/4011737667 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c40aa390 con 0x5620bcd19050 2024-03-30T01:02:31.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.375+0000 7f07bcc00700 1 -- 172.21.15.116:0/4011737667 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f07c40b2c90 con 0x5620bcd19050 2024-03-30T01:02:31.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.405+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/4011737667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5620bcc29400 con 0x5620bcd19050 2024-03-30T01:02:31.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.417+0000 7f07bcc00700 1 -- 172.21.15.116:0/4011737667 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07c40b2e50 con 0x5620bcd19050 2024-03-30T01:02:31.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.437+0000 7f07ed7b2b80 10 client.16059 fetch_fsmap learned FSMap version 27 2024-03-30T01:02:31.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.437+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/4011737667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5620bca96be0 con 0x5620bcd19050 2024-03-30T01:02:31.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.439+0000 7f07bcc00700 1 -- 172.21.15.116:0/4011737667 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5620bcc29400 con 0x5620bcd19050 2024-03-30T01:02:31.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.439+0000 7f07bcc00700 1 -- 172.21.15.116:0/4011737667 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f07c40a9c00 con 0x5620bcd19050 2024-03-30T01:02:31.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.441+0000 7f07ed7b2b80 10 client.16059 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T01:02:31.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.441+0000 7f07ed7b2b80 10 client.16059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:02:31.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.441+0000 7f07ed7b2b80 20 client.16059 populate_metadata read hostname 'smithi116' 2024-03-30T01:02:31.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.442+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/4011737667 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5620bcd2d050 0x5620bcd2f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.442+0000 7f07ed7b2b80 4 client.16059 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:02:31.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.442+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/4011737667 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5620bcc61780 con 0x5620bcd2d050 2024-03-30T01:02:31.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.442+0000 7f07d2a00700 1 --2- 172.21.15.116:0/4011737667 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5620bcd2d050 0x5620bcd2f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:31.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.445+0000 7f07d2a00700 1 --2- 172.21.15.116:0/4011737667 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5620bcd2d050 0x5620bcd2f470 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fee51034e58e9fa server_cookie=33b885bedfc81e90 in_seq=0 out_seq=0 2024-03-30T01:02:31.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.446+0000 7f07bcc00700 10 client.16059 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:02:31.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.448+0000 7f07bcc00700 1 -- 172.21.15.116:0/4011737667 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5620bcc61780 con 0x5620bcd2d050 2024-03-30T01:02:31.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.448+0000 7f07bcc00700 10 client.16059 handle_command_reply: tid=0 2024-03-30T01:02:31.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.449+0000 7f07ed7b2b80 1 client.16059 shutdown 2024-03-30T01:02:31.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.476+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/4011737667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 msgr2=0x5620bcd28cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:31.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.476+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/4011737667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd28cc0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f07c40aac30 tx=0x7f07c40295f0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.477+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/4011737667 shutdown_connections 2024-03-30T01:02:31.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.477+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/4011737667 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5620bcd2d050 0x5620bcd2f470 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.477+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/4011737667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd28cc0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.477+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/4011737667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd17850 0x5620bcd21830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.477+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/4011737667 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd181b0 0x5620bcd21d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.477+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/4011737667 >> 172.21.15.116:0/4011737667 conn(0x5620bcc76720 msgr2=0x5620bcd161f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:31.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.477+0000 7f07bcc00700 0 client.16059 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:02:31.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.477+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/4011737667 shutdown_connections 2024-03-30T01:02:31.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.477+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/4011737667 wait complete. 2024-03-30T01:02:31.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.478+0000 7f07ed7b2b80 20 client.16059 trim_cache size 0 max 16384 2024-03-30T01:02:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.554+0000 7f07ed7b2b80 1 Processor -- start 2024-03-30T01:02:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.555+0000 7f07ed7b2b80 1 -- start start 2024-03-30T01:02:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.556+0000 7f07ed7b2b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd17850 0x5620bcd20d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.556+0000 7f07ed7b2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd181b0 0x5620bcd28c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.556+0000 7f07d2a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd17850 0x5620bcd20d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:31.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.556+0000 7f07d2a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd17850 0x5620bcd20d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:41364/0 (socket says 172.21.15.116:41364) 2024-03-30T01:02:31.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.557+0000 7f07d2a00700 1 -- 172.21.15.116:0/2315894268 learned_addr learned my addr 172.21.15.116:0/2315894268 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:31.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.557+0000 7f07d2000700 1 --2- 172.21.15.116:0/2315894268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd181b0 0x5620bcd28c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:31.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.557+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/2315894268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd12610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.559+0000 7f07d3400700 1 --2- 172.21.15.116:0/2315894268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd12610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:31.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.559+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2315894268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620bcc29400 con 0x5620bcd19050 2024-03-30T01:02:31.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.559+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2315894268 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620bcd18ed0 con 0x5620bcd181b0 2024-03-30T01:02:31.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.559+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2315894268 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620bcd13930 con 0x5620bcd17850 2024-03-30T01:02:31.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.560+0000 7f07d2000700 1 -- 172.21.15.116:0/2315894268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd17850 msgr2=0x5620bcd20d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:31.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.560+0000 7f07d2000700 1 --2- 172.21.15.116:0/2315894268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd17850 0x5620bcd20d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.560+0000 7f07d2000700 1 -- 172.21.15.116:0/2315894268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 msgr2=0x5620bcd12610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:31.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.560+0000 7f07d2000700 1 --2- 172.21.15.116:0/2315894268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd12610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.560+0000 7f07d2000700 1 -- 172.21.15.116:0/2315894268 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620bcc2ef20 con 0x5620bcd181b0 2024-03-30T01:02:31.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.560+0000 7f07d2000700 1 --2- 172.21.15.116:0/2315894268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd181b0 0x5620bcd28c80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f07c000b2e0 tx=0x7f07c0009400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f91cfdbb583c1c5d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:31.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.563+0000 7f07c8c00700 1 -- 172.21.15.116:0/2315894268 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c000f460 con 0x5620bcd181b0 2024-03-30T01:02:31.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.563+0000 7f07c8c00700 1 -- 172.21.15.116:0/2315894268 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f07c001d040 con 0x5620bcd181b0 2024-03-30T01:02:31.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.563+0000 7f07c8c00700 1 -- 172.21.15.116:0/2315894268 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c0005290 con 0x5620bcd181b0 2024-03-30T01:02:31.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.564+0000 7f07d3400700 1 --2- 172.21.15.116:0/2315894268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd12610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:02:31.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.565+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2315894268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd181b0 msgr2=0x5620bcd28c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:31.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.565+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/2315894268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd181b0 0x5620bcd28c80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f07c000b2e0 tx=0x7f07c0009400 comp rx=0 tx=0).stop 2024-03-30T01:02:31.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.568+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2315894268 shutdown_connections 2024-03-30T01:02:31.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.568+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/2315894268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd17850 0x5620bcd20d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.568+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/2315894268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd12610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.568+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/2315894268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd181b0 0x5620bcd28c80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.568+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2315894268 >> 172.21.15.116:0/2315894268 conn(0x5620bcc76720 msgr2=0x5620bcd0b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:31.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.569+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2315894268 shutdown_connections 2024-03-30T01:02:31.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.569+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2315894268 wait complete. 2024-03-30T01:02:31.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.570+0000 7f07ed7b2b80 1 -- start start 2024-03-30T01:02:31.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.572+0000 7f07ed7b2b80 1 Processor -- start 2024-03-30T01:02:31.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.575+0000 7f07ed7b2b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd17850 0x5620bcd27520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.576+0000 7f07d2a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd17850 0x5620bcd27520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:31.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.576+0000 7f07d2a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd17850 0x5620bcd27520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:41380/0 (socket says 172.21.15.116:41380) 2024-03-30T01:02:31.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.576+0000 7f07ed7b2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd181b0 0x5620bcd27a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.576+0000 7f07d2a00700 1 -- 172.21.15.116:0/2737944290 learned_addr learned my addr 172.21.15.116:0/2737944290 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:31.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.577+0000 7f07d2000700 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd181b0 0x5620bcd27a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:31.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.580+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd27fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.580+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2737944290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620bcd80e30 con 0x5620bcd19050 2024-03-30T01:02:31.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.580+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2737944290 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620bcd80f70 con 0x5620bcd181b0 2024-03-30T01:02:31.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.580+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2737944290 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620bcd810b0 con 0x5620bcd17850 2024-03-30T01:02:31.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.580+0000 7f07ed7b2b80 10 client.0 _ll_register_callbacks cb 0x5620bcc0e4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:02:31.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.580+0000 7f07d3400700 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd27fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:31.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.581+0000 7f07d3400700 1 -- 172.21.15.116:0/2737944290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd17850 msgr2=0x5620bcd27520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:31.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.581+0000 7f07d3400700 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd17850 0x5620bcd27520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.581+0000 7f07d3400700 1 -- 172.21.15.116:0/2737944290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd181b0 msgr2=0x5620bcd27a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:31.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.581+0000 7f07d3400700 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd181b0 0x5620bcd27a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.581+0000 7f07d3400700 1 -- 172.21.15.116:0/2737944290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d410e0a0 con 0x5620bcd19050 2024-03-30T01:02:31.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.581+0000 7f07d3400700 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd27fa0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f07c401cfd0 tx=0x7f07c40b2ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feabfc501fd7f823 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:31.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.582+0000 7f07bcc00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:02:31.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.582+0000 7f07bcc00700 1 -- 172.21.15.116:0/2737944290 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c40aa390 con 0x5620bcd19050 2024-03-30T01:02:31.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.582+0000 7f07bcc00700 1 -- 172.21.15.116:0/2737944290 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f07c40beb70 con 0x5620bcd19050 2024-03-30T01:02:31.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.582+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2737944290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f07c40b3120 con 0x5620bcd19050 2024-03-30T01:02:31.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.582+0000 7f07d2a00700 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd17850 0x5620bcd27520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:02:31.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.582+0000 7f07d2000700 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd181b0 0x5620bcd27a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:02:31.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.583+0000 7f07bcc00700 1 -- 172.21.15.116:0/2737944290 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07c40af9f0 con 0x5620bcd19050 2024-03-30T01:02:31.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.583+0000 7f07ed7b2b80 10 client.16062 fetch_fsmap learned FSMap version 27 2024-03-30T01:02:31.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.583+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2737944290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5620bcaeed50 con 0x5620bcd19050 2024-03-30T01:02:31.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.583+0000 7f07bcc00700 1 -- 172.21.15.116:0/2737944290 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f07c40b3120 con 0x5620bcd19050 2024-03-30T01:02:31.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.583+0000 7f07bcc00700 1 -- 172.21.15.116:0/2737944290 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f07c4004750 con 0x5620bcd19050 2024-03-30T01:02:31.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.585+0000 7f07ed7b2b80 10 client.16062 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T01:02:31.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.585+0000 7f07ed7b2b80 10 client.16062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:02:31.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.585+0000 7f07ed7b2b80 20 client.16062 populate_metadata read hostname 'smithi116' 2024-03-30T01:02:31.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.585+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5620bcd82600 0x5620bcd84a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:31.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.585+0000 7f07ed7b2b80 4 client.16062 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T01:02:31.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.585+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2737944290 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5620bcab7d70 con 0x5620bcd82600 2024-03-30T01:02:31.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.585+0000 7f07d2a00700 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5620bcd82600 0x5620bcd84a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:31.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.592+0000 7f07d2a00700 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5620bcd82600 0x5620bcd84a20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11c6d354e19de975 server_cookie=549afbb4439b7d0f in_seq=0 out_seq=0 2024-03-30T01:02:31.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.592+0000 7f07bcc00700 10 client.16062 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:02:31.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.594+0000 7f07bcc00700 1 -- 172.21.15.116:0/2737944290 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5620bcab7d70 con 0x5620bcd82600 2024-03-30T01:02:31.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.594+0000 7f07bcc00700 10 client.16062 handle_command_reply: tid=0 2024-03-30T01:02:31.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.594+0000 7f07ed7b2b80 1 client.16062 shutdown 2024-03-30T01:02:31.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.600+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2737944290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 msgr2=0x5620bcd27fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:31.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.600+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd27fa0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f07c401cfd0 tx=0x7f07c40b2ec0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.604+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2737944290 shutdown_connections 2024-03-30T01:02:31.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.605+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5620bcd82600 0x5620bcd84a20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.605+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5620bcd17850 0x5620bcd27520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.605+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bcd19050 0x5620bcd27fa0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.605+0000 7f07ed7b2b80 1 --2- 172.21.15.116:0/2737944290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620bcd181b0 0x5620bcd27a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.605+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2737944290 >> 172.21.15.116:0/2737944290 conn(0x5620bcc76720 msgr2=0x5620bcd20d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:31.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.605+0000 7f07bcc00700 0 client.16062 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:02:31.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.617+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2737944290 shutdown_connections 2024-03-30T01:02:31.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.621+0000 7f07ed7b2b80 1 -- 172.21.15.116:0/2737944290 wait complete. 2024-03-30T01:02:31.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.622+0000 7f07ed7b2b80 20 client.16062 trim_cache size 0 max 16384 2024-03-30T01:02:31.637 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:02:31.637 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-30T01:02:31.637 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "54c83d41-a408-4ce9-878d-bf046d416e5c", 2024-03-30T01:02:31.637 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-30T01:02:31.637 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:02:31.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.638+0000 7f07d9c00700 1 -- 172.21.15.116:0/2303433730 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f07b80a1320 msgr2=0x7f07b80a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:31.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.638+0000 7f07d9c00700 1 --2- 172.21.15.116:0/2303433730 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f07b80a1320 0x7f07b80a37e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009d60 tx=0x7f07cc009040 comp rx=0 tx=0).stop 2024-03-30T01:02:31.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.638+0000 7f07d9c00700 1 -- 172.21.15.116:0/2303433730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d410d0f0 msgr2=0x7f07d41a48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.638+0000 7f07d9c00700 1 --2- 172.21.15.116:0/2303433730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d410d0f0 0x7f07d41a48d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f07c401e9f0 tx=0x7f07c401edb0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.641+0000 7f07d9c00700 1 -- 172.21.15.116:0/2303433730 shutdown_connections 2024-03-30T01:02:31.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.641+0000 7f07d9c00700 1 --2- 172.21.15.116:0/2303433730 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f07b80a1320 0x7f07b80a37e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.641+0000 7f07d9c00700 1 --2- 172.21.15.116:0/2303433730 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07d4107db0 0x7f07d419d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.641+0000 7f07d9c00700 1 --2- 172.21.15.116:0/2303433730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d410d0f0 0x7f07d41a48d0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.641+0000 7f07d9c00700 1 --2- 172.21.15.116:0/2303433730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07d410a720 0x7f07d419d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:31.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.641+0000 7f07d9c00700 1 -- 172.21.15.116:0/2303433730 >> 172.21.15.116:0/2303433730 conn(0x7f07d406dd50 msgr2=0x7f07d410af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:31.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.645+0000 7f07d9c00700 1 -- 172.21.15.116:0/2303433730 shutdown_connections 2024-03-30T01:02:31.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:31.651+0000 7f07d9c00700 1 -- 172.21.15.116:0/2303433730 wait complete. 2024-03-30T01:02:31.750 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T01:02:32.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:32 smithi130 ceph-mon[47626]: pgmap v431: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 364 op/s 2024-03-30T01:02:32.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:32 smithi130 ceph-mon[47626]: scrub queued for path: / 2024-03-30T01:02:32.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:32 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:32.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:32 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:32.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:32 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:32.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:32 smithi116 ceph-mon[38086]: pgmap v431: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 364 op/s 2024-03-30T01:02:32.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:32 smithi116 ceph-mon[38086]: scrub queued for path: / 2024-03-30T01:02:32.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:32 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:32.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:32 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:32.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:32 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:33.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:32 smithi181 ceph-mon[47622]: pgmap v431: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 364 op/s 2024-03-30T01:02:33.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:32 smithi181 ceph-mon[47622]: scrub queued for path: / 2024-03-30T01:02:33.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:32 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:33.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:32 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:33.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:32 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:33.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:33 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:33.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:33 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:02:33.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:33 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:33.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:33 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:02:34.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:33 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:34.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:33 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:02:34.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.401+0000 7f4ae3800700 1 -- 172.21.15.116:0/538358095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4adc107db0 msgr2=0x7f4adc10a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:34.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.401+0000 7f4ae3800700 1 --2- 172.21.15.116:0/538358095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4adc107db0 0x7f4adc10a1e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4ac80099e0 tx=0x7f4ac8009cf0 comp rx=0 tx=0).stop 2024-03-30T01:02:34.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.473+0000 7f4ae3800700 1 -- 172.21.15.116:0/538358095 shutdown_connections 2024-03-30T01:02:34.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.473+0000 7f4ae3800700 1 --2- 172.21.15.116:0/538358095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc10d0f0 0x7f4adc10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:34.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.473+0000 7f4ae3800700 1 --2- 172.21.15.116:0/538358095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4adc107db0 0x7f4adc10a1e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:34.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.473+0000 7f4ae3800700 1 --2- 172.21.15.116:0/538358095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4adc10a720 0x7f4adc10cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:34.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.473+0000 7f4ae3800700 1 -- 172.21.15.116:0/538358095 >> 172.21.15.116:0/538358095 conn(0x7f4adc06dd50 msgr2=0x7f4adc070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:34.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.473+0000 7f4ae3800700 1 -- 172.21.15.116:0/538358095 shutdown_connections 2024-03-30T01:02:34.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.474+0000 7f4ae3800700 1 -- 172.21.15.116:0/538358095 wait complete. 2024-03-30T01:02:34.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.477+0000 7f4ae3800700 1 Processor -- start 2024-03-30T01:02:34.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.530+0000 7f4ae3800700 1 -- start start 2024-03-30T01:02:34.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.547+0000 7f4ae3800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc107db0 0x7f4adc114440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:34.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.547+0000 7f4ae3800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4adc10a720 0x7f4adc114980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:34.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.547+0000 7f4ae3800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4adc10d0f0 0x7f4adc1b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:34.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.547+0000 7f4ae3800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc114f50 con 0x7f4adc10d0f0 2024-03-30T01:02:34.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.547+0000 7f4ae3800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc1150c0 con 0x7f4adc10a720 2024-03-30T01:02:34.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.547+0000 7f4ae3800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc1b62f0 con 0x7f4adc107db0 2024-03-30T01:02:34.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.569+0000 7f4adbe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4adc10a720 0x7f4adc114980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.577+0000 7f4ae0e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc107db0 0x7f4adc114440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.577+0000 7f4ae0e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc107db0 0x7f4adc114440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:41388/0 (socket says 172.21.15.116:41388) 2024-03-30T01:02:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.577+0000 7f4ae0e00700 1 -- 172.21.15.116:0/681973519 learned_addr learned my addr 172.21.15.116:0/681973519 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.585+0000 7f4ae0e00700 1 -- 172.21.15.116:0/681973519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4adc10a720 msgr2=0x7f4adc114980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.585+0000 7f4ae0e00700 1 --2- 172.21.15.116:0/681973519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4adc10a720 0x7f4adc114980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.585+0000 7f4ae0e00700 1 -- 172.21.15.116:0/681973519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4adc10d0f0 msgr2=0x7f4adc1b5db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T01:02:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.585+0000 7f4ae0e00700 1 --2- 172.21.15.116:0/681973519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4adc10d0f0 0x7f4adc1b5db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.585+0000 7f4ae0e00700 1 -- 172.21.15.116:0/681973519 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac80095f0 con 0x7f4adc107db0 2024-03-30T01:02:34.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.623+0000 7f4ae0e00700 1 --2- 172.21.15.116:0/681973519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc107db0 0x7f4adc114440 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8006010 tx=0x7f4ac8003ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:34.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.636+0000 7f4ad9600700 1 -- 172.21.15.116:0/681973519 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac801d070 con 0x7f4adc107db0 2024-03-30T01:02:34.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.636+0000 7f4ad9600700 1 -- 172.21.15.116:0/681973519 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ac80053e0 con 0x7f4adc107db0 2024-03-30T01:02:34.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.636+0000 7f4ad9600700 1 -- 172.21.15.116:0/681973519 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac80173f0 con 0x7f4adc107db0 2024-03-30T01:02:34.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.754+0000 7f4ae3800700 1 -- 172.21.15.116:0/681973519 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4adc1b6570 con 0x7f4adc107db0 2024-03-30T01:02:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.754+0000 7f4ae3800700 1 -- 172.21.15.116:0/681973519 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4adc1b68e0 con 0x7f4adc107db0 2024-03-30T01:02:34.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:34 smithi116 ceph-mon[38086]: pgmap v432: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 228 op/s 2024-03-30T01:02:34.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.968+0000 7f4ad9600700 1 -- 172.21.15.116:0/681973519 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f4ac8005550 con 0x7f4adc107db0 2024-03-30T01:02:35.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.995+0000 7f4af7300b80 1 Processor -- start 2024-03-30T01:02:35.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:34.996+0000 7f4ad9600700 1 --2- 172.21.15.116:0/681973519 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4ac00a1320 0x7f4ac00a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:35.008 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:34 smithi181 ceph-mon[47622]: pgmap v432: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 228 op/s 2024-03-30T01:02:35.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.014+0000 7f4ae1800700 1 --2- 172.21.15.116:0/681973519 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4ac00a1320 0x7f4ac00a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:35.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.016+0000 7f4ad9600700 1 -- 172.21.15.116:0/681973519 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4ac80ce030 con 0x7f4adc107db0 2024-03-30T01:02:35.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.021+0000 7f4ae1800700 1 --2- 172.21.15.116:0/681973519 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4ac00a1320 0x7f4ac00a37e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400ab60 tx=0x7f4ad4009230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:35.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.065+0000 7f4af7300b80 1 -- start start 2024-03-30T01:02:35.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.066+0000 7f4af7300b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697b7d0 0x56362697bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:35.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.066+0000 7f4af7300b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697c130 0x56362697c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:35.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.066+0000 7f4af7300b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697cfd0 0x56362697d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:35.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.066+0000 7f4af7300b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636268d3990 con 0x56362697c130 2024-03-30T01:02:35.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.066+0000 7f4af7300b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636268d1350 con 0x56362697b7d0 2024-03-30T01:02:35.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.066+0000 7f4af7300b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636268ccc40 con 0x56362697cfd0 2024-03-30T01:02:35.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.066+0000 7f4adbe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697c130 0x56362697c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:35.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.066+0000 7f4adbe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697c130 0x56362697c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45026/0 (socket says 172.21.15.116:45026) 2024-03-30T01:02:35.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.066+0000 7f4adbe00700 1 -- 172.21.15.116:0/1060628006 learned_addr learned my addr 172.21.15.116:0/1060628006 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:35.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.067+0000 7f4adbe00700 1 -- 172.21.15.116:0/1060628006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697cfd0 msgr2=0x56362697d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:35.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.067+0000 7f4adbe00700 1 --2- 172.21.15.116:0/1060628006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697cfd0 0x56362697d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:35.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.067+0000 7f4adbe00700 1 -- 172.21.15.116:0/1060628006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697b7d0 msgr2=0x56362697bbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:02:35.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.067+0000 7f4adbe00700 1 --2- 172.21.15.116:0/1060628006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697b7d0 0x56362697bbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:35.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.067+0000 7f4adbe00700 1 -- 172.21.15.116:0/1060628006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56362674e3b0 con 0x56362697c130 2024-03-30T01:02:35.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.068+0000 7f4adbe00700 1 --2- 172.21.15.116:0/1060628006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697c130 0x56362697c9f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009ad0 tx=0x7f4acc009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7365bca66b2c69 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:35.098 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:34 smithi130 ceph-mon[47626]: pgmap v432: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 228 op/s 2024-03-30T01:02:35.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.134+0000 7f4ad2000700 1 -- 172.21.15.116:0/1060628006 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc0095d0 con 0x56362697c130 2024-03-30T01:02:35.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.134+0000 7f4ad2000700 1 -- 172.21.15.116:0/1060628006 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4acc00ee50 con 0x56362697c130 2024-03-30T01:02:35.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.134+0000 7f4ad2000700 1 -- 172.21.15.116:0/1060628006 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc0123d0 con 0x56362697c130 2024-03-30T01:02:35.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.168+0000 7f4af7300b80 1 -- 172.21.15.116:0/1060628006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697c130 msgr2=0x56362697c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:35.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.168+0000 7f4af7300b80 1 --2- 172.21.15.116:0/1060628006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697c130 0x56362697c9f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009ad0 tx=0x7f4acc009de0 comp rx=0 tx=0).stop 2024-03-30T01:02:35.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.239+0000 7f4af7300b80 1 -- 172.21.15.116:0/1060628006 shutdown_connections 2024-03-30T01:02:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.239+0000 7f4af7300b80 1 --2- 172.21.15.116:0/1060628006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697cfd0 0x56362697d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.239+0000 7f4af7300b80 1 --2- 172.21.15.116:0/1060628006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697b7d0 0x56362697bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.239+0000 7f4af7300b80 1 --2- 172.21.15.116:0/1060628006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697c130 0x56362697c9f0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.239+0000 7f4af7300b80 1 -- 172.21.15.116:0/1060628006 >> 172.21.15.116:0/1060628006 conn(0x5636268da6a0 msgr2=0x5636269769a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.248+0000 7f4af7300b80 1 -- 172.21.15.116:0/1060628006 shutdown_connections 2024-03-30T01:02:35.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.250+0000 7f4af7300b80 1 -- 172.21.15.116:0/1060628006 wait complete. 2024-03-30T01:02:35.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.258+0000 7f4af7300b80 1 -- start start 2024-03-30T01:02:35.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.258+0000 7f4af7300b80 1 Processor -- start 2024-03-30T01:02:35.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.271+0000 7f4af7300b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697b7d0 0x5636269857c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.272+0000 7f4af7300b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697c130 0x563626985d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.272+0000 7f4af7300b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697cfd0 0x56362698ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:35.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.273+0000 7f4af7300b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563626868d80 con 0x56362697cfd0 2024-03-30T01:02:35.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.273+0000 7f4af7300b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563626868a20 con 0x56362697c130 2024-03-30T01:02:35.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.273+0000 7f4af7300b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636267506a0 con 0x56362697b7d0 2024-03-30T01:02:35.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.273+0000 7f4af7300b80 10 client.0 _ll_register_callbacks cb 0x563626872470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:02:35.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.274+0000 7f4ae1800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697cfd0 0x56362698ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:35.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.274+0000 7f4ae1800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697cfd0 0x56362698ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45040/0 (socket says 172.21.15.116:45040) 2024-03-30T01:02:35.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.274+0000 7f4ae1800700 1 -- 172.21.15.116:0/1484974190 learned_addr learned my addr 172.21.15.116:0/1484974190 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:35.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.275+0000 7f4ae1800700 1 -- 172.21.15.116:0/1484974190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697b7d0 msgr2=0x5636269857c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:02:35.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.275+0000 7f4ae1800700 1 --2- 172.21.15.116:0/1484974190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697b7d0 0x5636269857c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.275+0000 7f4ae1800700 1 -- 172.21.15.116:0/1484974190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697c130 msgr2=0x563626985d00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:02:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.275+0000 7f4ae1800700 1 --2- 172.21.15.116:0/1484974190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697c130 0x563626985d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.275+0000 7f4ae1800700 1 -- 172.21.15.116:0/1484974190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4acc009730 con 0x56362697cfd0 2024-03-30T01:02:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.275+0000 7f4ae1800700 1 --2- 172.21.15.116:0/1484974190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697cfd0 0x56362698ccf0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400dcc0 tx=0x7f4ad4004b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b34976226eb0ff server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.276+0000 7f4ac6a00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:02:35.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.325+0000 7f4ac6a00700 1 -- 172.21.15.116:0/1484974190 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad4018ce0 con 0x56362697cfd0 2024-03-30T01:02:35.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.325+0000 7f4ac6a00700 1 -- 172.21.15.116:0/1484974190 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4ad401e070 con 0x56362697cfd0 2024-03-30T01:02:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.354+0000 7f4af7300b80 1 -- 172.21.15.116:0/1484974190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56362688d380 con 0x56362697cfd0 2024-03-30T01:02:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.356+0000 7f4ac6a00700 1 -- 172.21.15.116:0/1484974190 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ad400de60 con 0x56362697cfd0 2024-03-30T01:02:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.356+0000 7f4af7300b80 10 client.16071 fetch_fsmap learned FSMap version 27 2024-03-30T01:02:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.356+0000 7f4af7300b80 1 -- 172.21.15.116:0/1484974190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5636266fab60 con 0x56362697cfd0 2024-03-30T01:02:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.366+0000 7f4ac6a00700 1 -- 172.21.15.116:0/1484974190 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56362688d380 con 0x56362697cfd0 2024-03-30T01:02:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.366+0000 7f4ac6a00700 1 -- 172.21.15.116:0/1484974190 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f4ad402d510 con 0x56362697cfd0 2024-03-30T01:02:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.367+0000 7f4af7300b80 10 client.16071 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T01:02:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.367+0000 7f4af7300b80 10 client.16071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:02:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.367+0000 7f4af7300b80 20 client.16071 populate_metadata read hostname 'smithi116' 2024-03-30T01:02:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.367+0000 7f4af7300b80 1 --2- 172.21.15.116:0/1484974190 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x563626991590 0x5636269939b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.367+0000 7f4af7300b80 4 client.16071 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:02:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.367+0000 7f4af7300b80 1 -- 172.21.15.116:0/1484974190 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636268c5700 con 0x563626991590 2024-03-30T01:02:35.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.377+0000 7f4ae0e00700 1 --2- 172.21.15.116:0/1484974190 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x563626991590 0x5636269939b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.427+0000 7f4ae0e00700 1 --2- 172.21.15.116:0/1484974190 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x563626991590 0x5636269939b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2800035211b49808 server_cookie=b7e12049da9bfd32 in_seq=0 out_seq=0 2024-03-30T01:02:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.432+0000 7f4ac6a00700 10 client.16071 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:02:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.465+0000 7f4ac6a00700 1 -- 172.21.15.116:0/1484974190 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5636268c5700 con 0x563626991590 2024-03-30T01:02:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.465+0000 7f4ac6a00700 10 client.16071 handle_command_reply: tid=0 2024-03-30T01:02:35.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.468+0000 7f4af7300b80 1 client.16071 shutdown 2024-03-30T01:02:35.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.589+0000 7f4af7300b80 1 -- 172.21.15.116:0/1484974190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697cfd0 msgr2=0x56362698ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:35.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.589+0000 7f4af7300b80 1 --2- 172.21.15.116:0/1484974190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697cfd0 0x56362698ccf0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400dcc0 tx=0x7f4ad4004b10 comp rx=0 tx=0).stop 2024-03-30T01:02:35.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.619+0000 7f4af7300b80 1 -- 172.21.15.116:0/1484974190 shutdown_connections 2024-03-30T01:02:35.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.619+0000 7f4af7300b80 1 --2- 172.21.15.116:0/1484974190 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x563626991590 0x5636269939b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:35.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.619+0000 7f4af7300b80 1 --2- 172.21.15.116:0/1484974190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697b7d0 0x5636269857c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:35.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.619+0000 7f4af7300b80 1 --2- 172.21.15.116:0/1484974190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697cfd0 0x56362698ccf0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:35.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.619+0000 7f4af7300b80 1 --2- 172.21.15.116:0/1484974190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697c130 0x563626985d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:35.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.619+0000 7f4af7300b80 1 -- 172.21.15.116:0/1484974190 >> 172.21.15.116:0/1484974190 conn(0x5636268da6a0 msgr2=0x56362697a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:35.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.620+0000 7f4ac6a00700 0 client.16071 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:02:35.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.631+0000 7f4af7300b80 1 -- 172.21.15.116:0/1484974190 shutdown_connections 2024-03-30T01:02:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.631+0000 7f4af7300b80 1 -- 172.21.15.116:0/1484974190 wait complete. 2024-03-30T01:02:35.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.642+0000 7f4af7300b80 20 client.16071 trim_cache size 0 max 16384 2024-03-30T01:02:35.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.969+0000 7f4af7300b80 1 Processor -- start 2024-03-30T01:02:35.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.969+0000 7f4af7300b80 1 -- start start 2024-03-30T01:02:35.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.970+0000 7f4af7300b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697b7d0 0x563626984320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:35.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.970+0000 7f4af7300b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697c130 0x56362698cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:35.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.970+0000 7f4af7300b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697cfd0 0x563626975f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:35.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.970+0000 7f4ae0e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697b7d0 0x563626984320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:35.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.970+0000 7f4af7300b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636269774b0 con 0x56362697b7d0 2024-03-30T01:02:35.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.970+0000 7f4af7300b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56362698ea20 con 0x56362697c130 2024-03-30T01:02:35.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.970+0000 7f4ae0e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697b7d0 0x563626984320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45042/0 (socket says 172.21.15.116:45042) 2024-03-30T01:02:35.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.970+0000 7f4af7300b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56362697d7d0 con 0x56362697cfd0 2024-03-30T01:02:35.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.970+0000 7f4ae0e00700 1 -- 172.21.15.116:0/1370245459 learned_addr learned my addr 172.21.15.116:0/1370245459 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:35.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.972+0000 7f4ae0e00700 1 -- 172.21.15.116:0/1370245459 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697cfd0 msgr2=0x563626975f10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:02:35.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.972+0000 7f4ae0e00700 1 --2- 172.21.15.116:0/1370245459 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697cfd0 0x563626975f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:35.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.972+0000 7f4ae0e00700 1 -- 172.21.15.116:0/1370245459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697c130 msgr2=0x56362698cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:02:35.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.972+0000 7f4ae0e00700 1 --2- 172.21.15.116:0/1370245459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697c130 0x56362698cc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:35.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.972+0000 7f4ae0e00700 1 -- 172.21.15.116:0/1370245459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563626892ea0 con 0x56362697b7d0 2024-03-30T01:02:35.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.973+0000 7f4ae0e00700 1 --2- 172.21.15.116:0/1370245459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697b7d0 0x563626984320 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8097c10 tx=0x7f4ac80a1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738b6bed3e8979eb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:35.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.986+0000 7f4ad2a00700 1 -- 172.21.15.116:0/1370245459 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac809b1a0 con 0x56362697b7d0 2024-03-30T01:02:35.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.986+0000 7f4ad2a00700 1 -- 172.21.15.116:0/1370245459 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ac809efd0 con 0x56362697b7d0 2024-03-30T01:02:35.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:35.986+0000 7f4ad2a00700 1 -- 172.21.15.116:0/1370245459 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac809b660 con 0x56362697b7d0 2024-03-30T01:02:36.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.002+0000 7f4af7300b80 1 -- 172.21.15.116:0/1370245459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697b7d0 msgr2=0x563626984320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.002+0000 7f4af7300b80 1 --2- 172.21.15.116:0/1370245459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697b7d0 0x563626984320 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8097c10 tx=0x7f4ac80a1040 comp rx=0 tx=0).stop 2024-03-30T01:02:36.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.022+0000 7f4af7300b80 1 -- 172.21.15.116:0/1370245459 shutdown_connections 2024-03-30T01:02:36.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.022+0000 7f4af7300b80 1 --2- 172.21.15.116:0/1370245459 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697cfd0 0x563626975f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:36.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.022+0000 7f4af7300b80 1 --2- 172.21.15.116:0/1370245459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697b7d0 0x563626984320 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:36.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.022+0000 7f4af7300b80 1 --2- 172.21.15.116:0/1370245459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697c130 0x56362698cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:36.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.022+0000 7f4af7300b80 1 -- 172.21.15.116:0/1370245459 >> 172.21.15.116:0/1370245459 conn(0x5636268da6a0 msgr2=0x563626998350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.027+0000 7f4af7300b80 1 -- 172.21.15.116:0/1370245459 shutdown_connections 2024-03-30T01:02:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.027+0000 7f4af7300b80 1 -- 172.21.15.116:0/1370245459 wait complete. 2024-03-30T01:02:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.028+0000 7f4af7300b80 1 -- start start 2024-03-30T01:02:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.028+0000 7f4af7300b80 1 Processor -- start 2024-03-30T01:02:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.037+0000 7f4af7300b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697b7d0 0x56362698aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.037+0000 7f4af7300b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697c130 0x56362698af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.037+0000 7f4ae0e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697b7d0 0x56362698aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.037+0000 7f4ae0e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697b7d0 0x56362698aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:60380/0 (socket says 172.21.15.116:60380) 2024-03-30T01:02:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.038+0000 7f4ae0e00700 1 -- 172.21.15.116:0/4247557237 learned_addr learned my addr 172.21.15.116:0/4247557237 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:02:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.038+0000 7f4af7300b80 1 --2- 172.21.15.116:0/4247557237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697cfd0 0x56362698bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.038+0000 7f4adbe00700 1 --2- 172.21.15.116:0/4247557237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697c130 0x56362698af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.039+0000 7f4ae1800700 1 --2- 172.21.15.116:0/4247557237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697cfd0 0x56362698bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.039+0000 7f4af7300b80 1 -- 172.21.15.116:0/4247557237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56362698c480 con 0x56362697cfd0 2024-03-30T01:02:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.039+0000 7f4af7300b80 1 -- 172.21.15.116:0/4247557237 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56362698c5f0 con 0x56362697b7d0 2024-03-30T01:02:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.039+0000 7f4af7300b80 1 -- 172.21.15.116:0/4247557237 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56362698c7b0 con 0x56362697c130 2024-03-30T01:02:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.039+0000 7f4af7300b80 10 client.0 _ll_register_callbacks cb 0x563626872470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:02:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.041+0000 7f4adbe00700 1 -- 172.21.15.116:0/4247557237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697b7d0 msgr2=0x56362698aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.041+0000 7f4adbe00700 1 --2- 172.21.15.116:0/4247557237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697b7d0 0x56362698aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.041+0000 7f4adbe00700 1 -- 172.21.15.116:0/4247557237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697cfd0 msgr2=0x56362698bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:02:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.041+0000 7f4adbe00700 1 --2- 172.21.15.116:0/4247557237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697cfd0 0x56362698bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.041+0000 7f4adbe00700 1 -- 172.21.15.116:0/4247557237 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4adc1b68e0 con 0x56362697c130 2024-03-30T01:02:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.047+0000 7f4adbe00700 1 --2- 172.21.15.116:0/4247557237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697c130 0x56362698af80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009ad0 tx=0x7f4acc004320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2003cab331d2d4e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:02:36.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.081+0000 7f4ac6a00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:02:36.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.081+0000 7f4ac6a00700 1 -- 172.21.15.116:0/4247557237 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc006da0 con 0x56362697c130 2024-03-30T01:02:36.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.081+0000 7f4ac6a00700 1 -- 172.21.15.116:0/4247557237 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4acc003940 con 0x56362697c130 2024-03-30T01:02:36.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.143+0000 7f4af7300b80 1 -- 172.21.15.116:0/4247557237 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4acc00ee50 con 0x56362697c130 2024-03-30T01:02:36.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.144+0000 7f4ac6a00700 1 -- 172.21.15.116:0/4247557237 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4acc006bc0 con 0x56362697c130 2024-03-30T01:02:36.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.144+0000 7f4af7300b80 10 client.26074 fetch_fsmap learned FSMap version 27 2024-03-30T01:02:36.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.144+0000 7f4af7300b80 1 -- 172.21.15.116:0/4247557237 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563626752cd0 con 0x56362697c130 2024-03-30T01:02:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.145+0000 7f4ac6a00700 1 -- 172.21.15.116:0/4247557237 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4acc00ee50 con 0x56362697c130 2024-03-30T01:02:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.145+0000 7f4ac6a00700 1 -- 172.21.15.116:0/4247557237 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f4acc011070 con 0x56362697c130 2024-03-30T01:02:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.145+0000 7f4af7300b80 10 client.26074 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T01:02:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.145+0000 7f4af7300b80 10 client.26074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:02:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.145+0000 7f4af7300b80 20 client.26074 populate_metadata read hostname 'smithi116' 2024-03-30T01:02:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.145+0000 7f4af7300b80 1 --2- 172.21.15.116:0/4247557237 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5636269e6360 0x5636269e8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:02:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.145+0000 7f4af7300b80 4 client.26074 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T01:02:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.145+0000 7f4af7300b80 1 -- 172.21.15.116:0/4247557237 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56362671bcf0 con 0x5636269e6360 2024-03-30T01:02:36.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.146+0000 7f4ae0e00700 1 --2- 172.21.15.116:0/4247557237 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5636269e6360 0x5636269e8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:02:36.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.148+0000 7f4ae0e00700 1 --2- 172.21.15.116:0/4247557237 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5636269e6360 0x5636269e8780 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd921cb950f2ade3 server_cookie=423b7e8196f02078 in_seq=0 out_seq=0 2024-03-30T01:02:36.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.150+0000 7f4ac6a00700 10 client.26074 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:02:36.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.150+0000 7f4ac6a00700 1 -- 172.21.15.116:0/4247557237 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x56362671bcf0 con 0x5636269e6360 2024-03-30T01:02:36.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.150+0000 7f4ac6a00700 10 client.26074 handle_command_reply: tid=0 2024-03-30T01:02:36.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.150+0000 7f4af7300b80 1 client.26074 shutdown 2024-03-30T01:02:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.151+0000 7f4af7300b80 1 -- 172.21.15.116:0/4247557237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697c130 msgr2=0x56362698af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.151+0000 7f4af7300b80 1 --2- 172.21.15.116:0/4247557237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697c130 0x56362698af80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009ad0 tx=0x7f4acc004320 comp rx=0 tx=0).stop 2024-03-30T01:02:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.152+0000 7f4af7300b80 1 -- 172.21.15.116:0/4247557237 shutdown_connections 2024-03-30T01:02:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.152+0000 7f4af7300b80 1 --2- 172.21.15.116:0/4247557237 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x5636269e6360 0x5636269e8780 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.152+0000 7f4af7300b80 1 --2- 172.21.15.116:0/4247557237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56362697cfd0 0x56362698bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:36.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.152+0000 7f4af7300b80 1 --2- 172.21.15.116:0/4247557237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56362697b7d0 0x56362698aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:36.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.152+0000 7f4af7300b80 1 --2- 172.21.15.116:0/4247557237 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56362697c130 0x56362698af80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:36.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.152+0000 7f4af7300b80 1 -- 172.21.15.116:0/4247557237 >> 172.21.15.116:0/4247557237 conn(0x5636268da6a0 msgr2=0x56362696f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:36.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.153+0000 7f4ac6a00700 0 client.26074 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:02:36.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.153+0000 7f4af7300b80 1 -- 172.21.15.116:0/4247557237 shutdown_connections 2024-03-30T01:02:36.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.154+0000 7f4af7300b80 1 -- 172.21.15.116:0/4247557237 wait complete. 2024-03-30T01:02:36.165 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:02:36.165 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (9999 inodes in the stack)", 2024-03-30T01:02:36.165 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-30T01:02:36.165 INFO:teuthology.orchestra.run.smithi116.stdout: "54c83d41-a408-4ce9-878d-bf046d416e5c": { 2024-03-30T01:02:36.165 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-30T01:02:36.165 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "54c83d41-a408-4ce9-878d-bf046d416e5c", 2024-03-30T01:02:36.165 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-30T01:02:36.166 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:02:36.166 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:02:36.166 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:02:36.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.158+0000 7f4af7300b80 20 client.26074 trim_cache size 0 max 16384 2024-03-30T01:02:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.163+0000 7f4ae3800700 1 -- 172.21.15.116:0/681973519 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4ac00a1320 msgr2=0x7f4ac00a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.163+0000 7f4ae3800700 1 --2- 172.21.15.116:0/681973519 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4ac00a1320 0x7f4ac00a37e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400ab60 tx=0x7f4ad4009230 comp rx=0 tx=0).stop 2024-03-30T01:02:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.163+0000 7f4ae3800700 1 -- 172.21.15.116:0/681973519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc107db0 msgr2=0x7f4adc114440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:02:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.163+0000 7f4ae3800700 1 --2- 172.21.15.116:0/681973519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc107db0 0x7f4adc114440 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8006010 tx=0x7f4ac8003ee0 comp rx=0 tx=0).stop 2024-03-30T01:02:36.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.167+0000 7f4ae3800700 1 -- 172.21.15.116:0/681973519 shutdown_connections 2024-03-30T01:02:36.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.167+0000 7f4ae3800700 1 --2- 172.21.15.116:0/681973519 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4ac00a1320 0x7f4ac00a37e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:36.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.167+0000 7f4ae3800700 1 --2- 172.21.15.116:0/681973519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4adc107db0 0x7f4adc114440 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:36.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.167+0000 7f4ae3800700 1 --2- 172.21.15.116:0/681973519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4adc10d0f0 0x7f4adc1b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:36.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.167+0000 7f4ae3800700 1 --2- 172.21.15.116:0/681973519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4adc10a720 0x7f4adc114980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:02:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.167+0000 7f4ae3800700 1 -- 172.21.15.116:0/681973519 >> 172.21.15.116:0/681973519 conn(0x7f4adc06dd50 msgr2=0x7f4adc10bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:02:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.171+0000 7f4ae3800700 1 -- 172.21.15.116:0/681973519 shutdown_connections 2024-03-30T01:02:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:02:36.171+0000 7f4ae3800700 1 -- 172.21.15.116:0/681973519 wait complete. 2024-03-30T01:02:36.679 INFO:tasks.cephfs.filesystem:scrub status for tag:54c83d41-a408-4ce9-878d-bf046d416e5c - {'path': '/', 'tag': '54c83d41-a408-4ce9-878d-bf046d416e5c', 'options': 'recursive,force'} 2024-03-30T01:02:36.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:36 smithi116 ceph-mon[38086]: pgmap v433: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 233 op/s 2024-03-30T01:02:37.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:36 smithi181 ceph-mon[47622]: pgmap v433: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 233 op/s 2024-03-30T01:02:37.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:36 smithi130 ceph-mon[47626]: pgmap v433: 129 pgs: 129 active+clean; 1.2 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 233 op/s 2024-03-30T01:02:38.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:38 smithi116 ceph-mon[38086]: pgmap v434: 129 pgs: 129 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 241 op/s 2024-03-30T01:02:39.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:38 smithi181 ceph-mon[47622]: pgmap v434: 129 pgs: 129 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 241 op/s 2024-03-30T01:02:39.030 INFO:tasks.workunit.client.0.smithi116.stdout: 347 509594 763 362832 908 188523 1172 2024-03-30T01:02:39.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:38 smithi130 ceph-mon[47626]: pgmap v434: 129 pgs: 129 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 241 op/s 2024-03-30T01:02:40.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:39 smithi181 ceph-mon[47622]: pgmap v435: 129 pgs: 129 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 381 op/s 2024-03-30T01:02:40.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:39 smithi130 ceph-mon[47626]: pgmap v435: 129 pgs: 129 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 381 op/s 2024-03-30T01:02:40.179 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:39 smithi116 ceph-mon[38086]: pgmap v435: 129 pgs: 129 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 381 op/s 2024-03-30T01:02:42.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:42 smithi130 ceph-mon[47626]: pgmap v436: 129 pgs: 129 active+clean; 1.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 10 MiB/s wr, 552 op/s 2024-03-30T01:02:42.690 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:42 smithi116 ceph-mon[38086]: pgmap v436: 129 pgs: 129 active+clean; 1.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 10 MiB/s wr, 552 op/s 2024-03-30T01:02:42.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:42 smithi181 ceph-mon[47622]: pgmap v436: 129 pgs: 129 active+clean; 1.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 10 MiB/s wr, 552 op/s 2024-03-30T01:02:44.681 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:44 smithi116 ceph-mon[38086]: pgmap v437: 129 pgs: 129 active+clean; 1.4 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 11 MiB/s wr, 581 op/s 2024-03-30T01:02:44.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:44 smithi181 ceph-mon[47622]: pgmap v437: 129 pgs: 129 active+clean; 1.4 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 11 MiB/s wr, 581 op/s 2024-03-30T01:02:44.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:44 smithi130 ceph-mon[47626]: pgmap v437: 129 pgs: 129 active+clean; 1.4 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 11 MiB/s wr, 581 op/s 2024-03-30T01:02:46.691 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:46 smithi116 ceph-mon[38086]: pgmap v438: 129 pgs: 129 active+clean; 1.4 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 15 MiB/s wr, 786 op/s 2024-03-30T01:02:46.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:46 smithi181 ceph-mon[47622]: pgmap v438: 129 pgs: 129 active+clean; 1.4 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 15 MiB/s wr, 786 op/s 2024-03-30T01:02:46.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:46 smithi130 ceph-mon[47626]: pgmap v438: 129 pgs: 129 active+clean; 1.4 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 15 MiB/s wr, 786 op/s 2024-03-30T01:02:48.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:48 smithi116 ceph-mon[38086]: pgmap v439: 129 pgs: 129 active+clean; 1.4 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 16 MiB/s wr, 815 op/s 2024-03-30T01:02:48.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:48 smithi181 ceph-mon[47622]: pgmap v439: 129 pgs: 129 active+clean; 1.4 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 16 MiB/s wr, 815 op/s 2024-03-30T01:02:48.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:48 smithi130 ceph-mon[47626]: pgmap v439: 129 pgs: 129 active+clean; 1.4 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 16 MiB/s wr, 815 op/s 2024-03-30T01:02:49.062 INFO:tasks.workunit.client.0.smithi116.stdout: 361 405090 682 293200 769 153785 857 2024-03-30T01:02:50.673 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:50 smithi116 ceph-mon[38086]: pgmap v440: 129 pgs: 129 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 17 MiB/s wr, 910 op/s 2024-03-30T01:02:50.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:50 smithi181 ceph-mon[47622]: pgmap v440: 129 pgs: 129 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 17 MiB/s wr, 910 op/s 2024-03-30T01:02:50.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:50 smithi130 ceph-mon[47626]: pgmap v440: 129 pgs: 129 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 17 MiB/s wr, 910 op/s 2024-03-30T01:02:52.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:52 smithi181 ceph-mon[47622]: pgmap v441: 129 pgs: 129 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 17 MiB/s wr, 843 op/s 2024-03-30T01:02:52.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:52 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:02:52.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:52 smithi130 ceph-mon[47626]: pgmap v441: 129 pgs: 129 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 17 MiB/s wr, 843 op/s 2024-03-30T01:02:52.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:52 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:02:52.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:52 smithi116 ceph-mon[38086]: pgmap v441: 129 pgs: 129 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 17 MiB/s wr, 843 op/s 2024-03-30T01:02:52.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:52 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:02:53.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:53 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:53.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:02:53.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:53 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:02:53.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:53 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:53.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:02:53.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:53 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:02:53.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:53 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:53.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:02:53.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:53 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:02:54.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: pgmap v442: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 13 MiB/s wr, 669 op/s 2024-03-30T01:02:54.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:54.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.756 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: pgmap v442: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 13 MiB/s wr, 669 op/s 2024-03-30T01:02:54.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:54.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: pgmap v442: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 13 MiB/s wr, 669 op/s 2024-03-30T01:02:54.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:54.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:54.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: pgmap v443: 129 pgs: 129 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 12 MiB/s wr, 649 op/s 2024-03-30T01:02:56.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.754 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.755 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:56 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:56.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: pgmap v443: 129 pgs: 129 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 12 MiB/s wr, 649 op/s 2024-03-30T01:02:56.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:56 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:56.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: pgmap v443: 129 pgs: 129 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 12 MiB/s wr, 649 op/s 2024-03-30T01:02:56.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.944 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:56.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:56.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:56 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:57.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:57 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:57.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:57 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:57.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:57 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:57.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:57 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:57.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:57 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:57.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:57 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:57.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:57 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:57.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:57 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:02:57.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:57 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:57.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:57 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:57.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:57 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:57.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:57 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:57.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:57 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:57.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:57 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:57.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:57 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:57.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:57 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:57.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:57 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:02:57.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:57 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:57.977 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:57 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:57.977 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:57 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:57.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:57 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:57.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:57 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:57.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:57 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:57.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:57 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:57.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:57 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:57.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:57 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:02:57.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:57 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:58.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:58 smithi130 ceph-mon[47626]: pgmap v444: 129 pgs: 129 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.1 MiB/s wr, 448 op/s 2024-03-30T01:02:58.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:58 smithi116 ceph-mon[38086]: pgmap v444: 129 pgs: 129 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.1 MiB/s wr, 448 op/s 2024-03-30T01:02:59.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:58 smithi181 ceph-mon[47622]: pgmap v444: 129 pgs: 129 active+clean; 1.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.1 MiB/s wr, 448 op/s 2024-03-30T01:02:59.057 INFO:tasks.workunit.client.0.smithi116.stdout: 381 521730 1305 378254 1187 193640 1374 2024-03-30T01:02:59.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:02:59 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:02:59.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:02:59 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:03:00.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:02:59 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:03:00.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:00 smithi130 ceph-mon[47626]: pgmap v445: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.3 MiB/s wr, 438 op/s 2024-03-30T01:03:00.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:00 smithi116 ceph-mon[38086]: pgmap v445: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.3 MiB/s wr, 438 op/s 2024-03-30T01:03:01.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:00 smithi181 ceph-mon[47622]: pgmap v445: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.3 MiB/s wr, 438 op/s 2024-03-30T01:03:02.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:02 smithi130 ceph-mon[47626]: pgmap v446: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 345 op/s 2024-03-30T01:03:02.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:02 smithi116 ceph-mon[38086]: pgmap v446: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 345 op/s 2024-03-30T01:03:03.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:02 smithi181 ceph-mon[47622]: pgmap v446: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 345 op/s 2024-03-30T01:03:04.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:04 smithi130 ceph-mon[47626]: pgmap v447: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 285 op/s 2024-03-30T01:03:04.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:04 smithi116 ceph-mon[38086]: pgmap v447: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 285 op/s 2024-03-30T01:03:05.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:04 smithi181 ceph-mon[47622]: pgmap v447: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 285 op/s 2024-03-30T01:03:06.680 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T01:03:06.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:06 smithi130 ceph-mon[47626]: pgmap v448: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 358 op/s 2024-03-30T01:03:06.973 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:06 smithi116 ceph-mon[38086]: pgmap v448: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 358 op/s 2024-03-30T01:03:07.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:06 smithi181 ceph-mon[47622]: pgmap v448: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 358 op/s 2024-03-30T01:03:07.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:07 smithi116 ceph-mon[38086]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:03:07.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:07 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:03:08.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:07 smithi181 ceph-mon[47622]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:03:08.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:07 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:03:08.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:07 smithi130 ceph-mon[47626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:03:08.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:07 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:03:08.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:08 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:03:08.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:08 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:03:08.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:08 smithi116 ceph-mon[38086]: pgmap v449: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 318 op/s 2024-03-30T01:03:09.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:08 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:03:09.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:08 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:03:09.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:08 smithi181 ceph-mon[47622]: pgmap v449: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 318 op/s 2024-03-30T01:03:09.054 INFO:tasks.workunit.client.0.smithi116.stdout: 397 537229 1011 387507 879 200449 1634 2024-03-30T01:03:09.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:08 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:03:09.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:08 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:03:09.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:08 smithi130 ceph-mon[47626]: pgmap v449: 129 pgs: 129 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 318 op/s 2024-03-30T01:03:10.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:10 smithi116 ceph-mon[38086]: pgmap v450: 129 pgs: 129 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 430 op/s 2024-03-30T01:03:11.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:10 smithi181 ceph-mon[47622]: pgmap v450: 129 pgs: 129 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 430 op/s 2024-03-30T01:03:11.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:10 smithi130 ceph-mon[47626]: pgmap v450: 129 pgs: 129 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 430 op/s 2024-03-30T01:03:12.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:12 smithi116 ceph-mon[38086]: pgmap v451: 129 pgs: 129 active+clean; 1.6 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 10 MiB/s wr, 624 op/s 2024-03-30T01:03:13.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:12 smithi181 ceph-mon[47622]: pgmap v451: 129 pgs: 129 active+clean; 1.6 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 10 MiB/s wr, 624 op/s 2024-03-30T01:03:13.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:12 smithi130 ceph-mon[47626]: pgmap v451: 129 pgs: 129 active+clean; 1.6 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 10 MiB/s wr, 624 op/s 2024-03-30T01:03:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.136+0000 7f309de00700 1 -- 172.21.15.116:0/3727318812 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f309810ce90 msgr2=0x7f3098117cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.136+0000 7f309de00700 1 --2- 172.21.15.116:0/3727318812 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f309810ce90 0x7f3098117cf0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f309001c0d0 tx=0x7f309001c3e0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.160+0000 7f309de00700 1 -- 172.21.15.116:0/3727318812 shutdown_connections 2024-03-30T01:03:14.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.160+0000 7f309de00700 1 --2- 172.21.15.116:0/3727318812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3098072bb0 0x7f3098109dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.160+0000 7f309de00700 1 --2- 172.21.15.116:0/3727318812 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f309810ce90 0x7f3098117cf0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.160+0000 7f309de00700 1 --2- 172.21.15.116:0/3727318812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f309810a390 0x7f309810c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.160+0000 7f309de00700 1 -- 172.21.15.116:0/3727318812 >> 172.21.15.116:0/3727318812 conn(0x7f309806dc60 msgr2=0x7f30980700a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:14.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.167+0000 7f309de00700 1 -- 172.21.15.116:0/3727318812 shutdown_connections 2024-03-30T01:03:14.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.170+0000 7f309de00700 1 -- 172.21.15.116:0/3727318812 wait complete. 2024-03-30T01:03:14.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.173+0000 7f309de00700 1 Processor -- start 2024-03-30T01:03:14.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.189+0000 7f309de00700 1 -- start start 2024-03-30T01:03:14.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.198+0000 7f309de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3098072bb0 0x7f309819d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:14.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.198+0000 7f309de00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f309810a390 0x7f309819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:14.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.198+0000 7f309de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f309810ce90 0x7f3098079010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:14.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.198+0000 7f309de00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3098079610 con 0x7f3098072bb0 2024-03-30T01:03:14.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.198+0000 7f309de00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3098079780 con 0x7f309810ce90 2024-03-30T01:03:14.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.198+0000 7f309de00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30980798f0 con 0x7f309810a390 2024-03-30T01:03:14.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.201+0000 7f3096a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3098072bb0 0x7f309819d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:14.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.201+0000 7f3096a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3098072bb0 0x7f309819d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40914/0 (socket says 172.21.15.116:40914) 2024-03-30T01:03:14.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.201+0000 7f3096a00700 1 -- 172.21.15.116:0/2353590895 learned_addr learned my addr 172.21.15.116:0/2353590895 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:14.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.202+0000 7f3096a00700 1 -- 172.21.15.116:0/2353590895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f309810a390 msgr2=0x7f309819de10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T01:03:14.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.202+0000 7f3096a00700 1 --2- 172.21.15.116:0/2353590895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f309810a390 0x7f309819de10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.202+0000 7f3096a00700 1 -- 172.21.15.116:0/2353590895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f309810ce90 msgr2=0x7f3098079010 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T01:03:14.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.202+0000 7f3096a00700 1 --2- 172.21.15.116:0/2353590895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f309810ce90 0x7f3098079010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.202+0000 7f3096a00700 1 -- 172.21.15.116:0/2353590895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309001ad10 con 0x7f3098072bb0 2024-03-30T01:03:14.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.202+0000 7f3096a00700 1 --2- 172.21.15.116:0/2353590895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3098072bb0 0x7f309819d8d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f308800dac0 tx=0x7f308800de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:14.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.205+0000 7f309ca00700 1 -- 172.21.15.116:0/2353590895 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088009990 con 0x7f3098072bb0 2024-03-30T01:03:14.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.213+0000 7f309de00700 1 -- 172.21.15.116:0/2353590895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3098079bd0 con 0x7f3098072bb0 2024-03-30T01:03:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.213+0000 7f309de00700 1 -- 172.21.15.116:0/2353590895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309807a130 con 0x7f3098072bb0 2024-03-30T01:03:14.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.233+0000 7f309ca00700 1 -- 172.21.15.116:0/2353590895 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3088010460 con 0x7f3098072bb0 2024-03-30T01:03:14.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.234+0000 7f309ca00700 1 -- 172.21.15.116:0/2353590895 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308800f620 con 0x7f3098072bb0 2024-03-30T01:03:14.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.234+0000 7f309ca00700 1 -- 172.21.15.116:0/2353590895 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f308800f800 con 0x7f3098072bb0 2024-03-30T01:03:14.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.236+0000 7f309ca00700 1 --2- 172.21.15.116:0/2353590895 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f307c0a12f0 0x7f307c0a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:14.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.237+0000 7f309ca00700 1 -- 172.21.15.116:0/2353590895 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f30880cda10 con 0x7f3098072bb0 2024-03-30T01:03:14.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.237+0000 7f3096000700 1 --2- 172.21.15.116:0/2353590895 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f307c0a12f0 0x7f307c0a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:14.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.238+0000 7f3096000700 1 --2- 172.21.15.116:0/2353590895 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f307c0a12f0 0x7f307c0a37b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f309819e780 tx=0x7f308400b3f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.255+0000 7f30b1894b80 1 Processor -- start 2024-03-30T01:03:14.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.269+0000 7f30b1894b80 1 -- start start 2024-03-30T01:03:14.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.269+0000 7f30b1894b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7f99e0 0x55e86c7f9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:14.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.269+0000 7f30b1894b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fa340 0x55e86c7fabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:14.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.269+0000 7f30b1894b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fb190 0x55e86c7fb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:14.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.270+0000 7f3097400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fb190 0x55e86c7fb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:14.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.270+0000 7f3097400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fb190 0x55e86c7fb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:53830/0 (socket says 172.21.15.116:53830) 2024-03-30T01:03:14.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.270+0000 7f3097400700 1 -- 172.21.15.116:0/720848880 learned_addr learned my addr 172.21.15.116:0/720848880 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:14.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.271+0000 7f30b1894b80 1 -- 172.21.15.116:0/720848880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86c751990 con 0x55e86c7f99e0 2024-03-30T01:03:14.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.271+0000 7f30b1894b80 1 -- 172.21.15.116:0/720848880 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86c74f350 con 0x55e86c7fa340 2024-03-30T01:03:14.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.271+0000 7f30b1894b80 1 -- 172.21.15.116:0/720848880 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86c74ac40 con 0x55e86c7fb190 2024-03-30T01:03:14.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.273+0000 7f3097400700 1 -- 172.21.15.116:0/720848880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fa340 msgr2=0x55e86c7fabb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:03:14.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.273+0000 7f3097400700 1 --2- 172.21.15.116:0/720848880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fa340 0x55e86c7fabb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.273+0000 7f3097400700 1 -- 172.21.15.116:0/720848880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7f99e0 msgr2=0x55e86c7f9e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:03:14.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.273+0000 7f3097400700 1 --2- 172.21.15.116:0/720848880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7f99e0 0x55e86c7f9e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.273+0000 7f3097400700 1 -- 172.21.15.116:0/720848880 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e86c5cc3b0 con 0x55e86c7fb190 2024-03-30T01:03:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.274+0000 7f3097400700 1 --2- 172.21.15.116:0/720848880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fb190 0x55e86c7fb610 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f309001a860 tx=0x7f309001ac50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4da6a698b9985b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:14.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.287+0000 7f3083e00700 1 -- 172.21.15.116:0/720848880 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309001f040 con 0x55e86c7fb190 2024-03-30T01:03:14.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.289+0000 7f3083e00700 1 -- 172.21.15.116:0/720848880 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3090019e80 con 0x55e86c7fb190 2024-03-30T01:03:14.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.289+0000 7f3083e00700 1 -- 172.21.15.116:0/720848880 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309002c6c0 con 0x55e86c7fb190 2024-03-30T01:03:14.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.310+0000 7f30b1894b80 1 -- 172.21.15.116:0/720848880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fb190 msgr2=0x55e86c7fb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:14.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.310+0000 7f30b1894b80 1 --2- 172.21.15.116:0/720848880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fb190 0x55e86c7fb610 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f309001a860 tx=0x7f309001ac50 comp rx=0 tx=0).stop 2024-03-30T01:03:14.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.341+0000 7f30b1894b80 1 -- 172.21.15.116:0/720848880 shutdown_connections 2024-03-30T01:03:14.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.341+0000 7f30b1894b80 1 --2- 172.21.15.116:0/720848880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fb190 0x55e86c7fb610 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.341+0000 7f30b1894b80 1 --2- 172.21.15.116:0/720848880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7f99e0 0x55e86c7f9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.341+0000 7f30b1894b80 1 --2- 172.21.15.116:0/720848880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fa340 0x55e86c7fabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.341+0000 7f30b1894b80 1 -- 172.21.15.116:0/720848880 >> 172.21.15.116:0/720848880 conn(0x55e86c7586a0 msgr2=0x55e86c7f4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:14.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.349+0000 7f30b1894b80 1 -- 172.21.15.116:0/720848880 shutdown_connections 2024-03-30T01:03:14.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.349+0000 7f30b1894b80 1 -- 172.21.15.116:0/720848880 wait complete. 2024-03-30T01:03:14.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.351+0000 7f30b1894b80 1 -- start start 2024-03-30T01:03:14.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.351+0000 7f30b1894b80 1 Processor -- start 2024-03-30T01:03:14.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.364+0000 7f30b1894b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7f99e0 0x55e86c803940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.385+0000 7f30b1894b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fa340 0x55e86c803e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.385+0000 7f30b1894b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb190 0x55e86c80aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.388+0000 7f3096a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7f99e0 0x55e86c803940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.388+0000 7f3096a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7f99e0 0x55e86c803940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:53836/0 (socket says 172.21.15.116:53836) 2024-03-30T01:03:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.389+0000 7f3096000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fa340 0x55e86c803e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.389+0000 7f3096000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fa340 0x55e86c803e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40920/0 (socket says 172.21.15.116:40920) 2024-03-30T01:03:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.390+0000 7f3096a00700 1 -- 172.21.15.116:0/2279734087 learned_addr learned my addr 172.21.15.116:0/2279734087 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:14.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.390+0000 7f30b1894b80 1 -- 172.21.15.116:0/2279734087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86c6e6d80 con 0x55e86c7fa340 2024-03-30T01:03:14.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.390+0000 7f30b1894b80 1 -- 172.21.15.116:0/2279734087 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86c6e6a20 con 0x55e86c7fb190 2024-03-30T01:03:14.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.390+0000 7f30b1894b80 1 -- 172.21.15.116:0/2279734087 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86c5ce6a0 con 0x55e86c7f99e0 2024-03-30T01:03:14.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.391+0000 7f3097400700 1 --2- 172.21.15.116:0/2279734087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb190 0x55e86c80aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:14.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.394+0000 7f30b1894b80 10 client.0 _ll_register_callbacks cb 0x55e86c6f0470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:03:14.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.395+0000 7f3096a00700 1 -- 172.21.15.116:0/2279734087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb190 msgr2=0x55e86c80aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:14.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.396+0000 7f3096a00700 1 --2- 172.21.15.116:0/2279734087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb190 0x55e86c80aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.396+0000 7f3096a00700 1 -- 172.21.15.116:0/2279734087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fa340 msgr2=0x55e86c803e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:14.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.396+0000 7f3096a00700 1 --2- 172.21.15.116:0/2279734087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fa340 0x55e86c803e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.396+0000 7f3096a00700 1 -- 172.21.15.116:0/2279734087 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3098118750 con 0x55e86c7f99e0 2024-03-30T01:03:14.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.398+0000 7f3096a00700 1 --2- 172.21.15.116:0/2279734087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7f99e0 0x55e86c803940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f30880975e0 tx=0x7f308809b360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0979bc4008b04a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:14.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.411+0000 7f3080c00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:03:14.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.411+0000 7f3080c00700 1 -- 172.21.15.116:0/2279734087 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308809afd0 con 0x55e86c7f99e0 2024-03-30T01:03:14.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.411+0000 7f3080c00700 1 -- 172.21.15.116:0/2279734087 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f30880a2ff0 con 0x55e86c7f99e0 2024-03-30T01:03:14.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.411+0000 7f30b1894b80 1 -- 172.21.15.116:0/2279734087 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e86c70b380 con 0x55e86c7f99e0 2024-03-30T01:03:14.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.411+0000 7f3080c00700 1 -- 172.21.15.116:0/2279734087 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f308809a060 con 0x55e86c7f99e0 2024-03-30T01:03:14.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.411+0000 7f30b1894b80 10 client.26083 fetch_fsmap learned FSMap version 28 2024-03-30T01:03:14.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.412+0000 7f30b1894b80 1 -- 172.21.15.116:0/2279734087 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e86c578b60 con 0x55e86c7f99e0 2024-03-30T01:03:14.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.412+0000 7f3080c00700 1 -- 172.21.15.116:0/2279734087 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30880b2070 con 0x55e86c7f99e0 2024-03-30T01:03:14.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.416+0000 7f3080c00700 1 -- 172.21.15.116:0/2279734087 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f308809bb80 con 0x55e86c7f99e0 2024-03-30T01:03:14.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.419+0000 7f30b1894b80 10 client.26083 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:03:14.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.419+0000 7f30b1894b80 10 client.26083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:03:14.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.419+0000 7f30b1894b80 20 client.26083 populate_metadata read hostname 'smithi116' 2024-03-30T01:03:14.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.419+0000 7f30b1894b80 1 --2- 172.21.15.116:0/2279734087 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e86c80f740 0x55e86c811b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:14.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.419+0000 7f30b1894b80 4 client.26083 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:03:14.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.419+0000 7f30b1894b80 1 -- 172.21.15.116:0/2279734087 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e86c743700 con 0x55e86c80f740 2024-03-30T01:03:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.420+0000 7f3096000700 1 --2- 172.21.15.116:0/2279734087 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e86c80f740 0x55e86c811b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.440+0000 7f3096000700 1 --2- 172.21.15.116:0/2279734087 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e86c80f740 0x55e86c811b60 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0af18e9ec605830 server_cookie=4f7ac0d6ca2b1b46 in_seq=0 out_seq=0 2024-03-30T01:03:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.440+0000 7f3080c00700 10 client.26083 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:03:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.441+0000 7f3080c00700 1 -- 172.21.15.116:0/2279734087 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e86c743700 con 0x55e86c80f740 2024-03-30T01:03:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.441+0000 7f3080c00700 10 client.26083 handle_command_reply: tid=0 2024-03-30T01:03:14.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.451+0000 7f30b1894b80 1 client.26083 shutdown 2024-03-30T01:03:14.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.460+0000 7f30b1894b80 1 -- 172.21.15.116:0/2279734087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7f99e0 msgr2=0x55e86c803940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:14.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.460+0000 7f30b1894b80 1 --2- 172.21.15.116:0/2279734087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7f99e0 0x55e86c803940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f30880975e0 tx=0x7f308809b360 comp rx=0 tx=0).stop 2024-03-30T01:03:14.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.464+0000 7f30b1894b80 1 -- 172.21.15.116:0/2279734087 shutdown_connections 2024-03-30T01:03:14.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.464+0000 7f30b1894b80 1 --2- 172.21.15.116:0/2279734087 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e86c80f740 0x55e86c811b60 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.464+0000 7f30b1894b80 1 --2- 172.21.15.116:0/2279734087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7f99e0 0x55e86c803940 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f30880975e0 tx=0x7f308809b360 comp rx=0 tx=0).stop 2024-03-30T01:03:14.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.464+0000 7f30b1894b80 1 --2- 172.21.15.116:0/2279734087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb190 0x55e86c80aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.464+0000 7f30b1894b80 1 --2- 172.21.15.116:0/2279734087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fa340 0x55e86c803e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:14.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.464+0000 7f30b1894b80 1 -- 172.21.15.116:0/2279734087 >> 172.21.15.116:0/2279734087 conn(0x55e86c7586a0 msgr2=0x55e86c7f83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:14.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.464+0000 7f3080c00700 0 client.26083 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:03:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.485+0000 7f30b1894b80 1 -- 172.21.15.116:0/2279734087 shutdown_connections 2024-03-30T01:03:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.487+0000 7f30b1894b80 1 -- 172.21.15.116:0/2279734087 wait complete. 2024-03-30T01:03:14.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:14.489+0000 7f30b1894b80 20 client.26083 trim_cache size 0 max 16384 2024-03-30T01:03:14.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:14 smithi116 ceph-mon[38086]: pgmap v452: 129 pgs: 129 active+clean; 1.6 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 615 op/s 2024-03-30T01:03:15.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:14 smithi181 ceph-mon[47622]: pgmap v452: 129 pgs: 129 active+clean; 1.6 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 615 op/s 2024-03-30T01:03:15.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:14 smithi130 ceph-mon[47626]: pgmap v452: 129 pgs: 129 active+clean; 1.6 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 615 op/s 2024-03-30T01:03:15.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.182+0000 7f30b1894b80 1 Processor -- start 2024-03-30T01:03:15.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.196+0000 7f30b1894b80 1 -- start start 2024-03-30T01:03:15.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.197+0000 7f30b1894b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fa340 0x55e86c802550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:15.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.201+0000 7f3096a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fa340 0x55e86c802550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:15.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.201+0000 7f3096a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fa340 0x55e86c802550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45474/0 (socket says 172.21.15.116:45474) 2024-03-30T01:03:15.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.201+0000 7f30b1894b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fb190 0x55e86c80ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.202+0000 7f30b1894b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb570 0x55e86c802ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.206+0000 7f30b1894b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86c80c050 con 0x55e86c7fa340 2024-03-30T01:03:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.206+0000 7f30b1894b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86c802340 con 0x55e86c7fb570 2024-03-30T01:03:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.206+0000 7f30b1894b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86c7f56c0 con 0x55e86c7fb190 2024-03-30T01:03:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.206+0000 7f3096000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fb190 0x55e86c80ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.207+0000 7f3096000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fb190 0x55e86c80ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59782/0 (socket says 172.21.15.116:59782) 2024-03-30T01:03:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.207+0000 7f3096000700 1 -- 172.21.15.116:0/80310854 learned_addr learned my addr 172.21.15.116:0/80310854 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.208+0000 7f3097400700 1 --2- 172.21.15.116:0/80310854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb570 0x55e86c802ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.213+0000 7f3096a00700 1 -- 172.21.15.116:0/80310854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fb190 msgr2=0x55e86c80ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.213+0000 7f3096a00700 1 --2- 172.21.15.116:0/80310854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fb190 0x55e86c80ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.213+0000 7f3096a00700 1 -- 172.21.15.116:0/80310854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb570 msgr2=0x55e86c802ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:15.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.213+0000 7f3096a00700 1 --2- 172.21.15.116:0/80310854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb570 0x55e86c802ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.213+0000 7f3096a00700 1 -- 172.21.15.116:0/80310854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e86c710ea0 con 0x55e86c7fa340 2024-03-30T01:03:15.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.214+0000 7f3096a00700 1 --2- 172.21.15.116:0/80310854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fa340 0x55e86c802550 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3088014000 tx=0x7f3088096eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23af3c7d7c9c5cae server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.224+0000 7f308d200700 1 -- 172.21.15.116:0/80310854 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308809a940 con 0x55e86c7fa340 2024-03-30T01:03:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.224+0000 7f308d200700 1 -- 172.21.15.116:0/80310854 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30880b2070 con 0x55e86c7fa340 2024-03-30T01:03:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.224+0000 7f308d200700 1 -- 172.21.15.116:0/80310854 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308809a060 con 0x55e86c7fa340 2024-03-30T01:03:15.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.236+0000 7f30b1894b80 1 -- 172.21.15.116:0/80310854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fa340 msgr2=0x55e86c802550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:15.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.236+0000 7f30b1894b80 1 --2- 172.21.15.116:0/80310854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fa340 0x55e86c802550 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3088014000 tx=0x7f3088096eb0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.262+0000 7f30b1894b80 1 -- 172.21.15.116:0/80310854 shutdown_connections 2024-03-30T01:03:15.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.262+0000 7f30b1894b80 1 --2- 172.21.15.116:0/80310854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb570 0x55e86c802ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.262+0000 7f30b1894b80 1 --2- 172.21.15.116:0/80310854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fa340 0x55e86c802550 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.262+0000 7f30b1894b80 1 --2- 172.21.15.116:0/80310854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fb190 0x55e86c80ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.262+0000 7f30b1894b80 1 -- 172.21.15.116:0/80310854 >> 172.21.15.116:0/80310854 conn(0x55e86c7586a0 msgr2=0x55e86c8164c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.262+0000 7f30b1894b80 1 -- 172.21.15.116:0/80310854 shutdown_connections 2024-03-30T01:03:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.262+0000 7f30b1894b80 1 -- 172.21.15.116:0/80310854 wait complete. 2024-03-30T01:03:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.263+0000 7f30b1894b80 1 -- start start 2024-03-30T01:03:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.264+0000 7f30b1894b80 1 Processor -- start 2024-03-30T01:03:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.265+0000 7f30b1894b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fa340 0x55e86c80a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.265+0000 7f30b1894b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb190 0x55e86c80aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.265+0000 7f3096a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fa340 0x55e86c80a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.265+0000 7f3096a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fa340 0x55e86c80a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59786/0 (socket says 172.21.15.116:59786) 2024-03-30T01:03:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.266+0000 7f3096000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb190 0x55e86c80aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.266+0000 7f3096000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb190 0x55e86c80aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:47730/0 (socket says 172.21.15.116:47730) 2024-03-30T01:03:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.266+0000 7f3096000700 1 -- 172.21.15.116:0/432111573 learned_addr learned my addr 172.21.15.116:0/432111573 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.266+0000 7f30b1894b80 1 --2- 172.21.15.116:0/432111573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fb570 0x55e86c8091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.266+0000 7f3097400700 1 --2- 172.21.15.116:0/432111573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fb570 0x55e86c8091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.269+0000 7f30b1894b80 1 -- 172.21.15.116:0/432111573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86c8098b0 con 0x55e86c7fb570 2024-03-30T01:03:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.269+0000 7f30b1894b80 1 -- 172.21.15.116:0/432111573 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86c809a20 con 0x55e86c7fb190 2024-03-30T01:03:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.269+0000 7f30b1894b80 1 -- 172.21.15.116:0/432111573 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e86c809be0 con 0x55e86c7fa340 2024-03-30T01:03:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.269+0000 7f30b1894b80 10 client.0 _ll_register_callbacks cb 0x55e86c6f0470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:03:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.270+0000 7f3096a00700 1 -- 172.21.15.116:0/432111573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb190 msgr2=0x55e86c80aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.270+0000 7f3096a00700 1 --2- 172.21.15.116:0/432111573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb190 0x55e86c80aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.270+0000 7f3096a00700 1 -- 172.21.15.116:0/432111573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fb570 msgr2=0x55e86c8091d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.270+0000 7f3096a00700 1 --2- 172.21.15.116:0/432111573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fb570 0x55e86c8091d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.270+0000 7f3096a00700 1 -- 172.21.15.116:0/432111573 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e86c578b60 con 0x55e86c7fa340 2024-03-30T01:03:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.270+0000 7f3097400700 1 --2- 172.21.15.116:0/432111573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fb570 0x55e86c8091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:03:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.271+0000 7f3096a00700 1 --2- 172.21.15.116:0/432111573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fa340 0x55e86c80a9b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55e86c7fa840 tx=0x7f308809bb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3323a3fe0560971 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:15.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.279+0000 7f3080c00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:03:15.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.279+0000 7f3080c00700 1 -- 172.21.15.116:0/432111573 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308809b270 con 0x55e86c7fa340 2024-03-30T01:03:15.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.279+0000 7f3080c00700 1 -- 172.21.15.116:0/432111573 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f30880a2b60 con 0x55e86c7fa340 2024-03-30T01:03:15.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.281+0000 7f30b1894b80 1 -- 172.21.15.116:0/432111573 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3090019e80 con 0x55e86c7fa340 2024-03-30T01:03:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.288+0000 7f3080c00700 1 -- 172.21.15.116:0/432111573 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30880a2d90 con 0x55e86c7fa340 2024-03-30T01:03:15.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.305+0000 7f30b1894b80 10 client.26089 fetch_fsmap learned FSMap version 28 2024-03-30T01:03:15.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.305+0000 7f30b1894b80 1 -- 172.21.15.116:0/432111573 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e86c5d0cd0 con 0x55e86c7fa340 2024-03-30T01:03:15.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.307+0000 7f3080c00700 1 -- 172.21.15.116:0/432111573 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3090019e80 con 0x55e86c7fa340 2024-03-30T01:03:15.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.312+0000 7f3080c00700 1 -- 172.21.15.116:0/432111573 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f30880969f0 con 0x55e86c7fa340 2024-03-30T01:03:15.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.313+0000 7f30b1894b80 10 client.26089 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:03:15.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.313+0000 7f30b1894b80 10 client.26089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:03:15.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.313+0000 7f30b1894b80 20 client.26089 populate_metadata read hostname 'smithi116' 2024-03-30T01:03:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.313+0000 7f30b1894b80 1 --2- 172.21.15.116:0/432111573 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e86c864460 0x55e86c866850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.313+0000 7f30b1894b80 4 client.26089 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T01:03:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.313+0000 7f30b1894b80 1 -- 172.21.15.116:0/432111573 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e86c599cf0 con 0x55e86c864460 2024-03-30T01:03:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.313+0000 7f3096000700 1 --2- 172.21.15.116:0/432111573 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e86c864460 0x55e86c866850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:15.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.321+0000 7f3096000700 1 --2- 172.21.15.116:0/432111573 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e86c864460 0x55e86c866850 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19b47d6d4a5ba62c server_cookie=cfcc616126ec2ead in_seq=0 out_seq=0 2024-03-30T01:03:15.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.329+0000 7f3080c00700 10 client.26089 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:03:15.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.330+0000 7f3080c00700 1 -- 172.21.15.116:0/432111573 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e86c599cf0 con 0x55e86c864460 2024-03-30T01:03:15.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.330+0000 7f3080c00700 10 client.26089 handle_command_reply: tid=0 2024-03-30T01:03:15.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.330+0000 7f30b1894b80 1 client.26089 shutdown 2024-03-30T01:03:15.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.352+0000 7f30b1894b80 1 -- 172.21.15.116:0/432111573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fa340 msgr2=0x55e86c80a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:15.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.352+0000 7f30b1894b80 1 --2- 172.21.15.116:0/432111573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fa340 0x55e86c80a9b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55e86c7fa840 tx=0x7f308809bb80 comp rx=0 tx=0).stop 2024-03-30T01:03:15.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.352+0000 7f30b1894b80 1 -- 172.21.15.116:0/432111573 shutdown_connections 2024-03-30T01:03:15.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.352+0000 7f30b1894b80 1 --2- 172.21.15.116:0/432111573 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e86c864460 0x55e86c866850 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.352+0000 7f30b1894b80 1 --2- 172.21.15.116:0/432111573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e86c7fa340 0x55e86c80a9b0 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55e86c7fa840 tx=0x7f308809bb80 comp rx=0 tx=0).stop 2024-03-30T01:03:15.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.353+0000 7f30b1894b80 1 --2- 172.21.15.116:0/432111573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e86c7fb570 0x55e86c8091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.353+0000 7f30b1894b80 1 --2- 172.21.15.116:0/432111573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e86c7fb190 0x55e86c80aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.353+0000 7f30b1894b80 1 -- 172.21.15.116:0/432111573 >> 172.21.15.116:0/432111573 conn(0x55e86c7586a0 msgr2=0x55e86c7ed7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:15.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.353+0000 7f3080c00700 0 client.26089 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:03:15.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.354+0000 7f30b1894b80 1 -- 172.21.15.116:0/432111573 shutdown_connections 2024-03-30T01:03:15.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.354+0000 7f30b1894b80 1 -- 172.21.15.116:0/432111573 wait complete. 2024-03-30T01:03:15.400 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:03:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-30T01:03:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-30T01:03:15.400 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:03:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.390+0000 7f30b1894b80 20 client.26089 trim_cache size 0 max 16384 2024-03-30T01:03:15.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.413+0000 7f309de00700 1 -- 172.21.15.116:0/2353590895 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f307c0a12f0 msgr2=0x7f307c0a37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:15.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.413+0000 7f309de00700 1 --2- 172.21.15.116:0/2353590895 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f307c0a12f0 0x7f307c0a37b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f309819e780 tx=0x7f308400b3f0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.413+0000 7f309de00700 1 -- 172.21.15.116:0/2353590895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3098072bb0 msgr2=0x7f309819d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:15.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.413+0000 7f309de00700 1 --2- 172.21.15.116:0/2353590895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3098072bb0 0x7f309819d8d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f308800dac0 tx=0x7f308800de80 comp rx=0 tx=0).stop 2024-03-30T01:03:15.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.419+0000 7f309de00700 1 -- 172.21.15.116:0/2353590895 shutdown_connections 2024-03-30T01:03:15.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.419+0000 7f309de00700 1 --2- 172.21.15.116:0/2353590895 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f307c0a12f0 0x7f307c0a37b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.419+0000 7f309de00700 1 --2- 172.21.15.116:0/2353590895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f309810ce90 0x7f3098079010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.419+0000 7f309de00700 1 --2- 172.21.15.116:0/2353590895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3098072bb0 0x7f309819d8d0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.419+0000 7f309de00700 1 --2- 172.21.15.116:0/2353590895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f309810a390 0x7f309819de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:15.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.419+0000 7f309de00700 1 -- 172.21.15.116:0/2353590895 >> 172.21.15.116:0/2353590895 conn(0x7f309806dc60 msgr2=0x7f3098109170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:15.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.474+0000 7f309de00700 1 -- 172.21.15.116:0/2353590895 shutdown_connections 2024-03-30T01:03:15.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:15.499+0000 7f309de00700 1 -- 172.21.15.116:0/2353590895 wait complete. 2024-03-30T01:03:16.089 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T01:03:16.089 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T01:03:16.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:16 smithi116 ceph-mon[38086]: pgmap v453: 129 pgs: 129 active+clean; 1.6 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 13 MiB/s wr, 778 op/s 2024-03-30T01:03:17.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:16 smithi181 ceph-mon[47622]: pgmap v453: 129 pgs: 129 active+clean; 1.6 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 13 MiB/s wr, 778 op/s 2024-03-30T01:03:17.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:16 smithi130 ceph-mon[47626]: pgmap v453: 129 pgs: 129 active+clean; 1.6 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 13 MiB/s wr, 778 op/s 2024-03-30T01:03:18.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:18 smithi116 ceph-mon[38086]: pgmap v454: 129 pgs: 129 active+clean; 1.7 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 15 MiB/s wr, 779 op/s 2024-03-30T01:03:19.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:18 smithi181 ceph-mon[47622]: pgmap v454: 129 pgs: 129 active+clean; 1.7 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 15 MiB/s wr, 779 op/s 2024-03-30T01:03:19.052 INFO:tasks.workunit.client.0.smithi116.stdout: 403 395280 321 289813 358 151667 1516 2024-03-30T01:03:19.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:18 smithi130 ceph-mon[47626]: pgmap v454: 129 pgs: 129 active+clean; 1.7 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 15 MiB/s wr, 779 op/s 2024-03-30T01:03:20.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:20 smithi116 ceph-mon[38086]: pgmap v455: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 16 MiB/s wr, 853 op/s 2024-03-30T01:03:21.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:20 smithi181 ceph-mon[47622]: pgmap v455: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 16 MiB/s wr, 853 op/s 2024-03-30T01:03:21.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:20 smithi130 ceph-mon[47626]: pgmap v455: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 16 MiB/s wr, 853 op/s 2024-03-30T01:03:22.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:21 smithi181 ceph-mon[47622]: pgmap v456: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 17 MiB/s wr, 859 op/s 2024-03-30T01:03:22.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:21 smithi130 ceph-mon[47626]: pgmap v456: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 17 MiB/s wr, 859 op/s 2024-03-30T01:03:22.184 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:21 smithi116 ceph-mon[38086]: pgmap v456: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 17 MiB/s wr, 859 op/s 2024-03-30T01:03:22.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.740+0000 7f250e200700 1 -- 172.21.15.116:0/4111286621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2508108df0 msgr2=0x7f250810d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:22.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.740+0000 7f250e200700 1 --2- 172.21.15.116:0/4111286621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2508108df0 0x7f250810d690 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f24f4009940 tx=0x7f24f4009c50 comp rx=0 tx=0).stop 2024-03-30T01:03:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.756+0000 7f250e200700 1 -- 172.21.15.116:0/4111286621 shutdown_connections 2024-03-30T01:03:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.756+0000 7f250e200700 1 --2- 172.21.15.116:0/4111286621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2508108df0 0x7f250810d690 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.756+0000 7f250e200700 1 --2- 172.21.15.116:0/4111286621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2508072170 0x7f2508072590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.756+0000 7f250e200700 1 --2- 172.21.15.116:0/4111286621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2508108430 0x7f25081088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.756+0000 7f250e200700 1 -- 172.21.15.116:0/4111286621 >> 172.21.15.116:0/4111286621 conn(0x7f250806dc20 msgr2=0x7f2508070060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.757+0000 7f250e200700 1 -- 172.21.15.116:0/4111286621 shutdown_connections 2024-03-30T01:03:22.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.765+0000 7f250e200700 1 -- 172.21.15.116:0/4111286621 wait complete. 2024-03-30T01:03:22.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.786+0000 7f250e200700 1 Processor -- start 2024-03-30T01:03:22.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.800+0000 7f250e200700 1 -- start start 2024-03-30T01:03:22.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.802+0000 7f250e200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2508072170 0x7f25081183a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:22.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.802+0000 7f250e200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2508108430 0x7f25081188e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:22.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.802+0000 7f250e200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2508108df0 0x7f25081c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:22.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.802+0000 7f250e200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2508119220 con 0x7f2508072170 2024-03-30T01:03:22.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.802+0000 7f250e200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25081c6f90 con 0x7f2508108df0 2024-03-30T01:03:22.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.802+0000 7f250e200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25081c7150 con 0x7f2508108430 2024-03-30T01:03:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.808+0000 7f2507e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2508108430 0x7f25081188e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.808+0000 7f2507e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2508108430 0x7f25081188e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59800/0 (socket says 172.21.15.116:59800) 2024-03-30T01:03:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.808+0000 7f2507e00700 1 -- 172.21.15.116:0/3452288713 learned_addr learned my addr 172.21.15.116:0/3452288713 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.808+0000 7f250d800700 1 --2- 172.21.15.116:0/3452288713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2508108df0 0x7f25081c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.809+0000 7f250ce00700 1 --2- 172.21.15.116:0/3452288713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2508072170 0x7f25081183a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.809+0000 7f250d800700 1 -- 172.21.15.116:0/3452288713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2508108430 msgr2=0x7f25081188e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.809+0000 7f250d800700 1 --2- 172.21.15.116:0/3452288713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2508108430 0x7f25081188e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.809+0000 7f250d800700 1 -- 172.21.15.116:0/3452288713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2508072170 msgr2=0x7f25081183a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:22.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.809+0000 7f250d800700 1 --2- 172.21.15.116:0/3452288713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2508072170 0x7f25081183a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:22.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.809+0000 7f250d800700 1 -- 172.21.15.116:0/3452288713 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f4009520 con 0x7f2508108df0 2024-03-30T01:03:22.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.837+0000 7f250d800700 1 --2- 172.21.15.116:0/3452288713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2508108df0 0x7f25081c6a50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f24f4009860 tx=0x7f24f4005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:22.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.845+0000 7f2505600700 1 -- 172.21.15.116:0/3452288713 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f401c070 con 0x7f2508108df0 2024-03-30T01:03:22.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.845+0000 7f2505600700 1 -- 172.21.15.116:0/3452288713 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24f4005620 con 0x7f2508108df0 2024-03-30T01:03:22.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.845+0000 7f2505600700 1 -- 172.21.15.116:0/3452288713 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f4004e80 con 0x7f2508108df0 2024-03-30T01:03:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.870+0000 7f250e200700 1 -- 172.21.15.116:0/3452288713 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25081c73d0 con 0x7f2508108df0 2024-03-30T01:03:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.870+0000 7f250e200700 1 -- 172.21.15.116:0/3452288713 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25081c77a0 con 0x7f2508108df0 2024-03-30T01:03:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.897+0000 7f250e200700 1 -- 172.21.15.116:0/3452288713 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25081125d0 con 0x7f2508108df0 2024-03-30T01:03:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.930+0000 7f2505600700 1 -- 172.21.15.116:0/3452288713 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f24f400b7d0 con 0x7f2508108df0 2024-03-30T01:03:22.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.950+0000 7f2505600700 1 --2- 172.21.15.116:0/3452288713 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f24f00a1320 0x7f24f00a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:22.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.963+0000 7f250ce00700 1 --2- 172.21.15.116:0/3452288713 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f24f00a1320 0x7f24f00a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:22.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.980+0000 7f250ce00700 1 --2- 172.21.15.116:0/3452288713 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f24f00a1320 0x7f24f00a37e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f250001a5c0 tx=0x7f250001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:22.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.982+0000 7f2505600700 1 -- 172.21.15.116:0/3452288713 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f24f40cdc70 con 0x7f2508108df0 2024-03-30T01:03:22.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:22.983+0000 7f2505600700 1 -- 172.21.15.116:0/3452288713 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f24f40d2260 con 0x7f2508108df0 2024-03-30T01:03:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:24 smithi181 ceph-mon[47622]: pgmap v457: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 12 MiB/s wr, 641 op/s 2024-03-30T01:03:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:24 smithi130 ceph-mon[47626]: pgmap v457: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 12 MiB/s wr, 641 op/s 2024-03-30T01:03:24.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:24 smithi116 ceph-mon[38086]: pgmap v457: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 12 MiB/s wr, 641 op/s 2024-03-30T01:03:26.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:26 smithi181 ceph-mon[47622]: pgmap v458: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 11 MiB/s wr, 665 op/s 2024-03-30T01:03:26.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:26 smithi130 ceph-mon[47626]: pgmap v458: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 11 MiB/s wr, 665 op/s 2024-03-30T01:03:26.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:26 smithi116 ceph-mon[38086]: pgmap v458: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 11 MiB/s wr, 665 op/s 2024-03-30T01:03:28.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:28 smithi181 ceph-mon[47622]: pgmap v459: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.4 MiB/s wr, 558 op/s 2024-03-30T01:03:28.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:28 smithi130 ceph-mon[47626]: pgmap v459: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.4 MiB/s wr, 558 op/s 2024-03-30T01:03:28.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:28 smithi116 ceph-mon[38086]: pgmap v459: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.4 MiB/s wr, 558 op/s 2024-03-30T01:03:29.054 INFO:tasks.workunit.client.0.smithi116.stdout: 423 527331 1155 383860 1259 216511 2459 2024-03-30T01:03:30.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:30 smithi181 ceph-mon[47622]: pgmap v460: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.4 MiB/s wr, 510 op/s 2024-03-30T01:03:30.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:30 smithi130 ceph-mon[47626]: pgmap v460: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.4 MiB/s wr, 510 op/s 2024-03-30T01:03:30.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:30 smithi116 ceph-mon[38086]: pgmap v460: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.4 MiB/s wr, 510 op/s 2024-03-30T01:03:32.753 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:32 smithi181 ceph-mon[47622]: pgmap v461: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.9 MiB/s wr, 470 op/s 2024-03-30T01:03:32.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:32 smithi130 ceph-mon[47626]: pgmap v461: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.9 MiB/s wr, 470 op/s 2024-03-30T01:03:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:32 smithi116 ceph-mon[38086]: pgmap v461: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.9 MiB/s wr, 470 op/s 2024-03-30T01:03:34.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:34.199+0000 7f250e200700 1 -- 172.21.15.116:0/3452288713 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24e4005180 con 0x7f2508108df0 2024-03-30T01:03:34.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:34.227+0000 7f2505600700 1 -- 172.21.15.116:0/3452288713 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10893 (secure 0 0 0) 0x7f24f4097120 con 0x7f2508108df0 2024-03-30T01:03:34.236 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T01:03:34.236 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T01:03:06.538975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T01:03:34.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:34.273+0000 7f250e200700 1 -- 172.21.15.116:0/3452288713 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f24f00a1320 msgr2=0x7f24f00a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:34.273+0000 7f250e200700 1 --2- 172.21.15.116:0/3452288713 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f24f00a1320 0x7f24f00a37e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f250001a5c0 tx=0x7f250001a040 comp rx=0 tx=0).stop 2024-03-30T01:03:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:34.273+0000 7f250e200700 1 -- 172.21.15.116:0/3452288713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2508108df0 msgr2=0x7f25081c6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:34.273+0000 7f250e200700 1 --2- 172.21.15.116:0/3452288713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2508108df0 0x7f25081c6a50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f24f4009860 tx=0x7f24f4005270 comp rx=0 tx=0).stop 2024-03-30T01:03:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:34.274+0000 7f250e200700 1 -- 172.21.15.116:0/3452288713 shutdown_connections 2024-03-30T01:03:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:34.274+0000 7f250e200700 1 --2- 172.21.15.116:0/3452288713 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f24f00a1320 0x7f24f00a37e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:34.274+0000 7f250e200700 1 --2- 172.21.15.116:0/3452288713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2508108df0 0x7f25081c6a50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:34.274+0000 7f250e200700 1 --2- 172.21.15.116:0/3452288713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2508072170 0x7f25081183a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:34.274+0000 7f250e200700 1 --2- 172.21.15.116:0/3452288713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2508108430 0x7f25081188e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:34.274+0000 7f250e200700 1 -- 172.21.15.116:0/3452288713 >> 172.21.15.116:0/3452288713 conn(0x7f250806dc20 msgr2=0x7f250810b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:34.303+0000 7f250e200700 1 -- 172.21.15.116:0/3452288713 shutdown_connections 2024-03-30T01:03:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:34.303+0000 7f250e200700 1 -- 172.21.15.116:0/3452288713 wait complete. 2024-03-30T01:03:34.327 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-03-30T01:03:34.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:34 smithi181 ceph-mon[47622]: pgmap v462: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 352 op/s 2024-03-30T01:03:34.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:34 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3452288713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:03:34.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:34 smithi130 ceph-mon[47626]: pgmap v462: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 352 op/s 2024-03-30T01:03:34.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:34 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3452288713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:03:34.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:34 smithi116 ceph-mon[38086]: pgmap v462: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 352 op/s 2024-03-30T01:03:34.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:34 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3452288713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:03:35.088 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T01:03:36.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:36 smithi181 ceph-mon[47622]: pgmap v463: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 380 op/s 2024-03-30T01:03:36.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:36 smithi130 ceph-mon[47626]: pgmap v463: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 380 op/s 2024-03-30T01:03:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:36 smithi116 ceph-mon[38086]: pgmap v463: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 380 op/s 2024-03-30T01:03:38.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:38 smithi181 ceph-mon[47622]: pgmap v464: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 348 op/s 2024-03-30T01:03:38.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:38 smithi130 ceph-mon[47626]: pgmap v464: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 348 op/s 2024-03-30T01:03:38.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:38 smithi116 ceph-mon[38086]: pgmap v464: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 348 op/s 2024-03-30T01:03:39.056 INFO:tasks.workunit.client.0.smithi116.stdout: 446 534467 1179 388883 1111 223389 2414 2024-03-30T01:03:40.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.640+0000 7f6304c00700 1 -- 172.21.15.116:0/1946169755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f630010a390 msgr2=0x7f630010c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:40.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.640+0000 7f6304c00700 1 --2- 172.21.15.116:0/1946169755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f630010a390 0x7f630010c820 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f62ec01a9a0 tx=0x7f62ec01acb0 comp rx=0 tx=0).stop 2024-03-30T01:03:40.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.685+0000 7f6304c00700 1 -- 172.21.15.116:0/1946169755 shutdown_connections 2024-03-30T01:03:40.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.685+0000 7f6304c00700 1 --2- 172.21.15.116:0/1946169755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6300072b80 0x7f6300109dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:40.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.685+0000 7f6304c00700 1 --2- 172.21.15.116:0/1946169755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f630010ce40 0x7f6300117cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:40.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.685+0000 7f6304c00700 1 --2- 172.21.15.116:0/1946169755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f630010a390 0x7f630010c820 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:40.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.685+0000 7f6304c00700 1 -- 172.21.15.116:0/1946169755 >> 172.21.15.116:0/1946169755 conn(0x7f630006da50 msgr2=0x7f630006fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:40.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.703+0000 7f6304c00700 1 -- 172.21.15.116:0/1946169755 shutdown_connections 2024-03-30T01:03:40.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.713+0000 7f6304c00700 1 -- 172.21.15.116:0/1946169755 wait complete. 2024-03-30T01:03:40.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.714+0000 7f6304c00700 1 Processor -- start 2024-03-30T01:03:40.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.753+0000 7f6304c00700 1 -- start start 2024-03-30T01:03:40.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.753+0000 7f6304c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6300072b80 0x7f630019d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.753+0000 7f6304c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f630010a390 0x7f630019dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.753+0000 7f6304c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630010ce40 0x7f63001a4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.753+0000 7f6304c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63001a4c70 con 0x7f6300072b80 2024-03-30T01:03:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.753+0000 7f6304c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63001a4de0 con 0x7f630010a390 2024-03-30T01:03:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.753+0000 7f6304c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63001a4fa0 con 0x7f630010ce40 2024-03-30T01:03:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.754+0000 7f62ffe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630010ce40 0x7f63001a4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.754+0000 7f62ffe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630010ce40 0x7f63001a4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:48604/0 (socket says 172.21.15.116:48604) 2024-03-30T01:03:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.754+0000 7f62ffe00700 1 -- 172.21.15.116:0/417748641 learned_addr learned my addr 172.21.15.116:0/417748641 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.755+0000 7f62ffe00700 1 -- 172.21.15.116:0/417748641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f630010a390 msgr2=0x7f630019dde0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T01:03:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.755+0000 7f62ffe00700 1 --2- 172.21.15.116:0/417748641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f630010a390 0x7f630019dde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.755+0000 7f62ffe00700 1 -- 172.21.15.116:0/417748641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6300072b80 msgr2=0x7f630019d8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T01:03:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.755+0000 7f62ff400700 1 --2- 172.21.15.116:0/417748641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6300072b80 0x7f630019d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.755+0000 7f62ffe00700 1 --2- 172.21.15.116:0/417748641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6300072b80 0x7f630019d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.755+0000 7f62ffe00700 1 -- 172.21.15.116:0/417748641 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec01a600 con 0x7f630010ce40 2024-03-30T01:03:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.756+0000 7f62ffe00700 1 --2- 172.21.15.116:0/417748641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630010ce40 0x7f63001a4700 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f62f8007ee0 tx=0x7f62f800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.759+0000 7f62f7e00700 1 -- 172.21.15.116:0/417748641 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8017070 con 0x7f630010ce40 2024-03-30T01:03:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.759+0000 7f62f7e00700 1 -- 172.21.15.116:0/417748641 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62f800f040 con 0x7f630010ce40 2024-03-30T01:03:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.759+0000 7f62f7e00700 1 -- 172.21.15.116:0/417748641 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8005740 con 0x7f630010ce40 2024-03-30T01:03:40.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.767+0000 7f6304c00700 1 -- 172.21.15.116:0/417748641 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63001a5280 con 0x7f630010ce40 2024-03-30T01:03:40.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.768+0000 7f6304c00700 1 -- 172.21.15.116:0/417748641 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63001a5710 con 0x7f630010ce40 2024-03-30T01:03:40.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:40 smithi130 ceph-mon[47626]: pgmap v465: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.7 MiB/s wr, 313 op/s 2024-03-30T01:03:40.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.849+0000 7f62f7e00700 1 -- 172.21.15.116:0/417748641 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f62f800dcd0 con 0x7f630010ce40 2024-03-30T01:03:40.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.855+0000 7f62f7e00700 1 --2- 172.21.15.116:0/417748641 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f62e40a11f0 0x7f62e40a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:40.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.861+0000 7f62ff400700 1 --2- 172.21.15.116:0/417748641 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f62e40a11f0 0x7f62e40a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:40.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.863+0000 7f62ff400700 1 --2- 172.21.15.116:0/417748641 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f62e40a11f0 0x7f62e40a36b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f62f0005fd0 tx=0x7f62f0005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:40.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.872+0000 7f62f7e00700 1 -- 172.21.15.116:0/417748641 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f62f80cba60 con 0x7f630010ce40 2024-03-30T01:03:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.900+0000 7f6319d82b80 1 Processor -- start 2024-03-30T01:03:40.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:40 smithi116 ceph-mon[38086]: pgmap v465: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.7 MiB/s wr, 313 op/s 2024-03-30T01:03:40.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.948+0000 7f6319d82b80 1 -- start start 2024-03-30T01:03:40.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.948+0000 7f6319d82b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 0x5609c59a5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:40.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.948+0000 7f6319d82b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a6340 0x5609c59a6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:40.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.948+0000 7f6319d82b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a7190 0x5609c59a7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:40.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.948+0000 7f6319d82b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c58fd990 con 0x5609c59a7190 2024-03-30T01:03:40.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.948+0000 7f6319d82b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c58fb350 con 0x5609c59a6340 2024-03-30T01:03:40.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.948+0000 7f6319d82b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c58f6c40 con 0x5609c59a59e0 2024-03-30T01:03:40.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.949+0000 7f62ffe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a7190 0x5609c59a7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:40.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.949+0000 7f62fea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a6340 0x5609c59a6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:40.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.950+0000 7f62fea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a6340 0x5609c59a6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:39128/0 (socket says 172.21.15.116:39128) 2024-03-30T01:03:40.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.950+0000 7f62fea00700 1 -- 172.21.15.116:0/1855151524 learned_addr learned my addr 172.21.15.116:0/1855151524 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:40.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.950+0000 7f62fea00700 1 -- 172.21.15.116:0/1855151524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 msgr2=0x5609c59a5e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:03:40.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.950+0000 7f62fea00700 1 --2- 172.21.15.116:0/1855151524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 0x5609c59a5e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:40.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.950+0000 7f62fea00700 1 -- 172.21.15.116:0/1855151524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a7190 msgr2=0x5609c59a7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:40.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.950+0000 7f62fea00700 1 --2- 172.21.15.116:0/1855151524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a7190 0x5609c59a7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:40.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.950+0000 7f62fea00700 1 -- 172.21.15.116:0/1855151524 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609c57783b0 con 0x5609c59a6340 2024-03-30T01:03:40.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.951+0000 7f62fea00700 1 --2- 172.21.15.116:0/1855151524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a6340 0x5609c59a6bb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0038a0 tx=0x7f62ec005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ea663f601412ecb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.972+0000 7f62ebe00700 1 -- 172.21.15.116:0/1855151524 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec01c9f0 con 0x5609c59a6340 2024-03-30T01:03:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.973+0000 7f62ebe00700 1 -- 172.21.15.116:0/1855151524 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62ec005030 con 0x5609c59a6340 2024-03-30T01:03:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.973+0000 7f62ebe00700 1 -- 172.21.15.116:0/1855151524 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec004950 con 0x5609c59a6340 2024-03-30T01:03:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.990+0000 7f6319d82b80 1 -- 172.21.15.116:0/1855151524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a6340 msgr2=0x5609c59a6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:40.990+0000 7f6319d82b80 1 --2- 172.21.15.116:0/1855151524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a6340 0x5609c59a6bb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0038a0 tx=0x7f62ec005fd0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:40 smithi181 ceph-mon[47622]: pgmap v465: 129 pgs: 129 active+clean; 1.7 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.7 MiB/s wr, 313 op/s 2024-03-30T01:03:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.003+0000 7f6319d82b80 1 -- 172.21.15.116:0/1855151524 shutdown_connections 2024-03-30T01:03:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.003+0000 7f6319d82b80 1 --2- 172.21.15.116:0/1855151524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 0x5609c59a5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.003+0000 7f6319d82b80 1 --2- 172.21.15.116:0/1855151524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a7190 0x5609c59a7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.003+0000 7f6319d82b80 1 --2- 172.21.15.116:0/1855151524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a6340 0x5609c59a6bb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.003+0000 7f6319d82b80 1 -- 172.21.15.116:0/1855151524 >> 172.21.15.116:0/1855151524 conn(0x5609c59046a0 msgr2=0x5609c59a0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.004+0000 7f6319d82b80 1 -- 172.21.15.116:0/1855151524 shutdown_connections 2024-03-30T01:03:41.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.007+0000 7f6319d82b80 1 -- 172.21.15.116:0/1855151524 wait complete. 2024-03-30T01:03:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.014+0000 7f6319d82b80 1 -- start start 2024-03-30T01:03:41.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.026+0000 7f6319d82b80 1 Processor -- start 2024-03-30T01:03:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.049+0000 7f6319d82b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a59e0 0x5609c59af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.049+0000 7f6319d82b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a6340 0x5609c59afd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.049+0000 7f6319d82b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a7190 0x5609c59b6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.049+0000 7f6319d82b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c5892d80 con 0x5609c59a59e0 2024-03-30T01:03:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.049+0000 7f6319d82b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c5892a20 con 0x5609c59a7190 2024-03-30T01:03:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.049+0000 7f6319d82b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c577a6a0 con 0x5609c59a6340 2024-03-30T01:03:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.049+0000 7f6319d82b80 10 client.0 _ll_register_callbacks cb 0x5609c589c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:03:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.050+0000 7f62ff400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a59e0 0x5609c59af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.050+0000 7f62fea00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a6340 0x5609c59afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.051+0000 7f62fea00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a6340 0x5609c59afd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:48618/0 (socket says 172.21.15.116:48618) 2024-03-30T01:03:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.051+0000 7f62fea00700 1 -- 172.21.15.116:0/3224266904 learned_addr learned my addr 172.21.15.116:0/3224266904 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.051+0000 7f62ffe00700 1 --2- 172.21.15.116:0/3224266904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a7190 0x5609c59b6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:41.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.053+0000 7f62ffe00700 1 -- 172.21.15.116:0/3224266904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a6340 msgr2=0x5609c59afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:41.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.053+0000 7f62ffe00700 1 --2- 172.21.15.116:0/3224266904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a6340 0x5609c59afd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.053+0000 7f62ffe00700 1 -- 172.21.15.116:0/3224266904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a59e0 msgr2=0x5609c59af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:41.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.053+0000 7f62ffe00700 1 --2- 172.21.15.116:0/3224266904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a59e0 0x5609c59af820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.053+0000 7f62ffe00700 1 -- 172.21.15.116:0/3224266904 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6300118720 con 0x5609c59a7190 2024-03-30T01:03:41.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.059+0000 7f62ffe00700 1 --2- 172.21.15.116:0/3224266904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a7190 0x5609c59b6da0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f62f80c9250 tx=0x7f62f8099350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8eed219fa8c2b4c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:41.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.064+0000 7f62e8c00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T01:03:41.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.064+0000 7f62e8c00700 1 -- 172.21.15.116:0/3224266904 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f80a1020 con 0x5609c59a7190 2024-03-30T01:03:41.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.064+0000 7f62e8c00700 1 -- 172.21.15.116:0/3224266904 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f62f80b8040 con 0x5609c59a7190 2024-03-30T01:03:41.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.078+0000 7f6319d82b80 1 -- 172.21.15.116:0/3224266904 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5609c58b7380 con 0x5609c59a7190 2024-03-30T01:03:41.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.098+0000 7f62e8c00700 1 -- 172.21.15.116:0/3224266904 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62f8098090 con 0x5609c59a7190 2024-03-30T01:03:41.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.098+0000 7f6319d82b80 10 client.26102 fetch_fsmap learned FSMap version 28 2024-03-30T01:03:41.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.098+0000 7f6319d82b80 1 -- 172.21.15.116:0/3224266904 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5609c5724b60 con 0x5609c59a7190 2024-03-30T01:03:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.102+0000 7f62e8c00700 1 -- 172.21.15.116:0/3224266904 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5609c58b7380 con 0x5609c59a7190 2024-03-30T01:03:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.104+0000 7f62e8c00700 1 -- 172.21.15.116:0/3224266904 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f62f8099bb0 con 0x5609c59a7190 2024-03-30T01:03:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.104+0000 7f6319d82b80 10 client.26102 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:03:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.104+0000 7f6319d82b80 10 client.26102 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T01:03:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.104+0000 7f6319d82b80 20 client.26102 populate_metadata read hostname 'smithi116' 2024-03-30T01:03:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.104+0000 7f6319d82b80 1 --2- 172.21.15.116:0/3224266904 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5609c59bb610 0x5609c59bda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.104+0000 7f6319d82b80 4 client.26102 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:03:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.104+0000 7f6319d82b80 1 -- 172.21.15.116:0/3224266904 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5609c58ef700 con 0x5609c59bb610 2024-03-30T01:03:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.105+0000 7f62ff400700 1 --2- 172.21.15.116:0/3224266904 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5609c59bb610 0x5609c59bda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:41.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.116+0000 7f62ff400700 1 --2- 172.21.15.116:0/3224266904 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5609c59bb610 0x5609c59bda30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a17f81346dfba7e0 server_cookie=636640be0eb19f20 in_seq=0 out_seq=0 2024-03-30T01:03:41.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.135+0000 7f62e8c00700 10 client.26102 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T01:03:41.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.135+0000 7f62e8c00700 1 -- 172.21.15.116:0/3224266904 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5609c58ef700 con 0x5609c59bb610 2024-03-30T01:03:41.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.135+0000 7f62e8c00700 10 client.26102 handle_command_reply: tid=0 2024-03-30T01:03:41.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.142+0000 7f6319d82b80 1 client.26102 shutdown 2024-03-30T01:03:41.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.155+0000 7f6319d82b80 1 -- 172.21.15.116:0/3224266904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a7190 msgr2=0x5609c59b6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:41.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.155+0000 7f6319d82b80 1 --2- 172.21.15.116:0/3224266904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a7190 0x5609c59b6da0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f62f80c9250 tx=0x7f62f8099350 comp rx=0 tx=0).stop 2024-03-30T01:03:41.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.157+0000 7f6319d82b80 1 -- 172.21.15.116:0/3224266904 shutdown_connections 2024-03-30T01:03:41.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.157+0000 7f6319d82b80 1 --2- 172.21.15.116:0/3224266904 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5609c59bb610 0x5609c59bda30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.157+0000 7f6319d82b80 1 --2- 172.21.15.116:0/3224266904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a7190 0x5609c59b6da0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.157+0000 7f6319d82b80 1 --2- 172.21.15.116:0/3224266904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a59e0 0x5609c59af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.157+0000 7f6319d82b80 1 --2- 172.21.15.116:0/3224266904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a6340 0x5609c59afd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.157+0000 7f62e8c00700 0 client.26102 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T01:03:41.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.157+0000 7f6319d82b80 1 -- 172.21.15.116:0/3224266904 >> 172.21.15.116:0/3224266904 conn(0x5609c59046a0 msgr2=0x5609c59a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:41.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.181+0000 7f6319d82b80 1 -- 172.21.15.116:0/3224266904 shutdown_connections 2024-03-30T01:03:41.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.186+0000 7f6319d82b80 1 -- 172.21.15.116:0/3224266904 wait complete. 2024-03-30T01:03:41.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.201+0000 7f6319d82b80 20 client.26102 trim_cache size 0 max 16384 2024-03-30T01:03:41.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.587+0000 7f6319d82b80 1 Processor -- start 2024-03-30T01:03:41.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.599+0000 7f6319d82b80 1 -- start start 2024-03-30T01:03:41.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.599+0000 7f6319d82b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 0x5609c59b7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:41.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.600+0000 7f6319d82b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a6340 0x5609c59b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:41.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.600+0000 7f6319d82b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a7190 0x5609c59aed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:41.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.600+0000 7f6319d82b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c59a1f10 con 0x5609c59a7190 2024-03-30T01:03:41.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.600+0000 7f6319d82b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c59b8b00 con 0x5609c59a6340 2024-03-30T01:03:41.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.600+0000 7f6319d82b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c59bb440 con 0x5609c59a59e0 2024-03-30T01:03:41.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.603+0000 7f62fea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a6340 0x5609c59b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:41.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.603+0000 7f62ff400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 0x5609c59b7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:41.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.607+0000 7f62ff400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 0x5609c59b7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:48620/0 (socket says 172.21.15.116:48620) 2024-03-30T01:03:41.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.607+0000 7f62ff400700 1 -- 172.21.15.116:0/1658798838 learned_addr learned my addr 172.21.15.116:0/1658798838 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:41.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.607+0000 7f62ffe00700 1 --2- 172.21.15.116:0/1658798838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a7190 0x5609c59aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:41.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.611+0000 7f62ff400700 1 -- 172.21.15.116:0/1658798838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a6340 msgr2=0x5609c59b6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:41.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.611+0000 7f62ff400700 1 --2- 172.21.15.116:0/1658798838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a6340 0x5609c59b6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.611+0000 7f62ff400700 1 -- 172.21.15.116:0/1658798838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a7190 msgr2=0x5609c59aed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:41.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.611+0000 7f62ff400700 1 --2- 172.21.15.116:0/1658798838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a7190 0x5609c59aed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.611+0000 7f62ff400700 1 -- 172.21.15.116:0/1658798838 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609c58bcea0 con 0x5609c59a59e0 2024-03-30T01:03:41.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.618+0000 7f62ff400700 1 --2- 172.21.15.116:0/1658798838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 0x5609c59b7450 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f62f0013040 tx=0x7f62f0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a532e038c9970f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:41.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.636+0000 7f62f5200700 1 -- 172.21.15.116:0/1658798838 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0024070 con 0x5609c59a59e0 2024-03-30T01:03:41.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.636+0000 7f62f5200700 1 -- 172.21.15.116:0/1658798838 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62f000ee20 con 0x5609c59a59e0 2024-03-30T01:03:41.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.636+0000 7f62f5200700 1 -- 172.21.15.116:0/1658798838 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0011c50 con 0x5609c59a59e0 2024-03-30T01:03:41.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.647+0000 7f6319d82b80 1 -- 172.21.15.116:0/1658798838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 msgr2=0x5609c59b7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:41.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.647+0000 7f6319d82b80 1 --2- 172.21.15.116:0/1658798838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 0x5609c59b7450 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f62f0013040 tx=0x7f62f0014000 comp rx=0 tx=0).stop 2024-03-30T01:03:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.669+0000 7f6319d82b80 1 -- 172.21.15.116:0/1658798838 shutdown_connections 2024-03-30T01:03:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.669+0000 7f6319d82b80 1 --2- 172.21.15.116:0/1658798838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 0x5609c59b7450 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.669+0000 7f6319d82b80 1 --2- 172.21.15.116:0/1658798838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a7190 0x5609c59aed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.669+0000 7f6319d82b80 1 --2- 172.21.15.116:0/1658798838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a6340 0x5609c59b6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.669+0000 7f6319d82b80 1 -- 172.21.15.116:0/1658798838 >> 172.21.15.116:0/1658798838 conn(0x5609c59046a0 msgr2=0x5609c59997f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:41.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.677+0000 7f6319d82b80 1 -- 172.21.15.116:0/1658798838 shutdown_connections 2024-03-30T01:03:41.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.680+0000 7f6319d82b80 1 -- 172.21.15.116:0/1658798838 wait complete. 2024-03-30T01:03:41.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.683+0000 7f6319d82b80 1 -- start start 2024-03-30T01:03:41.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.683+0000 7f6319d82b80 1 Processor -- start 2024-03-30T01:03:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.692+0000 7f6319d82b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 0x5609c59b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.692+0000 7f6319d82b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a6340 0x5609c59b5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.692+0000 7f6319d82b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a7190 0x5609c59b5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.692+0000 7f6319d82b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c59b5b90 con 0x5609c59a6340 2024-03-30T01:03:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.692+0000 7f6319d82b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c59b5d00 con 0x5609c59a7190 2024-03-30T01:03:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.692+0000 7f6319d82b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c59b5ec0 con 0x5609c59a59e0 2024-03-30T01:03:41.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.693+0000 7f6319d82b80 10 client.0 _ll_register_callbacks cb 0x5609c589c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:03:41.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.693+0000 7f62fea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a6340 0x5609c59b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:41.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.694+0000 7f62ffe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a7190 0x5609c59b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:41.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.694+0000 7f62ffe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a7190 0x5609c59b5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:39148/0 (socket says 172.21.15.116:39148) 2024-03-30T01:03:41.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.694+0000 7f62ffe00700 1 -- 172.21.15.116:0/2527418100 learned_addr learned my addr 172.21.15.116:0/2527418100 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:41.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.694+0000 7f62ff400700 1 --2- 172.21.15.116:0/2527418100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 0x5609c59b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:41.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.695+0000 7f62ffe00700 1 -- 172.21.15.116:0/2527418100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 msgr2=0x5609c59b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:41.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.695+0000 7f62ffe00700 1 --2- 172.21.15.116:0/2527418100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 0x5609c59b6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.695+0000 7f62ffe00700 1 -- 172.21.15.116:0/2527418100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a6340 msgr2=0x5609c59b5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:41.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.695+0000 7f62ffe00700 1 --2- 172.21.15.116:0/2527418100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a6340 0x5609c59b5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.695+0000 7f62ffe00700 1 -- 172.21.15.116:0/2527418100 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f000eb50 con 0x5609c59a7190 2024-03-30T01:03:41.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.698+0000 7f62ffe00700 1 --2- 172.21.15.116:0/2527418100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a7190 0x5609c59b5650 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f62f8098b30 tx=0x7f62f8007470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=820c98328988270a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:41.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.719+0000 7f62e8c00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T01:03:41.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.719+0000 7f62e8c00700 1 -- 172.21.15.116:0/2527418100 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f80b1260 con 0x5609c59a7190 2024-03-30T01:03:41.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.719+0000 7f62e8c00700 1 -- 172.21.15.116:0/2527418100 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f62f80b8040 con 0x5609c59a7190 2024-03-30T01:03:41.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.723+0000 7f6319d82b80 1 -- 172.21.15.116:0/2527418100 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f62ec005030 con 0x5609c59a7190 2024-03-30T01:03:41.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.726+0000 7f62e8c00700 1 -- 172.21.15.116:0/2527418100 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62f80a1020 con 0x5609c59a7190 2024-03-30T01:03:41.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.726+0000 7f6319d82b80 10 client.26108 fetch_fsmap learned FSMap version 28 2024-03-30T01:03:41.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.726+0000 7f6319d82b80 1 -- 172.21.15.116:0/2527418100 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5609c577ccd0 con 0x5609c59a7190 2024-03-30T01:03:41.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.727+0000 7f62e8c00700 1 -- 172.21.15.116:0/2527418100 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62ec005030 con 0x5609c59a7190 2024-03-30T01:03:41.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.728+0000 7f62e8c00700 1 -- 172.21.15.116:0/2527418100 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f62f80a0b90 con 0x5609c59a7190 2024-03-30T01:03:41.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.728+0000 7f6319d82b80 10 client.26108 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:03:41.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.728+0000 7f6319d82b80 10 client.26108 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T01:03:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.728+0000 7f6319d82b80 20 client.26108 populate_metadata read hostname 'smithi116' 2024-03-30T01:03:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.728+0000 7f6319d82b80 1 --2- 172.21.15.116:0/2527418100 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5609c5a10660 0x5609c5a12a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.728+0000 7f6319d82b80 4 client.26108 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:03:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.728+0000 7f6319d82b80 1 -- 172.21.15.116:0/2527418100 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5609c5745cf0 con 0x5609c5a10660 2024-03-30T01:03:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.729+0000 7f62ff400700 1 --2- 172.21.15.116:0/2527418100 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5609c5a10660 0x5609c5a12a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:41.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.732+0000 7f62ff400700 1 --2- 172.21.15.116:0/2527418100 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5609c5a10660 0x5609c5a12a80 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dca185d6ee3127f3 server_cookie=ce03eb470e8abb55 in_seq=0 out_seq=0 2024-03-30T01:03:41.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.732+0000 7f62e8c00700 10 client.26108 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T01:03:41.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.733+0000 7f62e8c00700 1 -- 172.21.15.116:0/2527418100 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5609c5745cf0 con 0x5609c5a10660 2024-03-30T01:03:41.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.733+0000 7f62e8c00700 10 client.26108 handle_command_reply: tid=0 2024-03-30T01:03:41.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.736+0000 7f6319d82b80 1 client.26108 shutdown 2024-03-30T01:03:41.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.785+0000 7f6319d82b80 1 -- 172.21.15.116:0/2527418100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a7190 msgr2=0x5609c59b5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:41.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.785+0000 7f6319d82b80 1 --2- 172.21.15.116:0/2527418100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a7190 0x5609c59b5650 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f62f8098b30 tx=0x7f62f8007470 comp rx=0 tx=0).stop 2024-03-30T01:03:41.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.786+0000 7f6319d82b80 1 -- 172.21.15.116:0/2527418100 shutdown_connections 2024-03-30T01:03:41.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.786+0000 7f6319d82b80 1 --2- 172.21.15.116:0/2527418100 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x5609c5a10660 0x5609c5a12a80 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.786+0000 7f6319d82b80 1 --2- 172.21.15.116:0/2527418100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5609c59a59e0 0x5609c59b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.786+0000 7f6319d82b80 1 --2- 172.21.15.116:0/2527418100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5609c59a7190 0x5609c59b5650 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.786+0000 7f6319d82b80 1 --2- 172.21.15.116:0/2527418100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609c59a6340 0x5609c59b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.786+0000 7f6319d82b80 1 -- 172.21.15.116:0/2527418100 >> 172.21.15.116:0/2527418100 conn(0x5609c59046a0 msgr2=0x5609c59aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:41.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.788+0000 7f62e8c00700 0 client.26108 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T01:03:41.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.835+0000 7f6319d82b80 1 -- 172.21.15.116:0/2527418100 shutdown_connections 2024-03-30T01:03:41.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.843+0000 7f6319d82b80 1 -- 172.21.15.116:0/2527418100 wait complete. 2024-03-30T01:03:41.868 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:03:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.860+0000 7f6319d82b80 20 client.26108 trim_cache size 0 max 16384 2024-03-30T01:03:41.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.904+0000 7f6304c00700 1 -- 172.21.15.116:0/417748641 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f62e40a11f0 msgr2=0x7f62e40a36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:41.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.904+0000 7f6304c00700 1 --2- 172.21.15.116:0/417748641 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f62e40a11f0 0x7f62e40a36b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f62f0005fd0 tx=0x7f62f0005f60 comp rx=0 tx=0).stop 2024-03-30T01:03:41.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.904+0000 7f6304c00700 1 -- 172.21.15.116:0/417748641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630010ce40 msgr2=0x7f63001a4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:41.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.904+0000 7f6304c00700 1 --2- 172.21.15.116:0/417748641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630010ce40 0x7f63001a4700 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f62f8007ee0 tx=0x7f62f800d3b0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.917+0000 7f6304c00700 1 -- 172.21.15.116:0/417748641 shutdown_connections 2024-03-30T01:03:41.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.917+0000 7f6304c00700 1 --2- 172.21.15.116:0/417748641 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f62e40a11f0 0x7f62e40a36b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.917+0000 7f6304c00700 1 --2- 172.21.15.116:0/417748641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f630010ce40 0x7f63001a4700 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.917+0000 7f6304c00700 1 --2- 172.21.15.116:0/417748641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6300072b80 0x7f630019d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.917+0000 7f6304c00700 1 --2- 172.21.15.116:0/417748641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f630010a390 0x7f630019dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:41.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.918+0000 7f6304c00700 1 -- 172.21.15.116:0/417748641 >> 172.21.15.116:0/417748641 conn(0x7f630006da50 msgr2=0x7f630006de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:41.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.938+0000 7f6304c00700 1 -- 172.21.15.116:0/417748641 shutdown_connections 2024-03-30T01:03:41.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:41.942+0000 7f6304c00700 1 -- 172.21.15.116:0/417748641 wait complete. 2024-03-30T01:03:42.563 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T01:03:42.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:42 smithi130 ceph-mon[47626]: pgmap v466: 129 pgs: 129 active+clean; 1.7 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 6.5 MiB/s wr, 458 op/s 2024-03-30T01:03:42.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:42 smithi116 ceph-mon[38086]: pgmap v466: 129 pgs: 129 active+clean; 1.7 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 6.5 MiB/s wr, 458 op/s 2024-03-30T01:03:43.005 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:42 smithi181 ceph-mon[47622]: pgmap v466: 129 pgs: 129 active+clean; 1.7 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 6.5 MiB/s wr, 458 op/s 2024-03-30T01:03:44.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:44 smithi130 ceph-mon[47626]: pgmap v467: 129 pgs: 129 active+clean; 1.8 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.4 MiB/s wr, 469 op/s 2024-03-30T01:03:44.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:44 smithi116 ceph-mon[38086]: pgmap v467: 129 pgs: 129 active+clean; 1.8 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.4 MiB/s wr, 469 op/s 2024-03-30T01:03:45.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:44 smithi181 ceph-mon[47622]: pgmap v467: 129 pgs: 129 active+clean; 1.8 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.4 MiB/s wr, 469 op/s 2024-03-30T01:03:46.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:46 smithi130 ceph-mon[47626]: pgmap v468: 129 pgs: 129 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 10 MiB/s wr, 686 op/s 2024-03-30T01:03:46.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:46 smithi116 ceph-mon[38086]: pgmap v468: 129 pgs: 129 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 10 MiB/s wr, 686 op/s 2024-03-30T01:03:47.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:46 smithi181 ceph-mon[47622]: pgmap v468: 129 pgs: 129 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 10 MiB/s wr, 686 op/s 2024-03-30T01:03:48.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:48 smithi130 ceph-mon[47626]: pgmap v469: 129 pgs: 129 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 704 op/s 2024-03-30T01:03:48.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:48 smithi116 ceph-mon[38086]: pgmap v469: 129 pgs: 129 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 704 op/s 2024-03-30T01:03:49.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:48 smithi181 ceph-mon[47622]: pgmap v469: 129 pgs: 129 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 704 op/s 2024-03-30T01:03:49.063 INFO:tasks.workunit.client.0.smithi116.stdout: 449 393943 0 280849 115 163148 269 2024-03-30T01:03:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.283+0000 7ffb62e00700 1 -- 172.21.15.116:0/2915577576 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb5c10a720 msgr2=0x7ffb5c10cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.283+0000 7ffb62e00700 1 --2- 172.21.15.116:0/2915577576 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb5c10a720 0x7ffb5c10cbb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffb48009990 tx=0x7ffb48009ca0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.284+0000 7ffb60e00700 1 --2- 172.21.15.116:0/2915577576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb5c10d0f0 0x7ffb5c10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:03:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.333+0000 7ffb62e00700 1 -- 172.21.15.116:0/2915577576 shutdown_connections 2024-03-30T01:03:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.333+0000 7ffb62e00700 1 --2- 172.21.15.116:0/2915577576 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb5c107db0 0x7ffb5c10a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.333+0000 7ffb62e00700 1 --2- 172.21.15.116:0/2915577576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb5c10d0f0 0x7ffb5c10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.333+0000 7ffb62e00700 1 --2- 172.21.15.116:0/2915577576 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb5c10a720 0x7ffb5c10cbb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.333+0000 7ffb62e00700 1 -- 172.21.15.116:0/2915577576 >> 172.21.15.116:0/2915577576 conn(0x7ffb5c06dd50 msgr2=0x7ffb5c070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.334+0000 7ffb62e00700 1 -- 172.21.15.116:0/2915577576 shutdown_connections 2024-03-30T01:03:49.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.339+0000 7ffb62e00700 1 -- 172.21.15.116:0/2915577576 wait complete. 2024-03-30T01:03:49.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.340+0000 7ffb62e00700 1 Processor -- start 2024-03-30T01:03:49.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.383+0000 7ffb62e00700 1 -- start start 2024-03-30T01:03:49.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.383+0000 7ffb62e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb5c107db0 0x7ffb5c1a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:49.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.383+0000 7ffb62e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb5c10a720 0x7ffb5c1a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:49.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.383+0000 7ffb62e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb5c10d0f0 0x7ffb5c1acff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:49.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.383+0000 7ffb62e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c1ad5c0 con 0x7ffb5c10d0f0 2024-03-30T01:03:49.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.383+0000 7ffb62e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c10f4e0 con 0x7ffb5c107db0 2024-03-30T01:03:49.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.383+0000 7ffb62e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c10f670 con 0x7ffb5c10a720 2024-03-30T01:03:49.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.387+0000 7ffb60e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb5c10d0f0 0x7ffb5c1acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:49.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.387+0000 7ffb60e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb5c10d0f0 0x7ffb5c1acff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59122/0 (socket says 172.21.15.116:59122) 2024-03-30T01:03:49.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.387+0000 7ffb60e00700 1 -- 172.21.15.116:0/3510878686 learned_addr learned my addr 172.21.15.116:0/3510878686 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:49.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.389+0000 7ffb60e00700 1 -- 172.21.15.116:0/3510878686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb5c10a720 msgr2=0x7ffb5c1a5fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T01:03:49.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.389+0000 7ffb60e00700 1 --2- 172.21.15.116:0/3510878686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb5c10a720 0x7ffb5c1a5fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.389+0000 7ffb60e00700 1 -- 172.21.15.116:0/3510878686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb5c107db0 msgr2=0x7ffb5c1a5a90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T01:03:49.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.389+0000 7ffb60e00700 1 --2- 172.21.15.116:0/3510878686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb5c107db0 0x7ffb5c1a5a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.389+0000 7ffb60e00700 1 -- 172.21.15.116:0/3510878686 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb480095f0 con 0x7ffb5c10d0f0 2024-03-30T01:03:49.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.392+0000 7ffb60e00700 1 --2- 172.21.15.116:0/3510878686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb5c10d0f0 0x7ffb5c1acff0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ffb54007d70 tx=0x7ffb5400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:49.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.403+0000 7ffb58c00700 1 -- 172.21.15.116:0/3510878686 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5400ad40 con 0x7ffb5c10d0f0 2024-03-30T01:03:49.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.404+0000 7ffb58c00700 1 -- 172.21.15.116:0/3510878686 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb540128e0 con 0x7ffb5c10d0f0 2024-03-30T01:03:49.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.404+0000 7ffb58c00700 1 -- 172.21.15.116:0/3510878686 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb540117c0 con 0x7ffb5c10d0f0 2024-03-30T01:03:49.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.404+0000 7ffb62e00700 1 -- 172.21.15.116:0/3510878686 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5c10f950 con 0x7ffb5c10d0f0 2024-03-30T01:03:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.405+0000 7ffb62e00700 1 -- 172.21.15.116:0/3510878686 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5c10fe10 con 0x7ffb5c10d0f0 2024-03-30T01:03:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.443+0000 7ffb58c00700 1 -- 172.21.15.116:0/3510878686 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7ffb54012420 con 0x7ffb5c10d0f0 2024-03-30T01:03:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.451+0000 7ffb58c00700 1 --2- 172.21.15.116:0/3510878686 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ffb440a1320 0x7ffb440a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.458+0000 7ffb58c00700 1 -- 172.21.15.116:0/3510878686 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ffb540cce40 con 0x7ffb5c10d0f0 2024-03-30T01:03:49.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.462+0000 7ffb768ecb80 1 Processor -- start 2024-03-30T01:03:49.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.464+0000 7ffb5be00700 1 --2- 172.21.15.116:0/3510878686 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ffb440a1320 0x7ffb440a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.465+0000 7ffb5be00700 1 --2- 172.21.15.116:0/3510878686 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ffb440a1320 0x7ffb440a37e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000ba10 tx=0x7ffb5000b3f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:49.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.495+0000 7ffb768ecb80 1 -- start start 2024-03-30T01:03:49.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.496+0000 7ffb768ecb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a47d0 0x5564e71a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:49.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.496+0000 7ffb768ecb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5130 0x5564e71a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:49.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.496+0000 7ffb768ecb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5fd0 0x5564e71a6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:49.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.498+0000 7ffb5b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5130 0x5564e71a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:49.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.498+0000 7ffb768ecb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e70fc990 con 0x5564e71a5130 2024-03-30T01:03:49.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.498+0000 7ffb768ecb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e70fa350 con 0x5564e71a47d0 2024-03-30T01:03:49.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.498+0000 7ffb768ecb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e70f5c40 con 0x5564e71a5fd0 2024-03-30T01:03:49.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.500+0000 7ffb60e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5fd0 0x5564e71a6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:49.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.501+0000 7ffb60e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5fd0 0x5564e71a6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:43810/0 (socket says 172.21.15.116:43810) 2024-03-30T01:03:49.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.501+0000 7ffb60e00700 1 -- 172.21.15.116:0/1473959548 learned_addr learned my addr 172.21.15.116:0/1473959548 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:49.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.503+0000 7ffb60e00700 1 -- 172.21.15.116:0/1473959548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a47d0 msgr2=0x5564e71a4bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:03:49.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.503+0000 7ffb60e00700 1 --2- 172.21.15.116:0/1473959548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a47d0 0x5564e71a4bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.503+0000 7ffb60e00700 1 -- 172.21.15.116:0/1473959548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5130 msgr2=0x5564e71a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:49.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.503+0000 7ffb60e00700 1 --2- 172.21.15.116:0/1473959548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5130 0x5564e71a59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.503+0000 7ffb60e00700 1 -- 172.21.15.116:0/1473959548 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564e6f773b0 con 0x5564e71a5fd0 2024-03-30T01:03:49.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.508+0000 7ffb60e00700 1 --2- 172.21.15.116:0/1473959548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5fd0 0x5564e71a6450 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffb54099e60 tx=0x7ffb540193e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=407a1cbba001896f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:49.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.523+0000 7ffb4d600700 1 -- 172.21.15.116:0/1473959548 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5409dd50 con 0x5564e71a5fd0 2024-03-30T01:03:49.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.523+0000 7ffb4d600700 1 -- 172.21.15.116:0/1473959548 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb540a23a0 con 0x5564e71a5fd0 2024-03-30T01:03:49.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.523+0000 7ffb4d600700 1 -- 172.21.15.116:0/1473959548 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb540a2170 con 0x5564e71a5fd0 2024-03-30T01:03:49.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.540+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1473959548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5fd0 msgr2=0x5564e71a6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:49.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.540+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1473959548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5fd0 0x5564e71a6450 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffb54099e60 tx=0x7ffb540193e0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.549+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1473959548 shutdown_connections 2024-03-30T01:03:49.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.549+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1473959548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5fd0 0x5564e71a6450 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.549+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1473959548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a47d0 0x5564e71a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.549+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1473959548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5130 0x5564e71a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.549+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1473959548 >> 172.21.15.116:0/1473959548 conn(0x5564e71036a0 msgr2=0x5564e719f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:49.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.589+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1473959548 shutdown_connections 2024-03-30T01:03:49.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.600+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1473959548 wait complete. 2024-03-30T01:03:49.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.600+0000 7ffb768ecb80 1 -- start start 2024-03-30T01:03:49.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.601+0000 7ffb768ecb80 1 Processor -- start 2024-03-30T01:03:49.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.621+0000 7ffb768ecb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a47d0 0x5564e71ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:49.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.621+0000 7ffb768ecb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a5130 0x5564e71aec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:49.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.621+0000 7ffb768ecb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5fd0 0x5564e71b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:49.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.621+0000 7ffb768ecb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e7091d80 con 0x5564e71a47d0 2024-03-30T01:03:49.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.621+0000 7ffb768ecb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e7091a20 con 0x5564e71a5130 2024-03-30T01:03:49.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.621+0000 7ffb768ecb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e6f796a0 con 0x5564e71a5fd0 2024-03-30T01:03:49.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.621+0000 7ffb768ecb80 10 client.0 _ll_register_callbacks cb 0x5564e709b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:03:49.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.621+0000 7ffb5b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a5130 0x5564e71aec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:49.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.622+0000 7ffb5b400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a5130 0x5564e71aec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:35850/0 (socket says 172.21.15.116:35850) 2024-03-30T01:03:49.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.622+0000 7ffb5b400700 1 -- 172.21.15.116:0/1316262144 learned_addr learned my addr 172.21.15.116:0/1316262144 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:49.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.623+0000 7ffb60e00700 1 --2- 172.21.15.116:0/1316262144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5fd0 0x5564e71b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:49.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.623+0000 7ffb5b400700 1 -- 172.21.15.116:0/1316262144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5fd0 msgr2=0x5564e71b5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:49.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.623+0000 7ffb5b400700 1 --2- 172.21.15.116:0/1316262144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5fd0 0x5564e71b5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.623+0000 7ffb5b400700 1 -- 172.21.15.116:0/1316262144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a47d0 msgr2=0x5564e71ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:49.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.623+0000 7ffb5b400700 1 --2- 172.21.15.116:0/1316262144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a47d0 0x5564e71ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.623+0000 7ffb5b400700 1 -- 172.21.15.116:0/1316262144 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5c10fe10 con 0x5564e71a5130 2024-03-30T01:03:49.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.625+0000 7ffb5b400700 1 --2- 172.21.15.116:0/1316262144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a5130 0x5564e71aec70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffb4800b5c0 tx=0x7ffb48009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cf52756d1ff4381 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.639+0000 7ffb42000700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T01:03:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.639+0000 7ffb42000700 1 -- 172.21.15.116:0/1316262144 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4800f870 con 0x5564e71a5130 2024-03-30T01:03:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.639+0000 7ffb42000700 1 -- 172.21.15.116:0/1316262144 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ffb4800f9d0 con 0x5564e71a5130 2024-03-30T01:03:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.644+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1316262144 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5564e70b6380 con 0x5564e71a5130 2024-03-30T01:03:49.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.659+0000 7ffb42000700 1 -- 172.21.15.116:0/1316262144 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb4800fb90 con 0x5564e71a5130 2024-03-30T01:03:49.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.676+0000 7ffb768ecb80 10 client.26114 fetch_fsmap learned FSMap version 28 2024-03-30T01:03:49.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.676+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1316262144 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5564e6f23b60 con 0x5564e71a5130 2024-03-30T01:03:49.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.677+0000 7ffb42000700 1 -- 172.21.15.116:0/1316262144 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5564e70b6380 con 0x5564e71a5130 2024-03-30T01:03:49.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.690+0000 7ffb42000700 1 -- 172.21.15.116:0/1316262144 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7ffb48020070 con 0x5564e71a5130 2024-03-30T01:03:49.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.690+0000 7ffb768ecb80 10 client.26114 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:03:49.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.690+0000 7ffb768ecb80 10 client.26114 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T01:03:49.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.690+0000 7ffb768ecb80 20 client.26114 populate_metadata read hostname 'smithi116' 2024-03-30T01:03:49.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.690+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1316262144 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x5564e71ba610 0x5564e71bca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:49.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.690+0000 7ffb768ecb80 4 client.26114 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:03:49.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.690+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1316262144 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5564e70ee700 con 0x5564e71ba610 2024-03-30T01:03:49.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.691+0000 7ffb5be00700 1 --2- 172.21.15.116:0/1316262144 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x5564e71ba610 0x5564e71bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:49.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.708+0000 7ffb5be00700 1 --2- 172.21.15.116:0/1316262144 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x5564e71ba610 0x5564e71bca30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=33ed8b911116437a server_cookie=8d2ac44a30d0f7bc in_seq=0 out_seq=0 2024-03-30T01:03:49.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.708+0000 7ffb42000700 10 client.26114 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T01:03:49.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.709+0000 7ffb42000700 1 -- 172.21.15.116:0/1316262144 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5564e70ee700 con 0x5564e71ba610 2024-03-30T01:03:49.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.709+0000 7ffb42000700 10 client.26114 handle_command_reply: tid=0 2024-03-30T01:03:49.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.712+0000 7ffb768ecb80 1 client.26114 shutdown 2024-03-30T01:03:49.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.720+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1316262144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a5130 msgr2=0x5564e71aec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:49.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.720+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1316262144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a5130 0x5564e71aec70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffb4800b5c0 tx=0x7ffb48009ee0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.726+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1316262144 shutdown_connections 2024-03-30T01:03:49.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.726+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1316262144 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x5564e71ba610 0x5564e71bca30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.726+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1316262144 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5fd0 0x5564e71b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.726+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1316262144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a47d0 0x5564e71ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.726+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1316262144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a5130 0x5564e71aec70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:49.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.726+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1316262144 >> 172.21.15.116:0/1316262144 conn(0x5564e71036a0 msgr2=0x5564e71a3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:49.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.741+0000 7ffb42000700 0 client.26114 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T01:03:49.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.760+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1316262144 shutdown_connections 2024-03-30T01:03:49.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.761+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1316262144 wait complete. 2024-03-30T01:03:49.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:49.763+0000 7ffb768ecb80 20 client.26114 trim_cache size 0 max 16384 2024-03-30T01:03:50.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.202+0000 7ffb768ecb80 1 Processor -- start 2024-03-30T01:03:50.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.206+0000 7ffb768ecb80 1 -- start start 2024-03-30T01:03:50.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.213+0000 7ffb768ecb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a47d0 0x5564e71b6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:50.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.213+0000 7ffb768ecb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5130 0x5564e71a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:50.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.214+0000 7ffb5be00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a47d0 0x5564e71b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:50.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.214+0000 7ffb5be00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a47d0 0x5564e71b6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:43832/0 (socket says 172.21.15.116:43832) 2024-03-30T01:03:50.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.217+0000 7ffb768ecb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a5fd0 0x5564e71b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:50.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.217+0000 7ffb768ecb80 1 -- 172.21.15.116:0/2407346279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e71ad150 con 0x5564e71a5130 2024-03-30T01:03:50.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.217+0000 7ffb5be00700 1 -- 172.21.15.116:0/2407346279 learned_addr learned my addr 172.21.15.116:0/2407346279 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:50.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.217+0000 7ffb768ecb80 1 -- 172.21.15.116:0/2407346279 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e71b6110 con 0x5564e71a5fd0 2024-03-30T01:03:50.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.217+0000 7ffb768ecb80 1 -- 172.21.15.116:0/2407346279 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e71a08b0 con 0x5564e71a47d0 2024-03-30T01:03:50.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.217+0000 7ffb60e00700 1 --2- 172.21.15.116:0/2407346279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a5fd0 0x5564e71b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:50.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.218+0000 7ffb5be00700 1 -- 172.21.15.116:0/2407346279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a5fd0 msgr2=0x5564e71b5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:50.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.218+0000 7ffb5be00700 1 --2- 172.21.15.116:0/2407346279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a5fd0 0x5564e71b5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.218+0000 7ffb5be00700 1 -- 172.21.15.116:0/2407346279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5130 msgr2=0x5564e71a63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:50.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.218+0000 7ffb5be00700 1 --2- 172.21.15.116:0/2407346279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5130 0x5564e71a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.218+0000 7ffb5be00700 1 -- 172.21.15.116:0/2407346279 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564e70bbea0 con 0x5564e71a47d0 2024-03-30T01:03:50.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.219+0000 7ffb5be00700 1 --2- 172.21.15.116:0/2407346279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a47d0 0x5564e71b6360 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ffb50022c40 tx=0x7ffb50004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=233a1c7b6b5fd3fc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:50.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.240+0000 7ffb4e000700 1 -- 172.21.15.116:0/2407346279 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb50022350 con 0x5564e71a47d0 2024-03-30T01:03:50.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.240+0000 7ffb4e000700 1 -- 172.21.15.116:0/2407346279 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb500224b0 con 0x5564e71a47d0 2024-03-30T01:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.240+0000 7ffb4e000700 1 -- 172.21.15.116:0/2407346279 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5001e6b0 con 0x5564e71a47d0 2024-03-30T01:03:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.280+0000 7ffb768ecb80 1 -- 172.21.15.116:0/2407346279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a47d0 msgr2=0x5564e71b6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.280+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/2407346279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a47d0 0x5564e71b6360 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ffb50022c40 tx=0x7ffb50004750 comp rx=0 tx=0).stop 2024-03-30T01:03:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.288+0000 7ffb768ecb80 1 -- 172.21.15.116:0/2407346279 shutdown_connections 2024-03-30T01:03:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.288+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/2407346279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a47d0 0x5564e71b6360 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.288+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/2407346279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a5fd0 0x5564e71b5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.288+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/2407346279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5130 0x5564e71a63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.288+0000 7ffb768ecb80 1 -- 172.21.15.116:0/2407346279 >> 172.21.15.116:0/2407346279 conn(0x5564e71036a0 msgr2=0x5564e71985a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.290+0000 7ffb768ecb80 1 -- 172.21.15.116:0/2407346279 shutdown_connections 2024-03-30T01:03:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.290+0000 7ffb768ecb80 1 -- 172.21.15.116:0/2407346279 wait complete. 2024-03-30T01:03:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.292+0000 7ffb768ecb80 1 -- start start 2024-03-30T01:03:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.301+0000 7ffb768ecb80 1 Processor -- start 2024-03-30T01:03:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.320+0000 7ffb768ecb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a47d0 0x5564e71b3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.322+0000 7ffb768ecb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5130 0x5564e71b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.322+0000 7ffb768ecb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5fd0 0x5564e71b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.322+0000 7ffb768ecb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e71b53f0 con 0x5564e71a5fd0 2024-03-30T01:03:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.322+0000 7ffb768ecb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e71b5560 con 0x5564e71a47d0 2024-03-30T01:03:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.322+0000 7ffb768ecb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e71b5720 con 0x5564e71a5130 2024-03-30T01:03:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.322+0000 7ffb768ecb80 10 client.0 _ll_register_callbacks cb 0x5564e709b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:03:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.326+0000 7ffb60e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5fd0 0x5564e71b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.326+0000 7ffb60e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5fd0 0x5564e71b4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59156/0 (socket says 172.21.15.116:59156) 2024-03-30T01:03:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.326+0000 7ffb60e00700 1 -- 172.21.15.116:0/1099497424 learned_addr learned my addr 172.21.15.116:0/1099497424 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.326+0000 7ffb60e00700 1 -- 172.21.15.116:0/1099497424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5130 msgr2=0x5564e71b3ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:03:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.326+0000 7ffb60e00700 1 --2- 172.21.15.116:0/1099497424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5130 0x5564e71b3ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.326+0000 7ffb60e00700 1 -- 172.21.15.116:0/1099497424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a47d0 msgr2=0x5564e71b3960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:03:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.326+0000 7ffb60e00700 1 --2- 172.21.15.116:0/1099497424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a47d0 0x5564e71b3960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.326+0000 7ffb60e00700 1 -- 172.21.15.116:0/1099497424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5000d7d0 con 0x5564e71a5fd0 2024-03-30T01:03:50.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.337+0000 7ffb60e00700 1 --2- 172.21.15.116:0/1099497424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5fd0 0x5564e71b4eb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ffb54099eb0 tx=0x7ffb5409a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e95b05f30a1491 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:50.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.345+0000 7ffb42000700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:03:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.345+0000 7ffb42000700 1 -- 172.21.15.116:0/1099497424 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb540ca650 con 0x5564e71a5fd0 2024-03-30T01:03:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.345+0000 7ffb42000700 1 -- 172.21.15.116:0/1099497424 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ffb540ca7b0 con 0x5564e71a5fd0 2024-03-30T01:03:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.346+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1099497424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb540a23a0 con 0x5564e71a5fd0 2024-03-30T01:03:50.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.361+0000 7ffb42000700 1 -- 172.21.15.116:0/1099497424 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb54099410 con 0x5564e71a5fd0 2024-03-30T01:03:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.376+0000 7ffb768ecb80 10 client.16116 fetch_fsmap learned FSMap version 28 2024-03-30T01:03:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.376+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1099497424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5564e6f7bcd0 con 0x5564e71a5fd0 2024-03-30T01:03:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.386+0000 7ffb42000700 1 -- 172.21.15.116:0/1099497424 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb540a23a0 con 0x5564e71a5fd0 2024-03-30T01:03:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.388+0000 7ffb42000700 1 -- 172.21.15.116:0/1099497424 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7ffb54095eb0 con 0x5564e71a5fd0 2024-03-30T01:03:50.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.404+0000 7ffb768ecb80 10 client.16116 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:03:50.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.404+0000 7ffb768ecb80 10 client.16116 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T01:03:50.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.404+0000 7ffb768ecb80 20 client.16116 populate_metadata read hostname 'smithi116' 2024-03-30T01:03:50.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.404+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1099497424 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x5564e720efa0 0x5564e7211390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:50.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.404+0000 7ffb768ecb80 4 client.16116 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:03:50.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.404+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1099497424 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5564e6f44cf0 con 0x5564e720efa0 2024-03-30T01:03:50.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.404+0000 7ffb5be00700 1 --2- 172.21.15.116:0/1099497424 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x5564e720efa0 0x5564e7211390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:50.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.413+0000 7ffb5be00700 1 --2- 172.21.15.116:0/1099497424 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x5564e720efa0 0x5564e7211390 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dea797189ec6ed1b server_cookie=73359cae3cae545d in_seq=0 out_seq=0 2024-03-30T01:03:50.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.413+0000 7ffb42000700 10 client.16116 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T01:03:50.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.417+0000 7ffb42000700 1 -- 172.21.15.116:0/1099497424 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5564e6f44cf0 con 0x5564e720efa0 2024-03-30T01:03:50.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.417+0000 7ffb42000700 10 client.16116 handle_command_reply: tid=0 2024-03-30T01:03:50.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.417+0000 7ffb768ecb80 1 client.16116 shutdown 2024-03-30T01:03:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.424+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1099497424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5fd0 msgr2=0x5564e71b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.424+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1099497424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5fd0 0x5564e71b4eb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ffb54099eb0 tx=0x7ffb5409a0a0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.426+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1099497424 shutdown_connections 2024-03-30T01:03:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.426+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1099497424 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x5564e720efa0 0x5564e7211390 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.426+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1099497424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564e71a5fd0 0x5564e71b4eb0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.426+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1099497424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564e71a47d0 0x5564e71b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.426+0000 7ffb768ecb80 1 --2- 172.21.15.116:0/1099497424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5564e71a5130 0x5564e71b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.426+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1099497424 >> 172.21.15.116:0/1099497424 conn(0x5564e71036a0 msgr2=0x5564e71ac530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.426+0000 7ffb42000700 0 client.16116 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T01:03:50.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.440+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1099497424 shutdown_connections 2024-03-30T01:03:50.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.442+0000 7ffb768ecb80 1 -- 172.21.15.116:0/1099497424 wait complete. 2024-03-30T01:03:50.468 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:03:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.456+0000 7ffb768ecb80 20 client.16116 trim_cache size 0 max 16384 2024-03-30T01:03:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.487+0000 7ffb62e00700 1 -- 172.21.15.116:0/3510878686 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ffb440a1320 msgr2=0x7ffb440a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.487+0000 7ffb62e00700 1 --2- 172.21.15.116:0/3510878686 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ffb440a1320 0x7ffb440a37e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000ba10 tx=0x7ffb5000b3f0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.487+0000 7ffb62e00700 1 -- 172.21.15.116:0/3510878686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb5c10d0f0 msgr2=0x7ffb5c1acff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.487+0000 7ffb62e00700 1 --2- 172.21.15.116:0/3510878686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb5c10d0f0 0x7ffb5c1acff0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ffb54007d70 tx=0x7ffb5400a340 comp rx=0 tx=0).stop 2024-03-30T01:03:50.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.496+0000 7ffb62e00700 1 -- 172.21.15.116:0/3510878686 shutdown_connections 2024-03-30T01:03:50.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.496+0000 7ffb62e00700 1 --2- 172.21.15.116:0/3510878686 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ffb440a1320 0x7ffb440a37e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.496+0000 7ffb62e00700 1 --2- 172.21.15.116:0/3510878686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb5c10d0f0 0x7ffb5c1acff0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.496+0000 7ffb62e00700 1 --2- 172.21.15.116:0/3510878686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb5c107db0 0x7ffb5c1a5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.496+0000 7ffb62e00700 1 --2- 172.21.15.116:0/3510878686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb5c10a720 0x7ffb5c1a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:50.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.496+0000 7ffb62e00700 1 -- 172.21.15.116:0/3510878686 >> 172.21.15.116:0/3510878686 conn(0x7ffb5c06dd50 msgr2=0x7ffb5c109f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:50.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.500+0000 7ffb62e00700 1 -- 172.21.15.116:0/3510878686 shutdown_connections 2024-03-30T01:03:50.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:50.501+0000 7ffb62e00700 1 -- 172.21.15.116:0/3510878686 wait complete. 2024-03-30T01:03:50.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:50 smithi130 ceph-mon[47626]: pgmap v470: 129 pgs: 129 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 13 MiB/s wr, 822 op/s 2024-03-30T01:03:50.918 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:50 smithi116 ceph-mon[38086]: pgmap v470: 129 pgs: 129 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 13 MiB/s wr, 822 op/s 2024-03-30T01:03:50.973 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T01:03:51.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:50 smithi181 ceph-mon[47622]: pgmap v470: 129 pgs: 129 active+clean; 1.8 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 13 MiB/s wr, 822 op/s 2024-03-30T01:03:52.578 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:52 smithi116 ceph-mon[38086]: pgmap v471: 129 pgs: 129 active+clean; 1.9 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 14 MiB/s wr, 922 op/s 2024-03-30T01:03:52.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:52 smithi130 ceph-mon[47626]: pgmap v471: 129 pgs: 129 active+clean; 1.9 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 14 MiB/s wr, 922 op/s 2024-03-30T01:03:53.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:52 smithi181 ceph-mon[47622]: pgmap v471: 129 pgs: 129 active+clean; 1.9 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 14 MiB/s wr, 922 op/s 2024-03-30T01:03:54.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:54 smithi130 ceph-mon[47626]: pgmap v472: 129 pgs: 129 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 13 MiB/s wr, 793 op/s 2024-03-30T01:03:54.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:54 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:03:54.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:54 smithi116 ceph-mon[38086]: pgmap v472: 129 pgs: 129 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 13 MiB/s wr, 793 op/s 2024-03-30T01:03:54.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:54 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:03:55.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:54 smithi181 ceph-mon[47622]: pgmap v472: 129 pgs: 129 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 13 MiB/s wr, 793 op/s 2024-03-30T01:03:55.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:54 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:03:55.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.733+0000 7f1fb8000700 1 -- 172.21.15.116:0/894586895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fb010aad0 msgr2=0x7f1fb01138c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:55.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.733+0000 7f1fb8000700 1 --2- 172.21.15.116:0/894586895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fb010aad0 0x7f1fb01138c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40098f0 tx=0x7f1fa4009c00 comp rx=0 tx=0).stop 2024-03-30T01:03:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.749+0000 7f1fb8000700 1 -- 172.21.15.116:0/894586895 shutdown_connections 2024-03-30T01:03:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.749+0000 7f1fb8000700 1 --2- 172.21.15.116:0/894586895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fb010aad0 0x7f1fb01138c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.749+0000 7f1fb8000700 1 --2- 172.21.15.116:0/894586895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb0107b90 0x7f1fb0109fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.749+0000 7f1fb8000700 1 --2- 172.21.15.116:0/894586895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fb010a590 0x7f1fb01112d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.749+0000 7f1fb8000700 1 -- 172.21.15.116:0/894586895 >> 172.21.15.116:0/894586895 conn(0x7f1fb006dbc0 msgr2=0x7f1fb0070000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.759+0000 7f1fb8000700 1 -- 172.21.15.116:0/894586895 shutdown_connections 2024-03-30T01:03:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.760+0000 7f1fb8000700 1 -- 172.21.15.116:0/894586895 wait complete. 2024-03-30T01:03:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.762+0000 7f1fb8000700 1 Processor -- start 2024-03-30T01:03:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.766+0000 7f1fb8000700 1 -- start start 2024-03-30T01:03:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.774+0000 7f1fb8000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fb0107b90 0x7f1fb01ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:55.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.778+0000 7f1fb8000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb010a590 0x7f1fb01ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:55.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.781+0000 7f1fb8000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fb010aad0 0x7f1fb01b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:55.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.783+0000 7f1fb8000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb01b5ee0 con 0x7f1fb010a590 2024-03-30T01:03:55.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.783+0000 7f1fb8000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb01b6020 con 0x7f1fb0107b90 2024-03-30T01:03:55.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.783+0000 7f1fb8000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb01b61b0 con 0x7f1fb010aad0 2024-03-30T01:03:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.783+0000 7f1fb4c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb010a590 0x7f1fb01ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.783+0000 7f1fb4c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb010a590 0x7f1fb01ae8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41390/0 (socket says 172.21.15.116:41390) 2024-03-30T01:03:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.783+0000 7f1fb4c00700 1 -- 172.21.15.116:0/3733812864 learned_addr learned my addr 172.21.15.116:0/3733812864 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.784+0000 7f1fb6000700 1 --2- 172.21.15.116:0/3733812864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fb010aad0 0x7f1fb01b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.786+0000 7f1fb5600700 1 --2- 172.21.15.116:0/3733812864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fb0107b90 0x7f1fb01ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.786+0000 7f1fb5600700 1 -- 172.21.15.116:0/3733812864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fb010aad0 msgr2=0x7f1fb01b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.786+0000 7f1fb5600700 1 --2- 172.21.15.116:0/3733812864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fb010aad0 0x7f1fb01b5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.786+0000 7f1fb5600700 1 -- 172.21.15.116:0/3733812864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb010a590 msgr2=0x7f1fb01ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.786+0000 7f1fb5600700 1 --2- 172.21.15.116:0/3733812864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb010a590 0x7f1fb01ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.786+0000 7f1fb5600700 1 -- 172.21.15.116:0/3733812864 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa4009520 con 0x7f1fb0107b90 2024-03-30T01:03:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.790+0000 7f1fb5600700 1 --2- 172.21.15.116:0/3733812864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fb0107b90 0x7f1fb01ae3b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c005fd0 tx=0x7f1f9c00eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.791+0000 7f1fae000700 1 -- 172.21.15.116:0/3733812864 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c004220 con 0x7f1fb0107b90 2024-03-30T01:03:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.791+0000 7f1fae000700 1 -- 172.21.15.116:0/3733812864 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f9c004380 con 0x7f1fb0107b90 2024-03-30T01:03:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.791+0000 7f1fae000700 1 -- 172.21.15.116:0/3733812864 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c003be0 con 0x7f1fb0107b90 2024-03-30T01:03:55.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.803+0000 7f1fb8000700 1 -- 172.21.15.116:0/3733812864 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb01b6490 con 0x7f1fb0107b90 2024-03-30T01:03:55.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.804+0000 7f1fb8000700 1 -- 172.21.15.116:0/3733812864 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb01b68c0 con 0x7f1fb0107b90 2024-03-30T01:03:55.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.839+0000 7f1fcbba8b80 1 Processor -- start 2024-03-30T01:03:55.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.845+0000 7f1fae000700 1 -- 172.21.15.116:0/3733812864 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f1f9c003d40 con 0x7f1fb0107b90 2024-03-30T01:03:55.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.847+0000 7f1fae000700 1 --2- 172.21.15.116:0/3733812864 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1f980a12a0 0x7f1f980a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:55.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.849+0000 7f1fb6000700 1 --2- 172.21.15.116:0/3733812864 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1f980a12a0 0x7f1f980a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:55.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.850+0000 7f1fae000700 1 -- 172.21.15.116:0/3733812864 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1f9c014070 con 0x7f1fb0107b90 2024-03-30T01:03:55.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.851+0000 7f1fb6000700 1 --2- 172.21.15.116:0/3733812864 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1f980a12a0 0x7f1f980a3760 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009fd0 tx=0x7f1fa400b5f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:55.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.854+0000 7f1fcbba8b80 1 -- start start 2024-03-30T01:03:55.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.854+0000 7f1fcbba8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cce97d0 0x56338cce9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:55.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.854+0000 7f1fcbba8b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338ccea130 0x56338ccea9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:55.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.855+0000 7f1fcbba8b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cceafd0 0x56338cceb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:55.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.855+0000 7f1fb5600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cce97d0 0x56338cce9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:55.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.855+0000 7f1fb5600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cce97d0 0x56338cce9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:56230/0 (socket says 172.21.15.116:56230) 2024-03-30T01:03:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.855+0000 7f1fb5600700 1 -- 172.21.15.116:0/3733170721 learned_addr learned my addr 172.21.15.116:0/3733170721 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.855+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/3733170721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338cc41990 con 0x56338cceafd0 2024-03-30T01:03:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.855+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/3733170721 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338cc3f350 con 0x56338cce97d0 2024-03-30T01:03:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.855+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/3733170721 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338cc3ac40 con 0x56338ccea130 2024-03-30T01:03:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.855+0000 7f1fb6000700 1 --2- 172.21.15.116:0/3733170721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cceafd0 0x56338cceb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.857+0000 7f1fb5600700 1 -- 172.21.15.116:0/3733170721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338ccea130 msgr2=0x56338ccea9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:03:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.857+0000 7f1fb5600700 1 --2- 172.21.15.116:0/3733170721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338ccea130 0x56338ccea9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.857+0000 7f1fb5600700 1 -- 172.21.15.116:0/3733170721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cceafd0 msgr2=0x56338cceb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.857+0000 7f1fb5600700 1 --2- 172.21.15.116:0/3733170721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cceafd0 0x56338cceb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.857+0000 7f1fb5600700 1 -- 172.21.15.116:0/3733170721 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56338cabc3b0 con 0x56338cce97d0 2024-03-30T01:03:55.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.859+0000 7f1fb5600700 1 --2- 172.21.15.116:0/3733170721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cce97d0 0x56338cce9bf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0962b0 tx=0x7f1f9c009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=919db11fc93af429 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:55.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.859+0000 7f1fa2600700 1 -- 172.21.15.116:0/3733170721 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c0044f0 con 0x56338cce97d0 2024-03-30T01:03:55.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.859+0000 7f1fa2600700 1 -- 172.21.15.116:0/3733170721 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f9c09a4b0 con 0x56338cce97d0 2024-03-30T01:03:55.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.859+0000 7f1fa2600700 1 -- 172.21.15.116:0/3733170721 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c099500 con 0x56338cce97d0 2024-03-30T01:03:55.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.860+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/3733170721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cce97d0 msgr2=0x56338cce9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:55.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.860+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/3733170721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cce97d0 0x56338cce9bf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c0962b0 tx=0x7f1f9c009f70 comp rx=0 tx=0).stop 2024-03-30T01:03:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.905+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/3733170721 shutdown_connections 2024-03-30T01:03:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.905+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/3733170721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cceafd0 0x56338cceb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.905+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/3733170721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cce97d0 0x56338cce9bf0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.905+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/3733170721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338ccea130 0x56338ccea9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.905+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/3733170721 >> 172.21.15.116:0/3733170721 conn(0x56338cc486a0 msgr2=0x56338cce49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.942+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/3733170721 shutdown_connections 2024-03-30T01:03:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.950+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/3733170721 wait complete. 2024-03-30T01:03:55.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.951+0000 7f1fcbba8b80 1 -- start start 2024-03-30T01:03:55.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.951+0000 7f1fcbba8b80 1 Processor -- start 2024-03-30T01:03:55.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.957+0000 7f1fcbba8b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338cce97d0 0x56338ccf3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:55.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.957+0000 7f1fcbba8b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338ccea130 0x56338ccf3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:55.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.958+0000 7f1fb4c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338ccea130 0x56338ccf3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:55.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.958+0000 7f1fb4c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338ccea130 0x56338ccf3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41406/0 (socket says 172.21.15.116:41406) 2024-03-30T01:03:55.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.958+0000 7f1fb4c00700 1 -- 172.21.15.116:0/249515410 learned_addr learned my addr 172.21.15.116:0/249515410 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:55.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.961+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/249515410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cceafd0 0x56338ccfabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:55.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.961+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/249515410 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338cbd6d80 con 0x56338ccea130 2024-03-30T01:03:55.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.961+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/249515410 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338cbd6a20 con 0x56338cceafd0 2024-03-30T01:03:55.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.961+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/249515410 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338cabe6a0 con 0x56338cce97d0 2024-03-30T01:03:55.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.961+0000 7f1fcbba8b80 10 client.0 _ll_register_callbacks cb 0x56338cbe0470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:03:55.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.962+0000 7f1fb4c00700 1 -- 172.21.15.116:0/249515410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338cce97d0 msgr2=0x56338ccf3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:55.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.962+0000 7f1fb5600700 1 --2- 172.21.15.116:0/249515410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338cce97d0 0x56338ccf3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:55.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.962+0000 7f1fb4c00700 1 --2- 172.21.15.116:0/249515410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338cce97d0 0x56338ccf3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:55.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.963+0000 7f1fb4c00700 1 -- 172.21.15.116:0/249515410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cceafd0 msgr2=0x56338ccfabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:55.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.963+0000 7f1fb4c00700 1 --2- 172.21.15.116:0/249515410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cceafd0 0x56338ccfabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:55.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.963+0000 7f1fb4c00700 1 -- 172.21.15.116:0/249515410 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb01b68c0 con 0x56338ccea130 2024-03-30T01:03:55.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.963+0000 7f1fb5600700 1 --2- 172.21.15.116:0/249515410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338cce97d0 0x56338ccf3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:03:55.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.965+0000 7f1fb4c00700 1 --2- 172.21.15.116:0/249515410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338ccea130 0x56338ccf3ba0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1fa800e970 tx=0x7f1fa800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b2d0b3ee7ccc0b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:55.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.977+0000 7f1f96a00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:03:55.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.977+0000 7f1f96a00700 1 -- 172.21.15.116:0/249515410 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa800c730 con 0x56338ccea130 2024-03-30T01:03:55.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.977+0000 7f1f96a00700 1 -- 172.21.15.116:0/249515410 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1fa800c890 con 0x56338ccea130 2024-03-30T01:03:55.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.981+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/249515410 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56338cbfb380 con 0x56338ccea130 2024-03-30T01:03:55.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.983+0000 7f1f96a00700 1 -- 172.21.15.116:0/249515410 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fa8009e70 con 0x56338ccea130 2024-03-30T01:03:56.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.996+0000 7f1fcbba8b80 10 client.16128 fetch_fsmap learned FSMap version 28 2024-03-30T01:03:56.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:55.996+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/249515410 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56338ca68b60 con 0x56338ccea130 2024-03-30T01:03:56.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.008+0000 7f1f96a00700 1 -- 172.21.15.116:0/249515410 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56338cbfb380 con 0x56338ccea130 2024-03-30T01:03:56.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.008+0000 7f1f96a00700 1 -- 172.21.15.116:0/249515410 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f1fa801d3f0 con 0x56338ccea130 2024-03-30T01:03:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.008+0000 7f1fcbba8b80 10 client.16128 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:03:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.008+0000 7f1fcbba8b80 10 client.16128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:03:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.008+0000 7f1fcbba8b80 20 client.16128 populate_metadata read hostname 'smithi116' 2024-03-30T01:03:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.008+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/249515410 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56338ccff520 0x56338cd01940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.008+0000 7f1fcbba8b80 4 client.16128 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:03:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.008+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/249515410 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56338cc33700 con 0x56338ccff520 2024-03-30T01:03:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.009+0000 7f1fb5600700 1 --2- 172.21.15.116:0/249515410 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56338ccff520 0x56338cd01940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:56.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.015+0000 7f1fb5600700 1 --2- 172.21.15.116:0/249515410 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56338ccff520 0x56338cd01940 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c016ba02a7c461bf server_cookie=729c8f4e3e985600 in_seq=0 out_seq=0 2024-03-30T01:03:56.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.015+0000 7f1f96a00700 10 client.16128 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:03:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.016+0000 7f1f96a00700 1 -- 172.21.15.116:0/249515410 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56338cc33700 con 0x56338ccff520 2024-03-30T01:03:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.016+0000 7f1f96a00700 10 client.16128 handle_command_reply: tid=0 2024-03-30T01:03:56.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.020+0000 7f1fcbba8b80 1 client.16128 shutdown 2024-03-30T01:03:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.047+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/249515410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338ccea130 msgr2=0x56338ccf3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.047+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/249515410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338ccea130 0x56338ccf3ba0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1fa800e970 tx=0x7f1fa800ed30 comp rx=0 tx=0).stop 2024-03-30T01:03:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.047+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/249515410 shutdown_connections 2024-03-30T01:03:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.047+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/249515410 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56338ccff520 0x56338cd01940 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.047+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/249515410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338cce97d0 0x56338ccf3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.047+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/249515410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cceafd0 0x56338ccfabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.047+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/249515410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338ccea130 0x56338ccf3ba0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.047+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/249515410 >> 172.21.15.116:0/249515410 conn(0x56338cc486a0 msgr2=0x56338cce8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.047+0000 7f1f96a00700 0 client.16128 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:03:56.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.052+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/249515410 shutdown_connections 2024-03-30T01:03:56.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.052+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/249515410 wait complete. 2024-03-30T01:03:56.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.095+0000 7f1fcbba8b80 20 client.16128 trim_cache size 0 max 16384 2024-03-30T01:03:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.516+0000 7f1fcbba8b80 1 Processor -- start 2024-03-30T01:03:56.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.556+0000 7f1fcbba8b80 1 -- start start 2024-03-30T01:03:56.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.557+0000 7f1fcbba8b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cce97d0 0x56338ccfb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:56.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.557+0000 7f1fcbba8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338ccea130 0x56338ccfaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:56.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.557+0000 7f1fcbba8b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338cceafd0 0x56338cceb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:56.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.557+0000 7f1fcbba8b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338ccea510 con 0x56338cce97d0 2024-03-30T01:03:56.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.557+0000 7f1fcbba8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338cce9f70 con 0x56338ccea130 2024-03-30T01:03:56.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.557+0000 7f1fcbba8b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338cce58b0 con 0x56338cceafd0 2024-03-30T01:03:56.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.557+0000 7f1fb6000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338cceafd0 0x56338cceb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:56.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.557+0000 7f1fb6000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338cceafd0 0x56338cceb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:41212/0 (socket says 172.21.15.116:41212) 2024-03-30T01:03:56.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.557+0000 7f1fb6000700 1 -- 172.21.15.116:0/2597644745 learned_addr learned my addr 172.21.15.116:0/2597644745 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:56.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.558+0000 7f1fb4c00700 1 --2- 172.21.15.116:0/2597644745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338ccea130 0x56338ccfaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:56.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.558+0000 7f1fb4c00700 1 -- 172.21.15.116:0/2597644745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338cceafd0 msgr2=0x56338cceb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:56.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.559+0000 7f1fb4c00700 1 --2- 172.21.15.116:0/2597644745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338cceafd0 0x56338cceb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.559+0000 7f1fb4c00700 1 -- 172.21.15.116:0/2597644745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cce97d0 msgr2=0x56338ccfb290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:03:56.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.559+0000 7f1fb4c00700 1 --2- 172.21.15.116:0/2597644745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cce97d0 0x56338ccfb290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.559+0000 7f1fb4c00700 1 -- 172.21.15.116:0/2597644745 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56338cc00ea0 con 0x56338ccea130 2024-03-30T01:03:56.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.564+0000 7f1fb4c00700 1 --2- 172.21.15.116:0/2597644745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338ccea130 0x56338ccfaba0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8000f80 tx=0x7f1fa80094c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeaa0bb4012844fa server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.591+0000 7f1fa3000700 1 -- 172.21.15.116:0/2597644745 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8007000 con 0x56338ccea130 2024-03-30T01:03:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.591+0000 7f1fa3000700 1 -- 172.21.15.116:0/2597644745 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1fa800f040 con 0x56338ccea130 2024-03-30T01:03:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.591+0000 7f1fa3000700 1 -- 172.21.15.116:0/2597644745 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa800ca90 con 0x56338ccea130 2024-03-30T01:03:56.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.609+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2597644745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338ccea130 msgr2=0x56338ccfaba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:56.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.609+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/2597644745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338ccea130 0x56338ccfaba0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8000f80 tx=0x7f1fa80094c0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.639+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2597644745 shutdown_connections 2024-03-30T01:03:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.639+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/2597644745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338cceafd0 0x56338cceb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.639+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/2597644745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cce97d0 0x56338ccfb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.639+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/2597644745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338ccea130 0x56338ccfaba0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.639+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2597644745 >> 172.21.15.116:0/2597644745 conn(0x56338cc486a0 msgr2=0x56338ccf2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:56.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.649+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2597644745 shutdown_connections 2024-03-30T01:03:56.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.651+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2597644745 wait complete. 2024-03-30T01:03:56.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.654+0000 7f1fcbba8b80 1 -- start start 2024-03-30T01:03:56.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.655+0000 7f1fcbba8b80 1 Processor -- start 2024-03-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.672+0000 7f1fcbba8b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cce97d0 0x56338ccf8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.673+0000 7f1fb5600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cce97d0 0x56338ccf8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.674+0000 7f1fb5600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cce97d0 0x56338ccf8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41418/0 (socket says 172.21.15.116:41418) 2024-03-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.677+0000 7f1fb5600700 1 -- 172.21.15.116:0/2062288679 learned_addr learned my addr 172.21.15.116:0/2062288679 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.677+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338ccea130 0x56338ccf94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.678+0000 7f1fb4c00700 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338ccea130 0x56338ccf94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.679+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cceafd0 0x56338ccfa120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:56.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.681+0000 7f1fb6000700 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cceafd0 0x56338ccfa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.682+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2062288679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338ccfa740 con 0x56338cce97d0 2024-03-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.682+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2062288679 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338ccfa880 con 0x56338cceafd0 2024-03-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.682+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2062288679 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338ccfaa40 con 0x56338ccea130 2024-03-30T01:03:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.685+0000 7f1fcbba8b80 10 client.0 _ll_register_callbacks cb 0x56338cbe0470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.691+0000 7f1fb5600700 1 -- 172.21.15.116:0/2062288679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338ccea130 msgr2=0x56338ccf94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.691+0000 7f1fb5600700 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338ccea130 0x56338ccf94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.691+0000 7f1fb5600700 1 -- 172.21.15.116:0/2062288679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cceafd0 msgr2=0x56338ccfa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.691+0000 7f1fb5600700 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cceafd0 0x56338ccfa120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.691+0000 7f1fb5600700 1 -- 172.21.15.116:0/2062288679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56338ca68b60 con 0x56338cce97d0 2024-03-30T01:03:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.692+0000 7f1fb4c00700 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338ccea130 0x56338ccf94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:03:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.695+0000 7f1fb6000700 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cceafd0 0x56338ccfa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:03:56.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.701+0000 7f1fb5600700 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cce97d0 0x56338ccf8f70 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c003ff0 tx=0x7f1f9c09a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467faf0fb7822d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:03:56.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.701+0000 7f1f96a00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:03:56.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.704+0000 7f1f96a00700 1 -- 172.21.15.116:0/2062288679 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c0a8070 con 0x56338cce97d0 2024-03-30T01:03:56.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.704+0000 7f1f96a00700 1 -- 172.21.15.116:0/2062288679 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1f9c09dbe0 con 0x56338cce97d0 2024-03-30T01:03:56.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.723+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2062288679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f9c09a4b0 con 0x56338cce97d0 2024-03-30T01:03:56.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.724+0000 7f1f96a00700 1 -- 172.21.15.116:0/2062288679 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f9c099db0 con 0x56338cce97d0 2024-03-30T01:03:56.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.729+0000 7f1fcbba8b80 10 client.16131 fetch_fsmap learned FSMap version 28 2024-03-30T01:03:56.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.729+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2062288679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56338cac0cd0 con 0x56338cce97d0 2024-03-30T01:03:56.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.731+0000 7f1f96a00700 1 -- 172.21.15.116:0/2062288679 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f9c09a4b0 con 0x56338cce97d0 2024-03-30T01:03:56.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.731+0000 7f1f96a00700 1 -- 172.21.15.116:0/2062288679 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f1f9c0ab030 con 0x56338cce97d0 2024-03-30T01:03:56.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.733+0000 7f1fcbba8b80 10 client.16131 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:03:56.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.733+0000 7f1fcbba8b80 10 client.16131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:03:56.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.733+0000 7f1fcbba8b80 20 client.16131 populate_metadata read hostname 'smithi116' 2024-03-30T01:03:56.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.733+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56338cd54310 0x56338cd56730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:03:56.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.733+0000 7f1fcbba8b80 4 client.16131 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:03:56.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.733+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2062288679 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56338ca89cf0 con 0x56338cd54310 2024-03-30T01:03:56.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.734+0000 7f1fb4c00700 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56338cd54310 0x56338cd56730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:03:56.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.747+0000 7f1fb4c00700 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56338cd54310 0x56338cd56730 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8464182bfd4fd410 server_cookie=80c9bf8c779206 in_seq=0 out_seq=0 2024-03-30T01:03:56.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.757+0000 7f1f96a00700 10 client.16131 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:03:56.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.757+0000 7f1f96a00700 1 -- 172.21.15.116:0/2062288679 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56338ca89cf0 con 0x56338cd54310 2024-03-30T01:03:56.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.757+0000 7f1f96a00700 10 client.16131 handle_command_reply: tid=0 2024-03-30T01:03:56.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.761+0000 7f1fcbba8b80 1 client.16131 shutdown 2024-03-30T01:03:56.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.783+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2062288679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cce97d0 msgr2=0x56338ccf8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:56.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.783+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cce97d0 0x56338ccf8f70 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c003ff0 tx=0x7f1f9c09a620 comp rx=0 tx=0).stop 2024-03-30T01:03:56.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.797+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2062288679 shutdown_connections 2024-03-30T01:03:56.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.797+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56338cd54310 0x56338cd56730 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.798+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56338cceafd0 0x56338ccfa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.798+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338cce97d0 0x56338ccf8f70 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.798+0000 7f1fcbba8b80 1 --2- 172.21.15.116:0/2062288679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56338ccea130 0x56338ccf94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.798+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2062288679 >> 172.21.15.116:0/2062288679 conn(0x56338cc486a0 msgr2=0x56338ccf2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:56.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.798+0000 7f1f96a00700 0 client.16131 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:03:56.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.811+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2062288679 shutdown_connections 2024-03-30T01:03:56.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.811+0000 7f1fcbba8b80 1 -- 172.21.15.116:0/2062288679 wait complete. 2024-03-30T01:03:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.815+0000 7f1fcbba8b80 20 client.16131 trim_cache size 0 max 16384 2024-03-30T01:03:56.822 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:03:56.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:56 smithi130 ceph-mon[47626]: pgmap v473: 129 pgs: 129 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 13 MiB/s wr, 811 op/s 2024-03-30T01:03:56.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.875+0000 7f1fb8000700 1 -- 172.21.15.116:0/3733812864 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1f980a12a0 msgr2=0x7f1f980a3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:56.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.875+0000 7f1fb8000700 1 --2- 172.21.15.116:0/3733812864 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1f980a12a0 0x7f1f980a3760 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009fd0 tx=0x7f1fa400b5f0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.875+0000 7f1fb8000700 1 -- 172.21.15.116:0/3733812864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fb0107b90 msgr2=0x7f1fb01ae3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:03:56.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.875+0000 7f1fb8000700 1 --2- 172.21.15.116:0/3733812864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fb0107b90 0x7f1fb01ae3b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c005fd0 tx=0x7f1f9c00eca0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.878+0000 7f1fb8000700 1 -- 172.21.15.116:0/3733812864 shutdown_connections 2024-03-30T01:03:56.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.878+0000 7f1fb8000700 1 --2- 172.21.15.116:0/3733812864 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1f980a12a0 0x7f1f980a3760 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.878+0000 7f1fb8000700 1 --2- 172.21.15.116:0/3733812864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fb010aad0 0x7f1fb01b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.878+0000 7f1fb8000700 1 --2- 172.21.15.116:0/3733812864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fb0107b90 0x7f1fb01ae3b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.878+0000 7f1fb8000700 1 --2- 172.21.15.116:0/3733812864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb010a590 0x7f1fb01ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:03:56.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.878+0000 7f1fb8000700 1 -- 172.21.15.116:0/3733812864 >> 172.21.15.116:0/3733812864 conn(0x7f1fb006dbc0 msgr2=0x7f1fb006feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:03:56.929 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:56 smithi116 ceph-mon[38086]: pgmap v473: 129 pgs: 129 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 13 MiB/s wr, 811 op/s 2024-03-30T01:03:56.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.903+0000 7f1fb8000700 1 -- 172.21.15.116:0/3733812864 shutdown_connections 2024-03-30T01:03:56.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:03:56.910+0000 7f1fb8000700 1 -- 172.21.15.116:0/3733812864 wait complete. 2024-03-30T01:03:57.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:56 smithi181 ceph-mon[47622]: pgmap v473: 129 pgs: 129 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 13 MiB/s wr, 811 op/s 2024-03-30T01:03:58.589 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T01:03:58.589 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T01:03:58.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:03:58 smithi130 ceph-mon[47626]: pgmap v474: 129 pgs: 129 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 9.9 MiB/s wr, 640 op/s 2024-03-30T01:03:58.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:03:58 smithi116 ceph-mon[38086]: pgmap v474: 129 pgs: 129 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 9.9 MiB/s wr, 640 op/s 2024-03-30T01:03:59.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:03:58 smithi181 ceph-mon[47622]: pgmap v474: 129 pgs: 129 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 9.9 MiB/s wr, 640 op/s 2024-03-30T01:03:59.061 INFO:tasks.workunit.client.0.smithi116.stdout: 456 440923 493 315473 331 181303 709 2024-03-30T01:04:00.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:00 smithi130 ceph-mon[47626]: pgmap v475: 129 pgs: 129 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 9.3 MiB/s wr, 629 op/s 2024-03-30T01:04:00.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:00 smithi116 ceph-mon[38086]: pgmap v475: 129 pgs: 129 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 9.3 MiB/s wr, 629 op/s 2024-03-30T01:04:01.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:00 smithi181 ceph-mon[47622]: pgmap v475: 129 pgs: 129 active+clean; 1.9 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 9.3 MiB/s wr, 629 op/s 2024-03-30T01:04:02.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:02 smithi130 ceph-mon[47626]: pgmap v476: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.3 MiB/s wr, 521 op/s 2024-03-30T01:04:02.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:02 smithi116 ceph-mon[38086]: pgmap v476: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.3 MiB/s wr, 521 op/s 2024-03-30T01:04:03.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:02 smithi181 ceph-mon[47622]: pgmap v476: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.3 MiB/s wr, 521 op/s 2024-03-30T01:04:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.562+0000 7fbbb6000700 1 -- 172.21.15.116:0/3137499952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb010a720 msgr2=0x7fbbb010cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.562+0000 7fbbb6000700 1 --2- 172.21.15.116:0/3137499952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb010a720 0x7fbbb010cbb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fbb9801ac00 tx=0x7fbb9801af10 comp rx=0 tx=0).stop 2024-03-30T01:04:04.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.585+0000 7fbbb6000700 1 -- 172.21.15.116:0/3137499952 shutdown_connections 2024-03-30T01:04:04.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.585+0000 7fbbb6000700 1 --2- 172.21.15.116:0/3137499952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbb0107db0 0x7fbbb010a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.585+0000 7fbbb6000700 1 --2- 172.21.15.116:0/3137499952 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbb010d0f0 0x7fbbb010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.585+0000 7fbbb6000700 1 --2- 172.21.15.116:0/3137499952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb010a720 0x7fbbb010cbb0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.585+0000 7fbbb6000700 1 -- 172.21.15.116:0/3137499952 >> 172.21.15.116:0/3137499952 conn(0x7fbbb006dd50 msgr2=0x7fbbb0070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:04.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.585+0000 7fbbb6000700 1 -- 172.21.15.116:0/3137499952 shutdown_connections 2024-03-30T01:04:04.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.585+0000 7fbbb6000700 1 -- 172.21.15.116:0/3137499952 wait complete. 2024-03-30T01:04:04.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.593+0000 7fbbb6000700 1 Processor -- start 2024-03-30T01:04:04.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.597+0000 7fbbb6000700 1 -- start start 2024-03-30T01:04:04.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.602+0000 7fbbb6000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbb0107db0 0x7fbbb019d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:04.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.602+0000 7fbbb6000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbb010a720 0x7fbbb019d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.603+0000 7fbbb6000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb010d0f0 0x7fbbb01a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.603+0000 7fbbaf400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbb0107db0 0x7fbbb019d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.603+0000 7fbbaf400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbb0107db0 0x7fbbb019d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:41246/0 (socket says 172.21.15.116:41246) 2024-03-30T01:04:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.604+0000 7fbbaea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbb010a720 0x7fbbb019d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.606+0000 7fbbafe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb010d0f0 0x7fbbb01a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.607+0000 7fbbafe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb010d0f0 0x7fbbb01a4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41454/0 (socket says 172.21.15.116:41454) 2024-03-30T01:04:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.607+0000 7fbbaea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbb010a720 0x7fbbb019d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:56288/0 (socket says 172.21.15.116:56288) 2024-03-30T01:04:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.607+0000 7fbbaea00700 1 -- 172.21.15.116:0/1302772333 learned_addr learned my addr 172.21.15.116:0/1302772333 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.607+0000 7fbbb6000700 1 -- 172.21.15.116:0/1302772333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb01a4f60 con 0x7fbbb010d0f0 2024-03-30T01:04:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.607+0000 7fbbb6000700 1 -- 172.21.15.116:0/1302772333 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb01a50a0 con 0x7fbbb010a720 2024-03-30T01:04:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.607+0000 7fbbb6000700 1 -- 172.21.15.116:0/1302772333 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb01a5230 con 0x7fbbb0107db0 2024-03-30T01:04:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.609+0000 7fbbaea00700 1 -- 172.21.15.116:0/1302772333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbb0107db0 msgr2=0x7fbbb019d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.609+0000 7fbbaea00700 1 --2- 172.21.15.116:0/1302772333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbb0107db0 0x7fbbb019d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.609+0000 7fbbaea00700 1 -- 172.21.15.116:0/1302772333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb010d0f0 msgr2=0x7fbbb01a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.609+0000 7fbbaea00700 1 --2- 172.21.15.116:0/1302772333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb010d0f0 0x7fbbb01a4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.609+0000 7fbbaea00700 1 -- 172.21.15.116:0/1302772333 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9801a600 con 0x7fbbb010a720 2024-03-30T01:04:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.609+0000 7fbbafe00700 1 --2- 172.21.15.116:0/1302772333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb010d0f0 0x7fbbb01a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-30T01:04:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.619+0000 7fbbaea00700 1 --2- 172.21.15.116:0/1302772333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbb010a720 0x7fbbb019d970 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbb98005780 tx=0x7fbb980057b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:04.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.632+0000 7fbba7e00700 1 -- 172.21.15.116:0/1302772333 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9802d070 con 0x7fbbb010a720 2024-03-30T01:04:04.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.632+0000 7fbba7e00700 1 -- 172.21.15.116:0/1302772333 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb98031410 con 0x7fbbb010a720 2024-03-30T01:04:04.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.632+0000 7fbba7e00700 1 -- 172.21.15.116:0/1302772333 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb980283f0 con 0x7fbbb010a720 2024-03-30T01:04:04.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.649+0000 7fbbb6000700 1 -- 172.21.15.116:0/1302772333 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb01a54b0 con 0x7fbbb010a720 2024-03-30T01:04:04.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.649+0000 7fbbb6000700 1 -- 172.21.15.116:0/1302772333 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb01a5880 con 0x7fbbb010a720 2024-03-30T01:04:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.691+0000 7fbba7e00700 1 -- 172.21.15.116:0/1302772333 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fbb98031580 con 0x7fbbb010a720 2024-03-30T01:04:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.693+0000 7fbba7e00700 1 --2- 172.21.15.116:0/1302772333 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbb940a1320 0x7fbb940a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.693+0000 7fbba7e00700 1 -- 172.21.15.116:0/1302772333 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbb980de550 con 0x7fbbb010a720 2024-03-30T01:04:04.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.705+0000 7fbbaf400700 1 --2- 172.21.15.116:0/1302772333 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbb940a1320 0x7fbb940a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:04.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.706+0000 7fbbaf400700 1 --2- 172.21.15.116:0/1302772333 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbb940a1320 0x7fbb940a37e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbba800b4a0 tx=0x7fbba800d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.725+0000 7fbbc9b01b80 1 Processor -- start 2024-03-30T01:04:04.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.772+0000 7fbbc9b01b80 1 -- start start 2024-03-30T01:04:04.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.785+0000 7fbbc9b01b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aab8850 0x55e48aab8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:04.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.785+0000 7fbbc9b01b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab91b0 0x55e48aab9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:04.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.785+0000 7fbbc9b01b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aaba050 0x55e48aaba4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:04.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.785+0000 7fbbc9b01b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48aa10a10 con 0x55e48aab91b0 2024-03-30T01:04:04.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.785+0000 7fbbc9b01b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48aa0e3d0 con 0x55e48aaba050 2024-03-30T01:04:04.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.785+0000 7fbbc9b01b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48aa09cc0 con 0x55e48aab8850 2024-03-30T01:04:04.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.794+0000 7fbbafe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aaba050 0x55e48aaba4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:04.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.794+0000 7fbbafe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aaba050 0x55e48aaba4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:56304/0 (socket says 172.21.15.116:56304) 2024-03-30T01:04:04.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.794+0000 7fbbafe00700 1 -- 172.21.15.116:0/538962537 learned_addr learned my addr 172.21.15.116:0/538962537 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:04.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.796+0000 7fbbaea00700 1 --2- 172.21.15.116:0/538962537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab91b0 0x55e48aab9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.797+0000 7fbbaea00700 1 -- 172.21.15.116:0/538962537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aab8850 msgr2=0x55e48aab8c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:04:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.797+0000 7fbbaea00700 1 --2- 172.21.15.116:0/538962537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aab8850 0x55e48aab8c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.797+0000 7fbbaea00700 1 -- 172.21.15.116:0/538962537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aaba050 msgr2=0x55e48aaba4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.797+0000 7fbbaea00700 1 --2- 172.21.15.116:0/538962537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aaba050 0x55e48aaba4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.797+0000 7fbbaea00700 1 -- 172.21.15.116:0/538962537 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e48a88b430 con 0x55e48aab91b0 2024-03-30T01:04:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.797+0000 7fbbaea00700 1 --2- 172.21.15.116:0/538962537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab91b0 0x55e48aab9a70 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbb98005750 tx=0x7fbb980a7b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a6eb0642865558 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.797+0000 7fbb9fe00700 1 -- 172.21.15.116:0/538962537 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb980abdc0 con 0x55e48aab91b0 2024-03-30T01:04:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.798+0000 7fbb9fe00700 1 -- 172.21.15.116:0/538962537 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb980ab6c0 con 0x55e48aab91b0 2024-03-30T01:04:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.798+0000 7fbb9fe00700 1 -- 172.21.15.116:0/538962537 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb980af170 con 0x55e48aab91b0 2024-03-30T01:04:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.799+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/538962537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab91b0 msgr2=0x55e48aab9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.799+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/538962537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab91b0 0x55e48aab9a70 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbb98005750 tx=0x7fbb980a7b30 comp rx=0 tx=0).stop 2024-03-30T01:04:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.799+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/538962537 shutdown_connections 2024-03-30T01:04:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.799+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/538962537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aab8850 0x55e48aab8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.799+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/538962537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aaba050 0x55e48aaba4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.799+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/538962537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab91b0 0x55e48aab9a70 secure :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbb98005750 tx=0x7fbb980a7b30 comp rx=0 tx=0).stop 2024-03-30T01:04:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.799+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/538962537 >> 172.21.15.116:0/538962537 conn(0x55e48aa17720 msgr2=0x55e48aab3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.800+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/538962537 shutdown_connections 2024-03-30T01:04:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.800+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/538962537 wait complete. 2024-03-30T01:04:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.801+0000 7fbbc9b01b80 1 -- start start 2024-03-30T01:04:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.802+0000 7fbbc9b01b80 1 Processor -- start 2024-03-30T01:04:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.803+0000 7fbbc9b01b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aab8850 0x55e48aae4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.804+0000 7fbbaf400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aab8850 0x55e48aae4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.804+0000 7fbbaf400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aab8850 0x55e48aae4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:41258/0 (socket says 172.21.15.116:41258) 2024-03-30T01:04:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.804+0000 7fbbaf400700 1 -- 172.21.15.116:0/1805195296 learned_addr learned my addr 172.21.15.116:0/1805195296 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.804+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aaba050 0x55e48aae4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.805+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aae9b40 0x55e48aaebf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.805+0000 7fbbaea00700 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aaba050 0x55e48aae4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.805+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1805195296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48a9a5e00 con 0x55e48aaba050 2024-03-30T01:04:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.805+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1805195296 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48a9a5aa0 con 0x55e48aae9b40 2024-03-30T01:04:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.805+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1805195296 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48a88d720 con 0x55e48aab8850 2024-03-30T01:04:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.805+0000 7fbbaf400700 1 -- 172.21.15.116:0/1805195296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aae9b40 msgr2=0x55e48aaebf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.806+0000 7fbbafe00700 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aae9b40 0x55e48aaebf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.806+0000 7fbbaf400700 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aae9b40 0x55e48aaebf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.806+0000 7fbbaf400700 1 -- 172.21.15.116:0/1805195296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aaba050 msgr2=0x55e48aae4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:04.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.806+0000 7fbbaf400700 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aaba050 0x55e48aae4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.806+0000 7fbbaf400700 1 -- 172.21.15.116:0/1805195296 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb01a5880 con 0x55e48aab8850 2024-03-30T01:04:04.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.807+0000 7fbbafe00700 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aae9b40 0x55e48aaebf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:04:04.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.807+0000 7fbbaea00700 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aaba050 0x55e48aae4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:04:04.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.807+0000 7fbbc9b01b80 10 client.0 _ll_register_callbacks cb 0x55e48a9af4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:04.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.807+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1805195296 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e48a9ca400 con 0x55e48aab8850 2024-03-30T01:04:04.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.808+0000 7fbbaf400700 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aab8850 0x55e48aae4620 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbba8016e00 tx=0x7fbba8004b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=261cde9a4a41d0e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:04.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.808+0000 7fbb9cc00700 10 client.26131 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:04:04.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.816+0000 7fbb9cc00700 1 -- 172.21.15.116:0/1805195296 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba8010040 con 0x55e48aab8850 2024-03-30T01:04:04.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.816+0000 7fbb9cc00700 1 -- 172.21.15.116:0/1805195296 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbba8009c50 con 0x55e48aab8850 2024-03-30T01:04:04.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.816+0000 7fbb9cc00700 1 -- 172.21.15.116:0/1805195296 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbba802a450 con 0x55e48aab8850 2024-03-30T01:04:04.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.816+0000 7fbbc9b01b80 10 client.26131 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:04.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.816+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1805195296 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({fsmap=28}) v3 -- 0x55e48a837be0 con 0x55e48aab8850 2024-03-30T01:04:04.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.817+0000 7fbb9cc00700 1 -- 172.21.15.116:0/1805195296 <== mon.1 v2:172.21.15.181:3300/0 4 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7fbba801f070 con 0x55e48aab8850 2024-03-30T01:04:04.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.817+0000 7fbbc9b01b80 10 client.26131 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:04.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.817+0000 7fbbc9b01b80 10 client.26131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:04:04.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.817+0000 7fbbc9b01b80 20 client.26131 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:04.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.817+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e48aac50f0 0x55e48aac75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:04.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.817+0000 7fbbc9b01b80 4 client.26131 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:04:04.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.817+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1805195296 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e48aa02780 con 0x55e48aac50f0 2024-03-30T01:04:04.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.817+0000 7fbbaea00700 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e48aac50f0 0x55e48aac75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.831+0000 7fbbaea00700 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e48aac50f0 0x55e48aac75b0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f6da7181234ed02 server_cookie=6d5953dad1bd06b7 in_seq=0 out_seq=0 2024-03-30T01:04:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.835+0000 7fbb9cc00700 10 client.26131 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.835+0000 7fbb9cc00700 1 -- 172.21.15.116:0/1805195296 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e48aa02780 con 0x55e48aac50f0 2024-03-30T01:04:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.835+0000 7fbb9cc00700 10 client.26131 handle_command_reply: tid=0 2024-03-30T01:04:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.839+0000 7fbbc9b01b80 1 client.26131 shutdown 2024-03-30T01:04:04.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:04 smithi130 ceph-mon[47626]: pgmap v477: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.0 MiB/s wr, 379 op/s 2024-03-30T01:04:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.839+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1805195296 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aab8850 msgr2=0x55e48aae4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.839+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aab8850 0x55e48aae4620 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbba8016e00 tx=0x7fbba8004b80 comp rx=0 tx=0).stop 2024-03-30T01:04:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.840+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1805195296 shutdown_connections 2024-03-30T01:04:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.840+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e48aac50f0 0x55e48aac75b0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.840+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aab8850 0x55e48aae4620 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.840+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aae9b40 0x55e48aaebf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.840+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/1805195296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aaba050 0x55e48aae4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.840+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1805195296 >> 172.21.15.116:0/1805195296 conn(0x55e48aa17720 msgr2=0x55e48aab71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.840+0000 7fbb9cc00700 0 client.26131 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.840+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1805195296 shutdown_connections 2024-03-30T01:04:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.840+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1805195296 wait complete. 2024-03-30T01:04:04.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:04.841+0000 7fbbc9b01b80 20 client.26131 trim_cache size 0 max 16384 2024-03-30T01:04:04.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:04 smithi116 ceph-mon[38086]: pgmap v477: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.0 MiB/s wr, 379 op/s 2024-03-30T01:04:05.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:04 smithi181 ceph-mon[47622]: pgmap v477: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.0 MiB/s wr, 379 op/s 2024-03-30T01:04:05.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.074+0000 7fbbc9b01b80 1 Processor -- start 2024-03-30T01:04:05.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.131+0000 7fbbc9b01b80 1 -- start start 2024-03-30T01:04:05.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.135+0000 7fbbc9b01b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab8850 0x55e48aad4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:05.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.135+0000 7fbbc9b01b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aaba050 0x55e48aae43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:05.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.138+0000 7fbbc9b01b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aae9b40 0x55e48aae4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:05.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.138+0000 7fbbc9b01b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48aaaf750 con 0x55e48aab8850 2024-03-30T01:04:05.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.138+0000 7fbbc9b01b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48aa02780 con 0x55e48aae9b40 2024-03-30T01:04:05.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.138+0000 7fbbc9b01b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48aab4530 con 0x55e48aaba050 2024-03-30T01:04:05.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.141+0000 7fbbaf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab8850 0x55e48aad4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:05.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.142+0000 7fbbafe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aae9b40 0x55e48aae4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:05.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.142+0000 7fbbaf400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab8850 0x55e48aad4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52332/0 (socket says 172.21.15.116:52332) 2024-03-30T01:04:05.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.142+0000 7fbbaf400700 1 -- 172.21.15.116:0/1380315877 learned_addr learned my addr 172.21.15.116:0/1380315877 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:05.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.142+0000 7fbbaf400700 1 -- 172.21.15.116:0/1380315877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aaba050 msgr2=0x55e48aae43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:05.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.142+0000 7fbbaf400700 1 --2- 172.21.15.116:0/1380315877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aaba050 0x55e48aae43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.143+0000 7fbbaf400700 1 -- 172.21.15.116:0/1380315877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aae9b40 msgr2=0x55e48aae4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:05.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.143+0000 7fbbaf400700 1 --2- 172.21.15.116:0/1380315877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aae9b40 0x55e48aae4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.143+0000 7fbbaf400700 1 -- 172.21.15.116:0/1380315877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e48aaac8f0 con 0x55e48aab8850 2024-03-30T01:04:05.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.146+0000 7fbbaf400700 1 --2- 172.21.15.116:0/1380315877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab8850 0x55e48aad4dd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbba800bbf0 tx=0x7fbba8018070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0fee8f37ae7682a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:05.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.208+0000 7fbba5200700 1 -- 172.21.15.116:0/1380315877 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba8010040 con 0x55e48aab8850 2024-03-30T01:04:05.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.209+0000 7fbba5200700 1 -- 172.21.15.116:0/1380315877 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e48a9ca400 con 0x55e48aab8850 2024-03-30T01:04:05.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.209+0000 7fbba5200700 1 -- 172.21.15.116:0/1380315877 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba8020bd0 con 0x55e48aab8850 2024-03-30T01:04:05.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.239+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1380315877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab8850 msgr2=0x55e48aad4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:05.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.239+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/1380315877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab8850 0x55e48aad4dd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbba800bbf0 tx=0x7fbba8018070 comp rx=0 tx=0).stop 2024-03-30T01:04:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.291+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1380315877 shutdown_connections 2024-03-30T01:04:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.291+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/1380315877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aae9b40 0x55e48aae4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.291+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/1380315877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab8850 0x55e48aad4dd0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.291+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/1380315877 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aaba050 0x55e48aae43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.291+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1380315877 >> 172.21.15.116:0/1380315877 conn(0x55e48aa17720 msgr2=0x55e48aabd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.302+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1380315877 shutdown_connections 2024-03-30T01:04:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.302+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/1380315877 wait complete. 2024-03-30T01:04:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.302+0000 7fbbc9b01b80 1 -- start start 2024-03-30T01:04:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.303+0000 7fbbc9b01b80 1 Processor -- start 2024-03-30T01:04:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.329+0000 7fbbc9b01b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab8850 0x55e48ab1b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.329+0000 7fbbc9b01b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aaba050 0x55e48ab1bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.329+0000 7fbbc9b01b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aae9b40 0x55e48ab22b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.329+0000 7fbbc9b01b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48ab23080 con 0x55e48aab8850 2024-03-30T01:04:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.331+0000 7fbbc9b01b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48ab231c0 con 0x55e48aaba050 2024-03-30T01:04:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.331+0000 7fbbc9b01b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48ab23350 con 0x55e48aae9b40 2024-03-30T01:04:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.331+0000 7fbbc9b01b80 10 client.0 _ll_register_callbacks cb 0x55e48a9af4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.337+0000 7fbbaea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aaba050 0x55e48ab1bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.337+0000 7fbbaea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aaba050 0x55e48ab1bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:58702/0 (socket says 172.21.15.116:58702) 2024-03-30T01:04:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.337+0000 7fbbaea00700 1 -- 172.21.15.116:0/3087808600 learned_addr learned my addr 172.21.15.116:0/3087808600 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.337+0000 7fbbaf400700 1 --2- 172.21.15.116:0/3087808600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab8850 0x55e48ab1b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.343+0000 7fbbaea00700 1 -- 172.21.15.116:0/3087808600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aae9b40 msgr2=0x55e48ab22b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.343+0000 7fbbaea00700 1 --2- 172.21.15.116:0/3087808600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aae9b40 0x55e48ab22b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.343+0000 7fbbaea00700 1 -- 172.21.15.116:0/3087808600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab8850 msgr2=0x55e48ab1b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.343+0000 7fbbaea00700 1 --2- 172.21.15.116:0/3087808600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab8850 0x55e48ab1b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.343+0000 7fbbaea00700 1 -- 172.21.15.116:0/3087808600 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e48a837be0 con 0x55e48aaba050 2024-03-30T01:04:05.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.373+0000 7fbbaea00700 1 --2- 172.21.15.116:0/3087808600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aaba050 0x55e48ab1bb50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbb98005750 tx=0x7fbb980a7340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9677bd1ec465edd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:05.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.373+0000 7fbb9cc00700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T01:04:05.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.376+0000 7fbb9cc00700 1 -- 172.21.15.116:0/3087808600 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb980a9ae0 con 0x55e48aaba050 2024-03-30T01:04:05.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.376+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/3087808600 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb980ab6c0 con 0x55e48aaba050 2024-03-30T01:04:05.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.376+0000 7fbb9cc00700 1 -- 172.21.15.116:0/3087808600 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbb980af4f0 con 0x55e48aaba050 2024-03-30T01:04:05.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.378+0000 7fbb9cc00700 1 -- 172.21.15.116:0/3087808600 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb980b3ab0 con 0x55e48aaba050 2024-03-30T01:04:05.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.378+0000 7fbbc9b01b80 10 client.26135 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:05.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.378+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/3087808600 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e48a9cff20 con 0x55e48aaba050 2024-03-30T01:04:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.378+0000 7fbb9cc00700 1 -- 172.21.15.116:0/3087808600 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb980bf2a0 con 0x55e48aaba050 2024-03-30T01:04:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.378+0000 7fbb9cc00700 1 -- 172.21.15.116:0/3087808600 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7fbb980ab920 con 0x55e48aaba050 2024-03-30T01:04:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.379+0000 7fbbc9b01b80 10 client.26135 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.379+0000 7fbbc9b01b80 10 client.26135 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:04:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.379+0000 7fbbc9b01b80 20 client.26135 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.379+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/3087808600 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e48ab249d0 0x55e48ab26df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.379+0000 7fbbc9b01b80 4 client.26135 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T01:04:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.379+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/3087808600 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x7fbb980ab6c0 con 0x55e48ab249d0 2024-03-30T01:04:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.383+0000 7fbbaf400700 1 --2- 172.21.15.116:0/3087808600 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e48ab249d0 0x55e48ab26df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.392+0000 7fbbaf400700 1 --2- 172.21.15.116:0/3087808600 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e48ab249d0 0x55e48ab26df0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed24412d19d8a364 server_cookie=cb075100ee944e09 in_seq=0 out_seq=0 2024-03-30T01:04:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.392+0000 7fbb9cc00700 10 client.26135 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.426+0000 7fbb9cc00700 1 -- 172.21.15.116:0/3087808600 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7fbb980ab6c0 con 0x55e48ab249d0 2024-03-30T01:04:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.426+0000 7fbb9cc00700 10 client.26135 handle_command_reply: tid=0 2024-03-30T01:04:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.426+0000 7fbbc9b01b80 1 client.26135 shutdown 2024-03-30T01:04:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.427+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/3087808600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aaba050 msgr2=0x55e48ab1bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.427+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/3087808600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aaba050 0x55e48ab1bb50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbb98005750 tx=0x7fbb980a7340 comp rx=0 tx=0).stop 2024-03-30T01:04:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.427+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/3087808600 shutdown_connections 2024-03-30T01:04:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.427+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/3087808600 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55e48ab249d0 0x55e48ab26df0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.427+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/3087808600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e48aae9b40 0x55e48ab22b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.427+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/3087808600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e48aab8850 0x55e48ab1b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.427+0000 7fbbc9b01b80 1 --2- 172.21.15.116:0/3087808600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e48aaba050 0x55e48ab1bb50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.427+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/3087808600 >> 172.21.15.116:0/3087808600 conn(0x55e48aa17720 msgr2=0x55e48aae43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.428+0000 7fbb9cc00700 0 client.26135 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.428+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/3087808600 shutdown_connections 2024-03-30T01:04:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.428+0000 7fbbc9b01b80 1 -- 172.21.15.116:0/3087808600 wait complete. 2024-03-30T01:04:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.429+0000 7fbbc9b01b80 20 client.26135 trim_cache size 0 max 16384 2024-03-30T01:04:05.435 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:04:05.435 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-30T01:04:05.435 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "1ef8a851-5fc9-428f-905f-d08d42299f02", 2024-03-30T01:04:05.435 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-30T01:04:05.435 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:04:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.433+0000 7fbbb6000700 1 -- 172.21.15.116:0/1302772333 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbb940a1320 msgr2=0x7fbb940a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.433+0000 7fbbb6000700 1 --2- 172.21.15.116:0/1302772333 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbb940a1320 0x7fbb940a37e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbba800b4a0 tx=0x7fbba800d040 comp rx=0 tx=0).stop 2024-03-30T01:04:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.433+0000 7fbbb6000700 1 -- 172.21.15.116:0/1302772333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbb010a720 msgr2=0x7fbbb019d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.433+0000 7fbbb6000700 1 --2- 172.21.15.116:0/1302772333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbb010a720 0x7fbbb019d970 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbb98005780 tx=0x7fbb980057b0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.434+0000 7fbbb6000700 1 -- 172.21.15.116:0/1302772333 shutdown_connections 2024-03-30T01:04:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.434+0000 7fbbb6000700 1 --2- 172.21.15.116:0/1302772333 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbb940a1320 0x7fbb940a37e0 secure :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbba800b4a0 tx=0x7fbba800d040 comp rx=0 tx=0).stop 2024-03-30T01:04:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.434+0000 7fbbb6000700 1 --2- 172.21.15.116:0/1302772333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbb0107db0 0x7fbbb019d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.434+0000 7fbbb6000700 1 --2- 172.21.15.116:0/1302772333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb010d0f0 0x7fbbb01a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.434+0000 7fbbb6000700 1 --2- 172.21.15.116:0/1302772333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbb010a720 0x7fbbb019d970 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.434+0000 7fbbb6000700 1 -- 172.21.15.116:0/1302772333 >> 172.21.15.116:0/1302772333 conn(0x7fbbb006dd50 msgr2=0x7fbbb010afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.435+0000 7fbbb6000700 1 -- 172.21.15.116:0/1302772333 shutdown_connections 2024-03-30T01:04:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:05.435+0000 7fbbb6000700 1 -- 172.21.15.116:0/1302772333 wait complete. 2024-03-30T01:04:05.994 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T01:04:06.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:06 smithi116 ceph-mon[38086]: pgmap v478: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.9 MiB/s wr, 373 op/s 2024-03-30T01:04:06.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:06 smithi116 ceph-mon[38086]: scrub queued for path: / 2024-03-30T01:04:06.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:06 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:06.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:06 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:04:07.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:06 smithi181 ceph-mon[47622]: pgmap v478: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.9 MiB/s wr, 373 op/s 2024-03-30T01:04:07.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:06 smithi181 ceph-mon[47622]: scrub queued for path: / 2024-03-30T01:04:07.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:06 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:07.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:06 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:04:07.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:06 smithi130 ceph-mon[47626]: pgmap v478: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.9 MiB/s wr, 373 op/s 2024-03-30T01:04:07.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:06 smithi130 ceph-mon[47626]: scrub queued for path: / 2024-03-30T01:04:07.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:06 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:07.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:06 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:04:08.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:08 smithi116 ceph-mon[38086]: pgmap v479: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.4 MiB/s wr, 334 op/s 2024-03-30T01:04:08.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:08 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:04:08.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:08 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:09.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:08 smithi181 ceph-mon[47622]: pgmap v479: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.4 MiB/s wr, 334 op/s 2024-03-30T01:04:09.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:08 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:04:09.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:08 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:09.073 INFO:tasks.workunit.client.0.smithi116.stdout: 478 511269 1482 364525 1157 206956 1499 2024-03-30T01:04:09.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:08 smithi130 ceph-mon[47626]: pgmap v479: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.4 MiB/s wr, 334 op/s 2024-03-30T01:04:09.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:08 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:04:09.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:08 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:09.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:09 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:04:09.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:09 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:09.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:09 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:04:09.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:09 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:04:10.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:09 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:04:10.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:09 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:10.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:09 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:04:10.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:09 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:04:10.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:09 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:04:10.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:09 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:10.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:09 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:04:10.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:09 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:04:10.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:10 smithi116 ceph-mon[38086]: pgmap v480: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.2 MiB/s wr, 389 op/s 2024-03-30T01:04:11.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:10 smithi181 ceph-mon[47622]: pgmap v480: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.2 MiB/s wr, 389 op/s 2024-03-30T01:04:11.117 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:10 smithi130 ceph-mon[47626]: pgmap v480: 129 pgs: 129 active+clean; 1.9 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.2 MiB/s wr, 389 op/s 2024-03-30T01:04:12.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:12 smithi116 ceph-mon[38086]: pgmap v481: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.2 MiB/s wr, 395 op/s 2024-03-30T01:04:13.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:12 smithi181 ceph-mon[47622]: pgmap v481: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.2 MiB/s wr, 395 op/s 2024-03-30T01:04:13.221 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:12 smithi130 ceph-mon[47626]: pgmap v481: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.2 MiB/s wr, 395 op/s 2024-03-30T01:04:14.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.442+0000 7f3f63600700 1 -- 172.21.15.116:0/3217676890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f5c10d0f0 msgr2=0x7f3f5c10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:14.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.442+0000 7f3f63600700 1 --2- 172.21.15.116:0/3217676890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f5c10d0f0 0x7f3f5c10d570 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400b3e0 tx=0x7f3f5400b6f0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.465+0000 7f3f63600700 1 -- 172.21.15.116:0/3217676890 shutdown_connections 2024-03-30T01:04:14.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.465+0000 7f3f63600700 1 --2- 172.21.15.116:0/3217676890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f5c107db0 0x7f3f5c10a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.465+0000 7f3f63600700 1 --2- 172.21.15.116:0/3217676890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f5c10d0f0 0x7f3f5c10d570 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.465+0000 7f3f63600700 1 --2- 172.21.15.116:0/3217676890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f5c10a720 0x7f3f5c10cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.465+0000 7f3f63600700 1 -- 172.21.15.116:0/3217676890 >> 172.21.15.116:0/3217676890 conn(0x7f3f5c06dd50 msgr2=0x7f3f5c070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:14.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.491+0000 7f3f63600700 1 -- 172.21.15.116:0/3217676890 shutdown_connections 2024-03-30T01:04:14.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.512+0000 7f3f63600700 1 -- 172.21.15.116:0/3217676890 wait complete. 2024-03-30T01:04:14.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.514+0000 7f3f63600700 1 Processor -- start 2024-03-30T01:04:14.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.519+0000 7f3f63600700 1 -- start start 2024-03-30T01:04:14.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.532+0000 7f3f63600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f5c107db0 0x7f3f5c19d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:14.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.533+0000 7f3f63600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f5c10a720 0x7f3f5c19d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:14.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.534+0000 7f3f5be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f5c10a720 0x7f3f5c19d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:14.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.534+0000 7f3f5be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f5c10a720 0x7f3f5c19d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:58708/0 (socket says 172.21.15.116:58708) 2024-03-30T01:04:14.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.534+0000 7f3f5be00700 1 -- 172.21.15.116:0/1978147097 learned_addr learned my addr 172.21.15.116:0/1978147097 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:14.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.534+0000 7f3f63600700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f5c10d0f0 0x7f3f5c1a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:14.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.536+0000 7f3f60c00700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f5c107db0 0x7f3f5c19d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:14.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.540+0000 7f3f63600700 1 -- 172.21.15.116:0/1978147097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c1a4db0 con 0x7f3f5c107db0 2024-03-30T01:04:14.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.540+0000 7f3f63600700 1 -- 172.21.15.116:0/1978147097 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c1a4ef0 con 0x7f3f5c10a720 2024-03-30T01:04:14.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.540+0000 7f3f63600700 1 -- 172.21.15.116:0/1978147097 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c1a5080 con 0x7f3f5c10d0f0 2024-03-30T01:04:14.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.540+0000 7f3f61600700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f5c10d0f0 0x7f3f5c1a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:14.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.541+0000 7f3f61600700 1 -- 172.21.15.116:0/1978147097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f5c10a720 msgr2=0x7f3f5c19d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:14.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.541+0000 7f3f61600700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f5c10a720 0x7f3f5c19d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.541+0000 7f3f61600700 1 -- 172.21.15.116:0/1978147097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f5c107db0 msgr2=0x7f3f5c19d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:14.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.541+0000 7f3f61600700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f5c107db0 0x7f3f5c19d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.541+0000 7f3f61600700 1 -- 172.21.15.116:0/1978147097 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f54009c40 con 0x7f3f5c10d0f0 2024-03-30T01:04:14.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.548+0000 7f3f60c00700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f5c107db0 0x7f3f5c19d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:04:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.560+0000 7f3f61600700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f5c10d0f0 0x7f3f5c1a47e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400b3e0 tx=0x7f3f54005ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.576+0000 7f3f59600700 1 -- 172.21.15.116:0/1978147097 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5400e040 con 0x7f3f5c10d0f0 2024-03-30T01:04:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.576+0000 7f3f59600700 1 -- 172.21.15.116:0/1978147097 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f54012e80 con 0x7f3f5c10d0f0 2024-03-30T01:04:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.576+0000 7f3f59600700 1 -- 172.21.15.116:0/1978147097 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5401de90 con 0x7f3f5c10d0f0 2024-03-30T01:04:14.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.603+0000 7f3f63600700 1 -- 172.21.15.116:0/1978147097 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5c1a5300 con 0x7f3f5c10d0f0 2024-03-30T01:04:14.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.604+0000 7f3f63600700 1 -- 172.21.15.116:0/1978147097 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5c1a5730 con 0x7f3f5c10d0f0 2024-03-30T01:04:14.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.712+0000 7f3f59600700 1 -- 172.21.15.116:0/1978147097 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f3f5401d580 con 0x7f3f5c10d0f0 2024-03-30T01:04:14.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.721+0000 7f3f59600700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3f400a1320 0x7f3f400a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:14.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.724+0000 7f3f59600700 1 -- 172.21.15.116:0/1978147097 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3f540cebb0 con 0x7f3f5c10d0f0 2024-03-30T01:04:14.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.726+0000 7f3f60c00700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3f400a1320 0x7f3f400a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:14.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.728+0000 7f3f60c00700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3f400a1320 0x7f3f400a37e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c005fd0 tx=0x7f3f4c005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:14.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.804+0000 7f3f77089b80 1 Processor -- start 2024-03-30T01:04:14.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.821+0000 7f3f77089b80 1 -- start start 2024-03-30T01:04:14.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.822+0000 7f3f77089b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a07d0 0x561ecd1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.822+0000 7f3f77089b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 0x561ecd1a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.822+0000 7f3f77089b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1a2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.822+0000 7f3f77089b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ecd0f8990 con 0x561ecd1a1fd0 2024-03-30T01:04:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.822+0000 7f3f77089b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ecd0f6350 con 0x561ecd1a1130 2024-03-30T01:04:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.822+0000 7f3f77089b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ecd0f1c40 con 0x561ecd1a07d0 2024-03-30T01:04:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.823+0000 7f3f61600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.823+0000 7f3f61600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45576/0 (socket says 172.21.15.116:45576) 2024-03-30T01:04:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.823+0000 7f3f61600700 1 -- 172.21.15.116:0/765449135 learned_addr learned my addr 172.21.15.116:0/765449135 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:14.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.824+0000 7f3f61600700 1 -- 172.21.15.116:0/765449135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a07d0 msgr2=0x561ecd1a0bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:04:14.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.824+0000 7f3f61600700 1 --2- 172.21.15.116:0/765449135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a07d0 0x561ecd1a0bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.824+0000 7f3f61600700 1 -- 172.21.15.116:0/765449135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 msgr2=0x561ecd1a19f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T01:04:14.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.824+0000 7f3f61600700 1 --2- 172.21.15.116:0/765449135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 0x561ecd1a19f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.824+0000 7f3f61600700 1 -- 172.21.15.116:0/765449135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561eccf733b0 con 0x561ecd1a1fd0 2024-03-30T01:04:14.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.824+0000 7f3f61600700 1 --2- 172.21.15.116:0/765449135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1a2450 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3f5409ba20 tx=0x7f3f54009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2cf88bde31e72f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:14.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.857+0000 7f3f51c00700 1 -- 172.21.15.116:0/765449135 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5409bd20 con 0x561ecd1a1fd0 2024-03-30T01:04:14.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.857+0000 7f3f51c00700 1 -- 172.21.15.116:0/765449135 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f5409fb50 con 0x561ecd1a1fd0 2024-03-30T01:04:14.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.857+0000 7f3f51c00700 1 -- 172.21.15.116:0/765449135 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5409b470 con 0x561ecd1a1fd0 2024-03-30T01:04:14.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.871+0000 7f3f77089b80 1 -- 172.21.15.116:0/765449135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 msgr2=0x561ecd1a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:14.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.871+0000 7f3f77089b80 1 --2- 172.21.15.116:0/765449135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1a2450 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3f5409ba20 tx=0x7f3f54009920 comp rx=0 tx=0).stop 2024-03-30T01:04:14.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.884+0000 7f3f77089b80 1 -- 172.21.15.116:0/765449135 shutdown_connections 2024-03-30T01:04:14.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.884+0000 7f3f77089b80 1 --2- 172.21.15.116:0/765449135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a07d0 0x561ecd1a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.884+0000 7f3f77089b80 1 --2- 172.21.15.116:0/765449135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1a2450 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.884+0000 7f3f77089b80 1 --2- 172.21.15.116:0/765449135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 0x561ecd1a19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.884+0000 7f3f77089b80 1 -- 172.21.15.116:0/765449135 >> 172.21.15.116:0/765449135 conn(0x561ecd0ff6a0 msgr2=0x561ecd19b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:14.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.887+0000 7f3f77089b80 1 -- 172.21.15.116:0/765449135 shutdown_connections 2024-03-30T01:04:14.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.888+0000 7f3f77089b80 1 -- 172.21.15.116:0/765449135 wait complete. 2024-03-30T01:04:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.892+0000 7f3f77089b80 1 -- start start 2024-03-30T01:04:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.898+0000 7f3f77089b80 1 Processor -- start 2024-03-30T01:04:14.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.907+0000 7f3f77089b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a07d0 0x561ecd1aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:14.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:14 smithi116 ceph-mon[38086]: pgmap v482: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.6 MiB/s wr, 424 op/s 2024-03-30T01:04:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.911+0000 7f3f77089b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a1130 0x561ecd1aaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.911+0000 7f3f60c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a07d0 0x561ecd1aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.912+0000 7f3f60c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a07d0 0x561ecd1aa4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:54020/0 (socket says 172.21.15.116:54020) 2024-03-30T01:04:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.912+0000 7f3f60c00700 1 -- 172.21.15.116:0/1641480334 learned_addr learned my addr 172.21.15.116:0/1641480334 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.912+0000 7f3f5be00700 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a1130 0x561ecd1aaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.913+0000 7f3f77089b80 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.915+0000 7f3f77089b80 1 -- 172.21.15.116:0/1641480334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ecd08dd80 con 0x561ecd1a1fd0 2024-03-30T01:04:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.915+0000 7f3f77089b80 1 -- 172.21.15.116:0/1641480334 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ecd08da20 con 0x561ecd1a07d0 2024-03-30T01:04:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.915+0000 7f3f77089b80 1 -- 172.21.15.116:0/1641480334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eccf756a0 con 0x561ecd1a1130 2024-03-30T01:04:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.915+0000 7f3f77089b80 10 client.0 _ll_register_callbacks cb 0x561ecd097470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.916+0000 7f3f61600700 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.918+0000 7f3f61600700 1 -- 172.21.15.116:0/1641480334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a1130 msgr2=0x561ecd1aaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.918+0000 7f3f61600700 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a1130 0x561ecd1aaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.918+0000 7f3f61600700 1 -- 172.21.15.116:0/1641480334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a07d0 msgr2=0x561ecd1aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.918+0000 7f3f61600700 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a07d0 0x561ecd1aa4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.918+0000 7f3f61600700 1 -- 172.21.15.116:0/1641480334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5c1a5730 con 0x561ecd1a1fd0 2024-03-30T01:04:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.919+0000 7f3f61600700 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1b19b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3f5409ba20 tx=0x7f3f5409b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82110b6fd1c9adb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.920+0000 7f3f60c00700 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a07d0 0x561ecd1aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:04:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.920+0000 7f3f5be00700 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a1130 0x561ecd1aaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T01:04:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.920+0000 7f3f46000700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:04:14.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.941+0000 7f3f46000700 1 -- 172.21.15.116:0/1641480334 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5409c1d0 con 0x561ecd1a1fd0 2024-03-30T01:04:14.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.941+0000 7f3f46000700 1 -- 172.21.15.116:0/1641480334 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3f5409bd20 con 0x561ecd1a1fd0 2024-03-30T01:04:14.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.961+0000 7f3f77089b80 1 -- 172.21.15.116:0/1641480334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561ecd0b2380 con 0x561ecd1a1fd0 2024-03-30T01:04:14.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.962+0000 7f3f46000700 1 -- 172.21.15.116:0/1641480334 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f5409a0c0 con 0x561ecd1a1fd0 2024-03-30T01:04:14.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.962+0000 7f3f77089b80 10 client.16152 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:14.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.962+0000 7f3f77089b80 1 -- 172.21.15.116:0/1641480334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561eccf1fb60 con 0x561ecd1a1fd0 2024-03-30T01:04:14.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.963+0000 7f3f46000700 1 -- 172.21.15.116:0/1641480334 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561ecd0b2380 con 0x561ecd1a1fd0 2024-03-30T01:04:14.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.963+0000 7f3f46000700 1 -- 172.21.15.116:0/1641480334 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f3f5409ac80 con 0x561ecd1a1fd0 2024-03-30T01:04:14.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.964+0000 7f3f77089b80 10 client.16152 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:14.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.964+0000 7f3f77089b80 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:04:14.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.964+0000 7f3f77089b80 20 client.16152 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:14.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.965+0000 7f3f77089b80 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561ecd1b5d10 0x561ecd1b8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:14.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.965+0000 7f3f77089b80 4 client.16152 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:04:14.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.965+0000 7f3f77089b80 1 -- 172.21.15.116:0/1641480334 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ecd0ea700 con 0x561ecd1b5d10 2024-03-30T01:04:14.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.965+0000 7f3f60c00700 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561ecd1b5d10 0x561ecd1b8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:14.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.966+0000 7f3f60c00700 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561ecd1b5d10 0x561ecd1b8130 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f59306af3908acd server_cookie=daf32df0bad10bbe in_seq=0 out_seq=0 2024-03-30T01:04:14.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.967+0000 7f3f46000700 10 client.16152 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:14.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.969+0000 7f3f46000700 1 -- 172.21.15.116:0/1641480334 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561ecd0ea700 con 0x561ecd1b5d10 2024-03-30T01:04:14.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.969+0000 7f3f46000700 10 client.16152 handle_command_reply: tid=0 2024-03-30T01:04:14.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.969+0000 7f3f77089b80 1 client.16152 shutdown 2024-03-30T01:04:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.974+0000 7f3f77089b80 1 -- 172.21.15.116:0/1641480334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 msgr2=0x561ecd1b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.974+0000 7f3f77089b80 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1b19b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3f5409ba20 tx=0x7f3f5409b990 comp rx=0 tx=0).stop 2024-03-30T01:04:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.975+0000 7f3f77089b80 1 -- 172.21.15.116:0/1641480334 shutdown_connections 2024-03-30T01:04:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.975+0000 7f3f77089b80 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561ecd1b5d10 0x561ecd1b8130 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.975+0000 7f3f77089b80 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1b19b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.975+0000 7f3f77089b80 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a07d0 0x561ecd1aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.975+0000 7f3f77089b80 1 --2- 172.21.15.116:0/1641480334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a1130 0x561ecd1aaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.975+0000 7f3f77089b80 1 -- 172.21.15.116:0/1641480334 >> 172.21.15.116:0/1641480334 conn(0x561ecd0ff6a0 msgr2=0x561ecd19f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.978+0000 7f3f46000700 0 client.16152 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.978+0000 7f3f77089b80 1 -- 172.21.15.116:0/1641480334 shutdown_connections 2024-03-30T01:04:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.978+0000 7f3f77089b80 1 -- 172.21.15.116:0/1641480334 wait complete. 2024-03-30T01:04:14.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:14.983+0000 7f3f77089b80 20 client.16152 trim_cache size 0 max 16384 2024-03-30T01:04:15.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:14 smithi181 ceph-mon[47622]: pgmap v482: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.6 MiB/s wr, 424 op/s 2024-03-30T01:04:15.612 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:14 smithi130 ceph-mon[47626]: pgmap v482: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.6 MiB/s wr, 424 op/s 2024-03-30T01:04:15.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.613+0000 7f3f77089b80 1 Processor -- start 2024-03-30T01:04:15.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.653+0000 7f3f77089b80 1 -- start start 2024-03-30T01:04:15.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.654+0000 7f3f77089b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a07d0 0x561ecd19af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.654+0000 7f3f77089b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 0x561ecd1b1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.654+0000 7f3f77089b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1945a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.654+0000 7f3f77089b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ecd0b2380 con 0x561ecd1a1fd0 2024-03-30T01:04:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.654+0000 7f3f77089b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ecd19c4b0 con 0x561ecd1a1130 2024-03-30T01:04:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.654+0000 7f3f77089b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ecd1b2b30 con 0x561ecd1a07d0 2024-03-30T01:04:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.657+0000 7f3f61600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1945a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.657+0000 7f3f61600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1945a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45584/0 (socket says 172.21.15.116:45584) 2024-03-30T01:04:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.657+0000 7f3f61600700 1 -- 172.21.15.116:0/724179185 learned_addr learned my addr 172.21.15.116:0/724179185 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.677+0000 7f3f60c00700 1 --2- 172.21.15.116:0/724179185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a07d0 0x561ecd19af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.677+0000 7f3f5be00700 1 --2- 172.21.15.116:0/724179185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 0x561ecd1b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.680+0000 7f3f61600700 1 -- 172.21.15.116:0/724179185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a07d0 msgr2=0x561ecd19af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.680+0000 7f3f61600700 1 --2- 172.21.15.116:0/724179185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a07d0 0x561ecd19af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.680+0000 7f3f61600700 1 -- 172.21.15.116:0/724179185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 msgr2=0x561ecd1b1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.680+0000 7f3f61600700 1 --2- 172.21.15.116:0/724179185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 0x561ecd1b1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.680+0000 7f3f61600700 1 -- 172.21.15.116:0/724179185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ecd0b7ea0 con 0x561ecd1a1fd0 2024-03-30T01:04:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.680+0000 7f3f61600700 1 --2- 172.21.15.116:0/724179185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1945a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3f54098220 tx=0x7f3f54009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7b976c8e38fa9e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.704+0000 7f3f52600700 1 -- 172.21.15.116:0/724179185 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5409bd20 con 0x561ecd1a1fd0 2024-03-30T01:04:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.704+0000 7f3f52600700 1 -- 172.21.15.116:0/724179185 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f5409a0c0 con 0x561ecd1a1fd0 2024-03-30T01:04:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.704+0000 7f3f52600700 1 -- 172.21.15.116:0/724179185 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5409c250 con 0x561ecd1a1fd0 2024-03-30T01:04:15.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.725+0000 7f3f77089b80 1 -- 172.21.15.116:0/724179185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 msgr2=0x561ecd1945a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:15.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.725+0000 7f3f77089b80 1 --2- 172.21.15.116:0/724179185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1945a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3f54098220 tx=0x7f3f54009920 comp rx=0 tx=0).stop 2024-03-30T01:04:15.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.746+0000 7f3f77089b80 1 -- 172.21.15.116:0/724179185 shutdown_connections 2024-03-30T01:04:15.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.746+0000 7f3f77089b80 1 --2- 172.21.15.116:0/724179185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a07d0 0x561ecd19af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:15.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.746+0000 7f3f77089b80 1 --2- 172.21.15.116:0/724179185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a1fd0 0x561ecd1945a0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:15.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.746+0000 7f3f77089b80 1 --2- 172.21.15.116:0/724179185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 0x561ecd1b1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:15.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.746+0000 7f3f77089b80 1 -- 172.21.15.116:0/724179185 >> 172.21.15.116:0/724179185 conn(0x561ecd0ff6a0 msgr2=0x561ecd1a8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:15.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.777+0000 7f3f77089b80 1 -- 172.21.15.116:0/724179185 shutdown_connections 2024-03-30T01:04:15.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.784+0000 7f3f77089b80 1 -- 172.21.15.116:0/724179185 wait complete. 2024-03-30T01:04:15.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.785+0000 7f3f77089b80 1 -- start start 2024-03-30T01:04:15.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.801+0000 7f3f77089b80 1 Processor -- start 2024-03-30T01:04:15.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.807+0000 7f3f77089b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a07d0 0x561ecd1af630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:15.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.807+0000 7f3f77089b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 0x561ecd1afb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:15.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.807+0000 7f3f77089b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a1fd0 0x561ecd1b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:15.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.807+0000 7f3f77089b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ecd1b1020 con 0x561ecd1a07d0 2024-03-30T01:04:15.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.807+0000 7f3f77089b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ecd1b1190 con 0x561ecd1a1130 2024-03-30T01:04:15.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.807+0000 7f3f77089b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ecd1b1350 con 0x561ecd1a1fd0 2024-03-30T01:04:15.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.807+0000 7f3f77089b80 10 client.0 _ll_register_callbacks cb 0x561ecd097470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:15.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.808+0000 7f3f5be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 0x561ecd1afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:15.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.808+0000 7f3f5be00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 0x561ecd1afb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:54042/0 (socket says 172.21.15.116:54042) 2024-03-30T01:04:15.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.808+0000 7f3f5be00700 1 -- 172.21.15.116:0/1927690705 learned_addr learned my addr 172.21.15.116:0/1927690705 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:15.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.808+0000 7f3f60c00700 1 --2- 172.21.15.116:0/1927690705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a07d0 0x561ecd1af630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.809+0000 7f3f60c00700 1 -- 172.21.15.116:0/1927690705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a1fd0 msgr2=0x561ecd1b0ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T01:04:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.809+0000 7f3f60c00700 1 --2- 172.21.15.116:0/1927690705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a1fd0 0x561ecd1b0ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.809+0000 7f3f60c00700 1 -- 172.21.15.116:0/1927690705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 msgr2=0x561ecd1afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.809+0000 7f3f60c00700 1 --2- 172.21.15.116:0/1927690705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 0x561ecd1afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.809+0000 7f3f60c00700 1 -- 172.21.15.116:0/1927690705 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561eccf1fb60 con 0x561ecd1a07d0 2024-03-30T01:04:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.809+0000 7f3f60c00700 1 --2- 172.21.15.116:0/1927690705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a07d0 0x561ecd1af630 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00e680 tx=0x7f3f4c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16436082e9dd475b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:15.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.833+0000 7f3f46000700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:04:15.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.833+0000 7f3f46000700 1 -- 172.21.15.116:0/1927690705 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c005240 con 0x561ecd1a07d0 2024-03-30T01:04:15.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.833+0000 7f3f46000700 1 -- 172.21.15.116:0/1927690705 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3f4c0053a0 con 0x561ecd1a07d0 2024-03-30T01:04:15.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.834+0000 7f3f77089b80 1 -- 172.21.15.116:0/1927690705 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3f5409fb50 con 0x561ecd1a07d0 2024-03-30T01:04:15.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.837+0000 7f3f46000700 1 -- 172.21.15.116:0/1927690705 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f4c005560 con 0x561ecd1a07d0 2024-03-30T01:04:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.852+0000 7f3f77089b80 10 client.16158 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.852+0000 7f3f77089b80 1 -- 172.21.15.116:0/1927690705 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561eccf77cd0 con 0x561ecd1a07d0 2024-03-30T01:04:15.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.880+0000 7f3f46000700 1 -- 172.21.15.116:0/1927690705 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f5409fb50 con 0x561ecd1a07d0 2024-03-30T01:04:15.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.880+0000 7f3f46000700 1 -- 172.21.15.116:0/1927690705 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f3f4c01f450 con 0x561ecd1a07d0 2024-03-30T01:04:15.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.880+0000 7f3f77089b80 10 client.16158 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:15.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.880+0000 7f3f77089b80 10 client.16158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:04:15.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.880+0000 7f3f77089b80 20 client.16158 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:15.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.880+0000 7f3f77089b80 1 --2- 172.21.15.116:0/1927690705 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561ecd20a940 0x561ecd20cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:15.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.880+0000 7f3f77089b80 4 client.16158 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T01:04:15.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.880+0000 7f3f77089b80 1 -- 172.21.15.116:0/1927690705 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561eccf40cf0 con 0x561ecd20a940 2024-03-30T01:04:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.891+0000 7f3f5be00700 1 --2- 172.21.15.116:0/1927690705 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561ecd20a940 0x561ecd20cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.892+0000 7f3f5be00700 1 --2- 172.21.15.116:0/1927690705 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561ecd20a940 0x561ecd20cd60 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=718f72a0bb214245 server_cookie=bb4dcf4ff596aa71 in_seq=0 out_seq=0 2024-03-30T01:04:15.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.900+0000 7f3f46000700 10 client.16158 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:15.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.900+0000 7f3f46000700 1 -- 172.21.15.116:0/1927690705 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+269 (crc 0 0 0) 0x561eccf40cf0 con 0x561ecd20a940 2024-03-30T01:04:15.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.900+0000 7f3f46000700 10 client.16158 handle_command_reply: tid=0 2024-03-30T01:04:15.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.900+0000 7f3f77089b80 1 client.16158 shutdown 2024-03-30T01:04:15.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.934+0000 7f3f77089b80 1 -- 172.21.15.116:0/1927690705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a07d0 msgr2=0x561ecd1af630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:15.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.934+0000 7f3f77089b80 1 --2- 172.21.15.116:0/1927690705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a07d0 0x561ecd1af630 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00e680 tx=0x7f3f4c0047c0 comp rx=0 tx=0).stop 2024-03-30T01:04:15.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.934+0000 7f3f77089b80 1 -- 172.21.15.116:0/1927690705 shutdown_connections 2024-03-30T01:04:15.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.934+0000 7f3f77089b80 1 --2- 172.21.15.116:0/1927690705 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561ecd20a940 0x561ecd20cd60 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:15.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.934+0000 7f3f77089b80 1 --2- 172.21.15.116:0/1927690705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ecd1a1fd0 0x561ecd1b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:15.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.934+0000 7f3f77089b80 1 --2- 172.21.15.116:0/1927690705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ecd1a07d0 0x561ecd1af630 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:15.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.934+0000 7f3f77089b80 1 --2- 172.21.15.116:0/1927690705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561ecd1a1130 0x561ecd1afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:15.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.934+0000 7f3f77089b80 1 -- 172.21.15.116:0/1927690705 >> 172.21.15.116:0/1927690705 conn(0x561ecd0ff6a0 msgr2=0x561ecd1b7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:15.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.934+0000 7f3f46000700 0 client.16158 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:15.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.936+0000 7f3f77089b80 1 -- 172.21.15.116:0/1927690705 shutdown_connections 2024-03-30T01:04:15.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.936+0000 7f3f77089b80 1 -- 172.21.15.116:0/1927690705 wait complete. 2024-03-30T01:04:15.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:15.961+0000 7f3f77089b80 20 client.16158 trim_cache size 0 max 16384 2024-03-30T01:04:16.025 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:04:16.025 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (20428 inodes in the stack)", 2024-03-30T01:04:16.025 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-30T01:04:16.026 INFO:teuthology.orchestra.run.smithi116.stdout: "1ef8a851-5fc9-428f-905f-d08d42299f02": { 2024-03-30T01:04:16.026 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-30T01:04:16.026 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "1ef8a851-5fc9-428f-905f-d08d42299f02", 2024-03-30T01:04:16.026 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-30T01:04:16.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:04:16.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:04:16.026 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:04:16.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:16.018+0000 7f3f63600700 1 -- 172.21.15.116:0/1978147097 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3f400a1320 msgr2=0x7f3f400a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:16.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:16.018+0000 7f3f63600700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3f400a1320 0x7f3f400a37e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c005fd0 tx=0x7f3f4c005ee0 comp rx=0 tx=0).stop 2024-03-30T01:04:16.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:16.019+0000 7f3f63600700 1 -- 172.21.15.116:0/1978147097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f5c10d0f0 msgr2=0x7f3f5c1a47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:16.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:16.019+0000 7f3f63600700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f5c10d0f0 0x7f3f5c1a47e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400b3e0 tx=0x7f3f54005ad0 comp rx=0 tx=0).stop 2024-03-30T01:04:16.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:16.020+0000 7f3f63600700 1 -- 172.21.15.116:0/1978147097 shutdown_connections 2024-03-30T01:04:16.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:16.020+0000 7f3f63600700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f3f400a1320 0x7f3f400a37e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:16.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:16.020+0000 7f3f63600700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f5c10d0f0 0x7f3f5c1a47e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:16.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:16.020+0000 7f3f63600700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f5c107db0 0x7f3f5c19d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:16.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:16.020+0000 7f3f63600700 1 --2- 172.21.15.116:0/1978147097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f5c10a720 0x7f3f5c19d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:16.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:16.020+0000 7f3f63600700 1 -- 172.21.15.116:0/1978147097 >> 172.21.15.116:0/1978147097 conn(0x7f3f5c06dd50 msgr2=0x7f3f5c10aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:16.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:16.028+0000 7f3f63600700 1 -- 172.21.15.116:0/1978147097 shutdown_connections 2024-03-30T01:04:16.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:16.028+0000 7f3f63600700 1 -- 172.21.15.116:0/1978147097 wait complete. 2024-03-30T01:04:16.615 INFO:tasks.cephfs.filesystem:scrub status for tag:1ef8a851-5fc9-428f-905f-d08d42299f02 - {'path': '/', 'tag': '1ef8a851-5fc9-428f-905f-d08d42299f02', 'options': 'recursive,force'} 2024-03-30T01:04:16.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:16 smithi116 ceph-mon[38086]: pgmap v483: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 9.4 MiB/s wr, 609 op/s 2024-03-30T01:04:17.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:16 smithi181 ceph-mon[47622]: pgmap v483: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 9.4 MiB/s wr, 609 op/s 2024-03-30T01:04:17.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:16 smithi130 ceph-mon[47626]: pgmap v483: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 9.4 MiB/s wr, 609 op/s 2024-03-30T01:04:18.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:18 smithi116 ceph-mon[38086]: pgmap v484: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 10 MiB/s wr, 640 op/s 2024-03-30T01:04:18.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:18 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:04:18.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:18 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:04:18.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:18 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:04:19.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:18 smithi181 ceph-mon[47622]: pgmap v484: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 10 MiB/s wr, 640 op/s 2024-03-30T01:04:19.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:18 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:04:19.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:18 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:04:19.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:18 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:04:19.072 INFO:tasks.workunit.client.0.smithi116.stdout: 496 442757 711 316845 968 185032 1841 2024-03-30T01:04:19.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:18 smithi130 ceph-mon[47626]: pgmap v484: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 10 MiB/s wr, 640 op/s 2024-03-30T01:04:19.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:18 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:04:19.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:18 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:04:19.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:18 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:04:20.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:19 smithi181 ceph-mon[47622]: pgmap v485: 129 pgs: 129 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 10 MiB/s wr, 641 op/s 2024-03-30T01:04:20.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:19 smithi130 ceph-mon[47626]: pgmap v485: 129 pgs: 129 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 10 MiB/s wr, 641 op/s 2024-03-30T01:04:20.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:19 smithi116 ceph-mon[38086]: pgmap v485: 129 pgs: 129 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 10 MiB/s wr, 641 op/s 2024-03-30T01:04:22.675 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:22 smithi116 ceph-mon[38086]: pgmap v486: 129 pgs: 129 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.6 MiB/s wr, 556 op/s 2024-03-30T01:04:22.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:22 smithi181 ceph-mon[47622]: pgmap v486: 129 pgs: 129 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.6 MiB/s wr, 556 op/s 2024-03-30T01:04:22.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:22 smithi130 ceph-mon[47626]: pgmap v486: 129 pgs: 129 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.6 MiB/s wr, 556 op/s 2024-03-30T01:04:24.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:24 smithi181 ceph-mon[47622]: pgmap v487: 129 pgs: 129 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.4 MiB/s wr, 523 op/s 2024-03-30T01:04:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:24 smithi130 ceph-mon[47626]: pgmap v487: 129 pgs: 129 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.4 MiB/s wr, 523 op/s 2024-03-30T01:04:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:24 smithi116 ceph-mon[38086]: pgmap v487: 129 pgs: 129 active+clean; 2.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.4 MiB/s wr, 523 op/s 2024-03-30T01:04:25.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:25 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:25.921 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:25 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:25.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:25 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:26.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:26 smithi181 ceph-mon[47622]: pgmap v488: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.7 MiB/s wr, 580 op/s 2024-03-30T01:04:26.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:26 smithi130 ceph-mon[47626]: pgmap v488: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.7 MiB/s wr, 580 op/s 2024-03-30T01:04:26.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:26 smithi116 ceph-mon[38086]: pgmap v488: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.7 MiB/s wr, 580 op/s 2024-03-30T01:04:28.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:28 smithi130 ceph-mon[47626]: pgmap v489: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 472 op/s 2024-03-30T01:04:28.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:28 smithi116 ceph-mon[38086]: pgmap v489: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 472 op/s 2024-03-30T01:04:29.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:28 smithi181 ceph-mon[47622]: pgmap v489: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 472 op/s 2024-03-30T01:04:29.077 INFO:tasks.workunit.client.0.smithi116.stdout: 517 477684 870 341373 1166 201021 1904 2024-03-30T01:04:29.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:29 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:29.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:29 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:30.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:29 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:30.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:30 smithi130 ceph-mon[47626]: pgmap v490: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 477 op/s 2024-03-30T01:04:30.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:30 smithi116 ceph-mon[38086]: pgmap v490: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 477 op/s 2024-03-30T01:04:31.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:30 smithi181 ceph-mon[47622]: pgmap v490: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 477 op/s 2024-03-30T01:04:31.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:31 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:31.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:31 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:32.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:31 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:32.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:32 smithi130 ceph-mon[47626]: pgmap v491: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 471 op/s 2024-03-30T01:04:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:32 smithi116 ceph-mon[38086]: pgmap v491: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 471 op/s 2024-03-30T01:04:33.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:32 smithi181 ceph-mon[47622]: pgmap v491: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 471 op/s 2024-03-30T01:04:34.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:34 smithi130 ceph-mon[47626]: pgmap v492: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 517 op/s 2024-03-30T01:04:34.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:34 smithi116 ceph-mon[38086]: pgmap v492: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 517 op/s 2024-03-30T01:04:35.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:34 smithi181 ceph-mon[47622]: pgmap v492: 129 pgs: 129 active+clean; 2.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 517 op/s 2024-03-30T01:04:36.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:36 smithi130 ceph-mon[47626]: pgmap v493: 129 pgs: 129 active+clean; 2.1 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.4 MiB/s wr, 801 op/s 2024-03-30T01:04:36.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:36 smithi116 ceph-mon[38086]: pgmap v493: 129 pgs: 129 active+clean; 2.1 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.4 MiB/s wr, 801 op/s 2024-03-30T01:04:37.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:36 smithi181 ceph-mon[47622]: pgmap v493: 129 pgs: 129 active+clean; 2.1 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.4 MiB/s wr, 801 op/s 2024-03-30T01:04:37.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:37 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:04:37.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:37 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:04:38.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:37 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:04:38.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:38 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:04:38.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:38 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:04:38.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:38 smithi130 ceph-mon[47626]: pgmap v494: 129 pgs: 129 active+clean; 2.1 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 11 MiB/s wr, 861 op/s 2024-03-30T01:04:38.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:38 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:04:38.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:38 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:04:38.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:38 smithi116 ceph-mon[38086]: pgmap v494: 129 pgs: 129 active+clean; 2.1 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 11 MiB/s wr, 861 op/s 2024-03-30T01:04:39.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:38 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:04:39.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:38 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:04:39.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:38 smithi181 ceph-mon[47622]: pgmap v494: 129 pgs: 129 active+clean; 2.1 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 11 MiB/s wr, 861 op/s 2024-03-30T01:04:40.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:40 smithi130 ceph-mon[47626]: pgmap v495: 129 pgs: 129 active+clean; 2.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 14 MiB/s wr, 1.03k op/s 2024-03-30T01:04:40.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:40 smithi116 ceph-mon[38086]: pgmap v495: 129 pgs: 129 active+clean; 2.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 14 MiB/s wr, 1.03k op/s 2024-03-30T01:04:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:40 smithi181 ceph-mon[47622]: pgmap v495: 129 pgs: 129 active+clean; 2.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 14 MiB/s wr, 1.03k op/s 2024-03-30T01:04:42.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:42 smithi130 ceph-mon[47626]: pgmap v496: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 15 MiB/s wr, 1.04k op/s 2024-03-30T01:04:42.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:42 smithi116 ceph-mon[38086]: pgmap v496: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 15 MiB/s wr, 1.04k op/s 2024-03-30T01:04:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:42 smithi181 ceph-mon[47622]: pgmap v496: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 15 MiB/s wr, 1.04k op/s 2024-03-30T01:04:44.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:44 smithi130 ceph-mon[47626]: pgmap v497: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 15 MiB/s wr, 1.06k op/s 2024-03-30T01:04:44.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:44 smithi116 ceph-mon[38086]: pgmap v497: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 15 MiB/s wr, 1.06k op/s 2024-03-30T01:04:45.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:44 smithi181 ceph-mon[47622]: pgmap v497: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 15 MiB/s wr, 1.06k op/s 2024-03-30T01:04:46.616 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T01:04:46.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.832+0000 7f5e66800700 1 -- 172.21.15.116:0/1315092587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e60100d80 msgr2=0x7f5e601011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:46.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.832+0000 7f5e66800700 1 --2- 172.21.15.116:0/1315092587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e60100d80 0x7f5e601011a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5e500099c0 tx=0x7f5e50009cd0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.832+0000 7f5e66800700 1 -- 172.21.15.116:0/1315092587 shutdown_connections 2024-03-30T01:04:46.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.832+0000 7f5e66800700 1 --2- 172.21.15.116:0/1315092587 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e60102910 0x7f5e601091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.832+0000 7f5e66800700 1 --2- 172.21.15.116:0/1315092587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e60100d80 0x7f5e601011a0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.832+0000 7f5e66800700 1 --2- 172.21.15.116:0/1315092587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5e60101f70 0x7f5e601023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.832+0000 7f5e66800700 1 -- 172.21.15.116:0/1315092587 >> 172.21.15.116:0/1315092587 conn(0x7f5e600fc2f0 msgr2=0x7f5e600fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:46.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.832+0000 7f5e66800700 1 -- 172.21.15.116:0/1315092587 shutdown_connections 2024-03-30T01:04:46.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.832+0000 7f5e66800700 1 -- 172.21.15.116:0/1315092587 wait complete. 2024-03-30T01:04:46.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.833+0000 7f5e66800700 1 Processor -- start 2024-03-30T01:04:46.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.833+0000 7f5e66800700 1 -- start start 2024-03-30T01:04:46.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e66800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5e60100d80 0x7f5e60198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e66800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e60101f70 0x7f5e60199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e66800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e60102910 0x7f5e601a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e66800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e601a0a60 con 0x7f5e60101f70 2024-03-30T01:04:46.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e66800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e601a0ba0 con 0x7f5e60100d80 2024-03-30T01:04:46.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e66800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e601a0d30 con 0x7f5e60102910 2024-03-30T01:04:46.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e5ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e60101f70 0x7f5e60199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e5ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e60101f70 0x7f5e60199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51562/0 (socket says 172.21.15.116:51562) 2024-03-30T01:04:46.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e5ea00700 1 -- 172.21.15.116:0/3398081356 learned_addr learned my addr 172.21.15.116:0/3398081356 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:46.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e5f400700 1 --2- 172.21.15.116:0/3398081356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5e60100d80 0x7f5e60198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e5fe00700 1 --2- 172.21.15.116:0/3398081356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e60102910 0x7f5e601a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e5ea00700 1 -- 172.21.15.116:0/3398081356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e60102910 msgr2=0x7f5e601a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:46.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e5ea00700 1 --2- 172.21.15.116:0/3398081356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e60102910 0x7f5e601a0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e5ea00700 1 -- 172.21.15.116:0/3398081356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5e60100d80 msgr2=0x7f5e60198f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:46.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e5ea00700 1 --2- 172.21.15.116:0/3398081356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5e60100d80 0x7f5e60198f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.834+0000 7f5e5ea00700 1 -- 172.21.15.116:0/3398081356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e500095f0 con 0x7f5e60101f70 2024-03-30T01:04:46.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.835+0000 7f5e5ea00700 1 --2- 172.21.15.116:0/3398081356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e60101f70 0x7f5e60199470 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c00d980 tx=0x7f5e4c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:46.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.835+0000 7f5e57e00700 1 -- 172.21.15.116:0/3398081356 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c004220 con 0x7f5e60101f70 2024-03-30T01:04:46.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.835+0000 7f5e57e00700 1 -- 172.21.15.116:0/3398081356 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e4c007640 con 0x7f5e60101f70 2024-03-30T01:04:46.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.835+0000 7f5e66800700 1 -- 172.21.15.116:0/3398081356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e601a0fb0 con 0x7f5e60101f70 2024-03-30T01:04:46.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.835+0000 7f5e57e00700 1 -- 172.21.15.116:0/3398081356 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c005690 con 0x7f5e60101f70 2024-03-30T01:04:46.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.835+0000 7f5e66800700 1 -- 172.21.15.116:0/3398081356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e601a1490 con 0x7f5e60101f70 2024-03-30T01:04:46.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:46 smithi130 ceph-mon[47626]: pgmap v498: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 15 MiB/s wr, 1.02k op/s 2024-03-30T01:04:46.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.839+0000 7f5e57e00700 1 -- 172.21.15.116:0/3398081356 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f5e4c004750 con 0x7f5e60101f70 2024-03-30T01:04:46.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.840+0000 7f5e7a418b80 1 Processor -- start 2024-03-30T01:04:46.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.840+0000 7f5e7a418b80 1 -- start start 2024-03-30T01:04:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.840+0000 7f5e7a418b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da027d0 0x56043da02bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.840+0000 7f5e7a418b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03130 0x56043da039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e57e00700 1 --2- 172.21.15.116:0/3398081356 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5e440a1120 0x7f5e440a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e7a418b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03fd0 0x56043da04450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e7a418b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56043d95a990 con 0x56043da027d0 2024-03-30T01:04:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e7a418b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56043d958350 con 0x56043da03fd0 2024-03-30T01:04:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e7a418b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56043d953c40 con 0x56043da03130 2024-03-30T01:04:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e57e00700 1 -- 172.21.15.116:0/3398081356 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5e4c0cca40 con 0x7f5e60101f70 2024-03-30T01:04:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e5fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da027d0 0x56043da02bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e5ea00700 1 --2- 172.21.15.116:0/3398081356 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5e440a1120 0x7f5e440a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e5fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da027d0 0x56043da02bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51568/0 (socket says 172.21.15.116:51568) 2024-03-30T01:04:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e5fe00700 1 -- 172.21.15.116:0/3728272135 learned_addr learned my addr 172.21.15.116:0/3728272135 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e5f400700 1 --2- 172.21.15.116:0/3728272135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03130 0x56043da039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e5fe00700 1 --2- 172.21.15.116:0/3728272135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03fd0 0x56043da04450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e5fe00700 1 -- 172.21.15.116:0/3728272135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03130 msgr2=0x56043da039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e5fe00700 1 --2- 172.21.15.116:0/3728272135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03130 0x56043da039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e5fe00700 1 -- 172.21.15.116:0/3728272135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03fd0 msgr2=0x56043da04450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e5fe00700 1 --2- 172.21.15.116:0/3728272135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03fd0 0x56043da04450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e5fe00700 1 -- 172.21.15.116:0/3728272135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56043d7d53b0 con 0x56043da027d0 2024-03-30T01:04:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.841+0000 7f5e5f400700 1 --2- 172.21.15.116:0/3728272135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03130 0x56043da039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.842+0000 7f5e5ea00700 1 --2- 172.21.15.116:0/3398081356 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5e440a1120 0x7f5e440a35e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c005060 tx=0x7f5e4c004fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.842+0000 7f5e5fe00700 1 --2- 172.21.15.116:0/3728272135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da027d0 0x56043da02bf0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x56043da037b0 tx=0x7f5e58003b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08dbca83087608 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.842+0000 7f5e54c00700 1 -- 172.21.15.116:0/3728272135 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e58004930 con 0x56043da027d0 2024-03-30T01:04:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.842+0000 7f5e54c00700 1 -- 172.21.15.116:0/3728272135 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e5800f040 con 0x56043da027d0 2024-03-30T01:04:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.842+0000 7f5e54c00700 1 -- 172.21.15.116:0/3728272135 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e58017410 con 0x56043da027d0 2024-03-30T01:04:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.842+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3728272135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da027d0 msgr2=0x56043da02bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.842+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/3728272135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da027d0 0x56043da02bf0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x56043da037b0 tx=0x7f5e58003b30 comp rx=0 tx=0).stop 2024-03-30T01:04:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.843+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3728272135 shutdown_connections 2024-03-30T01:04:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.843+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/3728272135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03fd0 0x56043da04450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.843+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/3728272135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da027d0 0x56043da02bf0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.843+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/3728272135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03130 0x56043da039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.843+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3728272135 >> 172.21.15.116:0/3728272135 conn(0x56043d9616a0 msgr2=0x56043d9fd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.843+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3728272135 shutdown_connections 2024-03-30T01:04:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.843+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3728272135 wait complete. 2024-03-30T01:04:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.843+0000 7f5e7a418b80 1 -- start start 2024-03-30T01:04:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.843+0000 7f5e7a418b80 1 Processor -- start 2024-03-30T01:04:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.844+0000 7f5e7a418b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da027d0 0x56043da0c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.844+0000 7f5e7a418b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03130 0x56043da0cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.844+0000 7f5e7a418b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03fd0 0x56043da13c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.844+0000 7f5e7a418b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56043d8efd80 con 0x56043da027d0 2024-03-30T01:04:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.844+0000 7f5e7a418b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56043d8efa20 con 0x56043da03130 2024-03-30T01:04:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.844+0000 7f5e7a418b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56043d7d76a0 con 0x56043da03fd0 2024-03-30T01:04:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.844+0000 7f5e7a418b80 10 client.0 _ll_register_callbacks cb 0x56043d8f9470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.844+0000 7f5e5fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da027d0 0x56043da0c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.844+0000 7f5e5fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da027d0 0x56043da0c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51580/0 (socket says 172.21.15.116:51580) 2024-03-30T01:04:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.844+0000 7f5e5fe00700 1 -- 172.21.15.116:0/1091355844 learned_addr learned my addr 172.21.15.116:0/1091355844 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.844+0000 7f5e5f400700 1 --2- 172.21.15.116:0/1091355844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03130 0x56043da0cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.844+0000 7f5e5fe00700 1 --2- 172.21.15.116:0/1091355844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03fd0 0x56043da13c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.844+0000 7f5e5fe00700 1 -- 172.21.15.116:0/1091355844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03fd0 msgr2=0x56043da13c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.845+0000 7f5e5fe00700 1 --2- 172.21.15.116:0/1091355844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03fd0 0x56043da13c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.845+0000 7f5e5fe00700 1 -- 172.21.15.116:0/1091355844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03130 msgr2=0x56043da0cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.845+0000 7f5e5fe00700 1 --2- 172.21.15.116:0/1091355844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03130 0x56043da0cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.845+0000 7f5e5fe00700 1 -- 172.21.15.116:0/1091355844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e60109b10 con 0x56043da027d0 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.845+0000 7f5e5f400700 1 --2- 172.21.15.116:0/1091355844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03130 0x56043da0cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.845+0000 7f5e5fe00700 1 --2- 172.21.15.116:0/1091355844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da027d0 0x56043da0c690 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5e580093f0 tx=0x7f5e58009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9746b2db67f86caa server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.845+0000 7f5e49600700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.845+0000 7f5e49600700 1 -- 172.21.15.116:0/1091355844 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5801a660 con 0x56043da027d0 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.845+0000 7f5e49600700 1 -- 172.21.15.116:0/1091355844 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5e5801a7c0 con 0x56043da027d0 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.845+0000 7f5e7a418b80 1 -- 172.21.15.116:0/1091355844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56043d914380 con 0x56043da027d0 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.845+0000 7f5e49600700 1 -- 172.21.15.116:0/1091355844 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e5801a980 con 0x56043da027d0 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.845+0000 7f5e7a418b80 10 client.16170 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.845+0000 7f5e7a418b80 1 -- 172.21.15.116:0/1091355844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f5e580099c0 con 0x56043da027d0 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.846+0000 7f5e49600700 1 -- 172.21.15.116:0/1091355844 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e58032020 con 0x56043da027d0 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.846+0000 7f5e49600700 1 -- 172.21.15.116:0/1091355844 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f5e58012070 con 0x56043da027d0 2024-03-30T01:04:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.846+0000 7f5e7a418b80 10 client.16170 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.846+0000 7f5e7a418b80 10 client.16170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.846+0000 7f5e7a418b80 20 client.16170 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.846+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/1091355844 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56043da183f0 0x56043da1a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.846+0000 7f5e7a418b80 4 client.16170 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.846+0000 7f5e7a418b80 1 -- 172.21.15.116:0/1091355844 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56043d94c700 con 0x56043da183f0 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.846+0000 7f5e5f400700 1 --2- 172.21.15.116:0/1091355844 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56043da183f0 0x56043da1a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.847+0000 7f5e5f400700 1 --2- 172.21.15.116:0/1091355844 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56043da183f0 0x56043da1a810 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2051368de664d1d9 server_cookie=8ed44fdbee3f07a5 in_seq=0 out_seq=0 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.847+0000 7f5e49600700 10 client.16170 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.847+0000 7f5e49600700 1 -- 172.21.15.116:0/1091355844 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56043d94c700 con 0x56043da183f0 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.847+0000 7f5e49600700 10 client.16170 handle_command_reply: tid=0 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.848+0000 7f5e7a418b80 1 client.16170 shutdown 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.848+0000 7f5e7a418b80 1 -- 172.21.15.116:0/1091355844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da027d0 msgr2=0x56043da0c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.848+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/1091355844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da027d0 0x56043da0c690 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5e580093f0 tx=0x7f5e58009360 comp rx=0 tx=0).stop 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.848+0000 7f5e7a418b80 1 -- 172.21.15.116:0/1091355844 shutdown_connections 2024-03-30T01:04:46.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.848+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/1091355844 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56043da183f0 0x56043da1a810 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.848+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/1091355844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03fd0 0x56043da13c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.848+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/1091355844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da027d0 0x56043da0c690 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.848+0000 7f5e49600700 0 client.16170 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:46.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.848+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/1091355844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03130 0x56043da0cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.848+0000 7f5e7a418b80 1 -- 172.21.15.116:0/1091355844 >> 172.21.15.116:0/1091355844 conn(0x56043d9616a0 msgr2=0x56043da01170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:46.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.848+0000 7f5e7a418b80 1 -- 172.21.15.116:0/1091355844 shutdown_connections 2024-03-30T01:04:46.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.848+0000 7f5e7a418b80 1 -- 172.21.15.116:0/1091355844 wait complete. 2024-03-30T01:04:46.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.848+0000 7f5e7a418b80 20 client.16170 trim_cache size 0 max 16384 2024-03-30T01:04:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.865+0000 7f5e7a418b80 1 Processor -- start 2024-03-30T01:04:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.865+0000 7f5e7a418b80 1 -- start start 2024-03-30T01:04:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.865+0000 7f5e7a418b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da027d0 0x56043da02bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.865+0000 7f5e7a418b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03130 0x56043da13bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.865+0000 7f5e7a418b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da03fd0 0x56043da043b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.865+0000 7f5e7a418b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56043da1aba0 con 0x56043da03fd0 2024-03-30T01:04:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.865+0000 7f5e7a418b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56043da0d880 con 0x56043da03130 2024-03-30T01:04:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.865+0000 7f5e7a418b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56043da037b0 con 0x56043da027d0 2024-03-30T01:04:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.866+0000 7f5e5f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03130 0x56043da13bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.866+0000 7f5e5f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03130 0x56043da13bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38246/0 (socket says 172.21.15.116:38246) 2024-03-30T01:04:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.866+0000 7f5e5f400700 1 -- 172.21.15.116:0/3342725618 learned_addr learned my addr 172.21.15.116:0/3342725618 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:46.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.866+0000 7f5e5fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da03fd0 0x56043da043b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.866+0000 7f5e5fe00700 1 --2- 172.21.15.116:0/3342725618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da027d0 0x56043da02bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.866+0000 7f5e5f400700 1 -- 172.21.15.116:0/3342725618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da027d0 msgr2=0x56043da02bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:46.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.866+0000 7f5e5f400700 1 --2- 172.21.15.116:0/3342725618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da027d0 0x56043da02bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.866+0000 7f5e5f400700 1 -- 172.21.15.116:0/3342725618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da03fd0 msgr2=0x56043da043b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:46.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.866+0000 7f5e5f400700 1 --2- 172.21.15.116:0/3342725618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da03fd0 0x56043da043b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.866+0000 7f5e5f400700 1 -- 172.21.15.116:0/3342725618 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56043d781b60 con 0x56043da03130 2024-03-30T01:04:46.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.866+0000 7f5e5fe00700 1 --2- 172.21.15.116:0/3342725618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da03fd0 0x56043da043b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:46.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.866+0000 7f5e5f400700 1 --2- 172.21.15.116:0/3342725618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03130 0x56043da13bd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5e50000c00 tx=0x7f5e50009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c52774c42e3b4da server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:46.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.867+0000 7f5e55600700 1 -- 172.21.15.116:0/3342725618 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5001d070 con 0x56043da03130 2024-03-30T01:04:46.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.867+0000 7f5e55600700 1 -- 172.21.15.116:0/3342725618 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e5001e040 con 0x56043da03130 2024-03-30T01:04:46.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.867+0000 7f5e55600700 1 -- 172.21.15.116:0/3342725618 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50003df0 con 0x56043da03130 2024-03-30T01:04:46.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.867+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3342725618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03130 msgr2=0x56043da13bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:46.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.867+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/3342725618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03130 0x56043da13bd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5e50000c00 tx=0x7f5e50009310 comp rx=0 tx=0).stop 2024-03-30T01:04:46.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.867+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3342725618 shutdown_connections 2024-03-30T01:04:46.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.867+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/3342725618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da027d0 0x56043da02bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.867+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/3342725618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da03fd0 0x56043da043b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.867+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/3342725618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da03130 0x56043da13bd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.867+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3342725618 >> 172.21.15.116:0/3342725618 conn(0x56043d9616a0 msgr2=0x56043d9f65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.867+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3342725618 shutdown_connections 2024-03-30T01:04:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.867+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3342725618 wait complete. 2024-03-30T01:04:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.868+0000 7f5e7a418b80 1 -- start start 2024-03-30T01:04:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.868+0000 7f5e7a418b80 1 Processor -- start 2024-03-30T01:04:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.868+0000 7f5e7a418b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da027d0 0x56043da134a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.868+0000 7f5e7a418b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da03130 0x56043da139e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.868+0000 7f5e7a418b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03fd0 0x56043da11ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.868+0000 7f5e7a418b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56043da12030 con 0x56043da03130 2024-03-30T01:04:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.868+0000 7f5e7a418b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56043da121a0 con 0x56043da027d0 2024-03-30T01:04:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.868+0000 7f5e7a418b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56043da12360 con 0x56043da03fd0 2024-03-30T01:04:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.868+0000 7f5e7a418b80 10 client.0 _ll_register_callbacks cb 0x56043d8f9470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e5fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da027d0 0x56043da134a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e5f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da03130 0x56043da139e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e5f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da03130 0x56043da139e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51596/0 (socket says 172.21.15.116:51596) 2024-03-30T01:04:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e5fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da027d0 0x56043da134a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38256/0 (socket says 172.21.15.116:38256) 2024-03-30T01:04:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e5f400700 1 -- 172.21.15.116:0/3297545400 learned_addr learned my addr 172.21.15.116:0/3297545400 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e5fe00700 1 --2- 172.21.15.116:0/3297545400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03fd0 0x56043da11ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e5f400700 1 -- 172.21.15.116:0/3297545400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03fd0 msgr2=0x56043da11ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e5f400700 1 --2- 172.21.15.116:0/3297545400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03fd0 0x56043da11ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e5f400700 1 -- 172.21.15.116:0/3297545400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da027d0 msgr2=0x56043da134a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e5f400700 1 --2- 172.21.15.116:0/3297545400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da027d0 0x56043da134a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e5f400700 1 -- 172.21.15.116:0/3297545400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e580099c0 con 0x56043da03130 2024-03-30T01:04:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e5f400700 1 --2- 172.21.15.116:0/3297545400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da03130 0x56043da139e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5e50003a40 tx=0x7f5e500037c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe8fc152a57f323f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e49600700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:04:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e49600700 1 -- 172.21.15.116:0/3297545400 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5001d050 con 0x56043da03130 2024-03-30T01:04:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e49600700 1 -- 172.21.15.116:0/3297545400 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5e50004040 con 0x56043da03130 2024-03-30T01:04:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.869+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3297545400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e5800f040 con 0x56043da03130 2024-03-30T01:04:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.870+0000 7f5e49600700 1 -- 172.21.15.116:0/3297545400 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e50004200 con 0x56043da03130 2024-03-30T01:04:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.870+0000 7f5e7a418b80 10 client.16176 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.870+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3297545400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f5e500053b0 con 0x56043da03130 2024-03-30T01:04:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.870+0000 7f5e49600700 1 -- 172.21.15.116:0/3297545400 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e5800f040 con 0x56043da03130 2024-03-30T01:04:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.870+0000 7f5e49600700 1 -- 172.21.15.116:0/3297545400 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f5e5000f870 con 0x56043da03130 2024-03-30T01:04:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.870+0000 7f5e7a418b80 10 client.16176 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.870+0000 7f5e7a418b80 10 client.16176 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:04:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.870+0000 7f5e7a418b80 20 client.16176 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.870+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/3297545400 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56043da6d010 0x56043da6f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.870+0000 7f5e7a418b80 4 client.16176 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T01:04:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.870+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3297545400 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56043d7a2cf0 con 0x56043da6d010 2024-03-30T01:04:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.871+0000 7f5e5fe00700 1 --2- 172.21.15.116:0/3297545400 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56043da6d010 0x56043da6f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.871+0000 7f5e5fe00700 1 --2- 172.21.15.116:0/3297545400 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56043da6d010 0x56043da6f400 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ebd8af820e818fd server_cookie=8c528ec0ceec6245 in_seq=0 out_seq=0 2024-03-30T01:04:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.871+0000 7f5e49600700 10 client.16176 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.871+0000 7f5e49600700 1 -- 172.21.15.116:0/3297545400 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56043d7a2cf0 con 0x56043da6d010 2024-03-30T01:04:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.871+0000 7f5e49600700 10 client.16176 handle_command_reply: tid=0 2024-03-30T01:04:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.871+0000 7f5e7a418b80 1 client.16176 shutdown 2024-03-30T01:04:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.872+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3297545400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da03130 msgr2=0x56043da139e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.872+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/3297545400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da03130 0x56043da139e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5e50003a40 tx=0x7f5e500037c0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.872+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3297545400 shutdown_connections 2024-03-30T01:04:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.872+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/3297545400 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x56043da6d010 0x56043da6f400 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.872+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/3297545400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56043da03fd0 0x56043da11ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.872+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/3297545400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56043da027d0 0x56043da134a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.872+0000 7f5e7a418b80 1 --2- 172.21.15.116:0/3297545400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56043da03130 0x56043da139e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.872+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3297545400 >> 172.21.15.116:0/3297545400 conn(0x56043d9616a0 msgr2=0x56043da071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.872+0000 7f5e49600700 0 client.16176 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.872+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3297545400 shutdown_connections 2024-03-30T01:04:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.872+0000 7f5e7a418b80 1 -- 172.21.15.116:0/3297545400 wait complete. 2024-03-30T01:04:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.872+0000 7f5e7a418b80 20 client.16176 trim_cache size 0 max 16384 2024-03-30T01:04:46.883 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:04:46.883 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-30T01:04:46.883 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-30T01:04:46.883 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:04:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.873+0000 7f5e66800700 1 -- 172.21.15.116:0/3398081356 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5e440a1120 msgr2=0x7f5e440a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.873+0000 7f5e66800700 1 --2- 172.21.15.116:0/3398081356 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5e440a1120 0x7f5e440a35e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c005060 tx=0x7f5e4c004fd0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.873+0000 7f5e66800700 1 -- 172.21.15.116:0/3398081356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e60101f70 msgr2=0x7f5e60199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.873+0000 7f5e66800700 1 --2- 172.21.15.116:0/3398081356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e60101f70 0x7f5e60199470 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c00d980 tx=0x7f5e4c00dc90 comp rx=0 tx=0).stop 2024-03-30T01:04:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.873+0000 7f5e66800700 1 -- 172.21.15.116:0/3398081356 shutdown_connections 2024-03-30T01:04:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.873+0000 7f5e66800700 1 --2- 172.21.15.116:0/3398081356 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f5e440a1120 0x7f5e440a35e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.873+0000 7f5e66800700 1 --2- 172.21.15.116:0/3398081356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e60102910 0x7f5e601a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.873+0000 7f5e66800700 1 --2- 172.21.15.116:0/3398081356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5e60100d80 0x7f5e60198f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.873+0000 7f5e66800700 1 --2- 172.21.15.116:0/3398081356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e60101f70 0x7f5e60199470 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:46.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.873+0000 7f5e66800700 1 -- 172.21.15.116:0/3398081356 >> 172.21.15.116:0/3398081356 conn(0x7f5e600fc2f0 msgr2=0x7f5e60107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:46.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.873+0000 7f5e66800700 1 -- 172.21.15.116:0/3398081356 shutdown_connections 2024-03-30T01:04:46.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:46.873+0000 7f5e66800700 1 -- 172.21.15.116:0/3398081356 wait complete. 2024-03-30T01:04:46.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:46 smithi116 ceph-mon[38086]: pgmap v498: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 15 MiB/s wr, 1.02k op/s 2024-03-30T01:04:46.896 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T01:04:46.897 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T01:04:47.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:46 smithi181 ceph-mon[47622]: pgmap v498: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 15 MiB/s wr, 1.02k op/s 2024-03-30T01:04:47.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.082+0000 7fb0d3a00700 1 -- 172.21.15.116:0/3768308551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0cc100c80 msgr2=0x7fb0cc1010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.082+0000 7fb0d3a00700 1 --2- 172.21.15.116:0/3768308551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0cc100c80 0x7fb0cc1010a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb0c00099c0 tx=0x7fb0c0009cd0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.082+0000 7fb0d3a00700 1 -- 172.21.15.116:0/3768308551 shutdown_connections 2024-03-30T01:04:47.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.082+0000 7fb0d3a00700 1 --2- 172.21.15.116:0/3768308551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb0cc102810 0x7fb0cc1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.082+0000 7fb0d3a00700 1 --2- 172.21.15.116:0/3768308551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0cc100c80 0x7fb0cc1010a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.082+0000 7fb0d3a00700 1 --2- 172.21.15.116:0/3768308551 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0cc101e70 0x7fb0cc1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.082+0000 7fb0d3a00700 1 -- 172.21.15.116:0/3768308551 >> 172.21.15.116:0/3768308551 conn(0x7fb0cc0fc1f0 msgr2=0x7fb0cc0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.082+0000 7fb0d3a00700 1 -- 172.21.15.116:0/3768308551 shutdown_connections 2024-03-30T01:04:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.082+0000 7fb0d3a00700 1 -- 172.21.15.116:0/3768308551 wait complete. 2024-03-30T01:04:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.083+0000 7fb0d3a00700 1 Processor -- start 2024-03-30T01:04:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.084+0000 7fb0d3a00700 1 -- start start 2024-03-30T01:04:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.084+0000 7fb0d3a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0cc100c80 0x7fb0cc198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.084+0000 7fb0d3a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb0cc101e70 0x7fb0cc199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.084+0000 7fb0d3a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0cc102810 0x7fb0cc1a0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.084+0000 7fb0d3a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc1a0950 con 0x7fb0cc102810 2024-03-30T01:04:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.084+0000 7fb0d3a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc1a0a90 con 0x7fb0cc101e70 2024-03-30T01:04:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.084+0000 7fb0d3a00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc1a0c20 con 0x7fb0cc100c80 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.084+0000 7fb0d1a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0cc102810 0x7fb0cc1a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.084+0000 7fb0d1000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0cc100c80 0x7fb0cc198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.084+0000 7fb0d1000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0cc100c80 0x7fb0cc198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:51204/0 (socket says 172.21.15.116:51204) 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.084+0000 7fb0d1000700 1 -- 172.21.15.116:0/1067289186 learned_addr learned my addr 172.21.15.116:0/1067289186 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.084+0000 7fb0cbe00700 1 --2- 172.21.15.116:0/1067289186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb0cc101e70 0x7fb0cc199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.085+0000 7fb0d1a00700 1 -- 172.21.15.116:0/1067289186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0cc100c80 msgr2=0x7fb0cc198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.085+0000 7fb0d1a00700 1 --2- 172.21.15.116:0/1067289186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0cc100c80 0x7fb0cc198e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.085+0000 7fb0d1a00700 1 -- 172.21.15.116:0/1067289186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb0cc101e70 msgr2=0x7fb0cc199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.085+0000 7fb0d1a00700 1 --2- 172.21.15.116:0/1067289186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb0cc101e70 0x7fb0cc199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.085+0000 7fb0d1a00700 1 -- 172.21.15.116:0/1067289186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c00095f0 con 0x7fb0cc102810 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.085+0000 7fb0d1a00700 1 --2- 172.21.15.116:0/1067289186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0cc102810 0x7fb0cc1a0380 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400efc0 tx=0x7fb0c400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.085+0000 7fb0c9600700 1 -- 172.21.15.116:0/1067289186 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4018070 con 0x7fb0cc102810 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.085+0000 7fb0c9600700 1 -- 172.21.15.116:0/1067289186 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0c400f040 con 0x7fb0cc102810 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.085+0000 7fb0d3a00700 1 -- 172.21.15.116:0/1067289186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0cc1a0ea0 con 0x7fb0cc102810 2024-03-30T01:04:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.085+0000 7fb0c9600700 1 -- 172.21.15.116:0/1067289186 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c400cd40 con 0x7fb0cc102810 2024-03-30T01:04:47.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.085+0000 7fb0d3a00700 1 -- 172.21.15.116:0/1067289186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0cc1a1370 con 0x7fb0cc102810 2024-03-30T01:04:47.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.086+0000 7fb0d3a00700 1 -- 172.21.15.116:0/1067289186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0cc067a10 con 0x7fb0cc102810 2024-03-30T01:04:47.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.088+0000 7fb0c9600700 1 -- 172.21.15.116:0/1067289186 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fb0c4003a00 con 0x7fb0cc102810 2024-03-30T01:04:47.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.090+0000 7fb0c9600700 1 --2- 172.21.15.116:0/1067289186 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb0b40a11a0 0x7fb0b40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.090+0000 7fb0c9600700 1 -- 172.21.15.116:0/1067289186 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb0c40cc6d0 con 0x7fb0cc102810 2024-03-30T01:04:47.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.090+0000 7fb0d1000700 1 --2- 172.21.15.116:0/1067289186 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb0b40a11a0 0x7fb0b40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.091+0000 7fb0d1000700 1 --2- 172.21.15.116:0/1067289186 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb0b40a11a0 0x7fb0b40a3660 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0000c00 tx=0x7fb0c0005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:47.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.094+0000 7fb0c9600700 1 -- 172.21.15.116:0/1067289186 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb0c4095be0 con 0x7fb0cc102810 2024-03-30T01:04:47.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.349+0000 7fb0d3a00700 1 -- 172.21.15.116:0/1067289186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0cc10c4e0 con 0x7fb0cc102810 2024-03-30T01:04:47.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.349+0000 7fb0c9600700 1 -- 172.21.15.116:0/1067289186 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10893 (secure 0 0 0) 0x7fb0c401b320 con 0x7fb0cc102810 2024-03-30T01:04:47.356 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T01:04:47.356 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T01:03:06.538975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T01:04:47.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.352+0000 7fb0d3a00700 1 -- 172.21.15.116:0/1067289186 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb0b40a11a0 msgr2=0x7fb0b40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.352+0000 7fb0d3a00700 1 --2- 172.21.15.116:0/1067289186 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb0b40a11a0 0x7fb0b40a3660 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0000c00 tx=0x7fb0c0005a90 comp rx=0 tx=0).stop 2024-03-30T01:04:47.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.352+0000 7fb0d3a00700 1 -- 172.21.15.116:0/1067289186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0cc102810 msgr2=0x7fb0cc1a0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.352+0000 7fb0d3a00700 1 --2- 172.21.15.116:0/1067289186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0cc102810 0x7fb0cc1a0380 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400efc0 tx=0x7fb0c400c3a0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.352+0000 7fb0d3a00700 1 -- 172.21.15.116:0/1067289186 shutdown_connections 2024-03-30T01:04:47.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.352+0000 7fb0d3a00700 1 --2- 172.21.15.116:0/1067289186 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb0b40a11a0 0x7fb0b40a3660 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.352+0000 7fb0d3a00700 1 --2- 172.21.15.116:0/1067289186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0cc100c80 0x7fb0cc198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.352+0000 7fb0d3a00700 1 --2- 172.21.15.116:0/1067289186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0cc102810 0x7fb0cc1a0380 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.352+0000 7fb0d3a00700 1 --2- 172.21.15.116:0/1067289186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb0cc101e70 0x7fb0cc199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.352+0000 7fb0d3a00700 1 -- 172.21.15.116:0/1067289186 >> 172.21.15.116:0/1067289186 conn(0x7fb0cc0fc1f0 msgr2=0x7fb0cc107640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:47.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.352+0000 7fb0d3a00700 1 -- 172.21.15.116:0/1067289186 shutdown_connections 2024-03-30T01:04:47.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.352+0000 7fb0d3a00700 1 -- 172.21.15.116:0/1067289186 wait complete. 2024-03-30T01:04:47.360 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-03-30T01:04:47.377 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T01:04:47.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.568+0000 7f9178800700 1 -- 172.21.15.116:0/2033930841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170100d80 msgr2=0x7f91701011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.568+0000 7f9178800700 1 --2- 172.21.15.116:0/2033930841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170100d80 0x7f91701011a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f91640099c0 tx=0x7f9164009cd0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.568+0000 7f9178800700 1 -- 172.21.15.116:0/2033930841 shutdown_connections 2024-03-30T01:04:47.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.568+0000 7f9178800700 1 --2- 172.21.15.116:0/2033930841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9170102910 0x7f91701091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.568+0000 7f9178800700 1 --2- 172.21.15.116:0/2033930841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170100d80 0x7f91701011a0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.568+0000 7f9178800700 1 --2- 172.21.15.116:0/2033930841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9170101f70 0x7f91701023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.568+0000 7f9178800700 1 -- 172.21.15.116:0/2033930841 >> 172.21.15.116:0/2033930841 conn(0x7f91700fc2f0 msgr2=0x7f91700fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:47.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.568+0000 7f9178800700 1 -- 172.21.15.116:0/2033930841 shutdown_connections 2024-03-30T01:04:47.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.568+0000 7f9178800700 1 -- 172.21.15.116:0/2033930841 wait complete. 2024-03-30T01:04:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.569+0000 7f9178800700 1 Processor -- start 2024-03-30T01:04:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.569+0000 7f9178800700 1 -- start start 2024-03-30T01:04:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.569+0000 7f9178800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9170100d80 0x7f9170196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.569+0000 7f9178800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170101f70 0x7f91701972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.569+0000 7f9178800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9170102910 0x7f917019e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.569+0000 7f9178800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917019e830 con 0x7f9170101f70 2024-03-30T01:04:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.569+0000 7f9178800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917019e9a0 con 0x7f9170100d80 2024-03-30T01:04:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.569+0000 7f9178800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917019eb60 con 0x7f9170102910 2024-03-30T01:04:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.570+0000 7f9175400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170101f70 0x7f91701972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.570+0000 7f9175e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9170100d80 0x7f9170196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.570+0000 7f9175400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170101f70 0x7f91701972a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51618/0 (socket says 172.21.15.116:51618) 2024-03-30T01:04:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.570+0000 7f9175e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9170100d80 0x7f9170196d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38312/0 (socket says 172.21.15.116:38312) 2024-03-30T01:04:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.570+0000 7f9175400700 1 -- 172.21.15.116:0/1890143549 learned_addr learned my addr 172.21.15.116:0/1890143549 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.570+0000 7f9176800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9170102910 0x7f917019e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.570+0000 7f9175400700 1 -- 172.21.15.116:0/1890143549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9170102910 msgr2=0x7f917019e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.570+0000 7f9175400700 1 --2- 172.21.15.116:0/1890143549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9170102910 0x7f917019e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.570+0000 7f9175400700 1 -- 172.21.15.116:0/1890143549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9170100d80 msgr2=0x7f9170196d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.570+0000 7f9175400700 1 --2- 172.21.15.116:0/1890143549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9170100d80 0x7f9170196d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.570+0000 7f9175400700 1 -- 172.21.15.116:0/1890143549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91640095f0 con 0x7f9170101f70 2024-03-30T01:04:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.570+0000 7f9175e00700 1 --2- 172.21.15.116:0/1890143549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9170100d80 0x7f9170196d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.570+0000 7f9175400700 1 --2- 172.21.15.116:0/1890143549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170101f70 0x7f91701972a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f916000d950 tx=0x7f916000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.571+0000 7f916aa00700 1 -- 172.21.15.116:0/1890143549 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916000f890 con 0x7f9170101f70 2024-03-30T01:04:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.571+0000 7f916aa00700 1 -- 172.21.15.116:0/1890143549 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9160010460 con 0x7f9170101f70 2024-03-30T01:04:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.571+0000 7f9178800700 1 -- 172.21.15.116:0/1890143549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917019ee40 con 0x7f9170101f70 2024-03-30T01:04:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.571+0000 7f916aa00700 1 -- 172.21.15.116:0/1890143549 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916000e610 con 0x7f9170101f70 2024-03-30T01:04:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.571+0000 7f9178800700 1 -- 172.21.15.116:0/1890143549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f917019f2c0 con 0x7f9170101f70 2024-03-30T01:04:47.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.574+0000 7f916aa00700 1 -- 172.21.15.116:0/1890143549 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f91600108d0 con 0x7f9170101f70 2024-03-30T01:04:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.575+0000 7f918c3e4b80 1 Processor -- start 2024-03-30T01:04:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.575+0000 7f916aa00700 1 --2- 172.21.15.116:0/1890143549 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91580a11a0 0x7f91580a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.575+0000 7f918c3e4b80 1 -- start start 2024-03-30T01:04:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.575+0000 7f916aa00700 1 -- 172.21.15.116:0/1890143549 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f91600cccf0 con 0x7f9170101f70 2024-03-30T01:04:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.575+0000 7f9176800700 1 --2- 172.21.15.116:0/1890143549 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91580a11a0 0x7f91580a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.575+0000 7f918c3e4b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 0x556b0aa3cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.575+0000 7f918c3e4b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3d130 0x556b0aa3d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.575+0000 7f918c3e4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3dfd0 0x556b0aa3e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.575+0000 7f918c3e4b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b0a994990 con 0x556b0aa3d130 2024-03-30T01:04:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.576+0000 7f918c3e4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b0a992350 con 0x556b0aa3dfd0 2024-03-30T01:04:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.576+0000 7f918c3e4b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b0a98dc40 con 0x556b0aa3c7d0 2024-03-30T01:04:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.576+0000 7f9175400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3d130 0x556b0aa3d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.576+0000 7f9175400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3d130 0x556b0aa3d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51634/0 (socket says 172.21.15.116:51634) 2024-03-30T01:04:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.576+0000 7f9175400700 1 -- 172.21.15.116:0/4144996649 learned_addr learned my addr 172.21.15.116:0/4144996649 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.576+0000 7f9175e00700 1 --2- 172.21.15.116:0/4144996649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 0x556b0aa3cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.576+0000 7f9176800700 1 --2- 172.21.15.116:0/4144996649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3dfd0 0x556b0aa3e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.576+0000 7f9175400700 1 -- 172.21.15.116:0/4144996649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 msgr2=0x556b0aa3cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.576+0000 7f9175400700 1 --2- 172.21.15.116:0/4144996649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 0x556b0aa3cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.576+0000 7f9175400700 1 -- 172.21.15.116:0/4144996649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3dfd0 msgr2=0x556b0aa3e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.576+0000 7f9175400700 1 --2- 172.21.15.116:0/4144996649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3dfd0 0x556b0aa3e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.576+0000 7f9176800700 1 --2- 172.21.15.116:0/1890143549 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91580a11a0 0x7f91580a3660 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f916c003f20 tx=0x7f916c009ba0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.576+0000 7f9175400700 1 -- 172.21.15.116:0/4144996649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b0a80f3b0 con 0x556b0aa3d130 2024-03-30T01:04:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.576+0000 7f9175400700 1 --2- 172.21.15.116:0/4144996649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3d130 0x556b0aa3d9f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9160096840 tx=0x7f916009a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=623782aacbeab1c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.577+0000 7f915f000700 1 -- 172.21.15.116:0/4144996649 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916009dc00 con 0x556b0aa3d130 2024-03-30T01:04:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.577+0000 7f915f000700 1 -- 172.21.15.116:0/4144996649 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91600a2250 con 0x556b0aa3d130 2024-03-30T01:04:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.577+0000 7f915f000700 1 -- 172.21.15.116:0/4144996649 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91600992c0 con 0x556b0aa3d130 2024-03-30T01:04:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.577+0000 7f918c3e4b80 1 -- 172.21.15.116:0/4144996649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3d130 msgr2=0x556b0aa3d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.577+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/4144996649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3d130 0x556b0aa3d9f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9160096840 tx=0x7f916009a550 comp rx=0 tx=0).stop 2024-03-30T01:04:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.577+0000 7f918c3e4b80 1 -- 172.21.15.116:0/4144996649 shutdown_connections 2024-03-30T01:04:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.577+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/4144996649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 0x556b0aa3cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.577+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/4144996649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3dfd0 0x556b0aa3e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.577+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/4144996649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3d130 0x556b0aa3d9f0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.577+0000 7f918c3e4b80 1 -- 172.21.15.116:0/4144996649 >> 172.21.15.116:0/4144996649 conn(0x556b0a99b6a0 msgr2=0x556b0aa379a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.578+0000 7f918c3e4b80 1 -- 172.21.15.116:0/4144996649 shutdown_connections 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.578+0000 7f918c3e4b80 1 -- 172.21.15.116:0/4144996649 wait complete. 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.578+0000 7f918c3e4b80 1 -- start start 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.578+0000 7f918c3e4b80 1 Processor -- start 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f918c3e4b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 0x556b0aa4aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f918c3e4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3d130 0x556b0aa4af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f918c3e4b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3dfd0 0x556b0aa51f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f918c3e4b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b0a929d80 con 0x556b0aa3dfd0 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f918c3e4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b0a929a20 con 0x556b0aa3d130 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f918c3e4b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b0a8116a0 con 0x556b0aa3c7d0 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f918c3e4b80 10 client.0 _ll_register_callbacks cb 0x556b0a933470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f9176800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3dfd0 0x556b0aa51f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f9176800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3dfd0 0x556b0aa51f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51648/0 (socket says 172.21.15.116:51648) 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f9176800700 1 -- 172.21.15.116:0/3049717713 learned_addr learned my addr 172.21.15.116:0/3049717713 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f9175400700 1 --2- 172.21.15.116:0/3049717713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3d130 0x556b0aa4af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f9175e00700 1 --2- 172.21.15.116:0/3049717713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 0x556b0aa4aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f9176800700 1 -- 172.21.15.116:0/3049717713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 msgr2=0x556b0aa4aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f9176800700 1 --2- 172.21.15.116:0/3049717713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 0x556b0aa4aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f9176800700 1 -- 172.21.15.116:0/3049717713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3d130 msgr2=0x556b0aa4af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f9176800700 1 --2- 172.21.15.116:0/3049717713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3d130 0x556b0aa4af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f9176800700 1 -- 172.21.15.116:0/3049717713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f917019f2c0 con 0x556b0aa3dfd0 2024-03-30T01:04:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.579+0000 7f9175400700 1 --2- 172.21.15.116:0/3049717713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3d130 0x556b0aa4af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.580+0000 7f9176800700 1 --2- 172.21.15.116:0/3049717713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3dfd0 0x556b0aa51f70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f916c008890 tx=0x7f916c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea3dd29958c885a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.580+0000 7f9153400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:04:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.580+0000 7f9153400700 1 -- 172.21.15.116:0/3049717713 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c004bb0 con 0x556b0aa3dfd0 2024-03-30T01:04:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.580+0000 7f9153400700 1 -- 172.21.15.116:0/3049717713 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f916c004d10 con 0x556b0aa3dfd0 2024-03-30T01:04:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.580+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3049717713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556b0a94e380 con 0x556b0aa3dfd0 2024-03-30T01:04:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.580+0000 7f9153400700 1 -- 172.21.15.116:0/3049717713 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f916c023750 con 0x556b0aa3dfd0 2024-03-30T01:04:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.580+0000 7f918c3e4b80 10 client.16194 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.580+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3049717713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556b0a7bbb60 con 0x556b0aa3dfd0 2024-03-30T01:04:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.581+0000 7f9153400700 1 -- 172.21.15.116:0/3049717713 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556b0a94e380 con 0x556b0aa3dfd0 2024-03-30T01:04:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.581+0000 7f9153400700 1 -- 172.21.15.116:0/3049717713 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f916c018860 con 0x556b0aa3dfd0 2024-03-30T01:04:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.581+0000 7f918c3e4b80 10 client.16194 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.581+0000 7f918c3e4b80 10 client.16194 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T01:04:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.581+0000 7f918c3e4b80 20 client.16194 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.581+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/3049717713 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x556b0aa56950 0x556b0aa58d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.581+0000 7f918c3e4b80 4 client.16194 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:04:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.581+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3049717713 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b0a986700 con 0x556b0aa56950 2024-03-30T01:04:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.581+0000 7f9175e00700 1 --2- 172.21.15.116:0/3049717713 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x556b0aa56950 0x556b0aa58d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.582+0000 7f9175e00700 1 --2- 172.21.15.116:0/3049717713 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x556b0aa56950 0x556b0aa58d70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c11e3726c8f7fb35 server_cookie=926813ae942a5da9 in_seq=0 out_seq=0 2024-03-30T01:04:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.582+0000 7f9153400700 10 client.16194 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T01:04:47.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.582+0000 7f9153400700 1 -- 172.21.15.116:0/3049717713 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556b0a986700 con 0x556b0aa56950 2024-03-30T01:04:47.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.582+0000 7f9153400700 10 client.16194 handle_command_reply: tid=0 2024-03-30T01:04:47.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.582+0000 7f918c3e4b80 1 client.16194 shutdown 2024-03-30T01:04:47.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.583+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3049717713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3dfd0 msgr2=0x556b0aa51f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.583+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/3049717713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3dfd0 0x556b0aa51f70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f916c008890 tx=0x7f916c004b40 comp rx=0 tx=0).stop 2024-03-30T01:04:47.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.583+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3049717713 shutdown_connections 2024-03-30T01:04:47.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.583+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/3049717713 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x556b0aa56950 0x556b0aa58d70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.583+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/3049717713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 0x556b0aa4aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.583+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/3049717713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3dfd0 0x556b0aa51f70 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.583+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/3049717713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3d130 0x556b0aa4af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.583+0000 7f9153400700 0 client.16194 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T01:04:47.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.583+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3049717713 >> 172.21.15.116:0/3049717713 conn(0x556b0a99b6a0 msgr2=0x556b0aa3b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:47.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.583+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3049717713 shutdown_connections 2024-03-30T01:04:47.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.583+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3049717713 wait complete. 2024-03-30T01:04:47.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.583+0000 7f918c3e4b80 20 client.16194 trim_cache size 0 max 16384 2024-03-30T01:04:47.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.597+0000 7f918c3e4b80 1 Processor -- start 2024-03-30T01:04:47.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.597+0000 7f918c3e4b80 1 -- start start 2024-03-30T01:04:47.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.597+0000 7f918c3e4b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 0x556b0aa3cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.597+0000 7f918c3e4b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3d130 0x556b0aa51e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.597+0000 7f918c3e4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3dfd0 0x556b0aa45b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.597+0000 7f918c3e4b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b0aa59100 con 0x556b0aa3d130 2024-03-30T01:04:47.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.597+0000 7f918c3e4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b0aa38660 con 0x556b0aa3dfd0 2024-03-30T01:04:47.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.597+0000 7f918c3e4b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b0aa3d510 con 0x556b0aa3c7d0 2024-03-30T01:04:47.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.598+0000 7f9175400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3d130 0x556b0aa51e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.598+0000 7f9175400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3d130 0x556b0aa51e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51662/0 (socket says 172.21.15.116:51662) 2024-03-30T01:04:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.598+0000 7f9175400700 1 -- 172.21.15.116:0/1816830485 learned_addr learned my addr 172.21.15.116:0/1816830485 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.598+0000 7f9176800700 1 --2- 172.21.15.116:0/1816830485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3dfd0 0x556b0aa45b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.598+0000 7f9175e00700 1 --2- 172.21.15.116:0/1816830485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 0x556b0aa3cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.598+0000 7f9175400700 1 -- 172.21.15.116:0/1816830485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 msgr2=0x556b0aa3cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.598+0000 7f9175400700 1 --2- 172.21.15.116:0/1816830485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 0x556b0aa3cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.598+0000 7f9175400700 1 -- 172.21.15.116:0/1816830485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3dfd0 msgr2=0x556b0aa45b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.598+0000 7f9175400700 1 --2- 172.21.15.116:0/1816830485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3dfd0 0x556b0aa45b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.598+0000 7f9175400700 1 -- 172.21.15.116:0/1816830485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b0a953ea0 con 0x556b0aa3d130 2024-03-30T01:04:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.598+0000 7f9176800700 1 --2- 172.21.15.116:0/1816830485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3dfd0 0x556b0aa45b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:04:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.598+0000 7f9175400700 1 --2- 172.21.15.116:0/1816830485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3d130 0x556b0aa51e90 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9160099d10 tx=0x7f9160099f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8414e55b5f6a3054 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.599+0000 7f915fa00700 1 -- 172.21.15.116:0/1816830485 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916009dc00 con 0x556b0aa3d130 2024-03-30T01:04:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.599+0000 7f915fa00700 1 -- 172.21.15.116:0/1816830485 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9160098170 con 0x556b0aa3d130 2024-03-30T01:04:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.599+0000 7f915fa00700 1 -- 172.21.15.116:0/1816830485 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160099590 con 0x556b0aa3d130 2024-03-30T01:04:47.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.599+0000 7f918c3e4b80 1 -- 172.21.15.116:0/1816830485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3d130 msgr2=0x556b0aa51e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.599+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/1816830485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3d130 0x556b0aa51e90 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9160099d10 tx=0x7f9160099f10 comp rx=0 tx=0).stop 2024-03-30T01:04:47.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.599+0000 7f918c3e4b80 1 -- 172.21.15.116:0/1816830485 shutdown_connections 2024-03-30T01:04:47.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.599+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/1816830485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3c7d0 0x556b0aa3cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.599+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/1816830485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3dfd0 0x556b0aa45b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.599+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/1816830485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3d130 0x556b0aa51e90 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.599+0000 7f918c3e4b80 1 -- 172.21.15.116:0/1816830485 >> 172.21.15.116:0/1816830485 conn(0x556b0a99b6a0 msgr2=0x556b0aa44540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:47.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.599+0000 7f918c3e4b80 1 -- 172.21.15.116:0/1816830485 shutdown_connections 2024-03-30T01:04:47.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.599+0000 7f918c3e4b80 1 -- 172.21.15.116:0/1816830485 wait complete. 2024-03-30T01:04:47.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.600+0000 7f918c3e4b80 1 -- start start 2024-03-30T01:04:47.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.600+0000 7f918c3e4b80 1 Processor -- start 2024-03-30T01:04:47.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.600+0000 7f918c3e4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3c7d0 0x556b0aa4f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.600+0000 7f918c3e4b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3d130 0x556b0aa4fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.600+0000 7f918c3e4b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3dfd0 0x556b0aa50c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f918c3e4b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b0aa511c0 con 0x556b0aa3dfd0 2024-03-30T01:04:47.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f918c3e4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b0aa51330 con 0x556b0aa3c7d0 2024-03-30T01:04:47.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f918c3e4b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b0aa514f0 con 0x556b0aa3d130 2024-03-30T01:04:47.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f918c3e4b80 10 client.0 _ll_register_callbacks cb 0x556b0a933470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f9176800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3dfd0 0x556b0aa50c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f9175e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3c7d0 0x556b0aa4f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f9176800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3dfd0 0x556b0aa50c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51666/0 (socket says 172.21.15.116:51666) 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f9175e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3c7d0 0x556b0aa4f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38332/0 (socket says 172.21.15.116:38332) 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f9176800700 1 -- 172.21.15.116:0/3084066260 learned_addr learned my addr 172.21.15.116:0/3084066260 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f9175400700 1 --2- 172.21.15.116:0/3084066260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3d130 0x556b0aa4fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f9175e00700 1 -- 172.21.15.116:0/3084066260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3d130 msgr2=0x556b0aa4fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f9175e00700 1 --2- 172.21.15.116:0/3084066260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3d130 0x556b0aa4fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f9175e00700 1 -- 172.21.15.116:0/3084066260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3dfd0 msgr2=0x556b0aa50c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f9175e00700 1 --2- 172.21.15.116:0/3084066260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3dfd0 0x556b0aa50c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f9175e00700 1 -- 172.21.15.116:0/3084066260 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b0a80f3b0 con 0x556b0aa3c7d0 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f9176800700 1 --2- 172.21.15.116:0/3084066260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3dfd0 0x556b0aa50c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.601+0000 7f9175400700 1 --2- 172.21.15.116:0/3084066260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3d130 0x556b0aa4fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.602+0000 7f9175e00700 1 --2- 172.21.15.116:0/3084066260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3c7d0 0x556b0aa4f870 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f91640036f0 tx=0x7f916400b540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9583209ad19922b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.602+0000 7f9153400700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T01:04:47.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.602+0000 7f9153400700 1 -- 172.21.15.116:0/3084066260 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916401d040 con 0x556b0aa3c7d0 2024-03-30T01:04:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.602+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3084066260 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f91600a2250 con 0x556b0aa3c7d0 2024-03-30T01:04:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.602+0000 7f9153400700 1 -- 172.21.15.116:0/3084066260 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9164005320 con 0x556b0aa3c7d0 2024-03-30T01:04:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.602+0000 7f9153400700 1 -- 172.21.15.116:0/3084066260 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9164005500 con 0x556b0aa3c7d0 2024-03-30T01:04:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.602+0000 7f918c3e4b80 10 client.26189 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.602+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3084066260 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556b0a813cd0 con 0x556b0aa3c7d0 2024-03-30T01:04:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.603+0000 7f9153400700 1 -- 172.21.15.116:0/3084066260 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91600a2250 con 0x556b0aa3c7d0 2024-03-30T01:04:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.603+0000 7f9153400700 1 -- 172.21.15.116:0/3084066260 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f91640099f0 con 0x556b0aa3c7d0 2024-03-30T01:04:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.603+0000 7f918c3e4b80 10 client.26189 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.603+0000 7f918c3e4b80 10 client.26189 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T01:04:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.603+0000 7f918c3e4b80 20 client.26189 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.603+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/3084066260 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x556b0aaab140 0x556b0aaad560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.603+0000 7f918c3e4b80 4 client.26189 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:04:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.603+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3084066260 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556b0a7dccf0 con 0x556b0aaab140 2024-03-30T01:04:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.603+0000 7f9175400700 1 --2- 172.21.15.116:0/3084066260 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x556b0aaab140 0x556b0aaad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f9175400700 1 --2- 172.21.15.116:0/3084066260 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x556b0aaab140 0x556b0aaad560 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ec84fbcc6b51473 server_cookie=4a526f262ab5e5d1 in_seq=0 out_seq=0 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f9153400700 10 client.26189 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f9153400700 1 -- 172.21.15.116:0/3084066260 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556b0a7dccf0 con 0x556b0aaab140 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f9153400700 10 client.26189 handle_command_reply: tid=0 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f918c3e4b80 1 client.26189 shutdown 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3084066260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3c7d0 msgr2=0x556b0aa4f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/3084066260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3c7d0 0x556b0aa4f870 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f91640036f0 tx=0x7f916400b540 comp rx=0 tx=0).stop 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3084066260 shutdown_connections 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/3084066260 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x556b0aaab140 0x556b0aaad560 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/3084066260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b0aa3dfd0 0x556b0aa50c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/3084066260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b0aa3c7d0 0x556b0aa4f870 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f918c3e4b80 1 --2- 172.21.15.116:0/3084066260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556b0aa3d130 0x556b0aa4fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f9153400700 0 client.26189 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3084066260 >> 172.21.15.116:0/3084066260 conn(0x556b0a99b6a0 msgr2=0x556b0aa45a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3084066260 shutdown_connections 2024-03-30T01:04:47.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f918c3e4b80 1 -- 172.21.15.116:0/3084066260 wait complete. 2024-03-30T01:04:47.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.604+0000 7f918c3e4b80 20 client.26189 trim_cache size 0 max 16384 2024-03-30T01:04:47.616 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:04:47.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.605+0000 7f9178800700 1 -- 172.21.15.116:0/1890143549 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91580a11a0 msgr2=0x7f91580a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.605+0000 7f9178800700 1 --2- 172.21.15.116:0/1890143549 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91580a11a0 0x7f91580a3660 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f916c003f20 tx=0x7f916c009ba0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.605+0000 7f9178800700 1 -- 172.21.15.116:0/1890143549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170101f70 msgr2=0x7f91701972a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.605+0000 7f9178800700 1 --2- 172.21.15.116:0/1890143549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170101f70 0x7f91701972a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f916000d950 tx=0x7f916000dc60 comp rx=0 tx=0).stop 2024-03-30T01:04:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.605+0000 7f9178800700 1 -- 172.21.15.116:0/1890143549 shutdown_connections 2024-03-30T01:04:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.605+0000 7f9178800700 1 --2- 172.21.15.116:0/1890143549 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f91580a11a0 0x7f91580a3660 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.605+0000 7f9178800700 1 --2- 172.21.15.116:0/1890143549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9170102910 0x7f917019e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.605+0000 7f9178800700 1 --2- 172.21.15.116:0/1890143549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9170100d80 0x7f9170196d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.605+0000 7f9178800700 1 --2- 172.21.15.116:0/1890143549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170101f70 0x7f91701972a0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.605+0000 7f9178800700 1 -- 172.21.15.116:0/1890143549 >> 172.21.15.116:0/1890143549 conn(0x7f91700fc2f0 msgr2=0x7f9170105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.605+0000 7f9178800700 1 -- 172.21.15.116:0/1890143549 shutdown_connections 2024-03-30T01:04:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.605+0000 7f9178800700 1 -- 172.21.15.116:0/1890143549 wait complete. 2024-03-30T01:04:47.626 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:47 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1067289186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:04:47.629 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T01:04:47.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.817+0000 7f4ce0600700 1 -- 172.21.15.116:0/2011549074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd8101f70 msgr2=0x7f4cd81023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.817+0000 7f4ce0600700 1 --2- 172.21.15.116:0/2011549074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd8101f70 0x7f4cd81023d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4cc40099c0 tx=0x7f4cc4009cd0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.818+0000 7f4ce0600700 1 -- 172.21.15.116:0/2011549074 shutdown_connections 2024-03-30T01:04:47.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.818+0000 7f4ce0600700 1 --2- 172.21.15.116:0/2011549074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cd8102910 0x7f4cd81091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.818+0000 7f4ce0600700 1 --2- 172.21.15.116:0/2011549074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cd8100d80 0x7f4cd81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.818+0000 7f4ce0600700 1 --2- 172.21.15.116:0/2011549074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd8101f70 0x7f4cd81023d0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.818+0000 7f4ce0600700 1 -- 172.21.15.116:0/2011549074 >> 172.21.15.116:0/2011549074 conn(0x7f4cd80fc2f0 msgr2=0x7f4cd80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:47.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.818+0000 7f4ce0600700 1 -- 172.21.15.116:0/2011549074 shutdown_connections 2024-03-30T01:04:47.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.818+0000 7f4ce0600700 1 -- 172.21.15.116:0/2011549074 wait complete. 2024-03-30T01:04:47.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.819+0000 7f4ce0600700 1 Processor -- start 2024-03-30T01:04:47.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.819+0000 7f4ce0600700 1 -- start start 2024-03-30T01:04:47.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.819+0000 7f4ce0600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cd8100d80 0x7f4cd8198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.819+0000 7f4ce0600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd8101f70 0x7f4cd8199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.819+0000 7f4ce0600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cd8102910 0x7f4cd81a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.819+0000 7f4ce0600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd81a0a20 con 0x7f4cd8101f70 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.819+0000 7f4ce0600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd81a0b60 con 0x7f4cd8100d80 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.819+0000 7f4ce0600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd81a0cf0 con 0x7f4cd8102910 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.820+0000 7f4cdd200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd8101f70 0x7f4cd8199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.820+0000 7f4cddc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cd8100d80 0x7f4cd8198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.820+0000 7f4cdd200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd8101f70 0x7f4cd8199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51692/0 (socket says 172.21.15.116:51692) 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.820+0000 7f4cdd200700 1 -- 172.21.15.116:0/1797037743 learned_addr learned my addr 172.21.15.116:0/1797037743 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.820+0000 7f4cddc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cd8100d80 0x7f4cd8198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38340/0 (socket says 172.21.15.116:38340) 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.820+0000 7f4cde600700 1 --2- 172.21.15.116:0/1797037743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cd8102910 0x7f4cd81a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.820+0000 7f4cddc00700 1 -- 172.21.15.116:0/1797037743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cd8102910 msgr2=0x7f4cd81a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.820+0000 7f4cddc00700 1 --2- 172.21.15.116:0/1797037743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cd8102910 0x7f4cd81a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.820+0000 7f4cddc00700 1 -- 172.21.15.116:0/1797037743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd8101f70 msgr2=0x7f4cd8199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.820+0000 7f4cddc00700 1 --2- 172.21.15.116:0/1797037743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd8101f70 0x7f4cd8199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.820+0000 7f4cddc00700 1 -- 172.21.15.116:0/1797037743 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc40095f0 con 0x7f4cd8100d80 2024-03-30T01:04:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.820+0000 7f4cdd200700 1 --2- 172.21.15.116:0/1797037743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd8101f70 0x7f4cd8199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:47.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.820+0000 7f4cddc00700 1 --2- 172.21.15.116:0/1797037743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cd8100d80 0x7f4cd8198ef0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc00d720 tx=0x7f4ccc00da30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:47.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.821+0000 7f4cd6000700 1 -- 172.21.15.116:0/1797037743 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc00f690 con 0x7f4cd8100d80 2024-03-30T01:04:47.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.821+0000 7f4ce0600700 1 -- 172.21.15.116:0/1797037743 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd81a0fd0 con 0x7f4cd8100d80 2024-03-30T01:04:47.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.821+0000 7f4cd6000700 1 -- 172.21.15.116:0/1797037743 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ccc00fcb0 con 0x7f4cd8100d80 2024-03-30T01:04:47.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.821+0000 7f4cd6000700 1 -- 172.21.15.116:0/1797037743 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc010e60 con 0x7f4cd8100d80 2024-03-30T01:04:47.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.821+0000 7f4ce0600700 1 -- 172.21.15.116:0/1797037743 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd81a1400 con 0x7f4cd8100d80 2024-03-30T01:04:47.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.826+0000 7f4cd6000700 1 -- 172.21.15.116:0/1797037743 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f4ccc00f7f0 con 0x7f4cd8100d80 2024-03-30T01:04:47.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.826+0000 7f4cf407bb80 1 Processor -- start 2024-03-30T01:04:47.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.826+0000 7f4cf407bb80 1 -- start start 2024-03-30T01:04:47.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.826+0000 7f4cf407bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 0x55c9131a8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.826+0000 7f4cf407bb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131a99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.827+0000 7f4cf407bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 0x55c9131aa450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.827+0000 7f4cf407bb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c913100990 con 0x55c9131a9130 2024-03-30T01:04:47.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.827+0000 7f4cf407bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9130fe350 con 0x55c9131a9fd0 2024-03-30T01:04:47.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.827+0000 7f4cf407bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9130f9c40 con 0x55c9131a87d0 2024-03-30T01:04:47.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.827+0000 7f4cddc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131a99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.827+0000 7f4cde600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 0x55c9131a8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.827+0000 7f4cddc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131a99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51702/0 (socket says 172.21.15.116:51702) 2024-03-30T01:04:47.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.827+0000 7f4cddc00700 1 -- 172.21.15.116:0/907742466 learned_addr learned my addr 172.21.15.116:0/907742466 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:47.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.827+0000 7f4cdd200700 1 --2- 172.21.15.116:0/907742466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 0x55c9131aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.827+0000 7f4cddc00700 1 -- 172.21.15.116:0/907742466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 msgr2=0x55c9131a8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.827+0000 7f4cddc00700 1 --2- 172.21.15.116:0/907742466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 0x55c9131a8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.827+0000 7f4cddc00700 1 -- 172.21.15.116:0/907742466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 msgr2=0x55c9131aa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.827+0000 7f4cddc00700 1 --2- 172.21.15.116:0/907742466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 0x55c9131aa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.827+0000 7f4cddc00700 1 -- 172.21.15.116:0/907742466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c912f7b3b0 con 0x55c9131a9130 2024-03-30T01:04:47.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.828+0000 7f4cddc00700 1 --2- 172.21.15.116:0/907742466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131a99f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0d2a80 tx=0x7f4ccc0d2d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff83720a1f5e2ed server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:47.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.828+0000 7f4ccaa00700 1 -- 172.21.15.116:0/907742466 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc0d4bc0 con 0x55c9131a9130 2024-03-30T01:04:47.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.828+0000 7f4ccaa00700 1 -- 172.21.15.116:0/907742466 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ccc0d1030 con 0x55c9131a9130 2024-03-30T01:04:47.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.828+0000 7f4ccaa00700 1 -- 172.21.15.116:0/907742466 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc0ce270 con 0x55c9131a9130 2024-03-30T01:04:47.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.828+0000 7f4cd6000700 1 --2- 172.21.15.116:0/1797037743 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4cc00a1120 0x7f4cc00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.828+0000 7f4cd6000700 1 -- 172.21.15.116:0/1797037743 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4ccc0cc830 con 0x7f4cd8100d80 2024-03-30T01:04:47.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.828+0000 7f4cde600700 1 --2- 172.21.15.116:0/1797037743 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4cc00a1120 0x7f4cc00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.829+0000 7f4cf407bb80 1 -- 172.21.15.116:0/907742466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 msgr2=0x55c9131a99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.829+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/907742466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131a99f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0d2a80 tx=0x7f4ccc0d2d90 comp rx=0 tx=0).stop 2024-03-30T01:04:47.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.829+0000 7f4cde600700 1 --2- 172.21.15.116:0/1797037743 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4cc00a1120 0x7f4cc00a35e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009710 tx=0x7f4cd0009100 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:47.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.829+0000 7f4cf407bb80 1 -- 172.21.15.116:0/907742466 shutdown_connections 2024-03-30T01:04:47.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.829+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/907742466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 0x55c9131a8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.829+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/907742466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 0x55c9131aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.829+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/907742466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131a99f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.829+0000 7f4cf407bb80 1 -- 172.21.15.116:0/907742466 >> 172.21.15.116:0/907742466 conn(0x55c9131076a0 msgr2=0x55c9131a39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:47.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.829+0000 7f4cf407bb80 1 -- 172.21.15.116:0/907742466 shutdown_connections 2024-03-30T01:04:47.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.829+0000 7f4cf407bb80 1 -- 172.21.15.116:0/907742466 wait complete. 2024-03-30T01:04:47.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.829+0000 7f4cf407bb80 1 -- start start 2024-03-30T01:04:47.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.830+0000 7f4cf407bb80 1 Processor -- start 2024-03-30T01:04:47.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.830+0000 7f4cf407bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a87d0 0x55c9131b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.830+0000 7f4cf407bb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.830+0000 7f4cf407bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a9fd0 0x55c9131b9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.830+0000 7f4cf407bb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c913095d80 con 0x55c9131a9130 2024-03-30T01:04:47.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.830+0000 7f4cf407bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c913095a20 con 0x55c9131a87d0 2024-03-30T01:04:47.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.830+0000 7f4cf407bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c912f7d6a0 con 0x55c9131a9fd0 2024-03-30T01:04:47.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.830+0000 7f4cf407bb80 10 client.0 _ll_register_callbacks cb 0x55c91309f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:47.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.830+0000 7f4cddc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a87d0 0x55c9131b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.830+0000 7f4cddc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a87d0 0x55c9131b26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38356/0 (socket says 172.21.15.116:38356) 2024-03-30T01:04:47.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.830+0000 7f4cdd200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.830+0000 7f4cddc00700 1 -- 172.21.15.116:0/2106323582 learned_addr learned my addr 172.21.15.116:0/2106323582 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:47.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.830+0000 7f4cde600700 1 --2- 172.21.15.116:0/2106323582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a9fd0 0x55c9131b9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.831+0000 7f4cddc00700 1 -- 172.21.15.116:0/2106323582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a9fd0 msgr2=0x55c9131b9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.831+0000 7f4cddc00700 1 --2- 172.21.15.116:0/2106323582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a9fd0 0x55c9131b9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.831+0000 7f4cddc00700 1 -- 172.21.15.116:0/2106323582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 msgr2=0x55c9131b2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.831+0000 7f4cddc00700 1 --2- 172.21.15.116:0/2106323582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.831+0000 7f4cddc00700 1 -- 172.21.15.116:0/2106323582 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd81a1400 con 0x55c9131a87d0 2024-03-30T01:04:47.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.831+0000 7f4cddc00700 1 --2- 172.21.15.116:0/2106323582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a87d0 0x55c9131b26e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc095e70 tx=0x7f4ccc0ccaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=832252c0f84700d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:47.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.831+0000 7f4cbf400700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T01:04:47.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.831+0000 7f4cbf400700 1 -- 172.21.15.116:0/2106323582 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc0d7a10 con 0x55c9131a87d0 2024-03-30T01:04:47.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.831+0000 7f4cf407bb80 1 -- 172.21.15.116:0/2106323582 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c9130ba380 con 0x55c9131a87d0 2024-03-30T01:04:47.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.831+0000 7f4cbf400700 1 -- 172.21.15.116:0/2106323582 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4ccc09a070 con 0x55c9131a87d0 2024-03-30T01:04:47.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.831+0000 7f4cbf400700 1 -- 172.21.15.116:0/2106323582 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ccc095cb0 con 0x55c9131a87d0 2024-03-30T01:04:47.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.831+0000 7f4cf407bb80 10 client.26201 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:47.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.831+0000 7f4cf407bb80 1 -- 172.21.15.116:0/2106323582 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c912f27b60 con 0x55c9131a87d0 2024-03-30T01:04:47.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.832+0000 7f4cbf400700 1 -- 172.21.15.116:0/2106323582 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c9130ba380 con 0x55c9131a87d0 2024-03-30T01:04:47.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.832+0000 7f4cbf400700 1 -- 172.21.15.116:0/2106323582 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f4ccc096080 con 0x55c9131a87d0 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.832+0000 7f4cf407bb80 10 client.26201 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.832+0000 7f4cf407bb80 10 client.26201 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.832+0000 7f4cf407bb80 20 client.26201 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.832+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/2106323582 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55c9131bdf20 0x55c9131c0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.832+0000 7f4cf407bb80 4 client.26201 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.832+0000 7f4cf407bb80 1 -- 172.21.15.116:0/2106323582 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c9130f2700 con 0x55c9131bdf20 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.832+0000 7f4cdd200700 1 --2- 172.21.15.116:0/2106323582 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55c9131bdf20 0x55c9131c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.833+0000 7f4cbf400700 10 client.26201 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.833+0000 7f4cdd200700 1 --2- 172.21.15.116:0/2106323582 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55c9131bdf20 0x55c9131c0340 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7eed9417c376e161 server_cookie=137e53d14768c4b5 in_seq=0 out_seq=0 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.833+0000 7f4cbf400700 1 -- 172.21.15.116:0/2106323582 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c9130f2700 con 0x55c9131bdf20 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.833+0000 7f4cbf400700 10 client.26201 handle_command_reply: tid=0 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.833+0000 7f4cf407bb80 1 client.26201 shutdown 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.833+0000 7f4cf407bb80 1 -- 172.21.15.116:0/2106323582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a87d0 msgr2=0x55c9131b26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.833+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/2106323582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a87d0 0x55c9131b26e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc095e70 tx=0x7f4ccc0ccaf0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.834+0000 7f4cf407bb80 1 -- 172.21.15.116:0/2106323582 shutdown_connections 2024-03-30T01:04:47.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.834+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/2106323582 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55c9131bdf20 0x55c9131c0340 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.834+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/2106323582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a9fd0 0x55c9131b9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.834+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/2106323582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a87d0 0x55c9131b26e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.834+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/2106323582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.834+0000 7f4cbf400700 0 client.26201 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T01:04:47.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.834+0000 7f4cf407bb80 1 -- 172.21.15.116:0/2106323582 >> 172.21.15.116:0/2106323582 conn(0x55c9131076a0 msgr2=0x55c9131a7170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:47.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.834+0000 7f4cf407bb80 1 -- 172.21.15.116:0/2106323582 shutdown_connections 2024-03-30T01:04:47.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.834+0000 7f4cf407bb80 1 -- 172.21.15.116:0/2106323582 wait complete. 2024-03-30T01:04:47.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.834+0000 7f4cf407bb80 20 client.26201 trim_cache size 0 max 16384 2024-03-30T01:04:47.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.848+0000 7f4cf407bb80 1 Processor -- start 2024-03-30T01:04:47.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.848+0000 7f4cf407bb80 1 -- start start 2024-03-30T01:04:47.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.848+0000 7f4cf407bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 0x55c9131ba270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.848+0000 7f4cf407bb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.848+0000 7f4cf407bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 0x55c9131b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.848+0000 7f4cf407bb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9131a9880 con 0x55c9131a9130 2024-03-30T01:04:47.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.848+0000 7f4cf407bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9131bdd10 con 0x55c9131a9fd0 2024-03-30T01:04:47.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.848+0000 7f4cf407bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9131a48b0 con 0x55c9131a87d0 2024-03-30T01:04:47.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.848+0000 7f4cdd200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.848+0000 7f4cdd200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51720/0 (socket says 172.21.15.116:51720) 2024-03-30T01:04:47.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.848+0000 7f4cdd200700 1 -- 172.21.15.116:0/2819422372 learned_addr learned my addr 172.21.15.116:0/2819422372 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:47.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.848+0000 7f4cde600700 1 --2- 172.21.15.116:0/2819422372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 0x55c9131b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.849+0000 7f4cddc00700 1 --2- 172.21.15.116:0/2819422372 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 0x55c9131ba270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.849+0000 7f4cdd200700 1 -- 172.21.15.116:0/2819422372 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 msgr2=0x55c9131ba270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.849+0000 7f4cdd200700 1 --2- 172.21.15.116:0/2819422372 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 0x55c9131ba270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.849+0000 7f4cdd200700 1 -- 172.21.15.116:0/2819422372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 msgr2=0x55c9131b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.849+0000 7f4cdd200700 1 --2- 172.21.15.116:0/2819422372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 0x55c9131b1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.849+0000 7f4cdd200700 1 -- 172.21.15.116:0/2819422372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9130bfea0 con 0x55c9131a9130 2024-03-30T01:04:47.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.849+0000 7f4cdd200700 1 --2- 172.21.15.116:0/2819422372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b9b80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4009f50 tx=0x7f4cc4005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4240a2883a175c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:47.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.849+0000 7f4ccb400700 1 -- 172.21.15.116:0/2819422372 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc401d040 con 0x55c9131a9130 2024-03-30T01:04:47.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.849+0000 7f4ccb400700 1 -- 172.21.15.116:0/2819422372 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4cc401e040 con 0x55c9131a9130 2024-03-30T01:04:47.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.849+0000 7f4ccb400700 1 -- 172.21.15.116:0/2819422372 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc40038c0 con 0x55c9131a9130 2024-03-30T01:04:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.850+0000 7f4cf407bb80 1 -- 172.21.15.116:0/2819422372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 msgr2=0x55c9131b9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.850+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/2819422372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b9b80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4009f50 tx=0x7f4cc4005be0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.850+0000 7f4cf407bb80 1 -- 172.21.15.116:0/2819422372 shutdown_connections 2024-03-30T01:04:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.850+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/2819422372 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 0x55c9131ba270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.850+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/2819422372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 0x55c9131b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.850+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/2819422372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b9b80 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.850+0000 7f4cf407bb80 1 -- 172.21.15.116:0/2819422372 >> 172.21.15.116:0/2819422372 conn(0x55c9131076a0 msgr2=0x55c91319c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.850+0000 7f4cf407bb80 1 -- 172.21.15.116:0/2819422372 shutdown_connections 2024-03-30T01:04:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.850+0000 7f4cf407bb80 1 -- 172.21.15.116:0/2819422372 wait complete. 2024-03-30T01:04:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.850+0000 7f4cf407bb80 1 -- start start 2024-03-30T01:04:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.851+0000 7f4cf407bb80 1 Processor -- start 2024-03-30T01:04:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.851+0000 7f4cf407bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 0x55c9131b7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.851+0000 7f4cf407bb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.851+0000 7f4cf407bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 0x55c9131b8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.851+0000 7f4cf407bb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9131b94e0 con 0x55c9131a9130 2024-03-30T01:04:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.851+0000 7f4cf407bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9131b9650 con 0x55c9131a9fd0 2024-03-30T01:04:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.851+0000 7f4cf407bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9131b9810 con 0x55c9131a87d0 2024-03-30T01:04:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.851+0000 7f4cf407bb80 10 client.0 _ll_register_callbacks cb 0x55c91309f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.851+0000 7f4cdd200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.851+0000 7f4cddc00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 0x55c9131b7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.851+0000 7f4cdd200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51724/0 (socket says 172.21.15.116:51724) 2024-03-30T01:04:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.851+0000 7f4cdd200700 1 -- 172.21.15.116:0/473456412 learned_addr learned my addr 172.21.15.116:0/473456412 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.851+0000 7f4cde600700 1 --2- 172.21.15.116:0/473456412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 0x55c9131b8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.852+0000 7f4cdd200700 1 -- 172.21.15.116:0/473456412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 msgr2=0x55c9131b7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.852+0000 7f4cdd200700 1 --2- 172.21.15.116:0/473456412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 0x55c9131b7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.852+0000 7f4cdd200700 1 -- 172.21.15.116:0/473456412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 msgr2=0x55c9131b8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.852+0000 7f4cdd200700 1 --2- 172.21.15.116:0/473456412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 0x55c9131b8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.852+0000 7f4cdd200700 1 -- 172.21.15.116:0/473456412 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd8109b10 con 0x55c9131a9130 2024-03-30T01:04:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.852+0000 7f4cddc00700 1 --2- 172.21.15.116:0/473456412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 0x55c9131b7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.852+0000 7f4cdd200700 1 --2- 172.21.15.116:0/473456412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b80d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400bf10 tx=0x7f4cc400bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75c151e7f7075c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.852+0000 7f4cbf400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:04:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.852+0000 7f4cbf400700 1 -- 172.21.15.116:0/473456412 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc401d040 con 0x55c9131a9130 2024-03-30T01:04:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.852+0000 7f4cbf400700 1 -- 172.21.15.116:0/473456412 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4cc4029cf0 con 0x55c9131a9130 2024-03-30T01:04:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.852+0000 7f4cf407bb80 1 -- 172.21.15.116:0/473456412 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ccc0d1030 con 0x55c9131a9130 2024-03-30T01:04:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.853+0000 7f4cbf400700 1 -- 172.21.15.116:0/473456412 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4cc4029150 con 0x55c9131a9130 2024-03-30T01:04:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.853+0000 7f4cf407bb80 10 client.16218 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.853+0000 7f4cf407bb80 1 -- 172.21.15.116:0/473456412 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c912f7fcd0 con 0x55c9131a9130 2024-03-30T01:04:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.853+0000 7f4cbf400700 1 -- 172.21.15.116:0/473456412 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ccc0d1030 con 0x55c9131a9130 2024-03-30T01:04:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.853+0000 7f4cbf400700 1 -- 172.21.15.116:0/473456412 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f4cc40099f0 con 0x55c9131a9130 2024-03-30T01:04:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.853+0000 7f4cf407bb80 10 client.16218 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.853+0000 7f4cf407bb80 10 client.16218 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T01:04:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.853+0000 7f4cf407bb80 20 client.16218 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.853+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/473456412 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55c913212950 0x55c913214d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.853+0000 7f4cf407bb80 4 client.16218 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:04:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.853+0000 7f4cf407bb80 1 -- 172.21.15.116:0/473456412 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c912f48cf0 con 0x55c913212950 2024-03-30T01:04:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.853+0000 7f4cddc00700 1 --2- 172.21.15.116:0/473456412 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55c913212950 0x55c913214d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cddc00700 1 --2- 172.21.15.116:0/473456412 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55c913212950 0x55c913214d40 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=69331a930f609ba5 server_cookie=e03edcc77d295ead in_seq=0 out_seq=0 2024-03-30T01:04:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cbf400700 10 client.16218 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T01:04:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cbf400700 1 -- 172.21.15.116:0/473456412 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c912f48cf0 con 0x55c913212950 2024-03-30T01:04:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cbf400700 10 client.16218 handle_command_reply: tid=0 2024-03-30T01:04:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cf407bb80 1 client.16218 shutdown 2024-03-30T01:04:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cf407bb80 1 -- 172.21.15.116:0/473456412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 msgr2=0x55c9131b80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/473456412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b80d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400bf10 tx=0x7f4cc400bb40 comp rx=0 tx=0).stop 2024-03-30T01:04:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cf407bb80 1 -- 172.21.15.116:0/473456412 shutdown_connections 2024-03-30T01:04:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/473456412 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x55c913212950 0x55c913214d40 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/473456412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c9131a87d0 0x55c9131b7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cbf400700 0 client.16218 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T01:04:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/473456412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c9131a9fd0 0x55c9131b8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cf407bb80 1 --2- 172.21.15.116:0/473456412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9131a9130 0x55c9131b80d0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cf407bb80 1 -- 172.21.15.116:0/473456412 >> 172.21.15.116:0/473456412 conn(0x55c9131076a0 msgr2=0x55c9131ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.854+0000 7f4cf407bb80 1 -- 172.21.15.116:0/473456412 shutdown_connections 2024-03-30T01:04:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.855+0000 7f4cf407bb80 1 -- 172.21.15.116:0/473456412 wait complete. 2024-03-30T01:04:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.855+0000 7f4cf407bb80 20 client.16218 trim_cache size 0 max 16384 2024-03-30T01:04:47.866 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:04:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.855+0000 7f4ce0600700 1 -- 172.21.15.116:0/1797037743 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4cc00a1120 msgr2=0x7f4cc00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.855+0000 7f4ce0600700 1 --2- 172.21.15.116:0/1797037743 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4cc00a1120 0x7f4cc00a35e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009710 tx=0x7f4cd0009100 comp rx=0 tx=0).stop 2024-03-30T01:04:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.855+0000 7f4ce0600700 1 -- 172.21.15.116:0/1797037743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cd8100d80 msgr2=0x7f4cd8198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.855+0000 7f4ce0600700 1 --2- 172.21.15.116:0/1797037743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cd8100d80 0x7f4cd8198ef0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc00d720 tx=0x7f4ccc00da30 comp rx=0 tx=0).stop 2024-03-30T01:04:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.855+0000 7f4ce0600700 1 -- 172.21.15.116:0/1797037743 shutdown_connections 2024-03-30T01:04:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.855+0000 7f4ce0600700 1 --2- 172.21.15.116:0/1797037743 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f4cc00a1120 0x7f4cc00a35e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.855+0000 7f4ce0600700 1 --2- 172.21.15.116:0/1797037743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4cd8102910 0x7f4cd81a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.855+0000 7f4ce0600700 1 --2- 172.21.15.116:0/1797037743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cd8100d80 0x7f4cd8198ef0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.855+0000 7f4ce0600700 1 --2- 172.21.15.116:0/1797037743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cd8101f70 0x7f4cd8199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.855+0000 7f4ce0600700 1 -- 172.21.15.116:0/1797037743 >> 172.21.15.116:0/1797037743 conn(0x7f4cd80fc2f0 msgr2=0x7f4cd8072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.855+0000 7f4ce0600700 1 -- 172.21.15.116:0/1797037743 shutdown_connections 2024-03-30T01:04:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:47.855+0000 7f4ce0600700 1 -- 172.21.15.116:0/1797037743 wait complete. 2024-03-30T01:04:47.881 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T01:04:48.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:47 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1067289186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:04:48.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.071+0000 7f5728a00700 1 -- 172.21.15.116:0/3498880954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5724101f70 msgr2=0x7f57241023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:48.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.071+0000 7f5728a00700 1 --2- 172.21.15.116:0/3498880954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5724101f70 0x7f57241023d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f57100099c0 tx=0x7f5710009cd0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.071+0000 7f5728a00700 1 -- 172.21.15.116:0/3498880954 shutdown_connections 2024-03-30T01:04:48.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.071+0000 7f5728a00700 1 --2- 172.21.15.116:0/3498880954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5724100d80 0x7f57241011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.071+0000 7f5728a00700 1 --2- 172.21.15.116:0/3498880954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5724102910 0x7f57241091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.071+0000 7f5728a00700 1 --2- 172.21.15.116:0/3498880954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5724101f70 0x7f57241023d0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.071+0000 7f5728a00700 1 -- 172.21.15.116:0/3498880954 >> 172.21.15.116:0/3498880954 conn(0x7f57240fc2f0 msgr2=0x7f57240fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:48.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.071+0000 7f5728a00700 1 -- 172.21.15.116:0/3498880954 shutdown_connections 2024-03-30T01:04:48.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.071+0000 7f5728a00700 1 -- 172.21.15.116:0/3498880954 wait complete. 2024-03-30T01:04:48.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.072+0000 7f5728a00700 1 Processor -- start 2024-03-30T01:04:48.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.072+0000 7f5728a00700 1 -- start start 2024-03-30T01:04:48.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.072+0000 7f5728a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5724100d80 0x7f5724074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.072+0000 7f5728a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5724101f70 0x7f57240731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.072+0000 7f5728a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5724102910 0x7f57240736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.072+0000 7f5728a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5724073e80 con 0x7f5724100d80 2024-03-30T01:04:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.072+0000 7f5728a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5724073ff0 con 0x7f5724102910 2024-03-30T01:04:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.072+0000 7f5728a00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57241a0d70 con 0x7f5724101f70 2024-03-30T01:04:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.072+0000 7f5721e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5724100d80 0x7f5724074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f5721e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5724100d80 0x7f5724074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51738/0 (socket says 172.21.15.116:51738) 2024-03-30T01:04:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f5721e00700 1 -- 172.21.15.116:0/2796924082 learned_addr learned my addr 172.21.15.116:0/2796924082 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f5721400700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5724101f70 0x7f57240731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f5722800700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5724102910 0x7f57240736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f5721e00700 1 -- 172.21.15.116:0/2796924082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5724101f70 msgr2=0x7f57240731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f5721e00700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5724101f70 0x7f57240731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f5721e00700 1 -- 172.21.15.116:0/2796924082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5724102910 msgr2=0x7f57240736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f5721e00700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5724102910 0x7f57240736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f5721e00700 1 -- 172.21.15.116:0/2796924082 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57100095f0 con 0x7f5724100d80 2024-03-30T01:04:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f5722800700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5724102910 0x7f57240736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:48.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f5721e00700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5724100d80 0x7f5724074b10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f571400e970 tx=0x7f571400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:48.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f571aa00700 1 -- 172.21.15.116:0/2796924082 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5714009790 con 0x7f5724100d80 2024-03-30T01:04:48.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f571aa00700 1 -- 172.21.15.116:0/2796924082 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f571400c780 con 0x7f5724100d80 2024-03-30T01:04:48.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f5728a00700 1 -- 172.21.15.116:0/2796924082 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57241a0fa0 con 0x7f5724100d80 2024-03-30T01:04:48.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.073+0000 7f571aa00700 1 -- 172.21.15.116:0/2796924082 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5714003be0 con 0x7f5724100d80 2024-03-30T01:04:48.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.074+0000 7f5728a00700 1 -- 172.21.15.116:0/2796924082 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57241a1420 con 0x7f5724100d80 2024-03-30T01:04:48.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.076+0000 7f571aa00700 1 -- 172.21.15.116:0/2796924082 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f571400cbf0 con 0x7f5724100d80 2024-03-30T01:04:48.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.078+0000 7f573c4d6b80 1 Processor -- start 2024-03-30T01:04:48.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.078+0000 7f573c4d6b80 1 -- start start 2024-03-30T01:04:48.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.079+0000 7f571aa00700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f57080a1120 0x7f57080a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.079+0000 7f573c4d6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92b7d0 0x561b9b92bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.079+0000 7f571aa00700 1 -- 172.21.15.116:0/2796924082 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5714014070 con 0x7f5724100d80 2024-03-30T01:04:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.079+0000 7f573c4d6b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92c130 0x561b9b92c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.079+0000 7f573c4d6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 0x561b9b92d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.079+0000 7f5722800700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f57080a1120 0x7f57080a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.079+0000 7f573c4d6b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b9b883990 con 0x561b9b92c130 2024-03-30T01:04:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.079+0000 7f573c4d6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b9b881350 con 0x561b9b92b7d0 2024-03-30T01:04:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.079+0000 7f573c4d6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b9b87cc40 con 0x561b9b92cfd0 2024-03-30T01:04:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.079+0000 7f5721400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92c130 0x561b9b92c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.079+0000 7f5721400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92c130 0x561b9b92c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51742/0 (socket says 172.21.15.116:51742) 2024-03-30T01:04:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.079+0000 7f5721400700 1 -- 172.21.15.116:0/96628351 learned_addr learned my addr 172.21.15.116:0/96628351 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.079+0000 7f5721e00700 1 --2- 172.21.15.116:0/96628351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92b7d0 0x561b9b92bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.079+0000 7f5722800700 1 --2- 172.21.15.116:0/96628351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 0x561b9b92d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.080+0000 7f5721400700 1 -- 172.21.15.116:0/96628351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 msgr2=0x561b9b92d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.080+0000 7f5721400700 1 --2- 172.21.15.116:0/96628351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 0x561b9b92d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.080+0000 7f5721400700 1 -- 172.21.15.116:0/96628351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92b7d0 msgr2=0x561b9b92bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.080+0000 7f5721400700 1 --2- 172.21.15.116:0/96628351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92b7d0 0x561b9b92bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.080+0000 7f5721400700 1 -- 172.21.15.116:0/96628351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b9b6fe3b0 con 0x561b9b92c130 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.080+0000 7f5721400700 1 --2- 172.21.15.116:0/96628351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92c130 0x561b9b92c9f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f57100058e0 tx=0x7f5710005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad11ba4d1977fff9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.080+0000 7f5722800700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f57080a1120 0x7f57080a35e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f571c009ec0 tx=0x7f571c004570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.080+0000 7f570f000700 1 -- 172.21.15.116:0/96628351 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571000ba10 con 0x561b9b92c130 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.080+0000 7f570f000700 1 -- 172.21.15.116:0/96628351 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f571001e040 con 0x561b9b92c130 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.080+0000 7f570f000700 1 -- 172.21.15.116:0/96628351 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710003a80 con 0x561b9b92c130 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.081+0000 7f573c4d6b80 1 -- 172.21.15.116:0/96628351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92c130 msgr2=0x561b9b92c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.081+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/96628351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92c130 0x561b9b92c9f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f57100058e0 tx=0x7f5710005c30 comp rx=0 tx=0).stop 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.081+0000 7f573c4d6b80 1 -- 172.21.15.116:0/96628351 shutdown_connections 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.081+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/96628351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 0x561b9b92d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.081+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/96628351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92b7d0 0x561b9b92bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.081+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/96628351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92c130 0x561b9b92c9f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.081+0000 7f573c4d6b80 1 -- 172.21.15.116:0/96628351 >> 172.21.15.116:0/96628351 conn(0x561b9b88a6a0 msgr2=0x561b9b9269a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.081+0000 7f573c4d6b80 1 -- 172.21.15.116:0/96628351 shutdown_connections 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.081+0000 7f573c4d6b80 1 -- 172.21.15.116:0/96628351 wait complete. 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.081+0000 7f573c4d6b80 1 -- start start 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.082+0000 7f573c4d6b80 1 Processor -- start 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.082+0000 7f573c4d6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92b7d0 0x561b9b9355c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.082+0000 7f573c4d6b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92c130 0x561b9b935b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.082+0000 7f573c4d6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 0x561b9b93cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.082+0000 7f573c4d6b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b9b818d80 con 0x561b9b92c130 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.082+0000 7f573c4d6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b9b818a20 con 0x561b9b92b7d0 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.082+0000 7f573c4d6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b9b7006a0 con 0x561b9b92cfd0 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.082+0000 7f573c4d6b80 10 client.0 _ll_register_callbacks cb 0x561b9b822470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.082+0000 7f5721400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92c130 0x561b9b935b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.082+0000 7f5721400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92c130 0x561b9b935b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51758/0 (socket says 172.21.15.116:51758) 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.082+0000 7f5721e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92b7d0 0x561b9b9355c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.082+0000 7f5721400700 1 -- 172.21.15.116:0/1767504215 learned_addr learned my addr 172.21.15.116:0/1767504215 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.082+0000 7f5722800700 1 --2- 172.21.15.116:0/1767504215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 0x561b9b93cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.083+0000 7f5721400700 1 -- 172.21.15.116:0/1767504215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 msgr2=0x561b9b93cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.083+0000 7f5721400700 1 --2- 172.21.15.116:0/1767504215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 0x561b9b93cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.083+0000 7f5721400700 1 -- 172.21.15.116:0/1767504215 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92b7d0 msgr2=0x561b9b9355c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.083+0000 7f5721400700 1 --2- 172.21.15.116:0/1767504215 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92b7d0 0x561b9b9355c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.083+0000 7f5721400700 1 -- 172.21.15.116:0/1767504215 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5724109b10 con 0x561b9b92c130 2024-03-30T01:04:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.083+0000 7f5721e00700 1 --2- 172.21.15.116:0/1767504215 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92b7d0 0x561b9b9355c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.083+0000 7f5721400700 1 --2- 172.21.15.116:0/1767504215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92c130 0x561b9b935b00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f571000bd90 tx=0x7f5710010470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4daeac22d83b1b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.083+0000 7f5703400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:04:48.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.083+0000 7f5703400700 1 -- 172.21.15.116:0/1767504215 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57100054e0 con 0x561b9b92c130 2024-03-30T01:04:48.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.083+0000 7f5703400700 1 -- 172.21.15.116:0/1767504215 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5710005640 con 0x561b9b92c130 2024-03-30T01:04:48.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.083+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1767504215 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b9b83d380 con 0x561b9b92c130 2024-03-30T01:04:48.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.084+0000 7f5703400700 1 -- 172.21.15.116:0/1767504215 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5710003840 con 0x561b9b92c130 2024-03-30T01:04:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.084+0000 7f573c4d6b80 10 client.16230 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.084+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1767504215 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561b9b6aab60 con 0x561b9b92c130 2024-03-30T01:04:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.084+0000 7f5703400700 1 -- 172.21.15.116:0/1767504215 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561b9b83d380 con 0x561b9b92c130 2024-03-30T01:04:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.084+0000 7f5703400700 1 -- 172.21.15.116:0/1767504215 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f571002b430 con 0x561b9b92c130 2024-03-30T01:04:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.084+0000 7f573c4d6b80 10 client.16230 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.084+0000 7f573c4d6b80 10 client.16230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:04:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.084+0000 7f573c4d6b80 20 client.16230 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.084+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/1767504215 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561b9b940e50 0x561b9b943270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.084+0000 7f573c4d6b80 4 client.16230 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:04:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.084+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1767504215 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b9b875700 con 0x561b9b940e50 2024-03-30T01:04:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.085+0000 7f5721e00700 1 --2- 172.21.15.116:0/1767504215 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561b9b940e50 0x561b9b943270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.085+0000 7f5721e00700 1 --2- 172.21.15.116:0/1767504215 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561b9b940e50 0x561b9b943270 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b53bb235ef75ecb3 server_cookie=1265d128eb76c560 in_seq=0 out_seq=0 2024-03-30T01:04:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.085+0000 7f5703400700 10 client.16230 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f5703400700 1 -- 172.21.15.116:0/1767504215 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561b9b875700 con 0x561b9b940e50 2024-03-30T01:04:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f5703400700 10 client.16230 handle_command_reply: tid=0 2024-03-30T01:04:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f573c4d6b80 1 client.16230 shutdown 2024-03-30T01:04:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1767504215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92c130 msgr2=0x561b9b935b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/1767504215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92c130 0x561b9b935b00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f571000bd90 tx=0x7f5710010470 comp rx=0 tx=0).stop 2024-03-30T01:04:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1767504215 shutdown_connections 2024-03-30T01:04:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/1767504215 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561b9b940e50 0x561b9b943270 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/1767504215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 0x561b9b93cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/1767504215 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92b7d0 0x561b9b9355c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f5703400700 0 client.16230 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/1767504215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92c130 0x561b9b935b00 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1767504215 >> 172.21.15.116:0/1767504215 conn(0x561b9b88a6a0 msgr2=0x561b9b92a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1767504215 shutdown_connections 2024-03-30T01:04:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1767504215 wait complete. 2024-03-30T01:04:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.086+0000 7f573c4d6b80 20 client.16230 trim_cache size 0 max 16384 2024-03-30T01:04:48.098 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:47 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1067289186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:04:48.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.099+0000 7f573c4d6b80 1 Processor -- start 2024-03-30T01:04:48.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.100+0000 7f573c4d6b80 1 -- start start 2024-03-30T01:04:48.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.100+0000 7f573c4d6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92b7d0 0x561b9b93d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.100+0000 7f573c4d6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92c130 0x561b9b93cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.100+0000 7f573c4d6b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92cfd0 0x561b9b92d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.100+0000 7f573c4d6b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b9b926f20 con 0x561b9b92cfd0 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.100+0000 7f573c4d6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b9b935130 con 0x561b9b92c130 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.100+0000 7f573c4d6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b9b940c90 con 0x561b9b92b7d0 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.100+0000 7f5722800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92cfd0 0x561b9b92d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.100+0000 7f5722800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92cfd0 0x561b9b92d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51764/0 (socket says 172.21.15.116:51764) 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.100+0000 7f5721400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92c130 0x561b9b93cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.100+0000 7f5722800700 1 -- 172.21.15.116:0/71479193 learned_addr learned my addr 172.21.15.116:0/71479193 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.100+0000 7f5721e00700 1 --2- 172.21.15.116:0/71479193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92b7d0 0x561b9b93d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.101+0000 7f5721400700 1 -- 172.21.15.116:0/71479193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92b7d0 msgr2=0x561b9b93d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.101+0000 7f5721400700 1 --2- 172.21.15.116:0/71479193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92b7d0 0x561b9b93d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.101+0000 7f5721400700 1 -- 172.21.15.116:0/71479193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92cfd0 msgr2=0x561b9b92d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.101+0000 7f5721400700 1 --2- 172.21.15.116:0/71479193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92cfd0 0x561b9b92d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.101+0000 7f5721400700 1 -- 172.21.15.116:0/71479193 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b9b842ea0 con 0x561b9b92c130 2024-03-30T01:04:48.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.101+0000 7f5721400700 1 --2- 172.21.15.116:0/71479193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92c130 0x561b9b93cb00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f57100098e0 tx=0x7f5710005ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59aef7b475565a44 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:48.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.101+0000 7f570fa00700 1 -- 172.21.15.116:0/71479193 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571002bab0 con 0x561b9b92c130 2024-03-30T01:04:48.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.101+0000 7f570fa00700 1 -- 172.21.15.116:0/71479193 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f571002b430 con 0x561b9b92c130 2024-03-30T01:04:48.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.101+0000 7f570fa00700 1 -- 172.21.15.116:0/71479193 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710003e10 con 0x561b9b92c130 2024-03-30T01:04:48.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.101+0000 7f573c4d6b80 1 -- 172.21.15.116:0/71479193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92c130 msgr2=0x561b9b93cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:48.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.101+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/71479193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92c130 0x561b9b93cb00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f57100098e0 tx=0x7f5710005ae0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.102+0000 7f573c4d6b80 1 -- 172.21.15.116:0/71479193 shutdown_connections 2024-03-30T01:04:48.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.102+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/71479193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92b7d0 0x561b9b93d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.102+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/71479193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92cfd0 0x561b9b92d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.102+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/71479193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92c130 0x561b9b93cb00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.102+0000 7f573c4d6b80 1 -- 172.21.15.116:0/71479193 >> 172.21.15.116:0/71479193 conn(0x561b9b88a6a0 msgr2=0x561b9b934300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.102+0000 7f573c4d6b80 1 -- 172.21.15.116:0/71479193 shutdown_connections 2024-03-30T01:04:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.102+0000 7f573c4d6b80 1 -- 172.21.15.116:0/71479193 wait complete. 2024-03-30T01:04:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.102+0000 7f573c4d6b80 1 -- start start 2024-03-30T01:04:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.102+0000 7f573c4d6b80 1 Processor -- start 2024-03-30T01:04:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.102+0000 7f573c4d6b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92b7d0 0x561b9b93a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f573c4d6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92c130 0x561b9b93a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f573c4d6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 0x561b9b93b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f573c4d6b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b9b93bd90 con 0x561b9b92b7d0 2024-03-30T01:04:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f573c4d6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b9b93bf00 con 0x561b9b92c130 2024-03-30T01:04:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f573c4d6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b9b93c0c0 con 0x561b9b92cfd0 2024-03-30T01:04:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f573c4d6b80 10 client.0 _ll_register_callbacks cb 0x561b9b822470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f5721e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92b7d0 0x561b9b93a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f5721e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92b7d0 0x561b9b93a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51772/0 (socket says 172.21.15.116:51772) 2024-03-30T01:04:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f5721e00700 1 -- 172.21.15.116:0/1405085734 learned_addr learned my addr 172.21.15.116:0/1405085734 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f5721400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92c130 0x561b9b93a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f5722800700 1 --2- 172.21.15.116:0/1405085734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 0x561b9b93b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f5721e00700 1 -- 172.21.15.116:0/1405085734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 msgr2=0x561b9b93b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f5721e00700 1 --2- 172.21.15.116:0/1405085734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 0x561b9b93b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f5721e00700 1 -- 172.21.15.116:0/1405085734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92c130 msgr2=0x561b9b93a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f5721e00700 1 --2- 172.21.15.116:0/1405085734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92c130 0x561b9b93a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f5721e00700 1 -- 172.21.15.116:0/1405085734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b9b6aab60 con 0x561b9b92b7d0 2024-03-30T01:04:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f5721400700 1 --2- 172.21.15.116:0/1405085734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92c130 0x561b9b93a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f5721e00700 1 --2- 172.21.15.116:0/1405085734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92b7d0 0x561b9b93a440 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5714005950 tx=0x7f57140961e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997364d967f8dc51 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.103+0000 7f5703400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:04:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.104+0000 7f5703400700 1 -- 172.21.15.116:0/1405085734 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5714099ea0 con 0x561b9b92b7d0 2024-03-30T01:04:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.104+0000 7f5703400700 1 -- 172.21.15.116:0/1405085734 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f571409dcd0 con 0x561b9b92b7d0 2024-03-30T01:04:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.104+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1405085734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f571001e040 con 0x561b9b92b7d0 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.104+0000 7f5703400700 1 -- 172.21.15.116:0/1405085734 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57140a2320 con 0x561b9b92b7d0 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.104+0000 7f573c4d6b80 10 client.16236 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.104+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1405085734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561b9b702cd0 con 0x561b9b92b7d0 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.104+0000 7f5703400700 1 -- 172.21.15.116:0/1405085734 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f571001e040 con 0x561b9b92b7d0 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.104+0000 7f5703400700 1 -- 172.21.15.116:0/1405085734 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f5714099390 con 0x561b9b92b7d0 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.104+0000 7f573c4d6b80 10 client.16236 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.104+0000 7f573c4d6b80 10 client.16236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.104+0000 7f573c4d6b80 20 client.16236 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.104+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/1405085734 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561b9b995610 0x561b9b997ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.104+0000 7f573c4d6b80 4 client.16236 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.104+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1405085734 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561b9b6cbcf0 con 0x561b9b995610 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.105+0000 7f5721400700 1 --2- 172.21.15.116:0/1405085734 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561b9b995610 0x561b9b997ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.105+0000 7f5703400700 10 client.16236 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.105+0000 7f5721400700 1 --2- 172.21.15.116:0/1405085734 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561b9b995610 0x561b9b997ad0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e78d033a976d9e0f server_cookie=63677172aaf610ce in_seq=0 out_seq=0 2024-03-30T01:04:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.105+0000 7f5703400700 1 -- 172.21.15.116:0/1405085734 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561b9b6cbcf0 con 0x561b9b995610 2024-03-30T01:04:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.105+0000 7f5703400700 10 client.16236 handle_command_reply: tid=0 2024-03-30T01:04:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.105+0000 7f573c4d6b80 1 client.16236 shutdown 2024-03-30T01:04:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1405085734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92b7d0 msgr2=0x561b9b93a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/1405085734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92b7d0 0x561b9b93a440 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5714005950 tx=0x7f57140961e0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1405085734 shutdown_connections 2024-03-30T01:04:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/1405085734 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x561b9b995610 0x561b9b997ad0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/1405085734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561b9b92cfd0 0x561b9b93b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/1405085734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b9b92b7d0 0x561b9b93a440 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f573c4d6b80 1 --2- 172.21.15.116:0/1405085734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b9b92c130 0x561b9b93a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f5703400700 0 client.16236 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1405085734 >> 172.21.15.116:0/1405085734 conn(0x561b9b88a6a0 msgr2=0x561b9b91f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1405085734 shutdown_connections 2024-03-30T01:04:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f573c4d6b80 1 -- 172.21.15.116:0/1405085734 wait complete. 2024-03-30T01:04:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f573c4d6b80 20 client.16236 trim_cache size 0 max 16384 2024-03-30T01:04:48.118 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:04:48.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f5728a00700 1 -- 172.21.15.116:0/2796924082 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f57080a1120 msgr2=0x7f57080a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f5728a00700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f57080a1120 0x7f57080a35e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f571c009ec0 tx=0x7f571c004570 comp rx=0 tx=0).stop 2024-03-30T01:04:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f5728a00700 1 -- 172.21.15.116:0/2796924082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5724100d80 msgr2=0x7f5724074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f5728a00700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5724100d80 0x7f5724074b10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f571400e970 tx=0x7f571400ec80 comp rx=0 tx=0).stop 2024-03-30T01:04:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.106+0000 7f5728a00700 1 -- 172.21.15.116:0/2796924082 shutdown_connections 2024-03-30T01:04:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.107+0000 7f5728a00700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f57080a1120 0x7f57080a35e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.107+0000 7f5728a00700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5724102910 0x7f57240736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.107+0000 7f5728a00700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5724100d80 0x7f5724074b10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.107+0000 7f5728a00700 1 --2- 172.21.15.116:0/2796924082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5724101f70 0x7f57240731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.107+0000 7f5728a00700 1 -- 172.21.15.116:0/2796924082 >> 172.21.15.116:0/2796924082 conn(0x7f57240fc2f0 msgr2=0x7f5724105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.107+0000 7f5728a00700 1 -- 172.21.15.116:0/2796924082 shutdown_connections 2024-03-30T01:04:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:48.107+0000 7f5728a00700 1 -- 172.21.15.116:0/2796924082 wait complete. 2024-03-30T01:04:48.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:48 smithi116 ceph-mon[38086]: pgmap v499: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 10 MiB/s wr, 699 op/s 2024-03-30T01:04:49.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:48 smithi181 ceph-mon[47622]: pgmap v499: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 10 MiB/s wr, 699 op/s 2024-03-30T01:04:49.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:48 smithi130 ceph-mon[47626]: pgmap v499: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 10 MiB/s wr, 699 op/s 2024-03-30T01:04:49.131 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T01:04:49.132 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T01:04:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.336+0000 7f51ff800700 1 -- 172.21.15.116:0/147825828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f8102910 msgr2=0x7f51f81091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.336+0000 7f51ff800700 1 --2- 172.21.15.116:0/147825828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f8102910 0x7f51f81091c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f51f000b0d0 tx=0x7f51f000b3e0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.337+0000 7f51ff800700 1 -- 172.21.15.116:0/147825828 shutdown_connections 2024-03-30T01:04:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.337+0000 7f51ff800700 1 --2- 172.21.15.116:0/147825828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f8102910 0x7f51f81091c0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.337+0000 7f51ff800700 1 --2- 172.21.15.116:0/147825828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51f8100d80 0x7f51f81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.337+0000 7f51ff800700 1 --2- 172.21.15.116:0/147825828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51f8101f70 0x7f51f81023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.337+0000 7f51ff800700 1 -- 172.21.15.116:0/147825828 >> 172.21.15.116:0/147825828 conn(0x7f51f80fc2f0 msgr2=0x7f51f80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.337+0000 7f51ff800700 1 -- 172.21.15.116:0/147825828 shutdown_connections 2024-03-30T01:04:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.337+0000 7f51ff800700 1 -- 172.21.15.116:0/147825828 wait complete. 2024-03-30T01:04:49.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.338+0000 7f51ff800700 1 Processor -- start 2024-03-30T01:04:49.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.338+0000 7f51ff800700 1 -- start start 2024-03-30T01:04:49.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.338+0000 7f51ff800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51f8100d80 0x7f51f8199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.338+0000 7f51ff800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51f8101f70 0x7f51f81995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.338+0000 7f51ff800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f8102910 0x7f51f81a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.338+0000 7f51ff800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f81a0b50 con 0x7f51f8102910 2024-03-30T01:04:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.338+0000 7f51ff800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f81a0c90 con 0x7f51f8100d80 2024-03-30T01:04:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.338+0000 7f51ff800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f81a0e20 con 0x7f51f8101f70 2024-03-30T01:04:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.338+0000 7f51f7e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51f8101f70 0x7f51f81995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.338+0000 7f51f7e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51f8101f70 0x7f51f81995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:51384/0 (socket says 172.21.15.116:51384) 2024-03-30T01:04:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.338+0000 7f51f7e00700 1 -- 172.21.15.116:0/1372418671 learned_addr learned my addr 172.21.15.116:0/1372418671 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.338+0000 7f51fd800700 1 --2- 172.21.15.116:0/1372418671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f8102910 0x7f51f81a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.338+0000 7f51fce00700 1 --2- 172.21.15.116:0/1372418671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51f8100d80 0x7f51f8199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.339+0000 7f51f7e00700 1 -- 172.21.15.116:0/1372418671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51f8100d80 msgr2=0x7f51f8199070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.339+0000 7f51f7e00700 1 --2- 172.21.15.116:0/1372418671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51f8100d80 0x7f51f8199070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.339+0000 7f51f7e00700 1 -- 172.21.15.116:0/1372418671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f8102910 msgr2=0x7f51f81a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.339+0000 7f51f7e00700 1 --2- 172.21.15.116:0/1372418671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f8102910 0x7f51f81a0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.339+0000 7f51f7e00700 1 -- 172.21.15.116:0/1372418671 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f0009d00 con 0x7f51f8101f70 2024-03-30T01:04:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.339+0000 7f51f7e00700 1 --2- 172.21.15.116:0/1372418671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51f8101f70 0x7f51f81995b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f51e400e9a0 tx=0x7f51e400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.339+0000 7f51f5600700 1 -- 172.21.15.116:0/1372418671 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e4009790 con 0x7f51f8101f70 2024-03-30T01:04:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.339+0000 7f51ff800700 1 -- 172.21.15.116:0/1372418671 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f81a1100 con 0x7f51f8101f70 2024-03-30T01:04:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.340+0000 7f51f5600700 1 -- 172.21.15.116:0/1372418671 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51e4010430 con 0x7f51f8101f70 2024-03-30T01:04:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.340+0000 7f51f5600700 1 -- 172.21.15.116:0/1372418671 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e4018680 con 0x7f51f8101f70 2024-03-30T01:04:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.340+0000 7f51ff800700 1 -- 172.21.15.116:0/1372418671 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f81a1660 con 0x7f51f8101f70 2024-03-30T01:04:49.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.343+0000 7f51f5600700 1 -- 172.21.15.116:0/1372418671 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f51e40105a0 con 0x7f51f8101f70 2024-03-30T01:04:49.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.344+0000 7f5213272b80 1 Processor -- start 2024-03-30T01:04:49.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.344+0000 7f5213272b80 1 -- start start 2024-03-30T01:04:49.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.345+0000 7f5213272b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6ddf9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.345+0000 7f5213272b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfa1b0 0x564c6ddfaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.345+0000 7f5213272b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6ddfb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.345+0000 7f5213272b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6dd51a10 con 0x564c6ddfb050 2024-03-30T01:04:49.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.345+0000 7f5213272b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6dd4f3d0 con 0x564c6ddf9850 2024-03-30T01:04:49.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.345+0000 7f5213272b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6dd4acc0 con 0x564c6ddfa1b0 2024-03-30T01:04:49.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.345+0000 7f51f5600700 1 --2- 172.21.15.116:0/1372418671 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f51e00a1120 0x7f51e00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.345+0000 7f51fd800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6ddf9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.345+0000 7f51fd800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6ddf9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38438/0 (socket says 172.21.15.116:38438) 2024-03-30T01:04:49.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.345+0000 7f51fd800700 1 -- 172.21.15.116:0/3225304810 learned_addr learned my addr 172.21.15.116:0/3225304810 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:49.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.345+0000 7f51f7e00700 1 --2- 172.21.15.116:0/3225304810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6ddfb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.345+0000 7f51fce00700 1 --2- 172.21.15.116:0/3225304810 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfa1b0 0x564c6ddfaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.345+0000 7f51f5600700 1 -- 172.21.15.116:0/1372418671 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f51e4014070 con 0x7f51f8101f70 2024-03-30T01:04:49.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.345+0000 7f51fd800700 1 --2- 172.21.15.116:0/1372418671 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f51e00a1120 0x7f51e00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.346+0000 7f51fd800700 1 -- 172.21.15.116:0/3225304810 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfa1b0 msgr2=0x564c6ddfaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.346+0000 7f51fd800700 1 --2- 172.21.15.116:0/3225304810 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfa1b0 0x564c6ddfaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.346+0000 7f51fd800700 1 -- 172.21.15.116:0/3225304810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 msgr2=0x564c6ddfb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.346+0000 7f51fd800700 1 --2- 172.21.15.116:0/3225304810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6ddfb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.346+0000 7f51fd800700 1 -- 172.21.15.116:0/3225304810 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c6dbcc430 con 0x564c6ddf9850 2024-03-30T01:04:49.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.346+0000 7f51f7e00700 1 --2- 172.21.15.116:0/3225304810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6ddfb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:49.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.346+0000 7f51fd800700 1 --2- 172.21.15.116:0/3225304810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6ddf9c70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f51f0006090 tx=0x7f51f0008c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23c051313f243322 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:49.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.346+0000 7f51fd800700 1 --2- 172.21.15.116:0/1372418671 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f51e00a1120 0x7f51e00a35e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f51f0005000 tx=0x7f51f0026000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:49.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.346+0000 7f51ea000700 1 -- 172.21.15.116:0/3225304810 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f000e040 con 0x564c6ddf9850 2024-03-30T01:04:49.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.346+0000 7f51ea000700 1 -- 172.21.15.116:0/3225304810 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51f0009300 con 0x564c6ddf9850 2024-03-30T01:04:49.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.346+0000 7f51ea000700 1 -- 172.21.15.116:0/3225304810 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f0005410 con 0x564c6ddf9850 2024-03-30T01:04:49.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.347+0000 7f5213272b80 1 -- 172.21.15.116:0/3225304810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 msgr2=0x564c6ddf9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.347+0000 7f5213272b80 1 --2- 172.21.15.116:0/3225304810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6ddf9c70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f51f0006090 tx=0x7f51f0008c90 comp rx=0 tx=0).stop 2024-03-30T01:04:49.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.347+0000 7f5213272b80 1 -- 172.21.15.116:0/3225304810 shutdown_connections 2024-03-30T01:04:49.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.347+0000 7f5213272b80 1 --2- 172.21.15.116:0/3225304810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6ddfb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.347+0000 7f5213272b80 1 --2- 172.21.15.116:0/3225304810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6ddf9c70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.347+0000 7f5213272b80 1 --2- 172.21.15.116:0/3225304810 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfa1b0 0x564c6ddfaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.347+0000 7f5213272b80 1 -- 172.21.15.116:0/3225304810 >> 172.21.15.116:0/3225304810 conn(0x564c6dd58720 msgr2=0x564c6ddf4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.347+0000 7f5213272b80 1 -- 172.21.15.116:0/3225304810 shutdown_connections 2024-03-30T01:04:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.347+0000 7f5213272b80 1 -- 172.21.15.116:0/3225304810 wait complete. 2024-03-30T01:04:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.348+0000 7f5213272b80 1 -- start start 2024-03-30T01:04:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.348+0000 7f5213272b80 1 Processor -- start 2024-03-30T01:04:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f5213272b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6de03740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f5213272b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfa1b0 0x564c6de03c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f5213272b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6de0ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f5213272b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6dce6e00 con 0x564c6ddfb050 2024-03-30T01:04:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f5213272b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6dce6aa0 con 0x564c6ddf9850 2024-03-30T01:04:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f5213272b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6dbce720 con 0x564c6ddfa1b0 2024-03-30T01:04:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f5213272b80 10 client.0 _ll_register_callbacks cb 0x564c6dcf04f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f51fd800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6de0ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f51fd800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6de0ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51820/0 (socket says 172.21.15.116:51820) 2024-03-30T01:04:49.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f51fd800700 1 -- 172.21.15.116:0/91642772 learned_addr learned my addr 172.21.15.116:0/91642772 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:49.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f51fce00700 1 --2- 172.21.15.116:0/91642772 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6de03740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f51f7e00700 1 --2- 172.21.15.116:0/91642772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfa1b0 0x564c6de03c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f51fd800700 1 -- 172.21.15.116:0/91642772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfa1b0 msgr2=0x564c6de03c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f51fd800700 1 --2- 172.21.15.116:0/91642772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfa1b0 0x564c6de03c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f51fd800700 1 -- 172.21.15.116:0/91642772 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 msgr2=0x564c6de03740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f51fd800700 1 --2- 172.21.15.116:0/91642772 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6de03740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.349+0000 7f51fd800700 1 -- 172.21.15.116:0/91642772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f81a1660 con 0x564c6ddfb050 2024-03-30T01:04:49.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.350+0000 7f51fd800700 1 --2- 172.21.15.116:0/91642772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6de0ac70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f51f00061c0 tx=0x7f51f0005930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81aa81013b4f1dd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:49.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.350+0000 7f51dea00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:04:49.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.350+0000 7f51dea00700 1 -- 172.21.15.116:0/91642772 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f000baa0 con 0x564c6ddfb050 2024-03-30T01:04:49.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.350+0000 7f51dea00700 1 -- 172.21.15.116:0/91642772 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f51f000bc00 con 0x564c6ddfb050 2024-03-30T01:04:49.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.350+0000 7f5213272b80 1 -- 172.21.15.116:0/91642772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564c6dd0b400 con 0x564c6ddfb050 2024-03-30T01:04:49.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.350+0000 7f51dea00700 1 -- 172.21.15.116:0/91642772 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51f000bdc0 con 0x564c6ddfb050 2024-03-30T01:04:49.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.350+0000 7f5213272b80 10 client.16248 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:49.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.350+0000 7f5213272b80 1 -- 172.21.15.116:0/91642772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f51f8109b10 con 0x564c6ddfb050 2024-03-30T01:04:49.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.350+0000 7f51dea00700 1 -- 172.21.15.116:0/91642772 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564c6dd0b400 con 0x564c6ddfb050 2024-03-30T01:04:49.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.350+0000 7f51dea00700 1 -- 172.21.15.116:0/91642772 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f51f003b080 con 0x564c6ddfb050 2024-03-30T01:04:49.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.351+0000 7f5213272b80 10 client.16248 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:49.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.351+0000 7f5213272b80 10 client.16248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:04:49.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.351+0000 7f5213272b80 20 client.16248 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:49.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.351+0000 7f5213272b80 1 --2- 172.21.15.116:0/91642772 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564c6de0f1a0 0x564c6de115c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.351+0000 7f5213272b80 4 client.16248 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:04:49.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.351+0000 7f5213272b80 1 -- 172.21.15.116:0/91642772 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564c6dd43780 con 0x564c6de0f1a0 2024-03-30T01:04:49.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.351+0000 7f51fce00700 1 --2- 172.21.15.116:0/91642772 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564c6de0f1a0 0x564c6de115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.351+0000 7f51dea00700 10 client.16248 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:49.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.351+0000 7f51fce00700 1 --2- 172.21.15.116:0/91642772 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564c6de0f1a0 0x564c6de115c0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=524209f512309acb server_cookie=6cfcd0c95fed2ec9 in_seq=0 out_seq=0 2024-03-30T01:04:49.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.352+0000 7f51dea00700 1 -- 172.21.15.116:0/91642772 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564c6dd43780 con 0x564c6de0f1a0 2024-03-30T01:04:49.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.352+0000 7f51dea00700 10 client.16248 handle_command_reply: tid=0 2024-03-30T01:04:49.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.352+0000 7f5213272b80 1 client.16248 shutdown 2024-03-30T01:04:49.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.352+0000 7f5213272b80 1 -- 172.21.15.116:0/91642772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 msgr2=0x564c6de0ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.352+0000 7f5213272b80 1 --2- 172.21.15.116:0/91642772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6de0ac70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f51f00061c0 tx=0x7f51f0005930 comp rx=0 tx=0).stop 2024-03-30T01:04:49.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.352+0000 7f5213272b80 1 -- 172.21.15.116:0/91642772 shutdown_connections 2024-03-30T01:04:49.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.352+0000 7f5213272b80 1 --2- 172.21.15.116:0/91642772 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564c6de0f1a0 0x564c6de115c0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.352+0000 7f5213272b80 1 --2- 172.21.15.116:0/91642772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6de0ac70 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.352+0000 7f5213272b80 1 --2- 172.21.15.116:0/91642772 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6de03740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.352+0000 7f5213272b80 1 --2- 172.21.15.116:0/91642772 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfa1b0 0x564c6de03c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.352+0000 7f51dea00700 0 client.16248 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.353+0000 7f5213272b80 1 -- 172.21.15.116:0/91642772 >> 172.21.15.116:0/91642772 conn(0x564c6dd58720 msgr2=0x564c6ddf81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.353+0000 7f5213272b80 1 -- 172.21.15.116:0/91642772 shutdown_connections 2024-03-30T01:04:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.353+0000 7f5213272b80 1 -- 172.21.15.116:0/91642772 wait complete. 2024-03-30T01:04:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.353+0000 7f5213272b80 20 client.16248 trim_cache size 0 max 16384 2024-03-30T01:04:49.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.370+0000 7f5213272b80 1 Processor -- start 2024-03-30T01:04:49.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.370+0000 7f5213272b80 1 -- start start 2024-03-30T01:04:49.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.370+0000 7f5213272b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6ddedff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.370+0000 7f5213272b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfa1b0 0x564c6de0ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.370+0000 7f5213272b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfb050 0x564c6ddf3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.370+0000 7f5213272b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6de02200 con 0x564c6ddfa1b0 2024-03-30T01:04:49.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.370+0000 7f5213272b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6de0be20 con 0x564c6ddf9850 2024-03-30T01:04:49.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.370+0000 7f5213272b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6de01e30 con 0x564c6ddfb050 2024-03-30T01:04:49.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.371+0000 7f51f7e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfa1b0 0x564c6de0ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.371+0000 7f51fce00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6ddedff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.371+0000 7f51f7e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfa1b0 0x564c6de0ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51836/0 (socket says 172.21.15.116:51836) 2024-03-30T01:04:49.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.371+0000 7f51f7e00700 1 -- 172.21.15.116:0/3239050986 learned_addr learned my addr 172.21.15.116:0/3239050986 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:49.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.371+0000 7f51fd800700 1 --2- 172.21.15.116:0/3239050986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfb050 0x564c6ddf3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.371+0000 7f51fce00700 1 -- 172.21.15.116:0/3239050986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfb050 msgr2=0x564c6ddf3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.371+0000 7f51fce00700 1 --2- 172.21.15.116:0/3239050986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfb050 0x564c6ddf3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.371+0000 7f51fce00700 1 -- 172.21.15.116:0/3239050986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfa1b0 msgr2=0x564c6de0ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.371+0000 7f51fce00700 1 --2- 172.21.15.116:0/3239050986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfa1b0 0x564c6de0ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.371+0000 7f51fce00700 1 -- 172.21.15.116:0/3239050986 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c6de161b0 con 0x564c6ddf9850 2024-03-30T01:04:49.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.371+0000 7f51f7e00700 1 --2- 172.21.15.116:0/3239050986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfa1b0 0x564c6de0ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:49.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.371+0000 7f51fce00700 1 --2- 172.21.15.116:0/3239050986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6ddedff0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x564c6de11ba0 tx=0x7f51ec009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4623a7ed053e200d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:49.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.372+0000 7f51eaa00700 1 -- 172.21.15.116:0/3239050986 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec016030 con 0x564c6ddf9850 2024-03-30T01:04:49.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.372+0000 7f51eaa00700 1 -- 172.21.15.116:0/3239050986 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51ec017040 con 0x564c6ddf9850 2024-03-30T01:04:49.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.372+0000 7f51eaa00700 1 -- 172.21.15.116:0/3239050986 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec010690 con 0x564c6ddf9850 2024-03-30T01:04:49.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.372+0000 7f5213272b80 1 -- 172.21.15.116:0/3239050986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 msgr2=0x564c6ddedff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.372+0000 7f5213272b80 1 --2- 172.21.15.116:0/3239050986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6ddedff0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x564c6de11ba0 tx=0x7f51ec009260 comp rx=0 tx=0).stop 2024-03-30T01:04:49.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.372+0000 7f5213272b80 1 -- 172.21.15.116:0/3239050986 shutdown_connections 2024-03-30T01:04:49.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.372+0000 7f5213272b80 1 --2- 172.21.15.116:0/3239050986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddfb050 0x564c6ddf3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.372+0000 7f5213272b80 1 --2- 172.21.15.116:0/3239050986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddf9850 0x564c6ddedff0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.372+0000 7f5213272b80 1 --2- 172.21.15.116:0/3239050986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfa1b0 0x564c6de0ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.372+0000 7f5213272b80 1 -- 172.21.15.116:0/3239050986 >> 172.21.15.116:0/3239050986 conn(0x564c6dd58720 msgr2=0x564c6dded620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:49.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.372+0000 7f5213272b80 1 -- 172.21.15.116:0/3239050986 shutdown_connections 2024-03-30T01:04:49.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.372+0000 7f5213272b80 1 -- 172.21.15.116:0/3239050986 wait complete. 2024-03-30T01:04:49.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.373+0000 7f5213272b80 1 -- start start 2024-03-30T01:04:49.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.373+0000 7f5213272b80 1 Processor -- start 2024-03-30T01:04:49.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f5213272b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddf9850 0x564c6de0a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f5213272b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddfa1b0 0x564c6de0a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f5213272b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6de63500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f5213272b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6de63a40 con 0x564c6ddfb050 2024-03-30T01:04:49.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f5213272b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6de63b80 con 0x564c6ddfa1b0 2024-03-30T01:04:49.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f5213272b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6de63d10 con 0x564c6ddf9850 2024-03-30T01:04:49.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f5213272b80 10 client.0 _ll_register_callbacks cb 0x564c6dcf04f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:49.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f51fce00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddf9850 0x564c6de0a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f51fce00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddf9850 0x564c6de0a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:51408/0 (socket says 172.21.15.116:51408) 2024-03-30T01:04:49.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f51fce00700 1 -- 172.21.15.116:0/1226237892 learned_addr learned my addr 172.21.15.116:0/1226237892 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:49.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f51fd800700 1 --2- 172.21.15.116:0/1226237892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6de63500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f51f7e00700 1 --2- 172.21.15.116:0/1226237892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddfa1b0 0x564c6de0a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f51fce00700 1 -- 172.21.15.116:0/1226237892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddfa1b0 msgr2=0x564c6de0a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f51fce00700 1 --2- 172.21.15.116:0/1226237892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddfa1b0 0x564c6de0a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f51fce00700 1 -- 172.21.15.116:0/1226237892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 msgr2=0x564c6de63500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f51fce00700 1 --2- 172.21.15.116:0/1226237892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6de63500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.374+0000 7f51fce00700 1 -- 172.21.15.116:0/1226237892 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f81a1100 con 0x564c6ddf9850 2024-03-30T01:04:49.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.375+0000 7f51fce00700 1 --2- 172.21.15.116:0/1226237892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddf9850 0x564c6de0a250 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f51ec01dc90 tx=0x7f51ec0046a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ab07c366696250f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:49.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.375+0000 7f51dea00700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:04:49.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.375+0000 7f51dea00700 1 -- 172.21.15.116:0/1226237892 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec016070 con 0x564c6ddf9850 2024-03-30T01:04:49.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.375+0000 7f5213272b80 1 -- 172.21.15.116:0/1226237892 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f51f0009300 con 0x564c6ddf9850 2024-03-30T01:04:49.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.375+0000 7f51dea00700 1 -- 172.21.15.116:0/1226237892 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f51ec0039e0 con 0x564c6ddf9850 2024-03-30T01:04:49.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.375+0000 7f51dea00700 1 -- 172.21.15.116:0/1226237892 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51ec003ba0 con 0x564c6ddf9850 2024-03-30T01:04:49.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.375+0000 7f5213272b80 10 client.26236 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:49.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.375+0000 7f5213272b80 1 -- 172.21.15.116:0/1226237892 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f51ec004040 con 0x564c6ddf9850 2024-03-30T01:04:49.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.375+0000 7f51dea00700 1 -- 172.21.15.116:0/1226237892 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51f0009300 con 0x564c6ddf9850 2024-03-30T01:04:49.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.376+0000 7f51dea00700 1 -- 172.21.15.116:0/1226237892 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f51ec0049f0 con 0x564c6ddf9850 2024-03-30T01:04:49.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.376+0000 7f5213272b80 10 client.26236 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:49.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.376+0000 7f5213272b80 10 client.26236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:04:49.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.376+0000 7f5213272b80 20 client.26236 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:49.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.376+0000 7f5213272b80 1 --2- 172.21.15.116:0/1226237892 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564c6de65790 0x564c6de67bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.376+0000 7f5213272b80 4 client.26236 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T01:04:49.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.376+0000 7f5213272b80 1 -- 172.21.15.116:0/1226237892 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564c6db99d70 con 0x564c6de65790 2024-03-30T01:04:49.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.376+0000 7f51f7e00700 1 --2- 172.21.15.116:0/1226237892 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564c6de65790 0x564c6de67bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.376+0000 7f51f7e00700 1 --2- 172.21.15.116:0/1226237892 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564c6de65790 0x564c6de67bb0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67dfe4e098d8bd30 server_cookie=32c07f8bace9f82e in_seq=0 out_seq=0 2024-03-30T01:04:49.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.376+0000 7f51dea00700 10 client.26236 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:49.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f51dea00700 1 -- 172.21.15.116:0/1226237892 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564c6db99d70 con 0x564c6de65790 2024-03-30T01:04:49.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f51dea00700 10 client.26236 handle_command_reply: tid=0 2024-03-30T01:04:49.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f5213272b80 1 client.26236 shutdown 2024-03-30T01:04:49.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f5213272b80 1 -- 172.21.15.116:0/1226237892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddf9850 msgr2=0x564c6de0a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f5213272b80 1 --2- 172.21.15.116:0/1226237892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddf9850 0x564c6de0a250 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f51ec01dc90 tx=0x7f51ec0046a0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f5213272b80 1 -- 172.21.15.116:0/1226237892 shutdown_connections 2024-03-30T01:04:49.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f5213272b80 1 --2- 172.21.15.116:0/1226237892 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x564c6de65790 0x564c6de67bb0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f5213272b80 1 --2- 172.21.15.116:0/1226237892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564c6ddf9850 0x564c6de0a250 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f5213272b80 1 --2- 172.21.15.116:0/1226237892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c6ddfb050 0x564c6de63500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f5213272b80 1 --2- 172.21.15.116:0/1226237892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c6ddfa1b0 0x564c6de0a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f51dea00700 0 client.26236 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:49.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f5213272b80 1 -- 172.21.15.116:0/1226237892 >> 172.21.15.116:0/1226237892 conn(0x564c6dd58720 msgr2=0x564c6de036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:49.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f5213272b80 1 -- 172.21.15.116:0/1226237892 shutdown_connections 2024-03-30T01:04:49.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f5213272b80 1 -- 172.21.15.116:0/1226237892 wait complete. 2024-03-30T01:04:49.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.378+0000 7f5213272b80 20 client.26236 trim_cache size 0 max 16384 2024-03-30T01:04:49.390 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:04:49.390 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-30T01:04:49.390 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "79d9c520-6c1d-49a0-9b63-93c578965389", 2024-03-30T01:04:49.390 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-30T01:04:49.390 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:04:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.379+0000 7f51ff800700 1 -- 172.21.15.116:0/1372418671 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f51e00a1120 msgr2=0x7f51e00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.379+0000 7f51ff800700 1 --2- 172.21.15.116:0/1372418671 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f51e00a1120 0x7f51e00a35e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f51f0005000 tx=0x7f51f0026000 comp rx=0 tx=0).stop 2024-03-30T01:04:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.379+0000 7f51ff800700 1 -- 172.21.15.116:0/1372418671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51f8101f70 msgr2=0x7f51f81995b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.379+0000 7f51ff800700 1 --2- 172.21.15.116:0/1372418671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51f8101f70 0x7f51f81995b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f51e400e9a0 tx=0x7f51e400ecb0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.379+0000 7f51ff800700 1 -- 172.21.15.116:0/1372418671 shutdown_connections 2024-03-30T01:04:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.379+0000 7f51ff800700 1 --2- 172.21.15.116:0/1372418671 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f51e00a1120 0x7f51e00a35e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.379+0000 7f51ff800700 1 --2- 172.21.15.116:0/1372418671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f8102910 0x7f51f81a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.379+0000 7f51ff800700 1 --2- 172.21.15.116:0/1372418671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51f8100d80 0x7f51f8199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.379+0000 7f51ff800700 1 --2- 172.21.15.116:0/1372418671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51f8101f70 0x7f51f81995b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.379+0000 7f51ff800700 1 -- 172.21.15.116:0/1372418671 >> 172.21.15.116:0/1372418671 conn(0x7f51f80fc2f0 msgr2=0x7f51f8072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.379+0000 7f51ff800700 1 -- 172.21.15.116:0/1372418671 shutdown_connections 2024-03-30T01:04:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.379+0000 7f51ff800700 1 -- 172.21.15.116:0/1372418671 wait complete. 2024-03-30T01:04:49.404 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T01:04:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.598+0000 7ff686c00700 1 -- 172.21.15.116:0/1366339739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff680074df0 msgr2=0x7ff680073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.598+0000 7ff686c00700 1 --2- 172.21.15.116:0/1366339739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff680074df0 0x7ff680073240 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff670009990 tx=0x7ff670009ca0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.598+0000 7ff686c00700 1 -- 172.21.15.116:0/1366339739 shutdown_connections 2024-03-30T01:04:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.598+0000 7ff686c00700 1 --2- 172.21.15.116:0/1366339739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff680074df0 0x7ff680073240 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.598+0000 7ff686c00700 1 --2- 172.21.15.116:0/1366339739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff680108c40 0x7ff68010b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.598+0000 7ff686c00700 1 --2- 172.21.15.116:0/1366339739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff680073810 0x7ff680073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.598+0000 7ff686c00700 1 -- 172.21.15.116:0/1366339739 >> 172.21.15.116:0/1366339739 conn(0x7ff6800fc2f0 msgr2=0x7ff6800fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.598+0000 7ff686c00700 1 -- 172.21.15.116:0/1366339739 shutdown_connections 2024-03-30T01:04:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.598+0000 7ff686c00700 1 -- 172.21.15.116:0/1366339739 wait complete. 2024-03-30T01:04:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.599+0000 7ff686c00700 1 Processor -- start 2024-03-30T01:04:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.599+0000 7ff686c00700 1 -- start start 2024-03-30T01:04:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.599+0000 7ff686c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff680073810 0x7ff680198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.599+0000 7ff686c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff680074df0 0x7ff680199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.599+0000 7ff686c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff680108c40 0x7ff6801a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.599+0000 7ff686c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6801a0a40 con 0x7ff680073810 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.599+0000 7ff686c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6801a0b80 con 0x7ff680074df0 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.599+0000 7ff686c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6801a0d10 con 0x7ff680108c40 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.600+0000 7ff67fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff680073810 0x7ff680198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.600+0000 7ff67fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff680073810 0x7ff680198f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51848/0 (socket says 172.21.15.116:51848) 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.600+0000 7ff67f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff680074df0 0x7ff680199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.600+0000 7ff67fe00700 1 -- 172.21.15.116:0/2585059131 learned_addr learned my addr 172.21.15.116:0/2585059131 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.600+0000 7ff684c00700 1 --2- 172.21.15.116:0/2585059131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff680108c40 0x7ff6801a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.600+0000 7ff67fe00700 1 -- 172.21.15.116:0/2585059131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff680108c40 msgr2=0x7ff6801a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.600+0000 7ff67fe00700 1 --2- 172.21.15.116:0/2585059131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff680108c40 0x7ff6801a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.600+0000 7ff67fe00700 1 -- 172.21.15.116:0/2585059131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff680074df0 msgr2=0x7ff680199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.600+0000 7ff67fe00700 1 --2- 172.21.15.116:0/2585059131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff680074df0 0x7ff680199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.600+0000 7ff67fe00700 1 -- 172.21.15.116:0/2585059131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6700095f0 con 0x7ff680073810 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.600+0000 7ff67fe00700 1 --2- 172.21.15.116:0/2585059131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff680073810 0x7ff680198f10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff670005910 tx=0x7ff670009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.600+0000 7ff67cc00700 1 -- 172.21.15.116:0/2585059131 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67001d070 con 0x7ff680073810 2024-03-30T01:04:49.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.601+0000 7ff67cc00700 1 -- 172.21.15.116:0/2585059131 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff670021410 con 0x7ff680073810 2024-03-30T01:04:49.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.601+0000 7ff686c00700 1 -- 172.21.15.116:0/2585059131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6801a0f90 con 0x7ff680073810 2024-03-30T01:04:49.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.601+0000 7ff67cc00700 1 -- 172.21.15.116:0/2585059131 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff670005720 con 0x7ff680073810 2024-03-30T01:04:49.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.601+0000 7ff686c00700 1 -- 172.21.15.116:0/2585059131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6801a1350 con 0x7ff680073810 2024-03-30T01:04:49.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.605+0000 7ff67cc00700 1 -- 172.21.15.116:0/2585059131 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7ff670021580 con 0x7ff680073810 2024-03-30T01:04:49.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.606+0000 7ff69a6f1b80 1 Processor -- start 2024-03-30T01:04:49.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.606+0000 7ff67cc00700 1 --2- 172.21.15.116:0/2585059131 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff6640a11a0 0x7ff6640a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.606+0000 7ff69a6f1b80 1 -- start start 2024-03-30T01:04:49.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.606+0000 7ff67cc00700 1 -- 172.21.15.116:0/2585059131 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff6700cdee0 con 0x7ff680073810 2024-03-30T01:04:49.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff684c00700 1 --2- 172.21.15.116:0/2585059131 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff6640a11a0 0x7ff6640a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff69a6f1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d98104ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff69a6f1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 0x55d98104f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff69a6f1b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d981050450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff69a6f1b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d980fa6990 con 0x55d98104ffd0 2024-03-30T01:04:49.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff69a6f1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d980fa4350 con 0x55d98104f130 2024-03-30T01:04:49.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff69a6f1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d980f9fc40 con 0x55d98104e7d0 2024-03-30T01:04:49.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff67fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d98104ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff67fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d98104ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:51444/0 (socket says 172.21.15.116:51444) 2024-03-30T01:04:49.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff67fe00700 1 -- 172.21.15.116:0/534228291 learned_addr learned my addr 172.21.15.116:0/534228291 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:49.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff684c00700 1 --2- 172.21.15.116:0/534228291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d981050450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff67f400700 1 --2- 172.21.15.116:0/534228291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 0x55d98104f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff67fe00700 1 -- 172.21.15.116:0/534228291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 msgr2=0x55d98104f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff67fe00700 1 --2- 172.21.15.116:0/534228291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 0x55d98104f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff67fe00700 1 -- 172.21.15.116:0/534228291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 msgr2=0x55d981050450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff67fe00700 1 --2- 172.21.15.116:0/534228291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d981050450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff67fe00700 1 -- 172.21.15.116:0/534228291 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d980e213b0 con 0x55d98104e7d0 2024-03-30T01:04:49.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.607+0000 7ff684c00700 1 --2- 172.21.15.116:0/534228291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d981050450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:04:49.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.608+0000 7ff684c00700 1 --2- 172.21.15.116:0/2585059131 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff6640a11a0 0x7ff6640a3660 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff67800f870 tx=0x7ff67800fc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:49.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.608+0000 7ff67fe00700 1 --2- 172.21.15.116:0/534228291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d98104ebf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff670006010 tx=0x7ff67009aa10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4005cf1b7ad79c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:49.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.608+0000 7ff675200700 1 -- 172.21.15.116:0/534228291 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67009eee0 con 0x55d98104e7d0 2024-03-30T01:04:49.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.608+0000 7ff675200700 1 -- 172.21.15.116:0/534228291 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6700a34a0 con 0x55d98104e7d0 2024-03-30T01:04:49.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.608+0000 7ff675200700 1 -- 172.21.15.116:0/534228291 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67009b570 con 0x55d98104e7d0 2024-03-30T01:04:49.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.608+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/534228291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 msgr2=0x55d98104ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.608+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/534228291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d98104ebf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff670006010 tx=0x7ff67009aa10 comp rx=0 tx=0).stop 2024-03-30T01:04:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.609+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/534228291 shutdown_connections 2024-03-30T01:04:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.609+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/534228291 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d98104ebf0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.609+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/534228291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d981050450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.609+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/534228291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 0x55d98104f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.609+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/534228291 >> 172.21.15.116:0/534228291 conn(0x55d980fad6a0 msgr2=0x55d9810499a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.609+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/534228291 shutdown_connections 2024-03-30T01:04:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.609+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/534228291 wait complete. 2024-03-30T01:04:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.609+0000 7ff69a6f1b80 1 -- start start 2024-03-30T01:04:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.609+0000 7ff69a6f1b80 1 Processor -- start 2024-03-30T01:04:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff69a6f1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d981058320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff69a6f1b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104f130 0x55d981058860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff69a6f1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104ffd0 0x55d98105f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff69a6f1b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d980f3bd80 con 0x55d98104f130 2024-03-30T01:04:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff69a6f1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d980f3ba20 con 0x55d98104ffd0 2024-03-30T01:04:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff69a6f1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d980e236a0 con 0x55d98104e7d0 2024-03-30T01:04:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff69a6f1b80 10 client.0 _ll_register_callbacks cb 0x55d980f45470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff67fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d981058320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff67fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d981058320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:51460/0 (socket says 172.21.15.116:51460) 2024-03-30T01:04:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff67fe00700 1 -- 172.21.15.116:0/2986513572 learned_addr learned my addr 172.21.15.116:0/2986513572 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff684c00700 1 --2- 172.21.15.116:0/2986513572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104ffd0 0x55d98105f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff67f400700 1 --2- 172.21.15.116:0/2986513572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104f130 0x55d981058860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff67fe00700 1 -- 172.21.15.116:0/2986513572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104ffd0 msgr2=0x55d98105f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff67fe00700 1 --2- 172.21.15.116:0/2986513572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104ffd0 0x55d98105f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff67fe00700 1 -- 172.21.15.116:0/2986513572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104f130 msgr2=0x55d981058860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff67fe00700 1 --2- 172.21.15.116:0/2986513572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104f130 0x55d981058860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff67fe00700 1 -- 172.21.15.116:0/2986513572 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6801a1350 con 0x55d98104e7d0 2024-03-30T01:04:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.610+0000 7ff684c00700 1 --2- 172.21.15.116:0/2986513572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104ffd0 0x55d98105f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.611+0000 7ff67fe00700 1 --2- 172.21.15.116:0/2986513572 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d981058320 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff6700975b0 tx=0x7ff67009a930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1e137368a5e9316 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.611+0000 7ff669600700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:04:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.611+0000 7ff669600700 1 -- 172.21.15.116:0/2986513572 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff670099450 con 0x55d98104e7d0 2024-03-30T01:04:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.611+0000 7ff669600700 1 -- 172.21.15.116:0/2986513572 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff67009eee0 con 0x55d98104e7d0 2024-03-30T01:04:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.611+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/2986513572 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d980f60380 con 0x55d98104e7d0 2024-03-30T01:04:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.611+0000 7ff669600700 1 -- 172.21.15.116:0/2986513572 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff67009b520 con 0x55d98104e7d0 2024-03-30T01:04:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.611+0000 7ff69a6f1b80 10 client.26248 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.611+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/2986513572 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d980dcdb60 con 0x55d98104e7d0 2024-03-30T01:04:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.611+0000 7ff669600700 1 -- 172.21.15.116:0/2986513572 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6700a7080 con 0x55d98104e7d0 2024-03-30T01:04:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.611+0000 7ff669600700 1 -- 172.21.15.116:0/2986513572 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7ff6700b9070 con 0x55d98104e7d0 2024-03-30T01:04:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.612+0000 7ff69a6f1b80 10 client.26248 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.612+0000 7ff69a6f1b80 10 client.26248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:04:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.612+0000 7ff69a6f1b80 20 client.26248 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.612+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/2986513572 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d981063c00 0x55d981066020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.612+0000 7ff69a6f1b80 4 client.26248 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:04:49.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.612+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/2986513572 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d980f98700 con 0x55d981063c00 2024-03-30T01:04:49.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.612+0000 7ff67f400700 1 --2- 172.21.15.116:0/2986513572 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d981063c00 0x55d981066020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.612+0000 7ff67f400700 1 --2- 172.21.15.116:0/2986513572 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d981063c00 0x55d981066020 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a9f0d7715df6e99 server_cookie=5ea7377e6dbd2dc1 in_seq=0 out_seq=0 2024-03-30T01:04:49.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.612+0000 7ff669600700 10 client.26248 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:49.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff669600700 1 -- 172.21.15.116:0/2986513572 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d980f98700 con 0x55d981063c00 2024-03-30T01:04:49.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff669600700 10 client.26248 handle_command_reply: tid=0 2024-03-30T01:04:49.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff69a6f1b80 1 client.26248 shutdown 2024-03-30T01:04:49.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/2986513572 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 msgr2=0x55d981058320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/2986513572 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d981058320 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff6700975b0 tx=0x7ff67009a930 comp rx=0 tx=0).stop 2024-03-30T01:04:49.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/2986513572 shutdown_connections 2024-03-30T01:04:49.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/2986513572 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d981063c00 0x55d981066020 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/2986513572 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d981058320 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff669600700 0 client.26248 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:49.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/2986513572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104ffd0 0x55d98105f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/2986513572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104f130 0x55d981058860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/2986513572 >> 172.21.15.116:0/2986513572 conn(0x55d980fad6a0 msgr2=0x55d98104d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:49.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/2986513572 shutdown_connections 2024-03-30T01:04:49.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/2986513572 wait complete. 2024-03-30T01:04:49.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.616+0000 7ff69a6f1b80 20 client.26248 trim_cache size 0 max 16384 2024-03-30T01:04:49.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.634+0000 7ff69a6f1b80 1 Processor -- start 2024-03-30T01:04:49.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.634+0000 7ff69a6f1b80 1 -- start start 2024-03-30T01:04:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.634+0000 7ff69a6f1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d981048f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.634+0000 7ff69a6f1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 0x55d98105f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.634+0000 7ff69a6f1b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d981057830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.634+0000 7ff69a6f1b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d98104fe50 con 0x55d98104ffd0 2024-03-30T01:04:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.634+0000 7ff69a6f1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d98104e370 con 0x55d98104f130 2024-03-30T01:04:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.634+0000 7ff69a6f1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d981060a50 con 0x55d98104e7d0 2024-03-30T01:04:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.634+0000 7ff684c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d981057830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.634+0000 7ff67fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d981048f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.634+0000 7ff684c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d981057830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51884/0 (socket says 172.21.15.116:51884) 2024-03-30T01:04:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.634+0000 7ff67fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d981048f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:51476/0 (socket says 172.21.15.116:51476) 2024-03-30T01:04:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.634+0000 7ff684c00700 1 -- 172.21.15.116:0/131813018 learned_addr learned my addr 172.21.15.116:0/131813018 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.634+0000 7ff67f400700 1 --2- 172.21.15.116:0/131813018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 0x55d98105f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.635+0000 7ff684c00700 1 -- 172.21.15.116:0/131813018 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 msgr2=0x55d981048f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.635+0000 7ff684c00700 1 --2- 172.21.15.116:0/131813018 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d981048f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.635+0000 7ff684c00700 1 -- 172.21.15.116:0/131813018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 msgr2=0x55d98105f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.635+0000 7ff684c00700 1 --2- 172.21.15.116:0/131813018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 0x55d98105f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.635+0000 7ff684c00700 1 -- 172.21.15.116:0/131813018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d981057310 con 0x55d98104ffd0 2024-03-30T01:04:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.635+0000 7ff684c00700 1 --2- 172.21.15.116:0/131813018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d981057830 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff67800fee0 tx=0x7ff67801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=119f9004bed2842d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.635+0000 7ff675c00700 1 -- 172.21.15.116:0/131813018 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67800e040 con 0x55d98104ffd0 2024-03-30T01:04:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.635+0000 7ff675c00700 1 -- 172.21.15.116:0/131813018 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff67800f040 con 0x55d98104ffd0 2024-03-30T01:04:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.635+0000 7ff675c00700 1 -- 172.21.15.116:0/131813018 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff678004280 con 0x55d98104ffd0 2024-03-30T01:04:49.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.636+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/131813018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 msgr2=0x55d981057830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.636+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/131813018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d981057830 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff67800fee0 tx=0x7ff67801f000 comp rx=0 tx=0).stop 2024-03-30T01:04:49.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.636+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/131813018 shutdown_connections 2024-03-30T01:04:49.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.636+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/131813018 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d981048f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.636+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/131813018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d981057830 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.636+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/131813018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 0x55d98105f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.636+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/131813018 >> 172.21.15.116:0/131813018 conn(0x55d980fad6a0 msgr2=0x55d9810425a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:49.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.636+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/131813018 shutdown_connections 2024-03-30T01:04:49.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.636+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/131813018 wait complete. 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.636+0000 7ff69a6f1b80 1 -- start start 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.637+0000 7ff69a6f1b80 1 Processor -- start 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.637+0000 7ff69a6f1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d98105f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.637+0000 7ff69a6f1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 0x55d98105dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.637+0000 7ff69a6f1b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d98105e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.637+0000 7ff69a6f1b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d98105e970 con 0x55d98104ffd0 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.637+0000 7ff69a6f1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d98105eae0 con 0x55d98104f130 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.637+0000 7ff69a6f1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9810b7a20 con 0x55d98104e7d0 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff69a6f1b80 10 client.0 _ll_register_callbacks cb 0x55d980f45470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff67f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 0x55d98105dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff67fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d98105f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff67f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 0x55d98105dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38516/0 (socket says 172.21.15.116:38516) 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff67fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d98105f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:51488/0 (socket says 172.21.15.116:51488) 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff67f400700 1 -- 172.21.15.116:0/835985455 learned_addr learned my addr 172.21.15.116:0/835985455 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:04:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff684c00700 1 --2- 172.21.15.116:0/835985455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d98105e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff67fe00700 1 -- 172.21.15.116:0/835985455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 msgr2=0x55d98105dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff67fe00700 1 --2- 172.21.15.116:0/835985455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 0x55d98105dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff67fe00700 1 -- 172.21.15.116:0/835985455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 msgr2=0x55d98105e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:04:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff67fe00700 1 --2- 172.21.15.116:0/835985455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d98105e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff67fe00700 1 -- 172.21.15.116:0/835985455 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6780093f0 con 0x55d98104e7d0 2024-03-30T01:04:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff67f400700 1 --2- 172.21.15.116:0/835985455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 0x55d98105dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:04:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff67fe00700 1 --2- 172.21.15.116:0/835985455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d98105f6a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff67009a9a0 tx=0x7ff67009aca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=684c82bfcd077d15 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:04:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff669600700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2024-03-30T01:04:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.638+0000 7ff669600700 1 -- 172.21.15.116:0/835985455 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6700cbea0 con 0x55d98104e7d0 2024-03-30T01:04:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.639+0000 7ff669600700 1 -- 172.21.15.116:0/835985455 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff67009eee0 con 0x55d98104e7d0 2024-03-30T01:04:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.639+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/835985455 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff6700a34a0 con 0x55d98104e7d0 2024-03-30T01:04:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.639+0000 7ff669600700 1 -- 172.21.15.116:0/835985455 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff670021860 con 0x55d98104e7d0 2024-03-30T01:04:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.639+0000 7ff69a6f1b80 10 client.26254 fetch_fsmap learned FSMap version 28 2024-03-30T01:04:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.639+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/835985455 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d980f65ea0 con 0x55d98104e7d0 2024-03-30T01:04:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.639+0000 7ff669600700 1 -- 172.21.15.116:0/835985455 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6700a34a0 con 0x55d98104e7d0 2024-03-30T01:04:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.639+0000 7ff669600700 1 -- 172.21.15.116:0/835985455 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 28) v1 ==== 7490+0+0 (secure 0 0 0) 0x7ff670096fc0 con 0x55d98104e7d0 2024-03-30T01:04:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.639+0000 7ff69a6f1b80 10 client.26254 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T01:04:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.639+0000 7ff69a6f1b80 10 client.26254 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:04:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.639+0000 7ff69a6f1b80 20 client.26254 populate_metadata read hostname 'smithi116' 2024-03-30T01:04:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.639+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/835985455 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d9810b8d10 0x55d9810bb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:04:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.639+0000 7ff69a6f1b80 4 client.26254 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T01:04:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.639+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/835985455 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d980deecf0 con 0x55d9810b8d10 2024-03-30T01:04:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.639+0000 7ff67f400700 1 --2- 172.21.15.116:0/835985455 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d9810b8d10 0x55d9810bb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:04:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff67f400700 1 --2- 172.21.15.116:0/835985455 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d9810b8d10 0x55d9810bb130 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83fd5dbfce386ac9 server_cookie=8d9d23ef58d24e89 in_seq=0 out_seq=0 2024-03-30T01:04:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff669600700 10 client.26254 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff669600700 1 -- 172.21.15.116:0/835985455 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55d980deecf0 con 0x55d9810b8d10 2024-03-30T01:04:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff669600700 10 client.26254 handle_command_reply: tid=0 2024-03-30T01:04:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff69a6f1b80 1 client.26254 shutdown 2024-03-30T01:04:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/835985455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 msgr2=0x55d98105f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/835985455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d98105f6a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff67009a9a0 tx=0x7ff67009aca0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/835985455 shutdown_connections 2024-03-30T01:04:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/835985455 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x55d9810b8d10 0x55d9810bb130 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/835985455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d98104e7d0 0x55d98105f6a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/835985455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98104ffd0 0x55d98105e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff669600700 0 client.26254 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:04:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff69a6f1b80 1 --2- 172.21.15.116:0/835985455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d98104f130 0x55d98105dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/835985455 >> 172.21.15.116:0/835985455 conn(0x55d980fad6a0 msgr2=0x55d981057830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/835985455 shutdown_connections 2024-03-30T01:04:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.640+0000 7ff69a6f1b80 1 -- 172.21.15.116:0/835985455 wait complete. 2024-03-30T01:04:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.641+0000 7ff69a6f1b80 20 client.26254 trim_cache size 0 max 16384 2024-03-30T01:04:49.652 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:04:49.652 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (6690 inodes in the stack)", 2024-03-30T01:04:49.652 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-30T01:04:49.652 INFO:teuthology.orchestra.run.smithi116.stdout: "79d9c520-6c1d-49a0-9b63-93c578965389": { 2024-03-30T01:04:49.652 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-30T01:04:49.652 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "79d9c520-6c1d-49a0-9b63-93c578965389", 2024-03-30T01:04:49.652 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-30T01:04:49.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:04:49.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:04:49.652 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:04:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.641+0000 7ff686c00700 1 -- 172.21.15.116:0/2585059131 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff6640a11a0 msgr2=0x7ff6640a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.641+0000 7ff686c00700 1 --2- 172.21.15.116:0/2585059131 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff6640a11a0 0x7ff6640a3660 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff67800f870 tx=0x7ff67800fc30 comp rx=0 tx=0).stop 2024-03-30T01:04:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.641+0000 7ff686c00700 1 -- 172.21.15.116:0/2585059131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff680073810 msgr2=0x7ff680198f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:04:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.641+0000 7ff686c00700 1 --2- 172.21.15.116:0/2585059131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff680073810 0x7ff680198f10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff670005910 tx=0x7ff670009580 comp rx=0 tx=0).stop 2024-03-30T01:04:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.641+0000 7ff686c00700 1 -- 172.21.15.116:0/2585059131 shutdown_connections 2024-03-30T01:04:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.641+0000 7ff686c00700 1 --2- 172.21.15.116:0/2585059131 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff6640a11a0 0x7ff6640a3660 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.641+0000 7ff686c00700 1 --2- 172.21.15.116:0/2585059131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff680108c40 0x7ff6801a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.641+0000 7ff686c00700 1 --2- 172.21.15.116:0/2585059131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff680073810 0x7ff680198f10 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.641+0000 7ff686c00700 1 --2- 172.21.15.116:0/2585059131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff680074df0 0x7ff680199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:04:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.641+0000 7ff686c00700 1 -- 172.21.15.116:0/2585059131 >> 172.21.15.116:0/2585059131 conn(0x7ff6800fc2f0 msgr2=0x7ff680104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:04:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.641+0000 7ff686c00700 1 -- 172.21.15.116:0/2585059131 shutdown_connections 2024-03-30T01:04:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:04:49.641+0000 7ff686c00700 1 -- 172.21.15.116:0/2585059131 wait complete. 2024-03-30T01:04:49.665 INFO:tasks.cephfs.filesystem:scrub status for tag:79d9c520-6c1d-49a0-9b63-93c578965389 - {'path': '/', 'tag': '79d9c520-6c1d-49a0-9b63-93c578965389', 'options': 'recursive,force'} 2024-03-30T01:04:50.601 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 01:04:50 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-i[85770]: 2024-03-30T01:04:50.283+0000 7f5360a86700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000002f77 [2,head] /volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-50.jpg auth v1208 ap=1 s=27704 n(v0 rc2024-03-30T01:00:19.426069+0000 b27704 1=1+0) (ifile excl) (iversion lock) cr={15318=0-4194304@1} caps={15318=pAsLsXsFsxcrwb/pFscr@1},l=15318 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55d5677c9700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000002f77:[<0x100000007c0/picture-50.jpg v1208>,<0x100000003a7/blog-7 v175758>,<0x10000000208/blogtest_in v3133>,<0x10000000201/src v2386>,<0x10000000006/blogbench-1.0 v2066>,<0x10000000005/tmp v579>,<0x100000001fb/client.0 v566>,<0x100000001fa/2e70c87b-dd51-4566-ab0f-a850e31dba64 v556>,<0x10000000001/sv_1 v555>,<0x10000000000/qa v548>,<0x1/volumes v526>]//[3]","memoryvalue":"(4)0x10000002f77:[<0x100000007c0/picture-50.jpg v1208>,<0x100000003a7/blog-7 v182315>,<0x10000000208/blogtest_in v3210>,<0x10000000201/src v2429>,<0x10000000006/blogbench-1.0 v2093>,<0x10000000005/tmp v609>,<0x100000001fb/client.0 v593>,<0x100000001fa/2e70c87b-dd51-4566-ab0f-a850e31dba64 v558>,<0x10000000001/sv_1 v582>,<0x10000000000/qa v562>,<0x1/volumes v484>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-30T01:04:50.601 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 01:04:50 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-i[85770]: 2024-03-30T01:04:50.286+0000 7f5360a86700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000002f7c [2,head] /volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-58.xml auth v1206 ap=1 s=4887 n(v0 rc2024-03-30T01:00:19.432069+0000 b4887 1=1+0) (ifile excl) (iversion lock) cr={15318=0-4194304@1} caps={15318=pAsLsXsFsxcrwb/pFscr@1},l=15318 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55d569305180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000002f7c:[<0x100000007c0/article-58.xml v1206>,<0x100000003a7/blog-7 v175758>,<0x10000000208/blogtest_in v3133>,<0x10000000201/src v2386>,<0x10000000006/blogbench-1.0 v2066>,<0x10000000005/tmp v579>,<0x100000001fb/client.0 v566>,<0x100000001fa/2e70c87b-dd51-4566-ab0f-a850e31dba64 v556>,<0x10000000001/sv_1 v555>,<0x10000000000/qa v548>,<0x1/volumes v526>]//[3]","memoryvalue":"(4)0x10000002f7c:[<0x100000007c0/article-58.xml v1206>,<0x100000003a7/blog-7 v182315>,<0x10000000208/blogtest_in v3210>,<0x10000000201/src v2429>,<0x10000000006/blogbench-1.0 v2093>,<0x10000000005/tmp v609>,<0x100000001fb/client.0 v593>,<0x100000001fa/2e70c87b-dd51-4566-ab0f-a850e31dba64 v558>,<0x10000000001/sv_1 v582>,<0x10000000000/qa v562>,<0x1/volumes v484>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-30T01:04:50.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:50 smithi181 ceph-mon[47622]: pgmap v500: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.4 MiB/s wr, 484 op/s 2024-03-30T01:04:50.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:50 smithi181 ceph-mon[47622]: scrub queued for path: / 2024-03-30T01:04:50.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:50 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:50.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:50 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:04:50.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:50 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:04:50.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:50 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:50.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:50 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:04:50.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:50 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:50.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:50 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:04:50.602 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:50 smithi181 ceph-mon[47622]: scrub summary: active paths [/] 2024-03-30T01:04:50.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:50 smithi116 ceph-mon[38086]: pgmap v500: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.4 MiB/s wr, 484 op/s 2024-03-30T01:04:50.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:50 smithi116 ceph-mon[38086]: scrub queued for path: / 2024-03-30T01:04:50.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:50 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:50.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:50 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:04:50.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:50 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:04:50.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:50 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:50.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:50 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:04:50.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:50 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:50.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:50 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:04:50.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:50 smithi116 ceph-mon[38086]: scrub summary: active paths [/] 2024-03-30T01:04:51.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:50 smithi130 ceph-mon[47626]: pgmap v500: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.4 MiB/s wr, 484 op/s 2024-03-30T01:04:51.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:50 smithi130 ceph-mon[47626]: scrub queued for path: / 2024-03-30T01:04:51.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:50 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:51.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:50 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:04:51.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:50 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:04:51.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:50 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:51.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:50 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:04:51.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:50 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:51.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:50 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:04:51.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:50 smithi130 ceph-mon[47626]: scrub summary: active paths [/] 2024-03-30T01:04:51.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:51 smithi116 ceph-mon[38086]: Scrub error on inode 0x10000002f77 (/volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-50.jpg) see mds.i log and `damage ls` output for details 2024-03-30T01:04:51.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:51 smithi116 ceph-mon[38086]: Scrub error on inode 0x10000002f7c (/volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-58.xml) see mds.i log and `damage ls` output for details 2024-03-30T01:04:52.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:51 smithi181 ceph-mon[47622]: Scrub error on inode 0x10000002f77 (/volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-50.jpg) see mds.i log and `damage ls` output for details 2024-03-30T01:04:52.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:51 smithi181 ceph-mon[47622]: Scrub error on inode 0x10000002f7c (/volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-58.xml) see mds.i log and `damage ls` output for details 2024-03-30T01:04:52.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:51 smithi130 ceph-mon[47626]: Scrub error on inode 0x10000002f77 (/volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-50.jpg) see mds.i log and `damage ls` output for details 2024-03-30T01:04:52.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:51 smithi130 ceph-mon[47626]: Scrub error on inode 0x10000002f7c (/volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-58.xml) see mds.i log and `damage ls` output for details 2024-03-30T01:04:52.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:52 smithi116 ceph-mon[38086]: pgmap v501: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 229 op/s 2024-03-30T01:04:53.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:52 smithi181 ceph-mon[47622]: pgmap v501: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 229 op/s 2024-03-30T01:04:53.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:52 smithi130 ceph-mon[47626]: pgmap v501: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 229 op/s 2024-03-30T01:04:53.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:53 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-a[38063]: 2024-03-30T01:04:53.607+0000 7f366d000700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-30T01:04:53.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:53 smithi116 ceph-mon[38086]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-30T01:04:53.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:53 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] up:active 2024-03-30T01:04:53.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:53 smithi116 ceph-mon[38086]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:04:54.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:53 smithi181 ceph-mon[47622]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-30T01:04:54.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:53 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] up:active 2024-03-30T01:04:54.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:53 smithi181 ceph-mon[47622]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:04:54.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:53 smithi130 ceph-mon[47626]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-30T01:04:54.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:53 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] up:active 2024-03-30T01:04:54.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:53 smithi130 ceph-mon[47626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:04:54.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:54 smithi116 ceph-mon[38086]: pgmap v502: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 1.1 MiB/s wr, 179 op/s 2024-03-30T01:04:54.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:54 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:55.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:54 smithi181 ceph-mon[47622]: pgmap v502: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 1.1 MiB/s wr, 179 op/s 2024-03-30T01:04:55.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:54 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:55.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:54 smithi130 ceph-mon[47626]: pgmap v502: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 1.1 MiB/s wr, 179 op/s 2024-03-30T01:04:55.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:54 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:55.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:55 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:55.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:55 smithi116 ceph-mon[38086]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:56.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:55 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:56.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:55 smithi181 ceph-mon[47622]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:56.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:55 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:56.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:55 smithi130 ceph-mon[47626]: scrub summary: idle+waiting paths [/] 2024-03-30T01:04:57.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:56 smithi181 ceph-mon[47622]: pgmap v503: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 123 KiB/s wr, 267 op/s 2024-03-30T01:04:57.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:56 smithi130 ceph-mon[47626]: pgmap v503: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 123 KiB/s wr, 267 op/s 2024-03-30T01:04:57.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:56 smithi116 ceph-mon[38086]: pgmap v503: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 123 KiB/s wr, 267 op/s 2024-03-30T01:04:58.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:04:58 smithi181 ceph-mon[47622]: pgmap v504: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 63 KiB/s wr, 265 op/s 2024-03-30T01:04:58.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:04:58 smithi130 ceph-mon[47626]: pgmap v504: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 63 KiB/s wr, 265 op/s 2024-03-30T01:04:58.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:04:58 smithi116 ceph-mon[38086]: pgmap v504: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 63 KiB/s wr, 265 op/s 2024-03-30T01:05:00.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:00 smithi181 ceph-mon[47622]: pgmap v505: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 22 KiB/s wr, 258 op/s 2024-03-30T01:05:00.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:00 smithi130 ceph-mon[47626]: pgmap v505: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 22 KiB/s wr, 258 op/s 2024-03-30T01:05:00.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:00 smithi116 ceph-mon[38086]: pgmap v505: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 22 KiB/s wr, 258 op/s 2024-03-30T01:05:02.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:02 smithi181 ceph-mon[47622]: pgmap v506: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 23 KiB/s wr, 259 op/s 2024-03-30T01:05:02.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:02 smithi130 ceph-mon[47626]: pgmap v506: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 23 KiB/s wr, 259 op/s 2024-03-30T01:05:02.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:02 smithi116 ceph-mon[38086]: pgmap v506: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 23 KiB/s wr, 259 op/s 2024-03-30T01:05:04.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:04 smithi181 ceph-mon[47622]: pgmap v507: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 24 KiB/s wr, 255 op/s 2024-03-30T01:05:04.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:04 smithi130 ceph-mon[47626]: pgmap v507: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 24 KiB/s wr, 255 op/s 2024-03-30T01:05:04.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:04 smithi116 ceph-mon[38086]: pgmap v507: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 24 KiB/s wr, 255 op/s 2024-03-30T01:05:06.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:06 smithi181 ceph-mon[47622]: pgmap v508: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 573 KiB/s wr, 217 op/s 2024-03-30T01:05:06.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:06 smithi130 ceph-mon[47626]: pgmap v508: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 573 KiB/s wr, 217 op/s 2024-03-30T01:05:06.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:06 smithi116 ceph-mon[38086]: pgmap v508: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 573 KiB/s wr, 217 op/s 2024-03-30T01:05:07.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:07 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:05:07.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:07 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:05:07.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:07 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:05:07.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:07 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:05:07.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:07 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:05:07.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:07 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:05:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:08 smithi181 ceph-mon[47622]: scrub summary: idle 2024-03-30T01:05:08.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:08 smithi181 ceph-mon[47622]: pgmap v509: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 70 op/s 2024-03-30T01:05:08.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:08 smithi130 ceph-mon[47626]: scrub summary: idle 2024-03-30T01:05:08.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:08 smithi130 ceph-mon[47626]: pgmap v509: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 70 op/s 2024-03-30T01:05:08.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:08 smithi116 ceph-mon[38086]: scrub summary: idle 2024-03-30T01:05:08.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:08 smithi116 ceph-mon[38086]: pgmap v509: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 70 op/s 2024-03-30T01:05:10.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:10 smithi130 ceph-mon[47626]: pgmap v510: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 30 op/s 2024-03-30T01:05:10.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:10 smithi116 ceph-mon[38086]: pgmap v510: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 30 op/s 2024-03-30T01:05:11.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:10 smithi181 ceph-mon[47622]: pgmap v510: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 30 op/s 2024-03-30T01:05:12.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:12 smithi130 ceph-mon[47626]: pgmap v511: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 27 op/s 2024-03-30T01:05:12.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:12 smithi130 ceph-mon[47626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:12.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:12 smithi116 ceph-mon[38086]: pgmap v511: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 27 op/s 2024-03-30T01:05:12.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:12 smithi116 ceph-mon[38086]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:13.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:12 smithi181 ceph-mon[47622]: pgmap v511: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 27 op/s 2024-03-30T01:05:13.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:12 smithi181 ceph-mon[47622]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:14.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:14 smithi130 ceph-mon[47626]: pgmap v512: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 30 op/s 2024-03-30T01:05:14.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:14 smithi116 ceph-mon[38086]: pgmap v512: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 30 op/s 2024-03-30T01:05:15.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:14 smithi181 ceph-mon[47622]: pgmap v512: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 30 op/s 2024-03-30T01:05:16.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:16 smithi130 ceph-mon[47626]: pgmap v513: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 28 op/s 2024-03-30T01:05:16.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:16 smithi116 ceph-mon[38086]: pgmap v513: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 28 op/s 2024-03-30T01:05:17.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:16 smithi181 ceph-mon[47622]: pgmap v513: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 28 op/s 2024-03-30T01:05:18.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:18 smithi130 ceph-mon[47626]: pgmap v514: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 723 KiB/s wr, 22 op/s 2024-03-30T01:05:18.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:18 smithi116 ceph-mon[38086]: pgmap v514: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 723 KiB/s wr, 22 op/s 2024-03-30T01:05:19.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:18 smithi181 ceph-mon[47622]: pgmap v514: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 723 KiB/s wr, 22 op/s 2024-03-30T01:05:19.666 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T01:05:19.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.870+0000 7fbc07e00700 1 -- 172.21.15.116:0/460048115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc00102910 msgr2=0x7fbc001091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:19.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.870+0000 7fbc07e00700 1 --2- 172.21.15.116:0/460048115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc00102910 0x7fbc001091c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800b0d0 tx=0x7fbbf800b3e0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.870+0000 7fbc07e00700 1 -- 172.21.15.116:0/460048115 shutdown_connections 2024-03-30T01:05:19.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.870+0000 7fbc07e00700 1 --2- 172.21.15.116:0/460048115 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc00100d80 0x7fbc001011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.870+0000 7fbc07e00700 1 --2- 172.21.15.116:0/460048115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc00102910 0x7fbc001091c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.870+0000 7fbc07e00700 1 --2- 172.21.15.116:0/460048115 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc00101f70 0x7fbc001023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.870+0000 7fbc07e00700 1 -- 172.21.15.116:0/460048115 >> 172.21.15.116:0/460048115 conn(0x7fbc000fc2f0 msgr2=0x7fbc000fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:19.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.870+0000 7fbc07e00700 1 -- 172.21.15.116:0/460048115 shutdown_connections 2024-03-30T01:05:19.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.870+0000 7fbc07e00700 1 -- 172.21.15.116:0/460048115 wait complete. 2024-03-30T01:05:19.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.872+0000 7fbc07e00700 1 Processor -- start 2024-03-30T01:05:19.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.872+0000 7fbc07e00700 1 -- start start 2024-03-30T01:05:19.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.872+0000 7fbc07e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc00100d80 0x7fbc00074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.872+0000 7fbc07e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc00101f70 0x7fbc000731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.872+0000 7fbc07e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc00102910 0x7fbc000736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.872+0000 7fbc07e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00073e80 con 0x7fbc00101f70 2024-03-30T01:05:19.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.872+0000 7fbc07e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00073ff0 con 0x7fbc00102910 2024-03-30T01:05:19.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.872+0000 7fbc07e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc001a0e50 con 0x7fbc00100d80 2024-03-30T01:05:19.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.873+0000 7fbc04a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc00101f70 0x7fbc000731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.873+0000 7fbc04a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc00101f70 0x7fbc000731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47770/0 (socket says 172.21.15.116:47770) 2024-03-30T01:05:19.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.873+0000 7fbc04a00700 1 -- 172.21.15.116:0/2919170741 learned_addr learned my addr 172.21.15.116:0/2919170741 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:19.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.873+0000 7fbc05400700 1 --2- 172.21.15.116:0/2919170741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc00100d80 0x7fbc00074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.873+0000 7fbc05e00700 1 --2- 172.21.15.116:0/2919170741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc00102910 0x7fbc000736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.873+0000 7fbc04a00700 1 -- 172.21.15.116:0/2919170741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc00100d80 msgr2=0x7fbc00074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:19.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.873+0000 7fbc04a00700 1 --2- 172.21.15.116:0/2919170741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc00100d80 0x7fbc00074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.873+0000 7fbc04a00700 1 -- 172.21.15.116:0/2919170741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc00102910 msgr2=0x7fbc000736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:19.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.873+0000 7fbc04a00700 1 --2- 172.21.15.116:0/2919170741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc00102910 0x7fbc000736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.873+0000 7fbc04a00700 1 -- 172.21.15.116:0/2919170741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf8009d00 con 0x7fbc00101f70 2024-03-30T01:05:19.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.873+0000 7fbc04a00700 1 --2- 172.21.15.116:0/2919170741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc00101f70 0x7fbc000731a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00e9a0 tx=0x7fbbec00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:19.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.874+0000 7fbbfe000700 1 -- 172.21.15.116:0/2919170741 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec009790 con 0x7fbc00101f70 2024-03-30T01:05:19.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.874+0000 7fbbfe000700 1 -- 172.21.15.116:0/2919170741 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbec00ce70 con 0x7fbc00101f70 2024-03-30T01:05:19.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.874+0000 7fbbfe000700 1 -- 172.21.15.116:0/2919170741 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec018420 con 0x7fbc00101f70 2024-03-30T01:05:19.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.874+0000 7fbc07e00700 1 -- 172.21.15.116:0/2919170741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc001a10e0 con 0x7fbc00101f70 2024-03-30T01:05:19.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.874+0000 7fbc07e00700 1 -- 172.21.15.116:0/2919170741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc001a15a0 con 0x7fbc00101f70 2024-03-30T01:05:19.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.877+0000 7fbbfe000700 1 -- 172.21.15.116:0/2919170741 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fbbec0106f0 con 0x7fbc00101f70 2024-03-30T01:05:19.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.878+0000 7fbc1b851b80 1 Processor -- start 2024-03-30T01:05:19.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.878+0000 7fbbfe000700 1 --2- 172.21.15.116:0/2919170741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbbe80a11a0 0x7fbbe80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.878+0000 7fbc1b851b80 1 -- start start 2024-03-30T01:05:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbbfe000700 1 -- 172.21.15.116:0/2919170741 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbbec014070 con 0x7fbc00101f70 2024-03-30T01:05:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbc05e00700 1 --2- 172.21.15.116:0/2919170741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbbe80a11a0 0x7fbbe80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbc1b851b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e828047d0 0x557e82804bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbc1b851b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e82805130 0x557e828059f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbc1b851b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 0x557e82806450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbc1b851b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e8275c990 con 0x557e828047d0 2024-03-30T01:05:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbc1b851b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e8275a350 con 0x557e82805fd0 2024-03-30T01:05:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbc1b851b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e82755c40 con 0x557e82805130 2024-03-30T01:05:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbc05400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e828047d0 0x557e82804bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbc04a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e82805130 0x557e828059f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbc05400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e828047d0 0x557e82804bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47778/0 (socket says 172.21.15.116:47778) 2024-03-30T01:05:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbc04a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e82805130 0x557e828059f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:33630/0 (socket says 172.21.15.116:33630) 2024-03-30T01:05:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbc05400700 1 -- 172.21.15.116:0/1760555422 learned_addr learned my addr 172.21.15.116:0/1760555422 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:19.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbc05e00700 1 --2- 172.21.15.116:0/2919170741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbbe80a11a0 0x7fbbe80a3660 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009fd0 tx=0x7fbbf800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:19.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.879+0000 7fbc05e00700 1 --2- 172.21.15.116:0/1760555422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 0x557e82806450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.880+0000 7fbc04a00700 1 -- 172.21.15.116:0/1760555422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 msgr2=0x557e82806450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:19.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.880+0000 7fbc04a00700 1 --2- 172.21.15.116:0/1760555422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 0x557e82806450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.880+0000 7fbc04a00700 1 -- 172.21.15.116:0/1760555422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e828047d0 msgr2=0x557e82804bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:19.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.880+0000 7fbc04a00700 1 --2- 172.21.15.116:0/1760555422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e828047d0 0x557e82804bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.880+0000 7fbc04a00700 1 -- 172.21.15.116:0/1760555422 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e825d73b0 con 0x557e82805130 2024-03-30T01:05:19.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.880+0000 7fbc05e00700 1 --2- 172.21.15.116:0/1760555422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 0x557e82806450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:05:19.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.880+0000 7fbc05400700 1 --2- 172.21.15.116:0/1760555422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e828047d0 0x557e82804bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:05:19.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.880+0000 7fbc04a00700 1 --2- 172.21.15.116:0/1760555422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e82805130 0x557e828059f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00c2d0 tx=0x7fbbec099a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71c361445d41e83d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:19.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.880+0000 7fbbf2600700 1 -- 172.21.15.116:0/1760555422 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec09dc40 con 0x557e82805130 2024-03-30T01:05:19.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.880+0000 7fbbf2600700 1 -- 172.21.15.116:0/1760555422 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbec0a2290 con 0x557e82805130 2024-03-30T01:05:19.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.880+0000 7fbbf2600700 1 -- 172.21.15.116:0/1760555422 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec0a1e00 con 0x557e82805130 2024-03-30T01:05:19.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.881+0000 7fbc1b851b80 1 -- 172.21.15.116:0/1760555422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e82805130 msgr2=0x557e828059f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:19.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.881+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/1760555422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e82805130 0x557e828059f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00c2d0 tx=0x7fbbec099a80 comp rx=0 tx=0).stop 2024-03-30T01:05:19.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.881+0000 7fbc1b851b80 1 -- 172.21.15.116:0/1760555422 shutdown_connections 2024-03-30T01:05:19.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.881+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/1760555422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 0x557e82806450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.881+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/1760555422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e828047d0 0x557e82804bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.881+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/1760555422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e82805130 0x557e828059f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.881+0000 7fbc1b851b80 1 -- 172.21.15.116:0/1760555422 >> 172.21.15.116:0/1760555422 conn(0x557e827636a0 msgr2=0x557e827ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:19.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.881+0000 7fbc1b851b80 1 -- 172.21.15.116:0/1760555422 shutdown_connections 2024-03-30T01:05:19.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.881+0000 7fbc1b851b80 1 -- 172.21.15.116:0/1760555422 wait complete. 2024-03-30T01:05:19.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.882+0000 7fbc1b851b80 1 -- start start 2024-03-30T01:05:19.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.882+0000 7fbc1b851b80 1 Processor -- start 2024-03-30T01:05:19.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.882+0000 7fbc1b851b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e828047d0 0x557e8280e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.882+0000 7fbc1b851b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e82805130 0x557e8280ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.882+0000 7fbc1b851b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 0x557e82815c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.882+0000 7fbc1b851b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e826f1d80 con 0x557e82805130 2024-03-30T01:05:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.882+0000 7fbc1b851b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e826f1a20 con 0x557e82805fd0 2024-03-30T01:05:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.882+0000 7fbc1b851b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e825d96a0 con 0x557e828047d0 2024-03-30T01:05:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.882+0000 7fbc1b851b80 10 client.0 _ll_register_callbacks cb 0x557e826fb470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:05:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbc04a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e82805130 0x557e8280ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbc05400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e828047d0 0x557e8280e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbc04a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e82805130 0x557e8280ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47786/0 (socket says 172.21.15.116:47786) 2024-03-30T01:05:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbc04a00700 1 -- 172.21.15.116:0/749237204 learned_addr learned my addr 172.21.15.116:0/749237204 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbc05e00700 1 --2- 172.21.15.116:0/749237204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 0x557e82815c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbc04a00700 1 -- 172.21.15.116:0/749237204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e828047d0 msgr2=0x557e8280e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbc04a00700 1 --2- 172.21.15.116:0/749237204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e828047d0 0x557e8280e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbc04a00700 1 -- 172.21.15.116:0/749237204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 msgr2=0x557e82815c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbc04a00700 1 --2- 172.21.15.116:0/749237204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 0x557e82815c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbc04a00700 1 -- 172.21.15.116:0/749237204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf4009960 con 0x557e82805130 2024-03-30T01:05:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbc05400700 1 --2- 172.21.15.116:0/749237204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e828047d0 0x557e8280e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:05:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbc05e00700 1 --2- 172.21.15.116:0/749237204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 0x557e82815c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:05:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbc04a00700 1 --2- 172.21.15.116:0/749237204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e82805130 0x557e8280ec00 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0a1d60 tx=0x7fbbec005ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e60d4a53cce3f4bf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbbe6a00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:05:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbbe6a00700 1 -- 172.21.15.116:0/749237204 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec09dc40 con 0x557e82805130 2024-03-30T01:05:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbbe6a00700 1 -- 172.21.15.116:0/749237204 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbbec0981b0 con 0x557e82805130 2024-03-30T01:05:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.883+0000 7fbc1b851b80 1 -- 172.21.15.116:0/749237204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557e82716380 con 0x557e82805130 2024-03-30T01:05:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.884+0000 7fbbe6a00700 1 -- 172.21.15.116:0/749237204 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbec099470 con 0x557e82805130 2024-03-30T01:05:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.884+0000 7fbc1b851b80 10 client.16284 fetch_fsmap learned FSMap version 30 2024-03-30T01:05:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.884+0000 7fbc1b851b80 1 -- 172.21.15.116:0/749237204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fbc001a15a0 con 0x557e82805130 2024-03-30T01:05:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.884+0000 7fbbe6a00700 1 -- 172.21.15.116:0/749237204 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557e82716380 con 0x557e82805130 2024-03-30T01:05:19.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.884+0000 7fbbe6a00700 1 -- 172.21.15.116:0/749237204 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fbbec098d70 con 0x557e82805130 2024-03-30T01:05:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.884+0000 7fbc1b851b80 10 client.16284 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T01:05:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.884+0000 7fbc1b851b80 10 client.16284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:05:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.884+0000 7fbc1b851b80 20 client.16284 populate_metadata read hostname 'smithi116' 2024-03-30T01:05:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.884+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/749237204 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557e8281a110 0x557e8281c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.884+0000 7fbc1b851b80 4 client.16284 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:05:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.884+0000 7fbc1b851b80 1 -- 172.21.15.116:0/749237204 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557e8274e700 con 0x557e8281a110 2024-03-30T01:05:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.885+0000 7fbc05400700 1 --2- 172.21.15.116:0/749237204 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557e8281a110 0x557e8281c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.885+0000 7fbc05400700 1 --2- 172.21.15.116:0/749237204 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557e8281a110 0x557e8281c530 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3027df06d2a7c1b8 server_cookie=777472aa9e58278f in_seq=0 out_seq=0 2024-03-30T01:05:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.885+0000 7fbbe6a00700 10 client.16284 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:05:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.886+0000 7fbbe6a00700 1 -- 172.21.15.116:0/749237204 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557e8274e700 con 0x557e8281a110 2024-03-30T01:05:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.886+0000 7fbbe6a00700 10 client.16284 handle_command_reply: tid=0 2024-03-30T01:05:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.886+0000 7fbc1b851b80 1 client.16284 shutdown 2024-03-30T01:05:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.886+0000 7fbc1b851b80 1 -- 172.21.15.116:0/749237204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e82805130 msgr2=0x557e8280ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.886+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/749237204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e82805130 0x557e8280ec00 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0a1d60 tx=0x7fbbec005ba0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.886+0000 7fbc1b851b80 1 -- 172.21.15.116:0/749237204 shutdown_connections 2024-03-30T01:05:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.886+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/749237204 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557e8281a110 0x557e8281c530 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.886+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/749237204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e828047d0 0x557e8280e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.886+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/749237204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 0x557e82815c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.886+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/749237204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e82805130 0x557e8280ec00 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.886+0000 7fbbe6a00700 0 client.16284 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:05:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.886+0000 7fbc1b851b80 1 -- 172.21.15.116:0/749237204 >> 172.21.15.116:0/749237204 conn(0x557e827636a0 msgr2=0x557e82803170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.886+0000 7fbc1b851b80 1 -- 172.21.15.116:0/749237204 shutdown_connections 2024-03-30T01:05:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.886+0000 7fbc1b851b80 1 -- 172.21.15.116:0/749237204 wait complete. 2024-03-30T01:05:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.887+0000 7fbc1b851b80 20 client.16284 trim_cache size 0 max 16384 2024-03-30T01:05:19.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.904+0000 7fbc1b851b80 1 Processor -- start 2024-03-30T01:05:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.904+0000 7fbc1b851b80 1 -- start start 2024-03-30T01:05:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.904+0000 7fbc1b851b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e828047d0 0x557e8280c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.904+0000 7fbc1b851b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e82805130 0x557e82815c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.904+0000 7fbc1b851b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 0x557e8280e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.904+0000 7fbc1b851b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e828008a0 con 0x557e82805130 2024-03-30T01:05:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.904+0000 7fbc1b851b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e827fb780 con 0x557e82805fd0 2024-03-30T01:05:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.904+0000 7fbc1b851b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e8280f340 con 0x557e828047d0 2024-03-30T01:05:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.904+0000 7fbc04a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e82805130 0x557e82815c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.905+0000 7fbc04a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e82805130 0x557e82815c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47788/0 (socket says 172.21.15.116:47788) 2024-03-30T01:05:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.905+0000 7fbc04a00700 1 -- 172.21.15.116:0/2948221495 learned_addr learned my addr 172.21.15.116:0/2948221495 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.905+0000 7fbc05e00700 1 --2- 172.21.15.116:0/2948221495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 0x557e8280e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.905+0000 7fbc05400700 1 --2- 172.21.15.116:0/2948221495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e828047d0 0x557e8280c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.905+0000 7fbc04a00700 1 -- 172.21.15.116:0/2948221495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e828047d0 msgr2=0x557e8280c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.905+0000 7fbc04a00700 1 --2- 172.21.15.116:0/2948221495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e828047d0 0x557e8280c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.905+0000 7fbc04a00700 1 -- 172.21.15.116:0/2948221495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 msgr2=0x557e8280e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:19.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.905+0000 7fbc04a00700 1 --2- 172.21.15.116:0/2948221495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 0x557e8280e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.905+0000 7fbc04a00700 1 -- 172.21.15.116:0/2948221495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e82583b60 con 0x557e82805130 2024-03-30T01:05:19.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.905+0000 7fbc04a00700 1 --2- 172.21.15.116:0/2948221495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e82805130 0x557e82815c00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbbec099b10 tx=0x7fbbec096220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8271999bd6b0532 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:19.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.905+0000 7fbbf3000700 1 -- 172.21.15.116:0/2948221495 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec09a510 con 0x557e82805130 2024-03-30T01:05:19.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.905+0000 7fbbf3000700 1 -- 172.21.15.116:0/2948221495 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbec09dc40 con 0x557e82805130 2024-03-30T01:05:19.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.905+0000 7fbbf3000700 1 -- 172.21.15.116:0/2948221495 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec099560 con 0x557e82805130 2024-03-30T01:05:19.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.906+0000 7fbc1b851b80 1 -- 172.21.15.116:0/2948221495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e82805130 msgr2=0x557e82815c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:19.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.906+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/2948221495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e82805130 0x557e82815c00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbbec099b10 tx=0x7fbbec096220 comp rx=0 tx=0).stop 2024-03-30T01:05:19.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.906+0000 7fbc1b851b80 1 -- 172.21.15.116:0/2948221495 shutdown_connections 2024-03-30T01:05:19.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.906+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/2948221495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e828047d0 0x557e8280c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.906+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/2948221495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805fd0 0x557e8280e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.906+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/2948221495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e82805130 0x557e82815c00 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.906+0000 7fbc1b851b80 1 -- 172.21.15.116:0/2948221495 >> 172.21.15.116:0/2948221495 conn(0x557e827636a0 msgr2=0x557e8280dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:19.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.906+0000 7fbc1b851b80 1 -- 172.21.15.116:0/2948221495 shutdown_connections 2024-03-30T01:05:19.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.906+0000 7fbc1b851b80 1 -- 172.21.15.116:0/2948221495 wait complete. 2024-03-30T01:05:19.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.907+0000 7fbc1b851b80 1 -- start start 2024-03-30T01:05:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.907+0000 7fbc1b851b80 1 Processor -- start 2024-03-30T01:05:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.907+0000 7fbc1b851b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e828047d0 0x557e82813a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.907+0000 7fbc1b851b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805130 0x557e82813f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.907+0000 7fbc1b851b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e82805fd0 0x557e82814f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.907+0000 7fbc1b851b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e828154b0 con 0x557e828047d0 2024-03-30T01:05:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.907+0000 7fbc1b851b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e82815620 con 0x557e82805130 2024-03-30T01:05:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.907+0000 7fbc1b851b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e828157e0 con 0x557e82805fd0 2024-03-30T01:05:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.907+0000 7fbc1b851b80 10 client.0 _ll_register_callbacks cb 0x557e826fb470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:05:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.907+0000 7fbc05400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e828047d0 0x557e82813a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.907+0000 7fbc05400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e828047d0 0x557e82813a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47790/0 (socket says 172.21.15.116:47790) 2024-03-30T01:05:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.907+0000 7fbc05400700 1 -- 172.21.15.116:0/2931275628 learned_addr learned my addr 172.21.15.116:0/2931275628 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:19.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.907+0000 7fbc05e00700 1 --2- 172.21.15.116:0/2931275628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e82805fd0 0x557e82814f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbc04a00700 1 --2- 172.21.15.116:0/2931275628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805130 0x557e82813f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbc05400700 1 -- 172.21.15.116:0/2931275628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e82805fd0 msgr2=0x557e82814f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:19.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbc05400700 1 --2- 172.21.15.116:0/2931275628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e82805fd0 0x557e82814f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbc05400700 1 -- 172.21.15.116:0/2931275628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805130 msgr2=0x557e82813f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:19.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbc05400700 1 --2- 172.21.15.116:0/2931275628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805130 0x557e82813f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbc05400700 1 -- 172.21.15.116:0/2931275628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc001a15a0 con 0x557e828047d0 2024-03-30T01:05:19.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbc04a00700 1 --2- 172.21.15.116:0/2931275628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805130 0x557e82813f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:05:19.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbc05400700 1 --2- 172.21.15.116:0/2931275628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e828047d0 0x557e82813a20 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x557e8281cda0 tx=0x7fbbf40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b01e697c797d43 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbbe6a00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:05:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbbe6a00700 1 -- 172.21.15.116:0/2931275628 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4017040 con 0x557e828047d0 2024-03-30T01:05:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbbe6a00700 1 -- 172.21.15.116:0/2931275628 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbbf40212b0 con 0x557e828047d0 2024-03-30T01:05:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbc1b851b80 1 -- 172.21.15.116:0/2931275628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbbec0a2290 con 0x557e828047d0 2024-03-30T01:05:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbbe6a00700 1 -- 172.21.15.116:0/2931275628 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbf4012ea0 con 0x557e828047d0 2024-03-30T01:05:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbc1b851b80 10 client.16290 fetch_fsmap learned FSMap version 30 2024-03-30T01:05:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.908+0000 7fbc1b851b80 1 -- 172.21.15.116:0/2931275628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557e8271bea0 con 0x557e828047d0 2024-03-30T01:05:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.909+0000 7fbbe6a00700 1 -- 172.21.15.116:0/2931275628 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbec0a2290 con 0x557e828047d0 2024-03-30T01:05:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.909+0000 7fbbe6a00700 1 -- 172.21.15.116:0/2931275628 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fbbf40039c0 con 0x557e828047d0 2024-03-30T01:05:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.909+0000 7fbc1b851b80 10 client.16290 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T01:05:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.909+0000 7fbc1b851b80 10 client.16290 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:05:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.909+0000 7fbc1b851b80 20 client.16290 populate_metadata read hostname 'smithi116' 2024-03-30T01:05:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.909+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/2931275628 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557e8286f090 0x557e828714b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.909+0000 7fbc1b851b80 4 client.16290 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T01:05:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.909+0000 7fbc1b851b80 1 -- 172.21.15.116:0/2931275628 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557e825a4cf0 con 0x557e8286f090 2024-03-30T01:05:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.909+0000 7fbc04a00700 1 --2- 172.21.15.116:0/2931275628 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557e8286f090 0x557e828714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbc04a00700 1 --2- 172.21.15.116:0/2931275628 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557e8286f090 0x557e828714b0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9fe333617b2dfb4 server_cookie=72af306ecc6e8af8 in_seq=0 out_seq=0 2024-03-30T01:05:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbbe6a00700 10 client.16290 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:05:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbbe6a00700 1 -- 172.21.15.116:0/2931275628 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x557e825a4cf0 con 0x557e8286f090 2024-03-30T01:05:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbbe6a00700 10 client.16290 handle_command_reply: tid=0 2024-03-30T01:05:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbc1b851b80 1 client.16290 shutdown 2024-03-30T01:05:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbc1b851b80 1 -- 172.21.15.116:0/2931275628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e828047d0 msgr2=0x557e82813a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/2931275628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e828047d0 0x557e82813a20 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x557e8281cda0 tx=0x7fbbf40095c0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbc1b851b80 1 -- 172.21.15.116:0/2931275628 shutdown_connections 2024-03-30T01:05:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/2931275628 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557e8286f090 0x557e828714b0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/2931275628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e82805fd0 0x557e82814f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/2931275628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e828047d0 0x557e82813a20 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbbe6a00700 0 client.16290 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:05:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbc1b851b80 1 --2- 172.21.15.116:0/2931275628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e82805130 0x557e82813f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbc1b851b80 1 -- 172.21.15.116:0/2931275628 >> 172.21.15.116:0/2931275628 conn(0x557e827636a0 msgr2=0x557e8280c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbc1b851b80 1 -- 172.21.15.116:0/2931275628 shutdown_connections 2024-03-30T01:05:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.910+0000 7fbc1b851b80 1 -- 172.21.15.116:0/2931275628 wait complete. 2024-03-30T01:05:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.911+0000 7fbc1b851b80 20 client.16290 trim_cache size 0 max 16384 2024-03-30T01:05:19.926 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:05:19.926 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-30T01:05:19.926 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-30T01:05:19.926 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:05:19.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.911+0000 7fbc07e00700 1 -- 172.21.15.116:0/2919170741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbbe80a11a0 msgr2=0x7fbbe80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:19.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.911+0000 7fbc07e00700 1 --2- 172.21.15.116:0/2919170741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbbe80a11a0 0x7fbbe80a3660 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009fd0 tx=0x7fbbf800b040 comp rx=0 tx=0).stop 2024-03-30T01:05:19.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.911+0000 7fbc07e00700 1 -- 172.21.15.116:0/2919170741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc00101f70 msgr2=0x7fbc000731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:19.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.911+0000 7fbc07e00700 1 --2- 172.21.15.116:0/2919170741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc00101f70 0x7fbc000731a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00e9a0 tx=0x7fbbec00ecb0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.911+0000 7fbc07e00700 1 -- 172.21.15.116:0/2919170741 shutdown_connections 2024-03-30T01:05:19.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.911+0000 7fbc07e00700 1 --2- 172.21.15.116:0/2919170741 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fbbe80a11a0 0x7fbbe80a3660 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.911+0000 7fbc07e00700 1 --2- 172.21.15.116:0/2919170741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc00100d80 0x7fbc00074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.911+0000 7fbc07e00700 1 --2- 172.21.15.116:0/2919170741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc00102910 0x7fbc000736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.911+0000 7fbc07e00700 1 --2- 172.21.15.116:0/2919170741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc00101f70 0x7fbc000731a0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:19.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.911+0000 7fbc07e00700 1 -- 172.21.15.116:0/2919170741 >> 172.21.15.116:0/2919170741 conn(0x7fbc000fc2f0 msgr2=0x7fbc00105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:19.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.911+0000 7fbc07e00700 1 -- 172.21.15.116:0/2919170741 shutdown_connections 2024-03-30T01:05:19.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:19.911+0000 7fbc07e00700 1 -- 172.21.15.116:0/2919170741 wait complete. 2024-03-30T01:05:19.935 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T01:05:19.935 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T01:05:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.123+0000 7fe6c6200700 1 -- 172.21.15.116:0/1821473845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0100c80 msgr2=0x7fe6c01010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.123+0000 7fe6c6200700 1 --2- 172.21.15.116:0/1821473845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0100c80 0x7fe6c01010a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe6b00099c0 tx=0x7fe6b0009cd0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.123+0000 7fe6c6200700 1 -- 172.21.15.116:0/1821473845 shutdown_connections 2024-03-30T01:05:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.123+0000 7fe6c6200700 1 --2- 172.21.15.116:0/1821473845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6c0102810 0x7fe6c01090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.123+0000 7fe6c6200700 1 --2- 172.21.15.116:0/1821473845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0100c80 0x7fe6c01010a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.123+0000 7fe6c6200700 1 --2- 172.21.15.116:0/1821473845 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6c0101e70 0x7fe6c01022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.123+0000 7fe6c6200700 1 -- 172.21.15.116:0/1821473845 >> 172.21.15.116:0/1821473845 conn(0x7fe6c00fc1f0 msgr2=0x7fe6c00fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.123+0000 7fe6c6200700 1 -- 172.21.15.116:0/1821473845 shutdown_connections 2024-03-30T01:05:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.123+0000 7fe6c6200700 1 -- 172.21.15.116:0/1821473845 wait complete. 2024-03-30T01:05:20.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.124+0000 7fe6c6200700 1 Processor -- start 2024-03-30T01:05:20.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.124+0000 7fe6c6200700 1 -- start start 2024-03-30T01:05:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.124+0000 7fe6c6200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6c0100c80 0x7fe6c0196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.124+0000 7fe6c6200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6c0101e70 0x7fe6c0197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.124+0000 7fe6c6200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0102810 0x7fe6c019e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.124+0000 7fe6c6200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c019e760 con 0x7fe6c0102810 2024-03-30T01:05:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.124+0000 7fe6c6200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c019e8a0 con 0x7fe6c0101e70 2024-03-30T01:05:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.124+0000 7fe6c6200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c019ea30 con 0x7fe6c0100c80 2024-03-30T01:05:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.125+0000 7fe6bfe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0102810 0x7fe6c019e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.125+0000 7fe6bf400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6c0100c80 0x7fe6c0196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.125+0000 7fe6bfe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0102810 0x7fe6c019e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47812/0 (socket says 172.21.15.116:47812) 2024-03-30T01:05:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.125+0000 7fe6bfe00700 1 -- 172.21.15.116:0/4219800625 learned_addr learned my addr 172.21.15.116:0/4219800625 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.125+0000 7fe6bea00700 1 --2- 172.21.15.116:0/4219800625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6c0101e70 0x7fe6c0197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.125+0000 7fe6bfe00700 1 -- 172.21.15.116:0/4219800625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6c0100c80 msgr2=0x7fe6c0196c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.125+0000 7fe6bfe00700 1 --2- 172.21.15.116:0/4219800625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6c0100c80 0x7fe6c0196c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.125+0000 7fe6bfe00700 1 -- 172.21.15.116:0/4219800625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6c0101e70 msgr2=0x7fe6c0197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.125+0000 7fe6bfe00700 1 --2- 172.21.15.116:0/4219800625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6c0101e70 0x7fe6c0197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.125+0000 7fe6bfe00700 1 -- 172.21.15.116:0/4219800625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b00095f0 con 0x7fe6c0102810 2024-03-30T01:05:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.125+0000 7fe6bf400700 1 --2- 172.21.15.116:0/4219800625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6c0100c80 0x7fe6c0196c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:05:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.125+0000 7fe6bfe00700 1 --2- 172.21.15.116:0/4219800625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0102810 0x7fe6c019e190 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800ef30 tx=0x7fe6b800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.126+0000 7fe6b7e00700 1 -- 172.21.15.116:0/4219800625 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8018070 con 0x7fe6c0102810 2024-03-30T01:05:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.126+0000 7fe6b7e00700 1 -- 172.21.15.116:0/4219800625 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6b800f040 con 0x7fe6c0102810 2024-03-30T01:05:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.126+0000 7fe6c6200700 1 -- 172.21.15.116:0/4219800625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c019ecb0 con 0x7fe6c0102810 2024-03-30T01:05:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.126+0000 7fe6b7e00700 1 -- 172.21.15.116:0/4219800625 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800cd00 con 0x7fe6c0102810 2024-03-30T01:05:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.126+0000 7fe6c6200700 1 -- 172.21.15.116:0/4219800625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c019f1c0 con 0x7fe6c0102810 2024-03-30T01:05:20.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.127+0000 7fe6c6200700 1 -- 172.21.15.116:0/4219800625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c0108150 con 0x7fe6c0102810 2024-03-30T01:05:20.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.129+0000 7fe6b7e00700 1 -- 172.21.15.116:0/4219800625 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fe6b8003a00 con 0x7fe6c0102810 2024-03-30T01:05:20.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.130+0000 7fe6b7e00700 1 --2- 172.21.15.116:0/4219800625 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe6a40a1120 0x7fe6a40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.130+0000 7fe6b7e00700 1 -- 172.21.15.116:0/4219800625 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe6b80cd0f0 con 0x7fe6c0102810 2024-03-30T01:05:20.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.130+0000 7fe6bf400700 1 --2- 172.21.15.116:0/4219800625 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe6a40a1120 0x7fe6a40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.131+0000 7fe6bf400700 1 --2- 172.21.15.116:0/4219800625 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe6a40a1120 0x7fe6a40a35e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0003730 tx=0x7fe6b000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:20.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.133+0000 7fe6b7e00700 1 -- 172.21.15.116:0/4219800625 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe6b80965a0 con 0x7fe6c0102810 2024-03-30T01:05:20.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.384+0000 7fe6c6200700 1 -- 172.21.15.116:0/4219800625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6c0067a10 con 0x7fe6c0102810 2024-03-30T01:05:20.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.385+0000 7fe6b7e00700 1 -- 172.21.15.116:0/4219800625 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10893 (secure 0 0 0) 0x7fe6b8025380 con 0x7fe6c0102810 2024-03-30T01:05:20.391 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T01:05:20.391 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T01:05:11.499260+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":140,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T01:05:20.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.387+0000 7fe6c6200700 1 -- 172.21.15.116:0/4219800625 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe6a40a1120 msgr2=0x7fe6a40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.387+0000 7fe6c6200700 1 --2- 172.21.15.116:0/4219800625 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe6a40a1120 0x7fe6a40a35e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0003730 tx=0x7fe6b000b540 comp rx=0 tx=0).stop 2024-03-30T01:05:20.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.387+0000 7fe6c6200700 1 -- 172.21.15.116:0/4219800625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0102810 msgr2=0x7fe6c019e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.387+0000 7fe6c6200700 1 --2- 172.21.15.116:0/4219800625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0102810 0x7fe6c019e190 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800ef30 tx=0x7fe6b800c3a0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.387+0000 7fe6c6200700 1 -- 172.21.15.116:0/4219800625 shutdown_connections 2024-03-30T01:05:20.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.387+0000 7fe6c6200700 1 --2- 172.21.15.116:0/4219800625 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fe6a40a1120 0x7fe6a40a35e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.387+0000 7fe6c6200700 1 --2- 172.21.15.116:0/4219800625 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6c0100c80 0x7fe6c0196c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.387+0000 7fe6c6200700 1 --2- 172.21.15.116:0/4219800625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6c0102810 0x7fe6c019e190 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.387+0000 7fe6c6200700 1 --2- 172.21.15.116:0/4219800625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6c0101e70 0x7fe6c0197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.387+0000 7fe6c6200700 1 -- 172.21.15.116:0/4219800625 >> 172.21.15.116:0/4219800625 conn(0x7fe6c00fc1f0 msgr2=0x7fe6c0105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:20.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.387+0000 7fe6c6200700 1 -- 172.21.15.116:0/4219800625 shutdown_connections 2024-03-30T01:05:20.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.388+0000 7fe6c6200700 1 -- 172.21.15.116:0/4219800625 wait complete. 2024-03-30T01:05:20.394 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-30T01:05:20.412 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T01:05:20.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.601+0000 7f9d17e00700 1 -- 172.21.15.116:0/27368338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d10101f70 msgr2=0x7f9d101023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.601+0000 7f9d17e00700 1 --2- 172.21.15.116:0/27368338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d10101f70 0x7f9d101023d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc0099c0 tx=0x7f9cfc009cd0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.601+0000 7f9d17e00700 1 -- 172.21.15.116:0/27368338 shutdown_connections 2024-03-30T01:05:20.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.601+0000 7f9d17e00700 1 --2- 172.21.15.116:0/27368338 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d10100d80 0x7f9d101011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.601+0000 7f9d17e00700 1 --2- 172.21.15.116:0/27368338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d10102910 0x7f9d101091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.601+0000 7f9d17e00700 1 --2- 172.21.15.116:0/27368338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d10101f70 0x7f9d101023d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.601+0000 7f9d17e00700 1 -- 172.21.15.116:0/27368338 >> 172.21.15.116:0/27368338 conn(0x7f9d100fc2f0 msgr2=0x7f9d100fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:20.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.601+0000 7f9d17e00700 1 -- 172.21.15.116:0/27368338 shutdown_connections 2024-03-30T01:05:20.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.602+0000 7f9d17e00700 1 -- 172.21.15.116:0/27368338 wait complete. 2024-03-30T01:05:20.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.603+0000 7f9d17e00700 1 Processor -- start 2024-03-30T01:05:20.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.603+0000 7f9d17e00700 1 -- start start 2024-03-30T01:05:20.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.603+0000 7f9d17e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d10100d80 0x7f9d10196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.603+0000 7f9d17e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d10101f70 0x7f9d101972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.603+0000 7f9d17e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d10102910 0x7f9d1019e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.603+0000 7f9d17e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1019e860 con 0x7f9d10100d80 2024-03-30T01:05:20.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.603+0000 7f9d17e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1019e9d0 con 0x7f9d10102910 2024-03-30T01:05:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.603+0000 7f9d17e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1019eb90 con 0x7f9d10101f70 2024-03-30T01:05:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.603+0000 7f9d15400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d10100d80 0x7f9d10196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.603+0000 7f9d15400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d10100d80 0x7f9d10196d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47816/0 (socket says 172.21.15.116:47816) 2024-03-30T01:05:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.603+0000 7f9d15400700 1 -- 172.21.15.116:0/2783953456 learned_addr learned my addr 172.21.15.116:0/2783953456 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.603+0000 7f9d14a00700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d10101f70 0x7f9d101972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.603+0000 7f9d15e00700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d10102910 0x7f9d1019e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.604+0000 7f9d15400700 1 -- 172.21.15.116:0/2783953456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d10101f70 msgr2=0x7f9d101972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.604+0000 7f9d15400700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d10101f70 0x7f9d101972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.604+0000 7f9d15400700 1 -- 172.21.15.116:0/2783953456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d10102910 msgr2=0x7f9d1019e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.604+0000 7f9d15400700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d10102910 0x7f9d1019e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.604+0000 7f9d15400700 1 -- 172.21.15.116:0/2783953456 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc0095f0 con 0x7f9d10100d80 2024-03-30T01:05:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.604+0000 7f9d14a00700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d10101f70 0x7f9d101972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:05:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.604+0000 7f9d15400700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d10100d80 0x7f9d10196d70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400e970 tx=0x7f9d0400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:20.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.604+0000 7f9d0e000700 1 -- 172.21.15.116:0/2783953456 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04009790 con 0x7f9d10100d80 2024-03-30T01:05:20.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.604+0000 7f9d0e000700 1 -- 172.21.15.116:0/2783953456 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d040044e0 con 0x7f9d10100d80 2024-03-30T01:05:20.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.604+0000 7f9d17e00700 1 -- 172.21.15.116:0/2783953456 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1019ee70 con 0x7f9d10100d80 2024-03-30T01:05:20.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.604+0000 7f9d0e000700 1 -- 172.21.15.116:0/2783953456 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04003e30 con 0x7f9d10100d80 2024-03-30T01:05:20.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.604+0000 7f9d17e00700 1 -- 172.21.15.116:0/2783953456 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1019f2a0 con 0x7f9d10100d80 2024-03-30T01:05:20.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.607+0000 7f9d0e000700 1 -- 172.21.15.116:0/2783953456 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f9d04004000 con 0x7f9d10100d80 2024-03-30T01:05:20.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.609+0000 7f9d0e000700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9cf80a11a0 0x7f9cf80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.609+0000 7f9d2ba15b80 1 Processor -- start 2024-03-30T01:05:20.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.609+0000 7f9d0e000700 1 -- 172.21.15.116:0/2783953456 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9d04014070 con 0x7f9d10100d80 2024-03-30T01:05:20.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.609+0000 7f9d15e00700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9cf80a11a0 0x7f9cf80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.609+0000 7f9d2ba15b80 1 -- start start 2024-03-30T01:05:20.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.609+0000 7f9d2ba15b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce78bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.609+0000 7f9d2ba15b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78c130 0x55dcce78c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.609+0000 7f9d2ba15b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78cfd0 0x55dcce78d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.609+0000 7f9d2ba15b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcce6e3990 con 0x55dcce78b7d0 2024-03-30T01:05:20.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.610+0000 7f9d2ba15b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcce6e1350 con 0x55dcce78c130 2024-03-30T01:05:20.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.610+0000 7f9d2ba15b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcce6dcc40 con 0x55dcce78cfd0 2024-03-30T01:05:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.610+0000 7f9d15400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce78bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.610+0000 7f9d15400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce78bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47830/0 (socket says 172.21.15.116:47830) 2024-03-30T01:05:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.610+0000 7f9d15400700 1 -- 172.21.15.116:0/1217931763 learned_addr learned my addr 172.21.15.116:0/1217931763 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.610+0000 7f9d15e00700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9cf80a11a0 0x7f9cf80a3660 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9d08009c90 tx=0x7f9d08009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.610+0000 7f9d14a00700 1 --2- 172.21.15.116:0/1217931763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78c130 0x55dcce78c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.610+0000 7f9d15e00700 1 --2- 172.21.15.116:0/1217931763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78cfd0 0x55dcce78d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.610+0000 7f9d15400700 1 -- 172.21.15.116:0/1217931763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78cfd0 msgr2=0x55dcce78d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.610+0000 7f9d15400700 1 --2- 172.21.15.116:0/1217931763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78cfd0 0x55dcce78d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.610+0000 7f9d15400700 1 -- 172.21.15.116:0/1217931763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78c130 msgr2=0x55dcce78c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.610+0000 7f9d15400700 1 --2- 172.21.15.116:0/1217931763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78c130 0x55dcce78c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.610+0000 7f9d15400700 1 -- 172.21.15.116:0/1217931763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dcce55e3b0 con 0x55dcce78b7d0 2024-03-30T01:05:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.610+0000 7f9d15400700 1 --2- 172.21.15.116:0/1217931763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce78bbf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9d04096450 tx=0x7f9d04099a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d059e9343ded22 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.611+0000 7f9d02600700 1 -- 172.21.15.116:0/1217931763 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d040b1070 con 0x55dcce78b7d0 2024-03-30T01:05:20.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.611+0000 7f9d02600700 1 -- 172.21.15.116:0/1217931763 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d04099f50 con 0x55dcce78b7d0 2024-03-30T01:05:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.611+0000 7f9d02600700 1 -- 172.21.15.116:0/1217931763 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d040982f0 con 0x55dcce78b7d0 2024-03-30T01:05:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.611+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/1217931763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 msgr2=0x55dcce78bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.611+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/1217931763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce78bbf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9d04096450 tx=0x7f9d04099a90 comp rx=0 tx=0).stop 2024-03-30T01:05:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.611+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/1217931763 shutdown_connections 2024-03-30T01:05:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.611+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/1217931763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78cfd0 0x55dcce78d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.611+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/1217931763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce78bbf0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.611+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/1217931763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78c130 0x55dcce78c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.611+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/1217931763 >> 172.21.15.116:0/1217931763 conn(0x55dcce6ea6a0 msgr2=0x55dcce7869a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.611+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/1217931763 shutdown_connections 2024-03-30T01:05:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.611+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/1217931763 wait complete. 2024-03-30T01:05:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.612+0000 7f9d2ba15b80 1 -- start start 2024-03-30T01:05:20.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.612+0000 7f9d2ba15b80 1 Processor -- start 2024-03-30T01:05:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.612+0000 7f9d2ba15b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce799ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d2ba15b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78c130 0x55dcce79a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d2ba15b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78cfd0 0x55dcce7a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d2ba15b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcce678d80 con 0x55dcce78b7d0 2024-03-30T01:05:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d2ba15b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcce678a20 con 0x55dcce78c130 2024-03-30T01:05:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d2ba15b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcce5606a0 con 0x55dcce78cfd0 2024-03-30T01:05:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d2ba15b80 10 client.0 _ll_register_callbacks cb 0x55dcce682470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:05:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d15400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce799ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d15400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce799ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47838/0 (socket says 172.21.15.116:47838) 2024-03-30T01:05:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d15400700 1 -- 172.21.15.116:0/843385438 learned_addr learned my addr 172.21.15.116:0/843385438 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d14a00700 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78c130 0x55dcce79a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d15e00700 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78cfd0 0x55dcce7a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d15400700 1 -- 172.21.15.116:0/843385438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78cfd0 msgr2=0x55dcce7a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d15400700 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78cfd0 0x55dcce7a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d15400700 1 -- 172.21.15.116:0/843385438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78c130 msgr2=0x55dcce79a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d15400700 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78c130 0x55dcce79a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d15400700 1 -- 172.21.15.116:0/843385438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1019f2a0 con 0x55dcce78b7d0 2024-03-30T01:05:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d15e00700 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78cfd0 0x55dcce7a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:05:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d14a00700 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78c130 0x55dcce79a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:05:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9d15400700 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce799ac0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400c5b0 tx=0x7f9d040ca570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0909419a5c9336 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.613+0000 7f9cf6a00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:05:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.614+0000 7f9cf6a00700 1 -- 172.21.15.116:0/843385438 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d040b8070 con 0x55dcce78b7d0 2024-03-30T01:05:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.614+0000 7f9cf6a00700 1 -- 172.21.15.116:0/843385438 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9d040b2070 con 0x55dcce78b7d0 2024-03-30T01:05:20.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.614+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/843385438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dcce69d380 con 0x55dcce78b7d0 2024-03-30T01:05:20.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.614+0000 7f9cf6a00700 1 -- 172.21.15.116:0/843385438 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d040b1030 con 0x55dcce78b7d0 2024-03-30T01:05:20.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.614+0000 7f9d2ba15b80 10 client.16308 fetch_fsmap learned FSMap version 30 2024-03-30T01:05:20.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.614+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/843385438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55dcce50ab60 con 0x55dcce78b7d0 2024-03-30T01:05:20.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.614+0000 7f9cf6a00700 1 -- 172.21.15.116:0/843385438 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dcce69d380 con 0x55dcce78b7d0 2024-03-30T01:05:20.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.614+0000 7f9cf6a00700 1 -- 172.21.15.116:0/843385438 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f9d04099440 con 0x55dcce78b7d0 2024-03-30T01:05:20.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.614+0000 7f9d2ba15b80 10 client.16308 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T01:05:20.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.614+0000 7f9d2ba15b80 10 client.16308 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T01:05:20.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.614+0000 7f9d2ba15b80 20 client.16308 populate_metadata read hostname 'smithi116' 2024-03-30T01:05:20.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.614+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55dcce7a5410 0x55dcce7a7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.614+0000 7f9d2ba15b80 4 client.16308 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:05:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.614+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/843385438 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dcce6d5700 con 0x55dcce7a5410 2024-03-30T01:05:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.615+0000 7f9d14a00700 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55dcce7a5410 0x55dcce7a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.615+0000 7f9d14a00700 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55dcce7a5410 0x55dcce7a7830 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=17d33a0479bf6ce4 server_cookie=904f92796ef6e331 in_seq=0 out_seq=0 2024-03-30T01:05:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.615+0000 7f9cf6a00700 10 client.16308 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T01:05:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.616+0000 7f9cf6a00700 1 -- 172.21.15.116:0/843385438 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55dcce6d5700 con 0x55dcce7a5410 2024-03-30T01:05:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.616+0000 7f9cf6a00700 10 client.16308 handle_command_reply: tid=0 2024-03-30T01:05:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.616+0000 7f9d2ba15b80 1 client.16308 shutdown 2024-03-30T01:05:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.616+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/843385438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 msgr2=0x55dcce799ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.616+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce799ac0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400c5b0 tx=0x7f9d040ca570 comp rx=0 tx=0).stop 2024-03-30T01:05:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.616+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/843385438 shutdown_connections 2024-03-30T01:05:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.616+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55dcce7a5410 0x55dcce7a7830 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.616+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78cfd0 0x55dcce7a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.616+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce799ac0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.616+0000 7f9cf6a00700 0 client.16308 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T01:05:20.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.616+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/843385438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78c130 0x55dcce79a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.616+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/843385438 >> 172.21.15.116:0/843385438 conn(0x55dcce6ea6a0 msgr2=0x55dcce78a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:20.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.616+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/843385438 shutdown_connections 2024-03-30T01:05:20.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.616+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/843385438 wait complete. 2024-03-30T01:05:20.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.617+0000 7f9d2ba15b80 20 client.16308 trim_cache size 0 max 16384 2024-03-30T01:05:20.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.631+0000 7f9d2ba15b80 1 Processor -- start 2024-03-30T01:05:20.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.631+0000 7f9d2ba15b80 1 -- start start 2024-03-30T01:05:20.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.631+0000 7f9d2ba15b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78b7d0 0x55dcce78d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.631+0000 7f9d2ba15b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78c130 0x55dcce7a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.631+0000 7f9d2ba15b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78cfd0 0x55dcce77f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.631+0000 7f9d2ba15b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcce787660 con 0x55dcce78c130 2024-03-30T01:05:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.631+0000 7f9d2ba15b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcce7878a0 con 0x55dcce78cfd0 2024-03-30T01:05:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.631+0000 7f9d2ba15b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcce78c7e0 con 0x55dcce78b7d0 2024-03-30T01:05:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.632+0000 7f9d14a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78c130 0x55dcce7a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.632+0000 7f9d15400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78b7d0 0x55dcce78d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.632+0000 7f9d15400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78b7d0 0x55dcce78d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:33730/0 (socket says 172.21.15.116:33730) 2024-03-30T01:05:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.632+0000 7f9d15400700 1 -- 172.21.15.116:0/1844653882 learned_addr learned my addr 172.21.15.116:0/1844653882 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.632+0000 7f9d15e00700 1 --2- 172.21.15.116:0/1844653882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78cfd0 0x55dcce77f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.632+0000 7f9d14a00700 1 -- 172.21.15.116:0/1844653882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78b7d0 msgr2=0x55dcce78d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.632+0000 7f9d14a00700 1 --2- 172.21.15.116:0/1844653882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78b7d0 0x55dcce78d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.632+0000 7f9d14a00700 1 -- 172.21.15.116:0/1844653882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78cfd0 msgr2=0x55dcce77f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.632+0000 7f9d14a00700 1 --2- 172.21.15.116:0/1844653882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78cfd0 0x55dcce77f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.632+0000 7f9d14a00700 1 -- 172.21.15.116:0/1844653882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dcce6a2ea0 con 0x55dcce78c130 2024-03-30T01:05:20.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.632+0000 7f9d14a00700 1 --2- 172.21.15.116:0/1844653882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78c130 0x55dcce7a0fb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009fd0 tx=0x7f9cfc009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c3113564330eb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:20.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.633+0000 7f9d03000700 1 -- 172.21.15.116:0/1844653882 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc029070 con 0x55dcce78c130 2024-03-30T01:05:20.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.633+0000 7f9d03000700 1 -- 172.21.15.116:0/1844653882 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cfc003c90 con 0x55dcce78c130 2024-03-30T01:05:20.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.633+0000 7f9d03000700 1 -- 172.21.15.116:0/1844653882 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc01d040 con 0x55dcce78c130 2024-03-30T01:05:20.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.634+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/1844653882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78c130 msgr2=0x55dcce7a0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.634+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/1844653882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78c130 0x55dcce7a0fb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009fd0 tx=0x7f9cfc009550 comp rx=0 tx=0).stop 2024-03-30T01:05:20.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.634+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/1844653882 shutdown_connections 2024-03-30T01:05:20.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.634+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/1844653882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78b7d0 0x55dcce78d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.634+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/1844653882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78cfd0 0x55dcce77f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.634+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/1844653882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78c130 0x55dcce7a0fb0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.634+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/1844653882 >> 172.21.15.116:0/1844653882 conn(0x55dcce6ea6a0 msgr2=0x55dcce7ac190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:20.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.634+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/1844653882 shutdown_connections 2024-03-30T01:05:20.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.634+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/1844653882 wait complete. 2024-03-30T01:05:20.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.634+0000 7f9d2ba15b80 1 -- start start 2024-03-30T01:05:20.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.634+0000 7f9d2ba15b80 1 Processor -- start 2024-03-30T01:05:20.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d2ba15b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce7a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d2ba15b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78c130 0x55dcce79f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d2ba15b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78cfd0 0x55dcce79f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d2ba15b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcce79fe80 con 0x55dcce78b7d0 2024-03-30T01:05:20.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d2ba15b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcce79fff0 con 0x55dcce78cfd0 2024-03-30T01:05:20.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d2ba15b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcce7a01b0 con 0x55dcce78c130 2024-03-30T01:05:20.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d2ba15b80 10 client.0 _ll_register_callbacks cb 0x55dcce682470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:05:20.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d15400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce7a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d15400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce7a0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47854/0 (socket says 172.21.15.116:47854) 2024-03-30T01:05:20.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d15400700 1 -- 172.21.15.116:0/99816995 learned_addr learned my addr 172.21.15.116:0/99816995 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:20.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d15e00700 1 --2- 172.21.15.116:0/99816995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78cfd0 0x55dcce79f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d14a00700 1 --2- 172.21.15.116:0/99816995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78c130 0x55dcce79f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d15400700 1 -- 172.21.15.116:0/99816995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78c130 msgr2=0x55dcce79f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d15400700 1 --2- 172.21.15.116:0/99816995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78c130 0x55dcce79f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d15400700 1 -- 172.21.15.116:0/99816995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78cfd0 msgr2=0x55dcce79f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d15400700 1 --2- 172.21.15.116:0/99816995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78cfd0 0x55dcce79f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d15400700 1 -- 172.21.15.116:0/99816995 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d10109b10 con 0x55dcce78b7d0 2024-03-30T01:05:20.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.635+0000 7f9d15e00700 1 --2- 172.21.15.116:0/99816995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78cfd0 0x55dcce79f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:05:20.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.636+0000 7f9d15400700 1 --2- 172.21.15.116:0/99816995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce7a0d60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9d040ca970 tx=0x7f9d04099dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c05f5788691ee1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:20.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.636+0000 7f9cf6a00700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:05:20.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.636+0000 7f9cf6a00700 1 -- 172.21.15.116:0/99816995 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d040b8040 con 0x55dcce78b7d0 2024-03-30T01:05:20.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.636+0000 7f9cf6a00700 1 -- 172.21.15.116:0/99816995 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9d040a2090 con 0x55dcce78b7d0 2024-03-30T01:05:20.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.636+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/99816995 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d04099f50 con 0x55dcce78b7d0 2024-03-30T01:05:20.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.636+0000 7f9cf6a00700 1 -- 172.21.15.116:0/99816995 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d040042b0 con 0x55dcce78b7d0 2024-03-30T01:05:20.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.636+0000 7f9d2ba15b80 10 client.16314 fetch_fsmap learned FSMap version 30 2024-03-30T01:05:20.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.636+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/99816995 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55dcce562cd0 con 0x55dcce78b7d0 2024-03-30T01:05:20.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.637+0000 7f9cf6a00700 1 -- 172.21.15.116:0/99816995 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d04099f50 con 0x55dcce78b7d0 2024-03-30T01:05:20.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.637+0000 7f9cf6a00700 1 -- 172.21.15.116:0/99816995 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f9d040832a0 con 0x55dcce78b7d0 2024-03-30T01:05:20.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.637+0000 7f9d2ba15b80 10 client.16314 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T01:05:20.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.637+0000 7f9d2ba15b80 10 client.16314 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-30T01:05:20.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.637+0000 7f9d2ba15b80 20 client.16314 populate_metadata read hostname 'smithi116' 2024-03-30T01:05:20.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.637+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/99816995 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55dcce7fa430 0x55dcce7fc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.637+0000 7f9d2ba15b80 4 client.16314 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:05:20.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.637+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/99816995 --> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dcce52bcf0 con 0x55dcce7fa430 2024-03-30T01:05:20.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.637+0000 7f9d14a00700 1 --2- 172.21.15.116:0/99816995 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55dcce7fa430 0x55dcce7fc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9cf6a00700 10 client.16314 ms_handle_connect on v2:172.21.15.181:6838/3114781386 2024-03-30T01:05:20.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9d14a00700 1 --2- 172.21.15.116:0/99816995 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55dcce7fa430 0x55dcce7fc820 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8a0a5c5c34441fbb server_cookie=74ef11a42323c598 in_seq=0 out_seq=0 2024-03-30T01:05:20.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9cf6a00700 1 -- 172.21.15.116:0/99816995 <== mds.2 v2:172.21.15.181:6838/3114781386 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+486 (crc 0 0 0) 0x55dcce52bcf0 con 0x55dcce7fa430 2024-03-30T01:05:20.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9cf6a00700 10 client.16314 handle_command_reply: tid=0 2024-03-30T01:05:20.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9d2ba15b80 1 client.16314 shutdown 2024-03-30T01:05:20.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/99816995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 msgr2=0x55dcce7a0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/99816995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce7a0d60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9d040ca970 tx=0x7f9d04099dd0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/99816995 shutdown_connections 2024-03-30T01:05:20.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/99816995 >> [v2:172.21.15.181:6838/3114781386,v1:172.21.15.181:6839/3114781386] conn(0x55dcce7fa430 0x55dcce7fc820 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/99816995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dcce78cfd0 0x55dcce79f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/99816995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcce78b7d0 0x55dcce7a0d60 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9cf6a00700 0 client.16314 ms_handle_reset on v2:172.21.15.181:6838/3114781386 2024-03-30T01:05:20.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9d2ba15b80 1 --2- 172.21.15.116:0/99816995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dcce78c130 0x55dcce79f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/99816995 >> 172.21.15.116:0/99816995 conn(0x55dcce6ea6a0 msgr2=0x55dcce77f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:20.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/99816995 shutdown_connections 2024-03-30T01:05:20.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.638+0000 7f9d2ba15b80 1 -- 172.21.15.116:0/99816995 wait complete. 2024-03-30T01:05:20.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.639+0000 7f9d2ba15b80 20 client.16314 trim_cache size 0 max 16384 2024-03-30T01:05:20.649 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-03-30T01:05:20.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:05:20.649 INFO:teuthology.orchestra.run.smithi116.stdout: "damage_type": "backtrace", 2024-03-30T01:05:20.649 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 439204740, 2024-03-30T01:05:20.649 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 1099511639927, 2024-03-30T01:05:20.649 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-50.jpg" 2024-03-30T01:05:20.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:05:20.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:05:20.650 INFO:teuthology.orchestra.run.smithi116.stdout: "damage_type": "backtrace", 2024-03-30T01:05:20.650 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 2519282214, 2024-03-30T01:05:20.650 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 1099511639932, 2024-03-30T01:05:20.650 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-58.xml" 2024-03-30T01:05:20.650 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:05:20.650 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-03-30T01:05:20.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.639+0000 7f9d17e00700 1 -- 172.21.15.116:0/2783953456 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9cf80a11a0 msgr2=0x7f9cf80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.639+0000 7f9d17e00700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9cf80a11a0 0x7f9cf80a3660 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9d08009c90 tx=0x7f9d08009040 comp rx=0 tx=0).stop 2024-03-30T01:05:20.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.639+0000 7f9d17e00700 1 -- 172.21.15.116:0/2783953456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d10100d80 msgr2=0x7f9d10196d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.639+0000 7f9d17e00700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d10100d80 0x7f9d10196d70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400e970 tx=0x7f9d0400ed30 comp rx=0 tx=0).stop 2024-03-30T01:05:20.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.640+0000 7f9d17e00700 1 -- 172.21.15.116:0/2783953456 shutdown_connections 2024-03-30T01:05:20.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.640+0000 7f9d17e00700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9cf80a11a0 0x7f9cf80a3660 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.640+0000 7f9d17e00700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d10102910 0x7f9d1019e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.640+0000 7f9d17e00700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d10100d80 0x7f9d10196d70 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.640+0000 7f9d17e00700 1 --2- 172.21.15.116:0/2783953456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d10101f70 0x7f9d101972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.640+0000 7f9d17e00700 1 -- 172.21.15.116:0/2783953456 >> 172.21.15.116:0/2783953456 conn(0x7f9d100fc2f0 msgr2=0x7f9d10105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:20.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.640+0000 7f9d17e00700 1 -- 172.21.15.116:0/2783953456 shutdown_connections 2024-03-30T01:05:20.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.640+0000 7f9d17e00700 1 -- 172.21.15.116:0/2783953456 wait complete. 2024-03-30T01:05:20.662 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:20 smithi116 ceph-mon[38086]: pgmap v515: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 57 KiB/s wr, 15 op/s 2024-03-30T01:05:20.662 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:20 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/4219800625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:05:20.664 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T01:05:20.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:20 smithi130 ceph-mon[47626]: pgmap v515: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 57 KiB/s wr, 15 op/s 2024-03-30T01:05:20.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:20 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/4219800625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:05:20.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.853+0000 7f7274400700 1 -- 172.21.15.116:0/2927132962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c101f70 msgr2=0x7f726c1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.853+0000 7f7274400700 1 --2- 172.21.15.116:0/2927132962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c101f70 0x7f726c1023d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f72580099c0 tx=0x7f7258009cd0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.853+0000 7f7274400700 1 -- 172.21.15.116:0/2927132962 shutdown_connections 2024-03-30T01:05:20.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.853+0000 7f7274400700 1 --2- 172.21.15.116:0/2927132962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f726c102910 0x7f726c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.853+0000 7f7274400700 1 --2- 172.21.15.116:0/2927132962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f726c100d80 0x7f726c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.853+0000 7f7274400700 1 --2- 172.21.15.116:0/2927132962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c101f70 0x7f726c1023d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.853+0000 7f7274400700 1 -- 172.21.15.116:0/2927132962 >> 172.21.15.116:0/2927132962 conn(0x7f726c0fc2f0 msgr2=0x7f726c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:20.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.853+0000 7f7274400700 1 -- 172.21.15.116:0/2927132962 shutdown_connections 2024-03-30T01:05:20.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.853+0000 7f7274400700 1 -- 172.21.15.116:0/2927132962 wait complete. 2024-03-30T01:05:20.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.854+0000 7f7274400700 1 Processor -- start 2024-03-30T01:05:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.854+0000 7f7274400700 1 -- start start 2024-03-30T01:05:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.854+0000 7f7274400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c100d80 0x7f726c198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.854+0000 7f7274400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f726c101f70 0x7f726c199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.854+0000 7f7274400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f726c102910 0x7f726c1a04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.854+0000 7f7274400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726c1a0a80 con 0x7f726c100d80 2024-03-30T01:05:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.854+0000 7f7274400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726c1a0bc0 con 0x7f726c102910 2024-03-30T01:05:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.854+0000 7f7274400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726c1a0d50 con 0x7f726c101f70 2024-03-30T01:05:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.855+0000 7f7271a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c100d80 0x7f726c198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.855+0000 7f7271a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c100d80 0x7f726c198f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47874/0 (socket says 172.21.15.116:47874) 2024-03-30T01:05:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.855+0000 7f7271a00700 1 -- 172.21.15.116:0/2847735169 learned_addr learned my addr 172.21.15.116:0/2847735169 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.855+0000 7f7272400700 1 --2- 172.21.15.116:0/2847735169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f726c102910 0x7f726c1a04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.855+0000 7f7271000700 1 --2- 172.21.15.116:0/2847735169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f726c101f70 0x7f726c199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.855+0000 7f7271a00700 1 -- 172.21.15.116:0/2847735169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f726c101f70 msgr2=0x7f726c199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.855+0000 7f7271a00700 1 --2- 172.21.15.116:0/2847735169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f726c101f70 0x7f726c199490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.855+0000 7f7271a00700 1 -- 172.21.15.116:0/2847735169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f726c102910 msgr2=0x7f726c1a04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.855+0000 7f7271a00700 1 --2- 172.21.15.116:0/2847735169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f726c102910 0x7f726c1a04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.855+0000 7f7271a00700 1 -- 172.21.15.116:0/2847735169 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72580095f0 con 0x7f726c100d80 2024-03-30T01:05:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.855+0000 7f7271a00700 1 --2- 172.21.15.116:0/2847735169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c100d80 0x7f726c198f50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f726000e970 tx=0x7f726000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.856+0000 7f726a000700 1 -- 172.21.15.116:0/2847735169 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260009790 con 0x7f726c100d80 2024-03-30T01:05:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.856+0000 7f726a000700 1 -- 172.21.15.116:0/2847735169 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f726000c7e0 con 0x7f726c100d80 2024-03-30T01:05:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.856+0000 7f7274400700 1 -- 172.21.15.116:0/2847735169 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f726c1a0fd0 con 0x7f726c100d80 2024-03-30T01:05:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.856+0000 7f726a000700 1 -- 172.21.15.116:0/2847735169 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260003c70 con 0x7f726c100d80 2024-03-30T01:05:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.856+0000 7f7274400700 1 -- 172.21.15.116:0/2847735169 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f726c1a1450 con 0x7f726c100d80 2024-03-30T01:05:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.859+0000 7f726a000700 1 -- 172.21.15.116:0/2847735169 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f726000cc50 con 0x7f726c100d80 2024-03-30T01:05:20.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.860+0000 7f726a000700 1 --2- 172.21.15.116:0/2847735169 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f72540a1120 0x7f72540a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.860+0000 7f728801db80 1 Processor -- start 2024-03-30T01:05:20.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.860+0000 7f726a000700 1 -- 172.21.15.116:0/2847735169 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7260014070 con 0x7f726c100d80 2024-03-30T01:05:20.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.860+0000 7f728801db80 1 -- start start 2024-03-30T01:05:20.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.860+0000 7f7271000700 1 --2- 172.21.15.116:0/2847735169 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f72540a1120 0x7f72540a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.860+0000 7f728801db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3acf7d0 0x558be3acfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.860+0000 7f728801db80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ad09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.860+0000 7f728801db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3ad0fd0 0x558be3ad1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.860+0000 7f728801db80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be3a27990 con 0x558be3ad0130 2024-03-30T01:05:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.860+0000 7f728801db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be3a25350 con 0x558be3ad0fd0 2024-03-30T01:05:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.860+0000 7f728801db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be3a20c40 con 0x558be3acf7d0 2024-03-30T01:05:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.861+0000 7f7271a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3acf7d0 0x558be3acfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.861+0000 7f7271a00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3acf7d0 0x558be3acfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:33760/0 (socket says 172.21.15.116:33760) 2024-03-30T01:05:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.861+0000 7f7271a00700 1 -- 172.21.15.116:0/5800794 learned_addr learned my addr 172.21.15.116:0/5800794 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.861+0000 7f7272400700 1 --2- 172.21.15.116:0/5800794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3ad0fd0 0x558be3ad1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.861+0000 7f7271000700 1 --2- 172.21.15.116:0/5800794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ad09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.861+0000 7f7271000700 1 --2- 172.21.15.116:0/2847735169 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f72540a1120 0x7f72540a35e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7258000c00 tx=0x7f725800bfb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.861+0000 7f7271a00700 1 -- 172.21.15.116:0/5800794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3ad0fd0 msgr2=0x558be3ad1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.861+0000 7f7271a00700 1 --2- 172.21.15.116:0/5800794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3ad0fd0 0x558be3ad1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.861+0000 7f7271a00700 1 -- 172.21.15.116:0/5800794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 msgr2=0x558be3ad09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.861+0000 7f7271a00700 1 --2- 172.21.15.116:0/5800794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ad09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.861+0000 7f7271a00700 1 -- 172.21.15.116:0/5800794 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558be38a23b0 con 0x558be3acf7d0 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.861+0000 7f7272400700 1 --2- 172.21.15.116:0/5800794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3ad0fd0 0x558be3ad1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.861+0000 7f7271a00700 1 --2- 172.21.15.116:0/5800794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3acf7d0 0x558be3acfbf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7260096960 tx=0x7f7260003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74cba016f5c9cbb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.862+0000 7f725ea00700 1 -- 172.21.15.116:0/5800794 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72600b1070 con 0x558be3acf7d0 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.862+0000 7f725ea00700 1 -- 172.21.15.116:0/5800794 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7260099ef0 con 0x558be3acf7d0 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.862+0000 7f725ea00700 1 -- 172.21.15.116:0/5800794 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726009a150 con 0x558be3acf7d0 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.862+0000 7f728801db80 1 -- 172.21.15.116:0/5800794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3acf7d0 msgr2=0x558be3acfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.862+0000 7f728801db80 1 --2- 172.21.15.116:0/5800794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3acf7d0 0x558be3acfbf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7260096960 tx=0x7f7260003680 comp rx=0 tx=0).stop 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.862+0000 7f728801db80 1 -- 172.21.15.116:0/5800794 shutdown_connections 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.862+0000 7f728801db80 1 --2- 172.21.15.116:0/5800794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3acf7d0 0x558be3acfbf0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.862+0000 7f728801db80 1 --2- 172.21.15.116:0/5800794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3ad0fd0 0x558be3ad1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.862+0000 7f728801db80 1 --2- 172.21.15.116:0/5800794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ad09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.862+0000 7f728801db80 1 -- 172.21.15.116:0/5800794 >> 172.21.15.116:0/5800794 conn(0x558be3a2e6a0 msgr2=0x558be3aca9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.862+0000 7f728801db80 1 -- 172.21.15.116:0/5800794 shutdown_connections 2024-03-30T01:05:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.863+0000 7f728801db80 1 -- 172.21.15.116:0/5800794 wait complete. 2024-03-30T01:05:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.863+0000 7f728801db80 1 -- start start 2024-03-30T01:05:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.863+0000 7f728801db80 1 Processor -- start 2024-03-30T01:05:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f728801db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3acf7d0 0x558be3ad9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f728801db80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ad9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f728801db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0fd0 0x558be3ae0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f728801db80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be39bcd80 con 0x558be3ad0130 2024-03-30T01:05:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f728801db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be39bca20 con 0x558be3acf7d0 2024-03-30T01:05:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f728801db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be38a46a0 con 0x558be3ad0fd0 2024-03-30T01:05:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f728801db80 10 client.0 _ll_register_callbacks cb 0x558be39c6470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:05:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f7271000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ad9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f7271000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ad9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47894/0 (socket says 172.21.15.116:47894) 2024-03-30T01:05:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f7271000700 1 -- 172.21.15.116:0/517291246 learned_addr learned my addr 172.21.15.116:0/517291246 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f7271a00700 1 --2- 172.21.15.116:0/517291246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3acf7d0 0x558be3ad9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f7272400700 1 --2- 172.21.15.116:0/517291246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0fd0 0x558be3ae0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f7271000700 1 -- 172.21.15.116:0/517291246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0fd0 msgr2=0x558be3ae0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f7271000700 1 --2- 172.21.15.116:0/517291246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0fd0 0x558be3ae0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f7271000700 1 -- 172.21.15.116:0/517291246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3acf7d0 msgr2=0x558be3ad9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f7271000700 1 --2- 172.21.15.116:0/517291246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3acf7d0 0x558be3ad9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.864+0000 7f7271000700 1 -- 172.21.15.116:0/517291246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7264011230 con 0x558be3ad0130 2024-03-30T01:05:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.865+0000 7f7271000700 1 --2- 172.21.15.116:0/517291246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ad9b90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f72580099c0 tx=0x7f7258009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d2015bf96a3885 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.865+0000 7f7253400700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.865+0000 7f7253400700 1 -- 172.21.15.116:0/517291246 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725800fdb0 con 0x558be3ad0130 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.865+0000 7f7253400700 1 -- 172.21.15.116:0/517291246 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f725802f480 con 0x558be3ad0130 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.865+0000 7f728801db80 1 -- 172.21.15.116:0/517291246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558be39e1380 con 0x558be3ad0130 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.865+0000 7f7253400700 1 -- 172.21.15.116:0/517291246 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f725802f640 con 0x558be3ad0130 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.865+0000 7f728801db80 10 client.16326 fetch_fsmap learned FSMap version 30 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.865+0000 7f728801db80 1 -- 172.21.15.116:0/517291246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f726c1a1450 con 0x558be3ad0130 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.865+0000 7f7253400700 1 -- 172.21.15.116:0/517291246 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558be39e1380 con 0x558be3ad0130 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.866+0000 7f7253400700 1 -- 172.21.15.116:0/517291246 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f72580258d0 con 0x558be3ad0130 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.866+0000 7f728801db80 10 client.16326 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.866+0000 7f728801db80 10 client.16326 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.866+0000 7f728801db80 20 client.16326 populate_metadata read hostname 'smithi116' 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.866+0000 7f728801db80 1 --2- 172.21.15.116:0/517291246 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x558be3ae5390 0x558be3ae77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.866+0000 7f728801db80 4 client.16326 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.866+0000 7f728801db80 1 -- 172.21.15.116:0/517291246 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558be3a19700 con 0x558be3ae5390 2024-03-30T01:05:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.866+0000 7f7271a00700 1 --2- 172.21.15.116:0/517291246 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x558be3ae5390 0x558be3ae77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.866+0000 7f7271a00700 1 --2- 172.21.15.116:0/517291246 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x558be3ae5390 0x558be3ae77b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=996971e54dc5428 server_cookie=4142fda12527fd14 in_seq=0 out_seq=0 2024-03-30T01:05:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.866+0000 7f7253400700 10 client.16326 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T01:05:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.867+0000 7f7253400700 1 -- 172.21.15.116:0/517291246 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558be3a19700 con 0x558be3ae5390 2024-03-30T01:05:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.867+0000 7f7253400700 10 client.16326 handle_command_reply: tid=0 2024-03-30T01:05:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.867+0000 7f728801db80 1 client.16326 shutdown 2024-03-30T01:05:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.867+0000 7f728801db80 1 -- 172.21.15.116:0/517291246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 msgr2=0x558be3ad9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.867+0000 7f728801db80 1 --2- 172.21.15.116:0/517291246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ad9b90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f72580099c0 tx=0x7f7258009cd0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.867+0000 7f728801db80 1 -- 172.21.15.116:0/517291246 shutdown_connections 2024-03-30T01:05:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.867+0000 7f728801db80 1 --2- 172.21.15.116:0/517291246 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x558be3ae5390 0x558be3ae77b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.867+0000 7f728801db80 1 --2- 172.21.15.116:0/517291246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0fd0 0x558be3ae0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.867+0000 7f7253400700 0 client.16326 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T01:05:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.867+0000 7f728801db80 1 --2- 172.21.15.116:0/517291246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3acf7d0 0x558be3ad9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.867+0000 7f728801db80 1 --2- 172.21.15.116:0/517291246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ad9b90 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.867+0000 7f728801db80 1 -- 172.21.15.116:0/517291246 >> 172.21.15.116:0/517291246 conn(0x558be3a2e6a0 msgr2=0x558be3ace170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.867+0000 7f728801db80 1 -- 172.21.15.116:0/517291246 shutdown_connections 2024-03-30T01:05:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.867+0000 7f728801db80 1 -- 172.21.15.116:0/517291246 wait complete. 2024-03-30T01:05:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.868+0000 7f728801db80 20 client.16326 trim_cache size 0 max 16384 2024-03-30T01:05:20.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.881+0000 7f728801db80 1 Processor -- start 2024-03-30T01:05:20.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.881+0000 7f728801db80 1 -- start start 2024-03-30T01:05:20.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.881+0000 7f728801db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3acf7d0 0x558be3ad8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.881+0000 7f728801db80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ae0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.881+0000 7f728801db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0fd0 0x558be3ad13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.881+0000 7f728801db80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be3ad0e50 con 0x558be3ad0130 2024-03-30T01:05:20.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.881+0000 7f728801db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be3ada550 con 0x558be3acf7d0 2024-03-30T01:05:20.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.882+0000 7f728801db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be3ae1d50 con 0x558be3ad0fd0 2024-03-30T01:05:20.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.882+0000 7f7271000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ae0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.882+0000 7f7271a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3acf7d0 0x558be3ad8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.882+0000 7f7271a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3acf7d0 0x558be3ad8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38426/0 (socket says 172.21.15.116:38426) 2024-03-30T01:05:20.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.882+0000 7f7271000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ae0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47906/0 (socket says 172.21.15.116:47906) 2024-03-30T01:05:20.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.882+0000 7f7272400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0fd0 0x558be3ad13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.882+0000 7f7271a00700 1 -- 172.21.15.116:0/2295377112 learned_addr learned my addr 172.21.15.116:0/2295377112 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:20.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.882+0000 7f7271a00700 1 -- 172.21.15.116:0/2295377112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0fd0 msgr2=0x558be3ad13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.882+0000 7f7271a00700 1 --2- 172.21.15.116:0/2295377112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0fd0 0x558be3ad13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.882+0000 7f7271a00700 1 -- 172.21.15.116:0/2295377112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 msgr2=0x558be3ae0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.882+0000 7f7271a00700 1 --2- 172.21.15.116:0/2295377112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ae0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.882+0000 7f7271a00700 1 -- 172.21.15.116:0/2295377112 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558be384eb60 con 0x558be3acf7d0 2024-03-30T01:05:20.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.882+0000 7f7271000700 1 --2- 172.21.15.116:0/2295377112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ae0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:05:20.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.882+0000 7f7271a00700 1 --2- 172.21.15.116:0/2295377112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3acf7d0 0x558be3ad8b60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7260099e80 tx=0x7f726000cf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d267d521f42d77f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.883+0000 7f725f400700 1 -- 172.21.15.116:0/2295377112 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72600b1040 con 0x558be3acf7d0 2024-03-30T01:05:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.883+0000 7f725f400700 1 -- 172.21.15.116:0/2295377112 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f726009dda0 con 0x558be3acf7d0 2024-03-30T01:05:20.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.883+0000 7f725f400700 1 -- 172.21.15.116:0/2295377112 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260099640 con 0x558be3acf7d0 2024-03-30T01:05:20.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.883+0000 7f728801db80 1 -- 172.21.15.116:0/2295377112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3acf7d0 msgr2=0x558be3ad8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.883+0000 7f728801db80 1 --2- 172.21.15.116:0/2295377112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3acf7d0 0x558be3ad8b60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7260099e80 tx=0x7f726000cf50 comp rx=0 tx=0).stop 2024-03-30T01:05:20.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.883+0000 7f728801db80 1 -- 172.21.15.116:0/2295377112 shutdown_connections 2024-03-30T01:05:20.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.883+0000 7f728801db80 1 --2- 172.21.15.116:0/2295377112 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0fd0 0x558be3ad13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.883+0000 7f728801db80 1 --2- 172.21.15.116:0/2295377112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3acf7d0 0x558be3ad8b60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.883+0000 7f728801db80 1 --2- 172.21.15.116:0/2295377112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3ad0130 0x558be3ae0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.883+0000 7f728801db80 1 -- 172.21.15.116:0/2295377112 >> 172.21.15.116:0/2295377112 conn(0x558be3a2e6a0 msgr2=0x558be3ac35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:20.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.883+0000 7f728801db80 1 -- 172.21.15.116:0/2295377112 shutdown_connections 2024-03-30T01:05:20.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.883+0000 7f728801db80 1 -- 172.21.15.116:0/2295377112 wait complete. 2024-03-30T01:05:20.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.884+0000 7f728801db80 1 -- start start 2024-03-30T01:05:20.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.884+0000 7f728801db80 1 Processor -- start 2024-03-30T01:05:20.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.884+0000 7f728801db80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3acf7d0 0x558be3adeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.884+0000 7f728801db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0130 0x558be3adf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.884+0000 7f728801db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3ad0fd0 0x558be3ae00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.884+0000 7f728801db80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be3ae05f0 con 0x558be3acf7d0 2024-03-30T01:05:20.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.884+0000 7f728801db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be3ae0760 con 0x558be3ad0fd0 2024-03-30T01:05:20.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.884+0000 7f728801db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be3ae0920 con 0x558be3ad0130 2024-03-30T01:05:20.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.884+0000 7f728801db80 10 client.0 _ll_register_callbacks cb 0x558be39c6470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:05:20.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.885+0000 7f7271a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3acf7d0 0x558be3adeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.885+0000 7f7272400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3ad0fd0 0x558be3ae00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.885+0000 7f7272400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3ad0fd0 0x558be3ae00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:38436/0 (socket says 172.21.15.116:38436) 2024-03-30T01:05:20.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.885+0000 7f7272400700 1 -- 172.21.15.116:0/388826896 learned_addr learned my addr 172.21.15.116:0/388826896 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:20.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.885+0000 7f7271000700 1 --2- 172.21.15.116:0/388826896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0130 0x558be3adf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.885+0000 7f7272400700 1 -- 172.21.15.116:0/388826896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0130 msgr2=0x558be3adf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.885+0000 7f7272400700 1 --2- 172.21.15.116:0/388826896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0130 0x558be3adf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.885+0000 7f7272400700 1 -- 172.21.15.116:0/388826896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3acf7d0 msgr2=0x558be3adeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:20.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.885+0000 7f7272400700 1 --2- 172.21.15.116:0/388826896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3acf7d0 0x558be3adeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.885+0000 7f7272400700 1 -- 172.21.15.116:0/388826896 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f726c1a1450 con 0x558be3ad0fd0 2024-03-30T01:05:20.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.885+0000 7f7271a00700 1 --2- 172.21.15.116:0/388826896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3acf7d0 0x558be3adeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:05:20.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.885+0000 7f7272400700 1 --2- 172.21.15.116:0/388826896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3ad0fd0 0x558be3ae00b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7264004660 tx=0x7f7264004970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a7b4d176054419 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:20.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.885+0000 7f7253400700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-30T01:05:20.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.886+0000 7f7253400700 1 -- 172.21.15.116:0/388826896 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726401b070 con 0x558be3ad0fd0 2024-03-30T01:05:20.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.886+0000 7f728801db80 1 -- 172.21.15.116:0/388826896 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7260099ef0 con 0x558be3ad0fd0 2024-03-30T01:05:20.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.886+0000 7f7253400700 1 -- 172.21.15.116:0/388826896 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7264003e50 con 0x558be3ad0fd0 2024-03-30T01:05:20.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.886+0000 7f7253400700 1 -- 172.21.15.116:0/388826896 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f726400e040 con 0x558be3ad0fd0 2024-03-30T01:05:20.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.886+0000 7f728801db80 10 client.26321 fetch_fsmap learned FSMap version 30 2024-03-30T01:05:20.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.886+0000 7f728801db80 1 -- 172.21.15.116:0/388826896 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x558be38a23b0 con 0x558be3ad0fd0 2024-03-30T01:05:20.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.886+0000 7f7253400700 1 -- 172.21.15.116:0/388826896 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7260099ef0 con 0x558be3ad0fd0 2024-03-30T01:05:20.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.886+0000 7f7253400700 1 -- 172.21.15.116:0/388826896 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f726401f7f0 con 0x558be3ad0fd0 2024-03-30T01:05:20.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.886+0000 7f728801db80 10 client.26321 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T01:05:20.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.886+0000 7f728801db80 10 client.26321 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T01:05:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.886+0000 7f728801db80 20 client.26321 populate_metadata read hostname 'smithi116' 2024-03-30T01:05:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.886+0000 7f728801db80 1 --2- 172.21.15.116:0/388826896 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x558be3b3a440 0x558be3b3c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.886+0000 7f728801db80 4 client.26321 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:05:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.886+0000 7f728801db80 1 -- 172.21.15.116:0/388826896 --> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558be386fcf0 con 0x558be3b3a440 2024-03-30T01:05:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.887+0000 7f7271a00700 1 --2- 172.21.15.116:0/388826896 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x558be3b3a440 0x558be3b3c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.887+0000 7f7253400700 10 client.26321 ms_handle_connect on v2:172.21.15.130:6836/3389690261 2024-03-30T01:05:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.887+0000 7f7271a00700 1 --2- 172.21.15.116:0/388826896 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x558be3b3a440 0x558be3b3c860 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cf2c476d0b5cd181 server_cookie=b50767479e00380c in_seq=0 out_seq=0 2024-03-30T01:05:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.887+0000 7f7253400700 1 -- 172.21.15.116:0/388826896 <== mds.1 v2:172.21.15.130:6836/3389690261 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558be386fcf0 con 0x558be3b3a440 2024-03-30T01:05:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.887+0000 7f7253400700 10 client.26321 handle_command_reply: tid=0 2024-03-30T01:05:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.887+0000 7f728801db80 1 client.26321 shutdown 2024-03-30T01:05:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.887+0000 7f728801db80 1 -- 172.21.15.116:0/388826896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3ad0fd0 msgr2=0x558be3ae00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.887+0000 7f728801db80 1 --2- 172.21.15.116:0/388826896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3ad0fd0 0x558be3ae00b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7264004660 tx=0x7f7264004970 comp rx=0 tx=0).stop 2024-03-30T01:05:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.888+0000 7f728801db80 1 -- 172.21.15.116:0/388826896 shutdown_connections 2024-03-30T01:05:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.888+0000 7f728801db80 1 --2- 172.21.15.116:0/388826896 >> [v2:172.21.15.130:6836/3389690261,v1:172.21.15.130:6837/3389690261] conn(0x558be3b3a440 0x558be3b3c860 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.888+0000 7f728801db80 1 --2- 172.21.15.116:0/388826896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558be3ad0fd0 0x558be3ae00b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.888+0000 7f728801db80 1 --2- 172.21.15.116:0/388826896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558be3acf7d0 0x558be3adeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.888+0000 7f728801db80 1 --2- 172.21.15.116:0/388826896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558be3ad0130 0x558be3adf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.888+0000 7f7253400700 0 client.26321 ms_handle_reset on v2:172.21.15.130:6836/3389690261 2024-03-30T01:05:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.888+0000 7f728801db80 1 -- 172.21.15.116:0/388826896 >> 172.21.15.116:0/388826896 conn(0x558be3a2e6a0 msgr2=0x558be3ad74d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.888+0000 7f728801db80 1 -- 172.21.15.116:0/388826896 shutdown_connections 2024-03-30T01:05:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.888+0000 7f728801db80 1 -- 172.21.15.116:0/388826896 wait complete. 2024-03-30T01:05:20.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.888+0000 7f728801db80 20 client.26321 trim_cache size 0 max 16384 2024-03-30T01:05:20.899 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:05:20.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.888+0000 7f7274400700 1 -- 172.21.15.116:0/2847735169 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f72540a1120 msgr2=0x7f72540a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.888+0000 7f7274400700 1 --2- 172.21.15.116:0/2847735169 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f72540a1120 0x7f72540a35e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7258000c00 tx=0x7f725800bfb0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.888+0000 7f7274400700 1 -- 172.21.15.116:0/2847735169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c100d80 msgr2=0x7f726c198f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.888+0000 7f7274400700 1 --2- 172.21.15.116:0/2847735169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c100d80 0x7f726c198f50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f726000e970 tx=0x7f726000ec80 comp rx=0 tx=0).stop 2024-03-30T01:05:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.889+0000 7f7274400700 1 -- 172.21.15.116:0/2847735169 shutdown_connections 2024-03-30T01:05:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.889+0000 7f7274400700 1 --2- 172.21.15.116:0/2847735169 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f72540a1120 0x7f72540a35e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.889+0000 7f7274400700 1 --2- 172.21.15.116:0/2847735169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f726c102910 0x7f726c1a04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.889+0000 7f7274400700 1 --2- 172.21.15.116:0/2847735169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c100d80 0x7f726c198f50 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.889+0000 7f7274400700 1 --2- 172.21.15.116:0/2847735169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f726c101f70 0x7f726c199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:20.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.889+0000 7f7274400700 1 -- 172.21.15.116:0/2847735169 >> 172.21.15.116:0/2847735169 conn(0x7f726c0fc2f0 msgr2=0x7f726c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:20.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.889+0000 7f7274400700 1 -- 172.21.15.116:0/2847735169 shutdown_connections 2024-03-30T01:05:20.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:20.889+0000 7f7274400700 1 -- 172.21.15.116:0/2847735169 wait complete. 2024-03-30T01:05:20.913 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T01:05:21.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:20 smithi181 ceph-mon[47622]: pgmap v515: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 57 KiB/s wr, 15 op/s 2024-03-30T01:05:21.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:20 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/4219800625' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:05:21.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.102+0000 7fb446400700 1 -- 172.21.15.116:0/3771112371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb440101f70 msgr2=0x7fb4401023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:21.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.102+0000 7fb446400700 1 --2- 172.21.15.116:0/3771112371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb440101f70 0x7fb4401023d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb42c0099c0 tx=0x7fb42c009cd0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.102+0000 7fb446400700 1 -- 172.21.15.116:0/3771112371 shutdown_connections 2024-03-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.102+0000 7fb446400700 1 --2- 172.21.15.116:0/3771112371 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb440100d80 0x7fb4401011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.102+0000 7fb446400700 1 --2- 172.21.15.116:0/3771112371 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb440102910 0x7fb4401091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.102+0000 7fb446400700 1 --2- 172.21.15.116:0/3771112371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb440101f70 0x7fb4401023d0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.102+0000 7fb446400700 1 -- 172.21.15.116:0/3771112371 >> 172.21.15.116:0/3771112371 conn(0x7fb4400fc2f0 msgr2=0x7fb4400fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.102+0000 7fb446400700 1 -- 172.21.15.116:0/3771112371 shutdown_connections 2024-03-30T01:05:21.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.102+0000 7fb446400700 1 -- 172.21.15.116:0/3771112371 wait complete. 2024-03-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.103+0000 7fb446400700 1 Processor -- start 2024-03-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.103+0000 7fb446400700 1 -- start start 2024-03-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.103+0000 7fb446400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb440100d80 0x7fb440074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.103+0000 7fb446400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb440101f70 0x7fb4400731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.103+0000 7fb446400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb440102910 0x7fb4400736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.103+0000 7fb446400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb440073e80 con 0x7fb440100d80 2024-03-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.103+0000 7fb446400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb440073ff0 con 0x7fb440101f70 2024-03-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.103+0000 7fb446400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4401a0d90 con 0x7fb440102910 2024-03-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.103+0000 7fb43f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb440100d80 0x7fb440074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb43f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb440100d80 0x7fb440074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47934/0 (socket says 172.21.15.116:47934) 2024-03-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb43f400700 1 -- 172.21.15.116:0/71823714 learned_addr learned my addr 172.21.15.116:0/71823714 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb43ea00700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb440101f70 0x7fb4400731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb43fe00700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb440102910 0x7fb4400736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb43f400700 1 -- 172.21.15.116:0/71823714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb440102910 msgr2=0x7fb4400736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb43f400700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb440102910 0x7fb4400736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb43f400700 1 -- 172.21.15.116:0/71823714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb440101f70 msgr2=0x7fb4400731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb43f400700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb440101f70 0x7fb4400731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb43f400700 1 -- 172.21.15.116:0/71823714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb430009520 con 0x7fb440100d80 2024-03-30T01:05:21.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb43fe00700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb440102910 0x7fb4400736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb43f400700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb440100d80 0x7fb440074b10 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fb43000ea50 tx=0x7fb43000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb437e00700 1 -- 172.21.15.116:0/71823714 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43000cb70 con 0x7fb440100d80 2024-03-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb437e00700 1 -- 172.21.15.116:0/71823714 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb43000ccd0 con 0x7fb440100d80 2024-03-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb446400700 1 -- 172.21.15.116:0/71823714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c0095f0 con 0x7fb440100d80 2024-03-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.104+0000 7fb437e00700 1 -- 172.21.15.116:0/71823714 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430018680 con 0x7fb440100d80 2024-03-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.105+0000 7fb446400700 1 -- 172.21.15.116:0/71823714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4401a12b0 con 0x7fb440100d80 2024-03-30T01:05:21.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.107+0000 7fb437e00700 1 -- 172.21.15.116:0/71823714 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fb4300188c0 con 0x7fb440100d80 2024-03-30T01:05:21.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.109+0000 7fb45a000b80 1 Processor -- start 2024-03-30T01:05:21.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.109+0000 7fb45a000b80 1 -- start start 2024-03-30T01:05:21.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.109+0000 7fb437e00700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4240a1120 0x7fb4240a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb45a000b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f77d0 0x557d5f0f7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb45a000b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f0f89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb437e00700 1 -- 172.21.15.116:0/71823714 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb430014070 con 0x7fb440100d80 2024-03-30T01:05:21.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb45a000b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f8fd0 0x557d5f0f9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb45a000b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5f04f990 con 0x557d5f0f8130 2024-03-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb45a000b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5f04d350 con 0x557d5f0f8fd0 2024-03-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb45a000b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5f048c40 con 0x557d5f0f77d0 2024-03-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb43fe00700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4240a1120 0x7fb4240a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb43ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f0f89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb43ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f0f89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47946/0 (socket says 172.21.15.116:47946) 2024-03-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb43ea00700 1 -- 172.21.15.116:0/2056043122 learned_addr learned my addr 172.21.15.116:0/2056043122 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb43f400700 1 --2- 172.21.15.116:0/2056043122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f77d0 0x557d5f0f7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb43fe00700 1 --2- 172.21.15.116:0/2056043122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f8fd0 0x557d5f0f9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb43ea00700 1 -- 172.21.15.116:0/2056043122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f77d0 msgr2=0x557d5f0f7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb43ea00700 1 --2- 172.21.15.116:0/2056043122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f77d0 0x557d5f0f7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb43ea00700 1 -- 172.21.15.116:0/2056043122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f8fd0 msgr2=0x557d5f0f9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb43ea00700 1 --2- 172.21.15.116:0/2056043122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f8fd0 0x557d5f0f9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.110+0000 7fb43ea00700 1 -- 172.21.15.116:0/2056043122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d5eeca3b0 con 0x557d5f0f8130 2024-03-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.111+0000 7fb43fe00700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4240a1120 0x7fb4240a35e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb438007770 tx=0x7fb438007b30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.111+0000 7fb43ea00700 1 --2- 172.21.15.116:0/2056043122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f0f89f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb42c0058e0 tx=0x7fb42c005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff19b783e68be79 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.111+0000 7fb434c00700 1 -- 172.21.15.116:0/2056043122 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c00ba10 con 0x557d5f0f8130 2024-03-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.111+0000 7fb434c00700 1 -- 172.21.15.116:0/2056043122 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb42c01e040 con 0x557d5f0f8130 2024-03-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.111+0000 7fb434c00700 1 -- 172.21.15.116:0/2056043122 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c003a30 con 0x557d5f0f8130 2024-03-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.112+0000 7fb45a000b80 1 -- 172.21.15.116:0/2056043122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 msgr2=0x557d5f0f89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.112+0000 7fb45a000b80 1 --2- 172.21.15.116:0/2056043122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f0f89f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb42c0058e0 tx=0x7fb42c005c30 comp rx=0 tx=0).stop 2024-03-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.112+0000 7fb45a000b80 1 -- 172.21.15.116:0/2056043122 shutdown_connections 2024-03-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.112+0000 7fb45a000b80 1 --2- 172.21.15.116:0/2056043122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f77d0 0x557d5f0f7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.112+0000 7fb45a000b80 1 --2- 172.21.15.116:0/2056043122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f8fd0 0x557d5f0f9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.112+0000 7fb45a000b80 1 --2- 172.21.15.116:0/2056043122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f0f89f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.112+0000 7fb45a000b80 1 -- 172.21.15.116:0/2056043122 >> 172.21.15.116:0/2056043122 conn(0x557d5f0566a0 msgr2=0x557d5f0f29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.112+0000 7fb45a000b80 1 -- 172.21.15.116:0/2056043122 shutdown_connections 2024-03-30T01:05:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.112+0000 7fb45a000b80 1 -- 172.21.15.116:0/2056043122 wait complete. 2024-03-30T01:05:21.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.112+0000 7fb45a000b80 1 -- start start 2024-03-30T01:05:21.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.113+0000 7fb45a000b80 1 Processor -- start 2024-03-30T01:05:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.113+0000 7fb45a000b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f77d0 0x557d5f1015d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.113+0000 7fb45a000b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f101b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.113+0000 7fb45a000b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f8fd0 0x557d5f108b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.113+0000 7fb45a000b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5efe4d80 con 0x557d5f0f8130 2024-03-30T01:05:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.113+0000 7fb45a000b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5efe4a20 con 0x557d5f0f77d0 2024-03-30T01:05:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.113+0000 7fb45a000b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5eecc6a0 con 0x557d5f0f8fd0 2024-03-30T01:05:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.113+0000 7fb45a000b80 10 client.0 _ll_register_callbacks cb 0x557d5efee470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:05:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.113+0000 7fb43ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f101b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.113+0000 7fb43ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f101b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47950/0 (socket says 172.21.15.116:47950) 2024-03-30T01:05:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.113+0000 7fb43ea00700 1 -- 172.21.15.116:0/2388130840 learned_addr learned my addr 172.21.15.116:0/2388130840 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.113+0000 7fb43f400700 1 --2- 172.21.15.116:0/2388130840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f77d0 0x557d5f1015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.113+0000 7fb43fe00700 1 --2- 172.21.15.116:0/2388130840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f8fd0 0x557d5f108b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.114+0000 7fb43ea00700 1 -- 172.21.15.116:0/2388130840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f8fd0 msgr2=0x557d5f108b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.114+0000 7fb43ea00700 1 --2- 172.21.15.116:0/2388130840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f8fd0 0x557d5f108b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.114+0000 7fb43ea00700 1 -- 172.21.15.116:0/2388130840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f77d0 msgr2=0x557d5f1015d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.114+0000 7fb43ea00700 1 --2- 172.21.15.116:0/2388130840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f77d0 0x557d5f1015d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.114+0000 7fb43ea00700 1 -- 172.21.15.116:0/2388130840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb440109b10 con 0x557d5f0f8130 2024-03-30T01:05:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.114+0000 7fb43ea00700 1 --2- 172.21.15.116:0/2388130840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f101b10 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00a010 tx=0x7fb42c004a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8acbaddc5c4b92 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.114+0000 7fb429600700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:05:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.114+0000 7fb429600700 1 -- 172.21.15.116:0/2388130840 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c005240 con 0x557d5f0f8130 2024-03-30T01:05:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.114+0000 7fb429600700 1 -- 172.21.15.116:0/2388130840 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb42c0053a0 con 0x557d5f0f8130 2024-03-30T01:05:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.114+0000 7fb45a000b80 1 -- 172.21.15.116:0/2388130840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d5f009380 con 0x557d5f0f8130 2024-03-30T01:05:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.114+0000 7fb429600700 1 -- 172.21.15.116:0/2388130840 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb42c005980 con 0x557d5f0f8130 2024-03-30T01:05:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.114+0000 7fb45a000b80 10 client.16344 fetch_fsmap learned FSMap version 30 2024-03-30T01:05:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.114+0000 7fb45a000b80 1 -- 172.21.15.116:0/2388130840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557d5ee76b60 con 0x557d5f0f8130 2024-03-30T01:05:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.115+0000 7fb429600700 1 -- 172.21.15.116:0/2388130840 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557d5f009380 con 0x557d5f0f8130 2024-03-30T01:05:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.115+0000 7fb429600700 1 -- 172.21.15.116:0/2388130840 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fb42c00f460 con 0x557d5f0f8130 2024-03-30T01:05:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.115+0000 7fb45a000b80 10 client.16344 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T01:05:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.115+0000 7fb45a000b80 10 client.16344 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:05:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.115+0000 7fb45a000b80 20 client.16344 populate_metadata read hostname 'smithi116' 2024-03-30T01:05:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.115+0000 7fb45a000b80 1 --2- 172.21.15.116:0/2388130840 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557d5f10cec0 0x557d5f10f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.115+0000 7fb45a000b80 4 client.16344 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T01:05:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.115+0000 7fb45a000b80 1 -- 172.21.15.116:0/2388130840 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d5f041700 con 0x557d5f10cec0 2024-03-30T01:05:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.115+0000 7fb43f400700 1 --2- 172.21.15.116:0/2388130840 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557d5f10cec0 0x557d5f10f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.116+0000 7fb43f400700 1 --2- 172.21.15.116:0/2388130840 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557d5f10cec0 0x557d5f10f2e0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c6e158e8cdb8c19 server_cookie=72d6c7dd1a5d9aec in_seq=0 out_seq=0 2024-03-30T01:05:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.116+0000 7fb429600700 10 client.16344 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:05:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.117+0000 7fb429600700 1 -- 172.21.15.116:0/2388130840 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557d5f041700 con 0x557d5f10cec0 2024-03-30T01:05:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.117+0000 7fb429600700 10 client.16344 handle_command_reply: tid=0 2024-03-30T01:05:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.117+0000 7fb45a000b80 1 client.16344 shutdown 2024-03-30T01:05:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.117+0000 7fb45a000b80 1 -- 172.21.15.116:0/2388130840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 msgr2=0x557d5f101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.117+0000 7fb45a000b80 1 --2- 172.21.15.116:0/2388130840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f101b10 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00a010 tx=0x7fb42c004a10 comp rx=0 tx=0).stop 2024-03-30T01:05:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.117+0000 7fb45a000b80 1 -- 172.21.15.116:0/2388130840 shutdown_connections 2024-03-30T01:05:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.117+0000 7fb45a000b80 1 --2- 172.21.15.116:0/2388130840 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557d5f10cec0 0x557d5f10f2e0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.117+0000 7fb45a000b80 1 --2- 172.21.15.116:0/2388130840 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f8fd0 0x557d5f108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.117+0000 7fb45a000b80 1 --2- 172.21.15.116:0/2388130840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f77d0 0x557d5f1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.117+0000 7fb45a000b80 1 --2- 172.21.15.116:0/2388130840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f101b10 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.117+0000 7fb429600700 0 client.16344 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:05:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.117+0000 7fb45a000b80 1 -- 172.21.15.116:0/2388130840 >> 172.21.15.116:0/2388130840 conn(0x557d5f0566a0 msgr2=0x557d5f0f6170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.118+0000 7fb45a000b80 1 -- 172.21.15.116:0/2388130840 shutdown_connections 2024-03-30T01:05:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.118+0000 7fb45a000b80 1 -- 172.21.15.116:0/2388130840 wait complete. 2024-03-30T01:05:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.118+0000 7fb45a000b80 20 client.16344 trim_cache size 0 max 16384 2024-03-30T01:05:21.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.131+0000 7fb45a000b80 1 Processor -- start 2024-03-30T01:05:21.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.131+0000 7fb45a000b80 1 -- start start 2024-03-30T01:05:21.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.131+0000 7fb45a000b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f77d0 0x557d5f109200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.131+0000 7fb45a000b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f108b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.131+0000 7fb45a000b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f8fd0 0x557d5f0eb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.131+0000 7fb45a000b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5f041700 con 0x557d5f0f8130 2024-03-30T01:05:21.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.131+0000 7fb45a000b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5f009380 con 0x557d5f0f77d0 2024-03-30T01:05:21.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.131+0000 7fb45a000b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5f0f8600 con 0x557d5f0f8fd0 2024-03-30T01:05:21.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.132+0000 7fb43ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f108b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.132+0000 7fb43fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f8fd0 0x557d5f0eb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.132+0000 7fb43fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f8fd0 0x557d5f0eb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:33832/0 (socket says 172.21.15.116:33832) 2024-03-30T01:05:21.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.132+0000 7fb43fe00700 1 -- 172.21.15.116:0/1578038594 learned_addr learned my addr 172.21.15.116:0/1578038594 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:21.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.132+0000 7fb43f400700 1 --2- 172.21.15.116:0/1578038594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f77d0 0x557d5f109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.132+0000 7fb43fe00700 1 -- 172.21.15.116:0/1578038594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f77d0 msgr2=0x557d5f109200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:21.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.132+0000 7fb43fe00700 1 --2- 172.21.15.116:0/1578038594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f77d0 0x557d5f109200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.132+0000 7fb43fe00700 1 -- 172.21.15.116:0/1578038594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 msgr2=0x557d5f108b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:21.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.132+0000 7fb43fe00700 1 --2- 172.21.15.116:0/1578038594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f108b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.132+0000 7fb43fe00700 1 -- 172.21.15.116:0/1578038594 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d5f00eea0 con 0x557d5f0f8fd0 2024-03-30T01:05:21.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.132+0000 7fb43ea00700 1 --2- 172.21.15.116:0/1578038594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:05:21.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.132+0000 7fb43fe00700 1 --2- 172.21.15.116:0/1578038594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f8fd0 0x557d5f0eb5a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb43801a730 tx=0x7fb43801d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc5b4bf2ea44bfc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:21.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.132+0000 7fb435600700 1 -- 172.21.15.116:0/1578038594 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43800e050 con 0x557d5f0f8fd0 2024-03-30T01:05:21.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.133+0000 7fb435600700 1 -- 172.21.15.116:0/1578038594 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb43800f040 con 0x557d5f0f8fd0 2024-03-30T01:05:21.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.133+0000 7fb435600700 1 -- 172.21.15.116:0/1578038594 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438027730 con 0x557d5f0f8fd0 2024-03-30T01:05:21.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.133+0000 7fb45a000b80 1 -- 172.21.15.116:0/1578038594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f8fd0 msgr2=0x557d5f0eb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:21.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.133+0000 7fb45a000b80 1 --2- 172.21.15.116:0/1578038594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f8fd0 0x557d5f0eb5a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb43801a730 tx=0x7fb43801d000 comp rx=0 tx=0).stop 2024-03-30T01:05:21.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.133+0000 7fb45a000b80 1 -- 172.21.15.116:0/1578038594 shutdown_connections 2024-03-30T01:05:21.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.133+0000 7fb45a000b80 1 --2- 172.21.15.116:0/1578038594 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f8fd0 0x557d5f0eb5a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.133+0000 7fb45a000b80 1 --2- 172.21.15.116:0/1578038594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f77d0 0x557d5f109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.133+0000 7fb45a000b80 1 --2- 172.21.15.116:0/1578038594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.133+0000 7fb45a000b80 1 -- 172.21.15.116:0/1578038594 >> 172.21.15.116:0/1578038594 conn(0x557d5f0566a0 msgr2=0x557d5f0ebf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:21.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.133+0000 7fb45a000b80 1 -- 172.21.15.116:0/1578038594 shutdown_connections 2024-03-30T01:05:21.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.133+0000 7fb45a000b80 1 -- 172.21.15.116:0/1578038594 wait complete. 2024-03-30T01:05:21.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.134+0000 7fb45a000b80 1 -- start start 2024-03-30T01:05:21.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.134+0000 7fb45a000b80 1 Processor -- start 2024-03-30T01:05:21.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.134+0000 7fb45a000b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f77d0 0x557d5f106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.134+0000 7fb45a000b80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f107050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.134+0000 7fb45a000b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f8fd0 0x557d5f108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.134+0000 7fb45a000b80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5f1085a0 con 0x557d5f0f8130 2024-03-30T01:05:21.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.134+0000 7fb45a000b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5f108710 con 0x557d5f0f8fd0 2024-03-30T01:05:21.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.134+0000 7fb45a000b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5f1088d0 con 0x557d5f0f77d0 2024-03-30T01:05:21.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.134+0000 7fb45a000b80 10 client.0 _ll_register_callbacks cb 0x557d5efee470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T01:05:21.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.134+0000 7fb43ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.134+0000 7fb43f400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f77d0 0x557d5f106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.134+0000 7fb43ea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f107050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47966/0 (socket says 172.21.15.116:47966) 2024-03-30T01:05:21.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.134+0000 7fb43ea00700 1 -- 172.21.15.116:0/847663567 learned_addr learned my addr 172.21.15.116:0/847663567 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:05:21.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.135+0000 7fb43fe00700 1 --2- 172.21.15.116:0/847663567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f8fd0 0x557d5f108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.135+0000 7fb43ea00700 1 -- 172.21.15.116:0/847663567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f77d0 msgr2=0x557d5f106b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:21.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.135+0000 7fb43ea00700 1 --2- 172.21.15.116:0/847663567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f77d0 0x557d5f106b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.135+0000 7fb43ea00700 1 -- 172.21.15.116:0/847663567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f8fd0 msgr2=0x557d5f108060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T01:05:21.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.135+0000 7fb43ea00700 1 --2- 172.21.15.116:0/847663567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f8fd0 0x557d5f108060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.135+0000 7fb43ea00700 1 -- 172.21.15.116:0/847663567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d5ee76b60 con 0x557d5f0f8130 2024-03-30T01:05:21.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.135+0000 7fb43fe00700 1 --2- 172.21.15.116:0/847663567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f8fd0 0x557d5f108060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:05:21.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.135+0000 7fb43f400700 1 --2- 172.21.15.116:0/847663567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f77d0 0x557d5f106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:05:21.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.135+0000 7fb43ea00700 1 --2- 172.21.15.116:0/847663567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f107050 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00b600 tx=0x7fb42c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cc4ac1d75040af server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:05:21.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.135+0000 7fb429600700 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-30T01:05:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.135+0000 7fb429600700 1 -- 172.21.15.116:0/847663567 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c00f670 con 0x557d5f0f8130 2024-03-30T01:05:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.135+0000 7fb429600700 1 -- 172.21.15.116:0/847663567 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb42c004b30 con 0x557d5f0f8130 2024-03-30T01:05:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.135+0000 7fb45a000b80 1 -- 172.21.15.116:0/847663567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb42c01e040 con 0x557d5f0f8130 2024-03-30T01:05:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.136+0000 7fb429600700 1 -- 172.21.15.116:0/847663567 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb42c02c5d0 con 0x557d5f0f8130 2024-03-30T01:05:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.136+0000 7fb45a000b80 10 client.16350 fetch_fsmap learned FSMap version 30 2024-03-30T01:05:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.136+0000 7fb45a000b80 1 -- 172.21.15.116:0/847663567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fb43801a470 con 0x557d5f0f8130 2024-03-30T01:05:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.136+0000 7fb429600700 1 -- 172.21.15.116:0/847663567 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb42c01e040 con 0x557d5f0f8130 2024-03-30T01:05:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.136+0000 7fb429600700 1 -- 172.21.15.116:0/847663567 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fb42c035040 con 0x557d5f0f8130 2024-03-30T01:05:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.136+0000 7fb45a000b80 10 client.16350 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T01:05:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.136+0000 7fb45a000b80 10 client.16350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T01:05:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.136+0000 7fb45a000b80 20 client.16350 populate_metadata read hostname 'smithi116' 2024-03-30T01:05:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.136+0000 7fb45a000b80 1 --2- 172.21.15.116:0/847663567 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557d5f161b70 0x557d5f163f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:05:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.136+0000 7fb45a000b80 4 client.16350 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T01:05:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.136+0000 7fb45a000b80 1 -- 172.21.15.116:0/847663567 --> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557d5ee97cf0 con 0x557d5f161b70 2024-03-30T01:05:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.136+0000 7fb43f400700 1 --2- 172.21.15.116:0/847663567 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557d5f161b70 0x557d5f163f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:05:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.137+0000 7fb43f400700 1 --2- 172.21.15.116:0/847663567 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557d5f161b70 0x557d5f163f90 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8b83de58cf215b0 server_cookie=ade3f67b6b0931c2 in_seq=0 out_seq=0 2024-03-30T01:05:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.137+0000 7fb429600700 10 client.16350 ms_handle_connect on v2:172.21.15.130:6838/1184545085 2024-03-30T01:05:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.137+0000 7fb429600700 1 -- 172.21.15.116:0/847663567 <== mds.0 v2:172.21.15.130:6838/1184545085 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557d5ee97cf0 con 0x557d5f161b70 2024-03-30T01:05:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.137+0000 7fb429600700 10 client.16350 handle_command_reply: tid=0 2024-03-30T01:05:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.137+0000 7fb45a000b80 1 client.16350 shutdown 2024-03-30T01:05:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.137+0000 7fb45a000b80 1 -- 172.21.15.116:0/847663567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 msgr2=0x557d5f107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.137+0000 7fb45a000b80 1 --2- 172.21.15.116:0/847663567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f107050 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00b600 tx=0x7fb42c0036f0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.137+0000 7fb45a000b80 1 -- 172.21.15.116:0/847663567 shutdown_connections 2024-03-30T01:05:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.137+0000 7fb45a000b80 1 --2- 172.21.15.116:0/847663567 >> [v2:172.21.15.130:6838/1184545085,v1:172.21.15.130:6839/1184545085] conn(0x557d5f161b70 0x557d5f163f90 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.137+0000 7fb45a000b80 1 --2- 172.21.15.116:0/847663567 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d5f0f77d0 0x557d5f106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.137+0000 7fb45a000b80 1 --2- 172.21.15.116:0/847663567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d5f0f8fd0 0x557d5f108060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.137+0000 7fb429600700 0 client.16350 ms_handle_reset on v2:172.21.15.130:6838/1184545085 2024-03-30T01:05:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.137+0000 7fb45a000b80 1 --2- 172.21.15.116:0/847663567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5f0f8130 0x557d5f107050 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.137+0000 7fb45a000b80 1 -- 172.21.15.116:0/847663567 >> 172.21.15.116:0/847663567 conn(0x557d5f0566a0 msgr2=0x557d5f0eb5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.138+0000 7fb45a000b80 1 -- 172.21.15.116:0/847663567 shutdown_connections 2024-03-30T01:05:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.138+0000 7fb45a000b80 1 -- 172.21.15.116:0/847663567 wait complete. 2024-03-30T01:05:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.138+0000 7fb45a000b80 20 client.16350 trim_cache size 0 max 16384 2024-03-30T01:05:21.149 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-30T01:05:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.138+0000 7fb446400700 1 -- 172.21.15.116:0/71823714 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4240a1120 msgr2=0x7fb4240a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.138+0000 7fb446400700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4240a1120 0x7fb4240a35e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb438007770 tx=0x7fb438007b30 comp rx=0 tx=0).stop 2024-03-30T01:05:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.138+0000 7fb446400700 1 -- 172.21.15.116:0/71823714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb440100d80 msgr2=0x7fb440074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:05:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.138+0000 7fb446400700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb440100d80 0x7fb440074b10 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fb43000ea50 tx=0x7fb43000ed60 comp rx=0 tx=0).stop 2024-03-30T01:05:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.138+0000 7fb446400700 1 -- 172.21.15.116:0/71823714 shutdown_connections 2024-03-30T01:05:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.138+0000 7fb446400700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb4240a1120 0x7fb4240a35e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.138+0000 7fb446400700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb440102910 0x7fb4400736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.138+0000 7fb446400700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb440100d80 0x7fb440074b10 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.138+0000 7fb446400700 1 --2- 172.21.15.116:0/71823714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb440101f70 0x7fb4400731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:05:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.138+0000 7fb446400700 1 -- 172.21.15.116:0/71823714 >> 172.21.15.116:0/71823714 conn(0x7fb4400fc2f0 msgr2=0x7fb440105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:05:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.139+0000 7fb446400700 1 -- 172.21.15.116:0/71823714 shutdown_connections 2024-03-30T01:05:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:05:21.139+0000 7fb446400700 1 -- 172.21.15.116:0/71823714 wait complete. 2024-03-30T01:05:21.162 ERROR:tasks.fwd_scrub:rank 2 damaged: [{'damage_type': 'backtrace', 'id': 439204740, 'ino': 1099511639927, 'path': '/volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/picture-50.jpg'}, {'damage_type': 'backtrace', 'id': 2519282214, 'ino': 1099511639932, 'path': '/volumes/qa/sv_1/2e70c87b-dd51-4566-ab0f-a850e31dba64/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-7/article-58.xml'}] 2024-03-30T01:05:21.162 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/fwd_scrub.py", line 90, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2024-03-30T01:05:22.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:22 smithi130 ceph-mon[47626]: pgmap v516: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 56 KiB/s wr, 17 op/s 2024-03-30T01:05:22.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:22 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:05:22.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:22 smithi116 ceph-mon[38086]: pgmap v516: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 56 KiB/s wr, 17 op/s 2024-03-30T01:05:22.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:22 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:05:23.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:22 smithi181 ceph-mon[47622]: pgmap v516: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 56 KiB/s wr, 17 op/s 2024-03-30T01:05:23.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:22 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:05:23.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi181 to 182.4M 2024-03-30T01:05:23.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: Unable to set osd_memory_target on smithi181 to 191343308: error parsing value: Value '191343308' is below minimum 939524096 2024-03-30T01:05:23.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.537 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.537 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.537 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.537 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.537 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi181 to 182.4M 2024-03-30T01:05:23.537 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: Unable to set osd_memory_target on smithi181 to 191343308: error parsing value: Value '191343308' is below minimum 939524096 2024-03-30T01:05:23.537 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.537 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.537 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.538 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.538 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.538 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.538 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.538 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.538 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:05:23.538 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:05:23.538 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:05:23.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi181 to 182.4M 2024-03-30T01:05:23.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: Unable to set osd_memory_target on smithi181 to 191343308: error parsing value: Value '191343308' is below minimum 939524096 2024-03-30T01:05:23.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:23.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:05:23.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:05:23.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:05:24.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:24.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:24.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:24.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:24.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:24.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:24.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T01:05:24.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:05:24.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:05:24.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:05:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:24 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi130 to 182.4M 2024-03-30T01:05:24.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:24 smithi130 ceph-mon[47626]: Unable to set osd_memory_target on smithi130 to 191356928: error parsing value: Value '191356928' is below minimum 939524096 2024-03-30T01:05:24.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:24 smithi130 ceph-mon[47626]: Adjusting osd_memory_target on smithi116 to 221.5M 2024-03-30T01:05:24.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:24 smithi130 ceph-mon[47626]: Unable to set osd_memory_target on smithi116 to 232352870: error parsing value: Value '232352870' is below minimum 939524096 2024-03-30T01:05:24.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:24 smithi130 ceph-mon[47626]: pgmap v517: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 57 KiB/s wr, 16 op/s 2024-03-30T01:05:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:24 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi130 to 182.4M 2024-03-30T01:05:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:24 smithi116 ceph-mon[38086]: Unable to set osd_memory_target on smithi130 to 191356928: error parsing value: Value '191356928' is below minimum 939524096 2024-03-30T01:05:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:24 smithi116 ceph-mon[38086]: Adjusting osd_memory_target on smithi116 to 221.5M 2024-03-30T01:05:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:24 smithi116 ceph-mon[38086]: Unable to set osd_memory_target on smithi116 to 232352870: error parsing value: Value '232352870' is below minimum 939524096 2024-03-30T01:05:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:24 smithi116 ceph-mon[38086]: pgmap v517: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 57 KiB/s wr, 16 op/s 2024-03-30T01:05:25.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:24 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi130 to 182.4M 2024-03-30T01:05:25.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:24 smithi181 ceph-mon[47622]: Unable to set osd_memory_target on smithi130 to 191356928: error parsing value: Value '191356928' is below minimum 939524096 2024-03-30T01:05:25.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:24 smithi181 ceph-mon[47622]: Adjusting osd_memory_target on smithi116 to 221.5M 2024-03-30T01:05:25.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:24 smithi181 ceph-mon[47622]: Unable to set osd_memory_target on smithi116 to 232352870: error parsing value: Value '232352870' is below minimum 939524096 2024-03-30T01:05:25.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:24 smithi181 ceph-mon[47622]: pgmap v517: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 57 KiB/s wr, 16 op/s 2024-03-30T01:05:26.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:26 smithi130 ceph-mon[47626]: pgmap v518: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-30T01:05:26.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:26 smithi116 ceph-mon[38086]: pgmap v518: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-30T01:05:27.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:26 smithi181 ceph-mon[47622]: pgmap v518: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-30T01:05:27.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:27 smithi130 ceph-mon[47626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:27.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:27 smithi116 ceph-mon[38086]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:28.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:27 smithi181 ceph-mon[47622]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:28.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:28 smithi130 ceph-mon[47626]: pgmap v519: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 26 KiB/s wr, 15 op/s 2024-03-30T01:05:28.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:28 smithi116 ceph-mon[38086]: pgmap v519: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 26 KiB/s wr, 15 op/s 2024-03-30T01:05:29.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:28 smithi181 ceph-mon[47622]: pgmap v519: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 26 KiB/s wr, 15 op/s 2024-03-30T01:05:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:30 smithi130 ceph-mon[47626]: pgmap v520: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 17 KiB/s wr, 24 op/s 2024-03-30T01:05:30.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:30 smithi116 ceph-mon[38086]: pgmap v520: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 17 KiB/s wr, 24 op/s 2024-03-30T01:05:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:30 smithi181 ceph-mon[47622]: pgmap v520: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 17 KiB/s wr, 24 op/s 2024-03-30T01:05:31.890 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-03-30T01:05:31.890 INFO:tasks.workunit.client.0.smithi116.stdout:Final score for writes: 517 2024-03-30T01:05:31.891 INFO:tasks.workunit.client.0.smithi116.stdout:Final score for reads : 296659 2024-03-30T01:05:31.891 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-03-30T01:05:31.892 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-30T01:05:31.892 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-30T01:05:32.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:32 smithi130 ceph-mon[47626]: pgmap v521: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 173 KiB/s wr, 42 op/s 2024-03-30T01:05:32.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:32 smithi130 ceph-mon[47626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:32.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:32 smithi116 ceph-mon[38086]: pgmap v521: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 173 KiB/s wr, 42 op/s 2024-03-30T01:05:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:32 smithi116 ceph-mon[38086]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:33.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:32 smithi181 ceph-mon[47622]: pgmap v521: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 173 KiB/s wr, 42 op/s 2024-03-30T01:05:33.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:32 smithi181 ceph-mon[47622]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:34.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:34 smithi116 ceph-mon[38086]: pgmap v522: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 237 KiB/s wr, 47 op/s 2024-03-30T01:05:35.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:34 smithi181 ceph-mon[47622]: pgmap v522: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 237 KiB/s wr, 47 op/s 2024-03-30T01:05:35.099 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:34 smithi130 ceph-mon[47626]: pgmap v522: 129 pgs: 129 active+clean; 2.2 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 237 KiB/s wr, 47 op/s 2024-03-30T01:05:36.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:36 smithi116 ceph-mon[38086]: pgmap v523: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 851 KiB/s wr, 211 op/s 2024-03-30T01:05:37.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:36 smithi181 ceph-mon[47622]: pgmap v523: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 851 KiB/s wr, 211 op/s 2024-03-30T01:05:37.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:36 smithi130 ceph-mon[47626]: pgmap v523: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 851 KiB/s wr, 211 op/s 2024-03-30T01:05:38.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:38 smithi116 ceph-mon[38086]: pgmap v524: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.1 MiB/s wr, 385 op/s 2024-03-30T01:05:39.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:38 smithi181 ceph-mon[47622]: pgmap v524: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.1 MiB/s wr, 385 op/s 2024-03-30T01:05:39.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:38 smithi130 ceph-mon[47626]: pgmap v524: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.1 MiB/s wr, 385 op/s 2024-03-30T01:05:40.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:40 smithi116 ceph-mon[38086]: pgmap v525: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.2 MiB/s wr, 707 op/s 2024-03-30T01:05:40.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:40 smithi116 ceph-mon[38086]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:40 smithi181 ceph-mon[47622]: pgmap v525: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.2 MiB/s wr, 707 op/s 2024-03-30T01:05:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:40 smithi181 ceph-mon[47622]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:41.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:40 smithi130 ceph-mon[47626]: pgmap v525: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.2 MiB/s wr, 707 op/s 2024-03-30T01:05:41.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:40 smithi130 ceph-mon[47626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:42.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:42 smithi116 ceph-mon[38086]: pgmap v526: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 791 op/s 2024-03-30T01:05:43.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:42 smithi181 ceph-mon[47622]: pgmap v526: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 791 op/s 2024-03-30T01:05:43.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:42 smithi130 ceph-mon[47626]: pgmap v526: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 791 op/s 2024-03-30T01:05:44.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:44 smithi116 ceph-mon[38086]: pgmap v527: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 831 op/s 2024-03-30T01:05:45.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:44 smithi181 ceph-mon[47622]: pgmap v527: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 831 op/s 2024-03-30T01:05:45.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:44 smithi130 ceph-mon[47626]: pgmap v527: 129 pgs: 129 active+clean; 2.2 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 831 op/s 2024-03-30T01:05:46.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:46 smithi116 ceph-mon[38086]: pgmap v528: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 970 op/s 2024-03-30T01:05:47.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:46 smithi181 ceph-mon[47622]: pgmap v528: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 970 op/s 2024-03-30T01:05:47.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:46 smithi130 ceph-mon[47626]: pgmap v528: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 970 op/s 2024-03-30T01:05:48.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:48 smithi116 ceph-mon[38086]: pgmap v529: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 877 op/s 2024-03-30T01:05:49.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:48 smithi181 ceph-mon[47622]: pgmap v529: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 877 op/s 2024-03-30T01:05:49.098 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:48 smithi130 ceph-mon[47626]: pgmap v529: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 877 op/s 2024-03-30T01:05:50.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:50 smithi116 ceph-mon[38086]: pgmap v530: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 865 KiB/s wr, 735 op/s 2024-03-30T01:05:50.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:50 smithi116 ceph-mon[38086]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:51.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:50 smithi181 ceph-mon[47622]: pgmap v530: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 865 KiB/s wr, 735 op/s 2024-03-30T01:05:51.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:50 smithi181 ceph-mon[47622]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:51.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:50 smithi130 ceph-mon[47626]: pgmap v530: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 865 KiB/s wr, 735 op/s 2024-03-30T01:05:51.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:50 smithi130 ceph-mon[47626]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-30T01:05:52.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:52 smithi116 ceph-mon[38086]: pgmap v531: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 589 op/s 2024-03-30T01:05:53.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:52 smithi181 ceph-mon[47622]: pgmap v531: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 589 op/s 2024-03-30T01:05:53.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:52 smithi130 ceph-mon[47626]: pgmap v531: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 589 op/s 2024-03-30T01:05:54.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:54 smithi116 ceph-mon[38086]: pgmap v532: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 564 op/s 2024-03-30T01:05:55.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:54 smithi181 ceph-mon[47622]: pgmap v532: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 564 op/s 2024-03-30T01:05:55.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:54 smithi130 ceph-mon[47626]: pgmap v532: 129 pgs: 129 active+clean; 2.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 564 op/s 2024-03-30T01:05:56.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:56 smithi116 ceph-mon[38086]: pgmap v533: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 551 op/s 2024-03-30T01:05:57.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:56 smithi181 ceph-mon[47622]: pgmap v533: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 551 op/s 2024-03-30T01:05:57.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:56 smithi130 ceph-mon[47626]: pgmap v533: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 551 op/s 2024-03-30T01:05:58.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:05:58 smithi116 ceph-mon[38086]: pgmap v534: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 411 op/s 2024-03-30T01:05:59.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:05:58 smithi181 ceph-mon[47622]: pgmap v534: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 411 op/s 2024-03-30T01:05:59.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:05:58 smithi130 ceph-mon[47626]: pgmap v534: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 411 op/s 2024-03-30T01:06:00.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:00 smithi116 ceph-mon[38086]: pgmap v535: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 342 op/s 2024-03-30T01:06:01.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:00 smithi181 ceph-mon[47622]: pgmap v535: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 342 op/s 2024-03-30T01:06:01.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:00 smithi130 ceph-mon[47626]: pgmap v535: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 342 op/s 2024-03-30T01:06:02.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:02 smithi116 ceph-mon[38086]: pgmap v536: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.8 MiB/s wr, 350 op/s 2024-03-30T01:06:03.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:02 smithi181 ceph-mon[47622]: pgmap v536: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.8 MiB/s wr, 350 op/s 2024-03-30T01:06:03.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:02 smithi130 ceph-mon[47626]: pgmap v536: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.8 MiB/s wr, 350 op/s 2024-03-30T01:06:04.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:04 smithi116 ceph-mon[38086]: pgmap v537: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.7 MiB/s wr, 222 op/s 2024-03-30T01:06:05.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:04 smithi181 ceph-mon[47622]: pgmap v537: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.7 MiB/s wr, 222 op/s 2024-03-30T01:06:05.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:04 smithi130 ceph-mon[47626]: pgmap v537: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.7 MiB/s wr, 222 op/s 2024-03-30T01:06:06.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:06 smithi116 ceph-mon[38086]: pgmap v538: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.1 MiB/s wr, 349 op/s 2024-03-30T01:06:07.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:06 smithi181 ceph-mon[47622]: pgmap v538: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.1 MiB/s wr, 349 op/s 2024-03-30T01:06:07.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:06 smithi130 ceph-mon[47626]: pgmap v538: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.1 MiB/s wr, 349 op/s 2024-03-30T01:06:08.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:08 smithi116 ceph-mon[38086]: pgmap v539: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.3 MiB/s wr, 377 op/s 2024-03-30T01:06:09.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:08 smithi181 ceph-mon[47622]: pgmap v539: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.3 MiB/s wr, 377 op/s 2024-03-30T01:06:09.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:08 smithi130 ceph-mon[47626]: pgmap v539: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.3 MiB/s wr, 377 op/s 2024-03-30T01:06:10.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:10 smithi116 ceph-mon[38086]: pgmap v540: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 382 op/s 2024-03-30T01:06:11.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:10 smithi181 ceph-mon[47622]: pgmap v540: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 382 op/s 2024-03-30T01:06:11.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:10 smithi130 ceph-mon[47626]: pgmap v540: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 382 op/s 2024-03-30T01:06:13.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:12 smithi181 ceph-mon[47622]: pgmap v541: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.9 MiB/s wr, 488 op/s 2024-03-30T01:06:13.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:12 smithi130 ceph-mon[47626]: pgmap v541: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.9 MiB/s wr, 488 op/s 2024-03-30T01:06:13.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:12 smithi116 ceph-mon[38086]: pgmap v541: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.9 MiB/s wr, 488 op/s 2024-03-30T01:06:15.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:14 smithi181 ceph-mon[47622]: pgmap v542: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 511 op/s 2024-03-30T01:06:15.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:14 smithi130 ceph-mon[47626]: pgmap v542: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 511 op/s 2024-03-30T01:06:15.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:14 smithi116 ceph-mon[38086]: pgmap v542: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 511 op/s 2024-03-30T01:06:16.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:15 smithi181 ceph-mon[47622]: pgmap v543: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 546 op/s 2024-03-30T01:06:16.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:15 smithi130 ceph-mon[47626]: pgmap v543: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 546 op/s 2024-03-30T01:06:16.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:15 smithi116 ceph-mon[38086]: pgmap v543: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 546 op/s 2024-03-30T01:06:18.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:18 smithi181 ceph-mon[47622]: pgmap v544: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 402 op/s 2024-03-30T01:06:18.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:18 smithi130 ceph-mon[47626]: pgmap v544: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 402 op/s 2024-03-30T01:06:18.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:18 smithi116 ceph-mon[38086]: pgmap v544: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 402 op/s 2024-03-30T01:06:20.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:20 smithi181 ceph-mon[47622]: pgmap v545: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 350 op/s 2024-03-30T01:06:20.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:20 smithi130 ceph-mon[47626]: pgmap v545: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 350 op/s 2024-03-30T01:06:20.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:20 smithi116 ceph-mon[38086]: pgmap v545: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 350 op/s 2024-03-30T01:06:22.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:22 smithi130 ceph-mon[47626]: pgmap v546: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 439 op/s 2024-03-30T01:06:22.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:22 smithi116 ceph-mon[38086]: pgmap v546: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 439 op/s 2024-03-30T01:06:23.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:22 smithi181 ceph-mon[47622]: pgmap v546: 129 pgs: 129 active+clean; 2.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 439 op/s 2024-03-30T01:06:23.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:23 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:06:23.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:23 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:06:24.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:23 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:06:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:24 smithi130 ceph-mon[47626]: pgmap v547: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 391 op/s 2024-03-30T01:06:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:24 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:06:24.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:24 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:06:24.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:24 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:06:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:24 smithi116 ceph-mon[38086]: pgmap v547: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 391 op/s 2024-03-30T01:06:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:24 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:06:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:24 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:06:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:24 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:06:25.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:24 smithi181 ceph-mon[47622]: pgmap v547: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 391 op/s 2024-03-30T01:06:25.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:24 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:06:25.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:24 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:06:25.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:24 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:06:26.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:26 smithi130 ceph-mon[47626]: pgmap v548: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 431 op/s 2024-03-30T01:06:26.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:26 smithi116 ceph-mon[38086]: pgmap v548: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 431 op/s 2024-03-30T01:06:27.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:26 smithi181 ceph-mon[47622]: pgmap v548: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 431 op/s 2024-03-30T01:06:28.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:28 smithi130 ceph-mon[47626]: pgmap v549: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 395 op/s 2024-03-30T01:06:28.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:28 smithi116 ceph-mon[38086]: pgmap v549: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 395 op/s 2024-03-30T01:06:29.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:28 smithi181 ceph-mon[47622]: pgmap v549: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 395 op/s 2024-03-30T01:06:30.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:30 smithi130 ceph-mon[47626]: pgmap v550: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 366 op/s 2024-03-30T01:06:30.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:30 smithi116 ceph-mon[38086]: pgmap v550: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 366 op/s 2024-03-30T01:06:31.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:30 smithi181 ceph-mon[47622]: pgmap v550: 129 pgs: 129 active+clean; 2.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 366 op/s 2024-03-30T01:06:32.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:32 smithi130 ceph-mon[47626]: pgmap v551: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 445 op/s 2024-03-30T01:06:32.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:32 smithi116 ceph-mon[38086]: pgmap v551: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 445 op/s 2024-03-30T01:06:33.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:32 smithi181 ceph-mon[47622]: pgmap v551: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 445 op/s 2024-03-30T01:06:34.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:34 smithi130 ceph-mon[47626]: pgmap v552: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 391 op/s 2024-03-30T01:06:34.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:34 smithi116 ceph-mon[38086]: pgmap v552: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 391 op/s 2024-03-30T01:06:35.006 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:34 smithi181 ceph-mon[47622]: pgmap v552: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 391 op/s 2024-03-30T01:06:36.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:36 smithi130 ceph-mon[47626]: pgmap v553: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 436 op/s 2024-03-30T01:06:36.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:36 smithi116 ceph-mon[38086]: pgmap v553: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 436 op/s 2024-03-30T01:06:37.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:36 smithi181 ceph-mon[47622]: pgmap v553: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 436 op/s 2024-03-30T01:06:38.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:38 smithi130 ceph-mon[47626]: pgmap v554: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 386 op/s 2024-03-30T01:06:38.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:38 smithi116 ceph-mon[38086]: pgmap v554: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 386 op/s 2024-03-30T01:06:39.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:38 smithi181 ceph-mon[47622]: pgmap v554: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 386 op/s 2024-03-30T01:06:40.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:40 smithi130 ceph-mon[47626]: pgmap v555: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 361 op/s 2024-03-30T01:06:40.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:40 smithi116 ceph-mon[38086]: pgmap v555: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 361 op/s 2024-03-30T01:06:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:40 smithi181 ceph-mon[47622]: pgmap v555: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 361 op/s 2024-03-30T01:06:42.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:42 smithi130 ceph-mon[47626]: pgmap v556: 129 pgs: 129 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 443 op/s 2024-03-30T01:06:42.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:42 smithi116 ceph-mon[38086]: pgmap v556: 129 pgs: 129 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 443 op/s 2024-03-30T01:06:43.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:42 smithi181 ceph-mon[47622]: pgmap v556: 129 pgs: 129 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 443 op/s 2024-03-30T01:06:44.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:44 smithi130 ceph-mon[47626]: pgmap v557: 129 pgs: 129 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 384 op/s 2024-03-30T01:06:44.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:44 smithi116 ceph-mon[38086]: pgmap v557: 129 pgs: 129 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 384 op/s 2024-03-30T01:06:45.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:44 smithi181 ceph-mon[47622]: pgmap v557: 129 pgs: 129 active+clean; 2.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 384 op/s 2024-03-30T01:06:46.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:46 smithi130 ceph-mon[47626]: pgmap v558: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 427 op/s 2024-03-30T01:06:46.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:46 smithi116 ceph-mon[38086]: pgmap v558: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 427 op/s 2024-03-30T01:06:47.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:46 smithi181 ceph-mon[47622]: pgmap v558: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 427 op/s 2024-03-30T01:06:48.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:48 smithi130 ceph-mon[47626]: pgmap v559: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 370 op/s 2024-03-30T01:06:48.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:48 smithi116 ceph-mon[38086]: pgmap v559: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 370 op/s 2024-03-30T01:06:49.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:48 smithi181 ceph-mon[47622]: pgmap v559: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 370 op/s 2024-03-30T01:06:50.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:50 smithi130 ceph-mon[47626]: pgmap v560: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 348 op/s 2024-03-30T01:06:50.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:50 smithi116 ceph-mon[38086]: pgmap v560: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 348 op/s 2024-03-30T01:06:51.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:50 smithi181 ceph-mon[47622]: pgmap v560: 129 pgs: 129 active+clean; 2.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 348 op/s 2024-03-30T01:06:52.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:52 smithi130 ceph-mon[47626]: pgmap v561: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 459 op/s 2024-03-30T01:06:52.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:52 smithi116 ceph-mon[38086]: pgmap v561: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 459 op/s 2024-03-30T01:06:53.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:52 smithi181 ceph-mon[47622]: pgmap v561: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 459 op/s 2024-03-30T01:06:54.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:54 smithi116 ceph-mon[38086]: pgmap v562: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 414 op/s 2024-03-30T01:06:55.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:54 smithi181 ceph-mon[47622]: pgmap v562: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 414 op/s 2024-03-30T01:06:55.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:54 smithi130 ceph-mon[47626]: pgmap v562: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 414 op/s 2024-03-30T01:06:56.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:56 smithi116 ceph-mon[38086]: pgmap v563: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 502 op/s 2024-03-30T01:06:57.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:56 smithi181 ceph-mon[47622]: pgmap v563: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 502 op/s 2024-03-30T01:06:57.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:56 smithi130 ceph-mon[47626]: pgmap v563: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 502 op/s 2024-03-30T01:06:58.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:06:58 smithi116 ceph-mon[38086]: pgmap v564: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 481 op/s 2024-03-30T01:06:59.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:06:58 smithi181 ceph-mon[47622]: pgmap v564: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 481 op/s 2024-03-30T01:06:59.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:06:58 smithi130 ceph-mon[47626]: pgmap v564: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 481 op/s 2024-03-30T01:07:00.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:00 smithi116 ceph-mon[38086]: pgmap v565: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 471 op/s 2024-03-30T01:07:01.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:00 smithi181 ceph-mon[47622]: pgmap v565: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 471 op/s 2024-03-30T01:07:01.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:00 smithi130 ceph-mon[47626]: pgmap v565: 129 pgs: 129 active+clean; 1.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 471 op/s 2024-03-30T01:07:02.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:02 smithi116 ceph-mon[38086]: pgmap v566: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 608 op/s 2024-03-30T01:07:03.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:02 smithi181 ceph-mon[47622]: pgmap v566: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 608 op/s 2024-03-30T01:07:03.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:02 smithi130 ceph-mon[47626]: pgmap v566: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 608 op/s 2024-03-30T01:07:04.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:04 smithi116 ceph-mon[38086]: pgmap v567: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 508 op/s 2024-03-30T01:07:05.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:04 smithi181 ceph-mon[47622]: pgmap v567: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 508 op/s 2024-03-30T01:07:05.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:04 smithi130 ceph-mon[47626]: pgmap v567: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 508 op/s 2024-03-30T01:07:06.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:06 smithi116 ceph-mon[38086]: pgmap v568: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 563 op/s 2024-03-30T01:07:07.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:06 smithi181 ceph-mon[47622]: pgmap v568: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 563 op/s 2024-03-30T01:07:07.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:06 smithi130 ceph-mon[47626]: pgmap v568: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 563 op/s 2024-03-30T01:07:08.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:08 smithi116 ceph-mon[38086]: pgmap v569: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 485 op/s 2024-03-30T01:07:09.007 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:08 smithi181 ceph-mon[47622]: pgmap v569: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 485 op/s 2024-03-30T01:07:09.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:08 smithi130 ceph-mon[47626]: pgmap v569: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 485 op/s 2024-03-30T01:07:10.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:10 smithi116 ceph-mon[38086]: pgmap v570: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 438 op/s 2024-03-30T01:07:11.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:10 smithi181 ceph-mon[47622]: pgmap v570: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 438 op/s 2024-03-30T01:07:11.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:10 smithi130 ceph-mon[47626]: pgmap v570: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 438 op/s 2024-03-30T01:07:12.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:12 smithi116 ceph-mon[38086]: pgmap v571: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 488 op/s 2024-03-30T01:07:13.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:12 smithi181 ceph-mon[47622]: pgmap v571: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 488 op/s 2024-03-30T01:07:13.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:12 smithi130 ceph-mon[47626]: pgmap v571: 129 pgs: 129 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 488 op/s 2024-03-30T01:07:14.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:14 smithi116 ceph-mon[38086]: pgmap v572: 129 pgs: 129 active+clean; 1.9 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 382 op/s 2024-03-30T01:07:15.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:14 smithi181 ceph-mon[47622]: pgmap v572: 129 pgs: 129 active+clean; 1.9 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 382 op/s 2024-03-30T01:07:15.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:14 smithi130 ceph-mon[47626]: pgmap v572: 129 pgs: 129 active+clean; 1.9 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 382 op/s 2024-03-30T01:07:16.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:16 smithi116 ceph-mon[38086]: pgmap v573: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 448 op/s 2024-03-30T01:07:17.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:16 smithi181 ceph-mon[47622]: pgmap v573: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 448 op/s 2024-03-30T01:07:17.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:16 smithi130 ceph-mon[47626]: pgmap v573: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 448 op/s 2024-03-30T01:07:18.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:18 smithi116 ceph-mon[38086]: pgmap v574: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 373 op/s 2024-03-30T01:07:19.017 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:18 smithi181 ceph-mon[47622]: pgmap v574: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 373 op/s 2024-03-30T01:07:19.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:18 smithi130 ceph-mon[47626]: pgmap v574: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 373 op/s 2024-03-30T01:07:20.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:20 smithi116 ceph-mon[38086]: pgmap v575: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 336 op/s 2024-03-30T01:07:21.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:20 smithi181 ceph-mon[47622]: pgmap v575: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 336 op/s 2024-03-30T01:07:21.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:20 smithi130 ceph-mon[47626]: pgmap v575: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 336 op/s 2024-03-30T01:07:23.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:22 smithi130 ceph-mon[47626]: pgmap v576: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 414 op/s 2024-03-30T01:07:23.104 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:22 smithi181 ceph-mon[47622]: pgmap v576: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 414 op/s 2024-03-30T01:07:23.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:22 smithi116 ceph-mon[38086]: pgmap v576: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 414 op/s 2024-03-30T01:07:25.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:24 smithi130 ceph-mon[47626]: pgmap v577: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 381 op/s 2024-03-30T01:07:25.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:24 smithi116 ceph-mon[38086]: pgmap v577: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 381 op/s 2024-03-30T01:07:25.307 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:24 smithi181 ceph-mon[47622]: pgmap v577: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 381 op/s 2024-03-30T01:07:26.005 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:25 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:07:26.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:25 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:07:26.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:25 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:07:27.008 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:26 smithi181 ceph-mon[47622]: pgmap v578: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 426 op/s 2024-03-30T01:07:27.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:26 smithi130 ceph-mon[47626]: pgmap v578: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 426 op/s 2024-03-30T01:07:27.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:26 smithi116 ceph-mon[38086]: pgmap v578: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 426 op/s 2024-03-30T01:07:28.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:28 smithi130 ceph-mon[47626]: pgmap v579: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 378 op/s 2024-03-30T01:07:28.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:28 smithi116 ceph-mon[38086]: pgmap v579: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 378 op/s 2024-03-30T01:07:29.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:28 smithi181 ceph-mon[47622]: pgmap v579: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 378 op/s 2024-03-30T01:07:30.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:30 smithi130 ceph-mon[47626]: pgmap v580: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 567 op/s 2024-03-30T01:07:30.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:30 smithi116 ceph-mon[38086]: pgmap v580: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 567 op/s 2024-03-30T01:07:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:30 smithi181 ceph-mon[47622]: pgmap v580: 129 pgs: 129 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 567 op/s 2024-03-30T01:07:32.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:32 smithi130 ceph-mon[47626]: pgmap v581: 129 pgs: 129 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 1.31k op/s 2024-03-30T01:07:32.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:07:32.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:07:32.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:07:32.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:32 smithi116 ceph-mon[38086]: pgmap v581: 129 pgs: 129 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 1.31k op/s 2024-03-30T01:07:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:07:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:07:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:07:33.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:32 smithi181 ceph-mon[47622]: pgmap v581: 129 pgs: 129 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 1.31k op/s 2024-03-30T01:07:33.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:07:33.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:07:33.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:07:34.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:34 smithi130 ceph-mon[47626]: pgmap v582: 129 pgs: 129 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 1.1 MiB/s wr, 1.41k op/s 2024-03-30T01:07:34.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:34 smithi116 ceph-mon[38086]: pgmap v582: 129 pgs: 129 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 1.1 MiB/s wr, 1.41k op/s 2024-03-30T01:07:35.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:34 smithi181 ceph-mon[47622]: pgmap v582: 129 pgs: 129 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 1.1 MiB/s wr, 1.41k op/s 2024-03-30T01:07:36.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:36 smithi130 ceph-mon[47626]: pgmap v583: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 1.47k op/s 2024-03-30T01:07:36.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:36 smithi116 ceph-mon[38086]: pgmap v583: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 1.47k op/s 2024-03-30T01:07:37.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:36 smithi181 ceph-mon[47622]: pgmap v583: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 1.47k op/s 2024-03-30T01:07:38.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:38 smithi130 ceph-mon[47626]: pgmap v584: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 1.1 MiB/s wr, 1.39k op/s 2024-03-30T01:07:38.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:38 smithi116 ceph-mon[38086]: pgmap v584: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 1.1 MiB/s wr, 1.39k op/s 2024-03-30T01:07:39.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:38 smithi181 ceph-mon[47622]: pgmap v584: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 1.1 MiB/s wr, 1.39k op/s 2024-03-30T01:07:40.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:40 smithi130 ceph-mon[47626]: pgmap v585: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.0 MiB/s wr, 1.38k op/s 2024-03-30T01:07:40.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:40 smithi116 ceph-mon[38086]: pgmap v585: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.0 MiB/s wr, 1.38k op/s 2024-03-30T01:07:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:40 smithi181 ceph-mon[47622]: pgmap v585: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.0 MiB/s wr, 1.38k op/s 2024-03-30T01:07:42.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:42 smithi130 ceph-mon[47626]: pgmap v586: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 1.26k op/s 2024-03-30T01:07:42.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:42 smithi116 ceph-mon[38086]: pgmap v586: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 1.26k op/s 2024-03-30T01:07:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:42 smithi181 ceph-mon[47622]: pgmap v586: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 1.26k op/s 2024-03-30T01:07:44.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:44 smithi130 ceph-mon[47626]: pgmap v587: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 531 op/s 2024-03-30T01:07:44.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:44 smithi116 ceph-mon[38086]: pgmap v587: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 531 op/s 2024-03-30T01:07:45.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:44 smithi181 ceph-mon[47622]: pgmap v587: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 531 op/s 2024-03-30T01:07:46.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:46 smithi130 ceph-mon[47626]: pgmap v588: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 404 op/s 2024-03-30T01:07:46.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:46 smithi116 ceph-mon[38086]: pgmap v588: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 404 op/s 2024-03-30T01:07:47.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:46 smithi181 ceph-mon[47622]: pgmap v588: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 404 op/s 2024-03-30T01:07:48.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:48 smithi130 ceph-mon[47626]: pgmap v589: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 318 op/s 2024-03-30T01:07:48.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:48 smithi116 ceph-mon[38086]: pgmap v589: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 318 op/s 2024-03-30T01:07:49.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:48 smithi181 ceph-mon[47622]: pgmap v589: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 318 op/s 2024-03-30T01:07:50.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:50 smithi130 ceph-mon[47626]: pgmap v590: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 346 op/s 2024-03-30T01:07:50.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:50 smithi116 ceph-mon[38086]: pgmap v590: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 346 op/s 2024-03-30T01:07:51.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:50 smithi181 ceph-mon[47622]: pgmap v590: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 346 op/s 2024-03-30T01:07:52.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:52 smithi116 ceph-mon[38086]: pgmap v591: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 446 op/s 2024-03-30T01:07:53.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:52 smithi181 ceph-mon[47622]: pgmap v591: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 446 op/s 2024-03-30T01:07:53.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:52 smithi130 ceph-mon[47626]: pgmap v591: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 446 op/s 2024-03-30T01:07:54.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:54 smithi116 ceph-mon[38086]: pgmap v592: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 403 op/s 2024-03-30T01:07:55.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:54 smithi181 ceph-mon[47622]: pgmap v592: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 403 op/s 2024-03-30T01:07:55.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:54 smithi130 ceph-mon[47626]: pgmap v592: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 403 op/s 2024-03-30T01:07:56.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:56 smithi116 ceph-mon[38086]: pgmap v593: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 490 op/s 2024-03-30T01:07:57.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:56 smithi181 ceph-mon[47622]: pgmap v593: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 490 op/s 2024-03-30T01:07:57.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:56 smithi130 ceph-mon[47626]: pgmap v593: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 490 op/s 2024-03-30T01:07:58.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:07:58 smithi116 ceph-mon[38086]: pgmap v594: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 462 op/s 2024-03-30T01:07:59.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:07:58 smithi181 ceph-mon[47622]: pgmap v594: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 462 op/s 2024-03-30T01:07:59.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:07:58 smithi130 ceph-mon[47626]: pgmap v594: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 462 op/s 2024-03-30T01:08:00.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:00 smithi116 ceph-mon[38086]: pgmap v595: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 474 op/s 2024-03-30T01:08:01.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:00 smithi181 ceph-mon[47622]: pgmap v595: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 474 op/s 2024-03-30T01:08:01.099 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:00 smithi130 ceph-mon[47626]: pgmap v595: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 474 op/s 2024-03-30T01:08:02.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:02 smithi116 ceph-mon[38086]: pgmap v596: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 548 op/s 2024-03-30T01:08:03.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:02 smithi181 ceph-mon[47622]: pgmap v596: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 548 op/s 2024-03-30T01:08:03.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:02 smithi130 ceph-mon[47626]: pgmap v596: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 548 op/s 2024-03-30T01:08:04.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:04 smithi116 ceph-mon[38086]: pgmap v597: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 536 op/s 2024-03-30T01:08:05.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:04 smithi181 ceph-mon[47622]: pgmap v597: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 536 op/s 2024-03-30T01:08:05.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:04 smithi130 ceph-mon[47626]: pgmap v597: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 536 op/s 2024-03-30T01:08:06.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:06 smithi116 ceph-mon[38086]: pgmap v598: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 651 op/s 2024-03-30T01:08:07.005 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:06 smithi181 ceph-mon[47622]: pgmap v598: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 651 op/s 2024-03-30T01:08:07.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:06 smithi130 ceph-mon[47626]: pgmap v598: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 651 op/s 2024-03-30T01:08:08.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:08 smithi116 ceph-mon[38086]: pgmap v599: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 603 op/s 2024-03-30T01:08:09.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:08 smithi181 ceph-mon[47622]: pgmap v599: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 603 op/s 2024-03-30T01:08:09.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:08 smithi130 ceph-mon[47626]: pgmap v599: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 603 op/s 2024-03-30T01:08:10.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:10 smithi116 ceph-mon[38086]: pgmap v600: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 625 op/s 2024-03-30T01:08:11.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:10 smithi181 ceph-mon[47622]: pgmap v600: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 625 op/s 2024-03-30T01:08:11.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:10 smithi130 ceph-mon[47626]: pgmap v600: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 625 op/s 2024-03-30T01:08:12.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:12 smithi116 ceph-mon[38086]: pgmap v601: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 686 op/s 2024-03-30T01:08:13.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:12 smithi181 ceph-mon[47622]: pgmap v601: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 686 op/s 2024-03-30T01:08:13.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:12 smithi130 ceph-mon[47626]: pgmap v601: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 686 op/s 2024-03-30T01:08:14.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:14 smithi116 ceph-mon[38086]: pgmap v602: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 638 op/s 2024-03-30T01:08:15.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:14 smithi181 ceph-mon[47622]: pgmap v602: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 638 op/s 2024-03-30T01:08:15.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:14 smithi130 ceph-mon[47626]: pgmap v602: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 638 op/s 2024-03-30T01:08:16.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:16 smithi116 ceph-mon[38086]: pgmap v603: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 609 op/s 2024-03-30T01:08:17.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:16 smithi181 ceph-mon[47622]: pgmap v603: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 609 op/s 2024-03-30T01:08:17.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:16 smithi130 ceph-mon[47626]: pgmap v603: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 609 op/s 2024-03-30T01:08:18.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:18 smithi116 ceph-mon[38086]: pgmap v604: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 467 op/s 2024-03-30T01:08:19.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:18 smithi181 ceph-mon[47622]: pgmap v604: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 467 op/s 2024-03-30T01:08:19.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:18 smithi130 ceph-mon[47626]: pgmap v604: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 467 op/s 2024-03-30T01:08:20.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:20 smithi116 ceph-mon[38086]: pgmap v605: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 468 op/s 2024-03-30T01:08:21.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:20 smithi181 ceph-mon[47622]: pgmap v605: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 468 op/s 2024-03-30T01:08:21.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:20 smithi130 ceph-mon[47626]: pgmap v605: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 468 op/s 2024-03-30T01:08:23.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:22 smithi181 ceph-mon[47622]: pgmap v606: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 504 op/s 2024-03-30T01:08:23.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:22 smithi130 ceph-mon[47626]: pgmap v606: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 504 op/s 2024-03-30T01:08:23.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:22 smithi116 ceph-mon[38086]: pgmap v606: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 504 op/s 2024-03-30T01:08:25.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:24 smithi181 ceph-mon[47622]: pgmap v607: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 453 op/s 2024-03-30T01:08:25.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:24 smithi130 ceph-mon[47626]: pgmap v607: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 453 op/s 2024-03-30T01:08:25.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:24 smithi116 ceph-mon[38086]: pgmap v607: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 453 op/s 2024-03-30T01:08:27.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:26 smithi181 ceph-mon[47622]: pgmap v608: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 449 op/s 2024-03-30T01:08:27.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:26 smithi130 ceph-mon[47626]: pgmap v608: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 449 op/s 2024-03-30T01:08:27.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:26 smithi116 ceph-mon[38086]: pgmap v608: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 449 op/s 2024-03-30T01:08:29.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:28 smithi181 ceph-mon[47622]: pgmap v609: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 378 op/s 2024-03-30T01:08:29.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:28 smithi130 ceph-mon[47626]: pgmap v609: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 378 op/s 2024-03-30T01:08:29.177 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:28 smithi116 ceph-mon[38086]: pgmap v609: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 378 op/s 2024-03-30T01:08:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:30 smithi181 ceph-mon[47622]: pgmap v610: 129 pgs: 129 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 397 op/s 2024-03-30T01:08:31.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:30 smithi130 ceph-mon[47626]: pgmap v610: 129 pgs: 129 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 397 op/s 2024-03-30T01:08:31.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:30 smithi116 ceph-mon[38086]: pgmap v610: 129 pgs: 129 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 397 op/s 2024-03-30T01:08:33.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:32 smithi181 ceph-mon[47622]: pgmap v611: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 390 op/s 2024-03-30T01:08:33.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:32 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:08:33.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:32 smithi130 ceph-mon[47626]: pgmap v611: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 390 op/s 2024-03-30T01:08:33.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:32 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:08:33.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:32 smithi116 ceph-mon[38086]: pgmap v611: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 390 op/s 2024-03-30T01:08:33.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:32 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:08:34.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:34 smithi130 ceph-mon[47626]: pgmap v612: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 340 op/s 2024-03-30T01:08:34.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:34.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:34.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:34.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:34 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:35.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:34 smithi181 ceph-mon[47622]: pgmap v612: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 340 op/s 2024-03-30T01:08:35.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:35.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:35.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:35.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:34 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:35.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:34 smithi116 ceph-mon[38086]: pgmap v612: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 340 op/s 2024-03-30T01:08:35.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:35.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:35.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:35.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:34 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:37.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:36 smithi181 ceph-mon[47622]: pgmap v613: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 354 op/s 2024-03-30T01:08:37.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:36 smithi130 ceph-mon[47626]: pgmap v613: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 354 op/s 2024-03-30T01:08:37.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:36 smithi116 ceph-mon[38086]: pgmap v613: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 354 op/s 2024-03-30T01:08:38.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:38 smithi130 ceph-mon[47626]: pgmap v614: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 979 KiB/s wr, 305 op/s 2024-03-30T01:08:38.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:38 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:38.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:38 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:38.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:38 smithi116 ceph-mon[38086]: pgmap v614: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 979 KiB/s wr, 305 op/s 2024-03-30T01:08:38.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:38 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:38.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:38 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:39.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:38 smithi181 ceph-mon[47622]: pgmap v614: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 979 KiB/s wr, 305 op/s 2024-03-30T01:08:39.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:38 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:39.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:38 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:39.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:39 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:08:39.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:39 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:08:39.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:39 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:39.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:39 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:08:39.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:39 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:08:39.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:39 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:40.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:39 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:08:40.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:39 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:08:40.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:39 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:08:40.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:40 smithi130 ceph-mon[47626]: pgmap v615: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 332 op/s 2024-03-30T01:08:40.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:40 smithi116 ceph-mon[38086]: pgmap v615: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 332 op/s 2024-03-30T01:08:41.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:40 smithi181 ceph-mon[47622]: pgmap v615: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 332 op/s 2024-03-30T01:08:42.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:42 smithi130 ceph-mon[47626]: pgmap v616: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 330 op/s 2024-03-30T01:08:42.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:42 smithi116 ceph-mon[38086]: pgmap v616: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 330 op/s 2024-03-30T01:08:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:42 smithi181 ceph-mon[47622]: pgmap v616: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 330 op/s 2024-03-30T01:08:44.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:44 smithi130 ceph-mon[47626]: pgmap v617: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 982 KiB/s wr, 316 op/s 2024-03-30T01:08:44.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:44 smithi116 ceph-mon[38086]: pgmap v617: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 982 KiB/s wr, 316 op/s 2024-03-30T01:08:45.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:44 smithi181 ceph-mon[47622]: pgmap v617: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 982 KiB/s wr, 316 op/s 2024-03-30T01:08:46.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:46 smithi130 ceph-mon[47626]: pgmap v618: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 357 op/s 2024-03-30T01:08:46.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:46 smithi116 ceph-mon[38086]: pgmap v618: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 357 op/s 2024-03-30T01:08:47.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:46 smithi181 ceph-mon[47622]: pgmap v618: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 357 op/s 2024-03-30T01:08:48.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:48 smithi116 ceph-mon[38086]: pgmap v619: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 336 op/s 2024-03-30T01:08:49.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:48 smithi181 ceph-mon[47622]: pgmap v619: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 336 op/s 2024-03-30T01:08:49.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:48 smithi130 ceph-mon[47626]: pgmap v619: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 336 op/s 2024-03-30T01:08:50.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:50 smithi116 ceph-mon[38086]: pgmap v620: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 367 op/s 2024-03-30T01:08:51.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:50 smithi181 ceph-mon[47622]: pgmap v620: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 367 op/s 2024-03-30T01:08:51.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:50 smithi130 ceph-mon[47626]: pgmap v620: 129 pgs: 129 active+clean; 1.4 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 367 op/s 2024-03-30T01:08:52.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:52 smithi116 ceph-mon[38086]: pgmap v621: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 363 op/s 2024-03-30T01:08:53.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:52 smithi181 ceph-mon[47622]: pgmap v621: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 363 op/s 2024-03-30T01:08:53.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:52 smithi130 ceph-mon[47626]: pgmap v621: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 363 op/s 2024-03-30T01:08:54.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:54 smithi116 ceph-mon[38086]: pgmap v622: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 352 op/s 2024-03-30T01:08:55.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:54 smithi181 ceph-mon[47622]: pgmap v622: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 352 op/s 2024-03-30T01:08:55.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:54 smithi130 ceph-mon[47626]: pgmap v622: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 352 op/s 2024-03-30T01:08:56.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:56 smithi116 ceph-mon[38086]: pgmap v623: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 375 op/s 2024-03-30T01:08:57.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:56 smithi181 ceph-mon[47622]: pgmap v623: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 375 op/s 2024-03-30T01:08:57.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:56 smithi130 ceph-mon[47626]: pgmap v623: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 375 op/s 2024-03-30T01:08:58.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:08:58 smithi116 ceph-mon[38086]: pgmap v624: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 988 KiB/s wr, 319 op/s 2024-03-30T01:08:59.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:08:58 smithi181 ceph-mon[47622]: pgmap v624: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 988 KiB/s wr, 319 op/s 2024-03-30T01:08:59.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:08:58 smithi130 ceph-mon[47626]: pgmap v624: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 988 KiB/s wr, 319 op/s 2024-03-30T01:09:00.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:00 smithi116 ceph-mon[38086]: pgmap v625: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 954 KiB/s wr, 325 op/s 2024-03-30T01:09:01.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:00 smithi181 ceph-mon[47622]: pgmap v625: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 954 KiB/s wr, 325 op/s 2024-03-30T01:09:01.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:00 smithi130 ceph-mon[47626]: pgmap v625: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 954 KiB/s wr, 325 op/s 2024-03-30T01:09:02.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:02 smithi116 ceph-mon[38086]: pgmap v626: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 325 op/s 2024-03-30T01:09:03.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:02 smithi181 ceph-mon[47622]: pgmap v626: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 325 op/s 2024-03-30T01:09:03.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:02 smithi130 ceph-mon[47626]: pgmap v626: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 325 op/s 2024-03-30T01:09:04.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:04 smithi116 ceph-mon[38086]: pgmap v627: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 951 KiB/s wr, 326 op/s 2024-03-30T01:09:05.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:04 smithi181 ceph-mon[47622]: pgmap v627: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 951 KiB/s wr, 326 op/s 2024-03-30T01:09:05.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:04 smithi130 ceph-mon[47626]: pgmap v627: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 951 KiB/s wr, 326 op/s 2024-03-30T01:09:06.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:06 smithi116 ceph-mon[38086]: pgmap v628: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 354 op/s 2024-03-30T01:09:07.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:06 smithi181 ceph-mon[47622]: pgmap v628: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 354 op/s 2024-03-30T01:09:07.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:06 smithi130 ceph-mon[47626]: pgmap v628: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 354 op/s 2024-03-30T01:09:09.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:08 smithi181 ceph-mon[47622]: pgmap v629: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 305 op/s 2024-03-30T01:09:09.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:08 smithi130 ceph-mon[47626]: pgmap v629: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 305 op/s 2024-03-30T01:09:09.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:08 smithi116 ceph-mon[38086]: pgmap v629: 129 pgs: 129 active+clean; 1.3 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 305 op/s 2024-03-30T01:09:11.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:10 smithi181 ceph-mon[47622]: pgmap v630: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 354 op/s 2024-03-30T01:09:11.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:10 smithi130 ceph-mon[47626]: pgmap v630: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 354 op/s 2024-03-30T01:09:11.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:10 smithi116 ceph-mon[38086]: pgmap v630: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 354 op/s 2024-03-30T01:09:13.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:12 smithi181 ceph-mon[47622]: pgmap v631: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 377 op/s 2024-03-30T01:09:13.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:12 smithi130 ceph-mon[47626]: pgmap v631: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 377 op/s 2024-03-30T01:09:13.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:12 smithi116 ceph-mon[38086]: pgmap v631: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 377 op/s 2024-03-30T01:09:15.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:14 smithi181 ceph-mon[47622]: pgmap v632: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 370 op/s 2024-03-30T01:09:15.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:14 smithi130 ceph-mon[47626]: pgmap v632: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 370 op/s 2024-03-30T01:09:15.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:14 smithi116 ceph-mon[38086]: pgmap v632: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 370 op/s 2024-03-30T01:09:17.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:16 smithi181 ceph-mon[47622]: pgmap v633: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 384 op/s 2024-03-30T01:09:17.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:16 smithi130 ceph-mon[47626]: pgmap v633: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 384 op/s 2024-03-30T01:09:17.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:16 smithi116 ceph-mon[38086]: pgmap v633: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 384 op/s 2024-03-30T01:09:19.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:18 smithi181 ceph-mon[47622]: pgmap v634: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 327 op/s 2024-03-30T01:09:19.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:18 smithi130 ceph-mon[47626]: pgmap v634: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 327 op/s 2024-03-30T01:09:19.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:18 smithi116 ceph-mon[38086]: pgmap v634: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 327 op/s 2024-03-30T01:09:20.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:19 smithi130 ceph-mon[47626]: pgmap v635: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 373 op/s 2024-03-30T01:09:20.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:19 smithi116 ceph-mon[38086]: pgmap v635: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 373 op/s 2024-03-30T01:09:20.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:19 smithi181 ceph-mon[47622]: pgmap v635: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 373 op/s 2024-03-30T01:09:22.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:22 smithi181 ceph-mon[47622]: pgmap v636: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 374 op/s 2024-03-30T01:09:22.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:22 smithi130 ceph-mon[47626]: pgmap v636: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 374 op/s 2024-03-30T01:09:22.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:22 smithi116 ceph-mon[38086]: pgmap v636: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 374 op/s 2024-03-30T01:09:24.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:24 smithi130 ceph-mon[47626]: pgmap v637: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 1012 KiB/s wr, 362 op/s 2024-03-30T01:09:24.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:24 smithi116 ceph-mon[38086]: pgmap v637: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 1012 KiB/s wr, 362 op/s 2024-03-30T01:09:25.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:24 smithi181 ceph-mon[47622]: pgmap v637: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 1012 KiB/s wr, 362 op/s 2024-03-30T01:09:26.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:26 smithi116 ceph-mon[38086]: pgmap v638: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 382 op/s 2024-03-30T01:09:27.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:26 smithi181 ceph-mon[47622]: pgmap v638: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 382 op/s 2024-03-30T01:09:27.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:26 smithi130 ceph-mon[47626]: pgmap v638: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 382 op/s 2024-03-30T01:09:28.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:28 smithi116 ceph-mon[38086]: pgmap v639: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 332 op/s 2024-03-30T01:09:29.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:28 smithi181 ceph-mon[47622]: pgmap v639: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 332 op/s 2024-03-30T01:09:29.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:28 smithi130 ceph-mon[47626]: pgmap v639: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 332 op/s 2024-03-30T01:09:30.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:30 smithi116 ceph-mon[38086]: pgmap v640: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 388 op/s 2024-03-30T01:09:31.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:30 smithi181 ceph-mon[47622]: pgmap v640: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 388 op/s 2024-03-30T01:09:31.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:30 smithi130 ceph-mon[47626]: pgmap v640: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 388 op/s 2024-03-30T01:09:32.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:32 smithi116 ceph-mon[38086]: pgmap v641: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 375 op/s 2024-03-30T01:09:33.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:32 smithi181 ceph-mon[47622]: pgmap v641: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 375 op/s 2024-03-30T01:09:33.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:32 smithi130 ceph-mon[47626]: pgmap v641: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 375 op/s 2024-03-30T01:09:34.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:34 smithi116 ceph-mon[38086]: pgmap v642: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 342 op/s 2024-03-30T01:09:35.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:34 smithi181 ceph-mon[47622]: pgmap v642: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 342 op/s 2024-03-30T01:09:35.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:34 smithi130 ceph-mon[47626]: pgmap v642: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 342 op/s 2024-03-30T01:09:36.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:36 smithi116 ceph-mon[38086]: pgmap v643: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 354 op/s 2024-03-30T01:09:37.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:36 smithi181 ceph-mon[47622]: pgmap v643: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 354 op/s 2024-03-30T01:09:37.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:36 smithi130 ceph-mon[47626]: pgmap v643: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 354 op/s 2024-03-30T01:09:38.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:38 smithi116 ceph-mon[38086]: pgmap v644: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 296 op/s 2024-03-30T01:09:39.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:38 smithi181 ceph-mon[47622]: pgmap v644: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 296 op/s 2024-03-30T01:09:39.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:38 smithi130 ceph-mon[47626]: pgmap v644: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 296 op/s 2024-03-30T01:09:39.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:39 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:09:40.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:39 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:09:40.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:39 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:09:40.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:40 smithi116 ceph-mon[38086]: pgmap v645: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 360 op/s 2024-03-30T01:09:40.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:40 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:09:40.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:40 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:09:40.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:40 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:09:41.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:40 smithi181 ceph-mon[47622]: pgmap v645: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 360 op/s 2024-03-30T01:09:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:40 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:09:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:40 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:09:41.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:40 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:09:41.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:40 smithi130 ceph-mon[47626]: pgmap v645: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 360 op/s 2024-03-30T01:09:41.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:40 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:09:41.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:40 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:09:41.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:40 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:09:42.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:42 smithi116 ceph-mon[38086]: pgmap v646: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 377 op/s 2024-03-30T01:09:43.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:42 smithi181 ceph-mon[47622]: pgmap v646: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 377 op/s 2024-03-30T01:09:43.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:42 smithi130 ceph-mon[47626]: pgmap v646: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 377 op/s 2024-03-30T01:09:44.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:44 smithi116 ceph-mon[38086]: pgmap v647: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1016 KiB/s wr, 386 op/s 2024-03-30T01:09:45.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:44 smithi181 ceph-mon[47622]: pgmap v647: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1016 KiB/s wr, 386 op/s 2024-03-30T01:09:45.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:44 smithi130 ceph-mon[47626]: pgmap v647: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1016 KiB/s wr, 386 op/s 2024-03-30T01:09:46.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:46 smithi116 ceph-mon[38086]: pgmap v648: 129 pgs: 129 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 408 op/s 2024-03-30T01:09:47.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:46 smithi181 ceph-mon[47622]: pgmap v648: 129 pgs: 129 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 408 op/s 2024-03-30T01:09:47.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:46 smithi130 ceph-mon[47626]: pgmap v648: 129 pgs: 129 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 408 op/s 2024-03-30T01:09:48.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:48 smithi116 ceph-mon[38086]: pgmap v649: 129 pgs: 129 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 358 op/s 2024-03-30T01:09:49.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:48 smithi181 ceph-mon[47622]: pgmap v649: 129 pgs: 129 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 358 op/s 2024-03-30T01:09:49.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:48 smithi130 ceph-mon[47626]: pgmap v649: 129 pgs: 129 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 358 op/s 2024-03-30T01:09:50.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:50 smithi116 ceph-mon[38086]: pgmap v650: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 423 op/s 2024-03-30T01:09:51.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:50 smithi181 ceph-mon[47622]: pgmap v650: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 423 op/s 2024-03-30T01:09:51.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:50 smithi130 ceph-mon[47626]: pgmap v650: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 423 op/s 2024-03-30T01:09:52.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:52 smithi116 ceph-mon[38086]: pgmap v651: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 424 op/s 2024-03-30T01:09:53.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:52 smithi181 ceph-mon[47622]: pgmap v651: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 424 op/s 2024-03-30T01:09:53.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:52 smithi130 ceph-mon[47626]: pgmap v651: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 424 op/s 2024-03-30T01:09:54.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:54 smithi116 ceph-mon[38086]: pgmap v652: 129 pgs: 129 active+clean; 1016 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 383 op/s 2024-03-30T01:09:55.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:54 smithi181 ceph-mon[47622]: pgmap v652: 129 pgs: 129 active+clean; 1016 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 383 op/s 2024-03-30T01:09:55.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:54 smithi130 ceph-mon[47626]: pgmap v652: 129 pgs: 129 active+clean; 1016 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 383 op/s 2024-03-30T01:09:57.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:56 smithi181 ceph-mon[47622]: pgmap v653: 129 pgs: 129 active+clean; 1002 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 398 op/s 2024-03-30T01:09:57.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:56 smithi130 ceph-mon[47626]: pgmap v653: 129 pgs: 129 active+clean; 1002 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 398 op/s 2024-03-30T01:09:57.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:56 smithi116 ceph-mon[38086]: pgmap v653: 129 pgs: 129 active+clean; 1002 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 398 op/s 2024-03-30T01:09:59.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:09:58 smithi181 ceph-mon[47622]: pgmap v654: 129 pgs: 129 active+clean; 999 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 351 op/s 2024-03-30T01:09:59.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:09:58 smithi130 ceph-mon[47626]: pgmap v654: 129 pgs: 129 active+clean; 999 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 351 op/s 2024-03-30T01:09:59.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:09:58 smithi116 ceph-mon[38086]: pgmap v654: 129 pgs: 129 active+clean; 999 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 351 op/s 2024-03-30T01:10:00.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:00 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-a[38063]: 2024-03-30T01:09:59.998+0000 7f366d000700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-30T01:10:00.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:00 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-a[38063]: 2024-03-30T01:09:59.998+0000 7f366d000700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-30T01:10:00.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:00 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-a[38063]: 2024-03-30T01:09:59.998+0000 7f366d000700 -1 log_channel(cluster) log [ERR] : mds.i(mds.2): Metadata damage detected 2024-03-30T01:10:00.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:00 smithi116 ceph-mon[38086]: pgmap v655: 129 pgs: 129 active+clean; 981 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 408 op/s 2024-03-30T01:10:00.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:00 smithi116 ceph-mon[38086]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-30T01:10:00.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:00 smithi116 ceph-mon[38086]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-30T01:10:00.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:00 smithi116 ceph-mon[38086]: mds.i(mds.2): Metadata damage detected 2024-03-30T01:10:01.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:00 smithi181 ceph-mon[47622]: pgmap v655: 129 pgs: 129 active+clean; 981 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 408 op/s 2024-03-30T01:10:01.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:00 smithi181 ceph-mon[47622]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-30T01:10:01.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:00 smithi181 ceph-mon[47622]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-30T01:10:01.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:00 smithi181 ceph-mon[47622]: mds.i(mds.2): Metadata damage detected 2024-03-30T01:10:01.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:00 smithi130 ceph-mon[47626]: pgmap v655: 129 pgs: 129 active+clean; 981 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 408 op/s 2024-03-30T01:10:01.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:00 smithi130 ceph-mon[47626]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-30T01:10:01.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:00 smithi130 ceph-mon[47626]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-30T01:10:01.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:00 smithi130 ceph-mon[47626]: mds.i(mds.2): Metadata damage detected 2024-03-30T01:10:03.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:02 smithi181 ceph-mon[47622]: pgmap v656: 129 pgs: 129 active+clean; 974 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 398 op/s 2024-03-30T01:10:03.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:02 smithi130 ceph-mon[47626]: pgmap v656: 129 pgs: 129 active+clean; 974 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 398 op/s 2024-03-30T01:10:03.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:02 smithi116 ceph-mon[38086]: pgmap v656: 129 pgs: 129 active+clean; 974 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 398 op/s 2024-03-30T01:10:05.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:04 smithi181 ceph-mon[47622]: pgmap v657: 129 pgs: 129 active+clean; 961 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 369 op/s 2024-03-30T01:10:05.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:04 smithi130 ceph-mon[47626]: pgmap v657: 129 pgs: 129 active+clean; 961 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 369 op/s 2024-03-30T01:10:05.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:04 smithi116 ceph-mon[38086]: pgmap v657: 129 pgs: 129 active+clean; 961 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 369 op/s 2024-03-30T01:10:07.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:06 smithi181 ceph-mon[47622]: pgmap v658: 129 pgs: 129 active+clean; 943 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 400 op/s 2024-03-30T01:10:07.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:06 smithi130 ceph-mon[47626]: pgmap v658: 129 pgs: 129 active+clean; 943 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 400 op/s 2024-03-30T01:10:07.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:06 smithi116 ceph-mon[38086]: pgmap v658: 129 pgs: 129 active+clean; 943 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 400 op/s 2024-03-30T01:10:09.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:08 smithi181 ceph-mon[47622]: pgmap v659: 129 pgs: 129 active+clean; 940 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 344 op/s 2024-03-30T01:10:09.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:08 smithi130 ceph-mon[47626]: pgmap v659: 129 pgs: 129 active+clean; 940 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 344 op/s 2024-03-30T01:10:09.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:08 smithi116 ceph-mon[38086]: pgmap v659: 129 pgs: 129 active+clean; 940 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 344 op/s 2024-03-30T01:10:11.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:10 smithi181 ceph-mon[47622]: pgmap v660: 129 pgs: 129 active+clean; 930 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 365 op/s 2024-03-30T01:10:11.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:10 smithi130 ceph-mon[47626]: pgmap v660: 129 pgs: 129 active+clean; 930 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 365 op/s 2024-03-30T01:10:11.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:10 smithi116 ceph-mon[38086]: pgmap v660: 129 pgs: 129 active+clean; 930 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 365 op/s 2024-03-30T01:10:13.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:12 smithi181 ceph-mon[47622]: pgmap v661: 129 pgs: 129 active+clean; 926 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 306 op/s 2024-03-30T01:10:13.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:12 smithi130 ceph-mon[47626]: pgmap v661: 129 pgs: 129 active+clean; 926 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 306 op/s 2024-03-30T01:10:13.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:12 smithi116 ceph-mon[38086]: pgmap v661: 129 pgs: 129 active+clean; 926 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 306 op/s 2024-03-30T01:10:15.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:14 smithi181 ceph-mon[47622]: pgmap v662: 129 pgs: 129 active+clean; 926 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 939 KiB/s wr, 273 op/s 2024-03-30T01:10:15.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:14 smithi130 ceph-mon[47626]: pgmap v662: 129 pgs: 129 active+clean; 926 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 939 KiB/s wr, 273 op/s 2024-03-30T01:10:15.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:14 smithi116 ceph-mon[38086]: pgmap v662: 129 pgs: 129 active+clean; 926 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 939 KiB/s wr, 273 op/s 2024-03-30T01:10:16.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:15 smithi130 ceph-mon[47626]: pgmap v663: 129 pgs: 129 active+clean; 914 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.0 MiB/s wr, 285 op/s 2024-03-30T01:10:16.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:15 smithi116 ceph-mon[38086]: pgmap v663: 129 pgs: 129 active+clean; 914 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.0 MiB/s wr, 285 op/s 2024-03-30T01:10:16.251 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:15 smithi181 ceph-mon[47622]: pgmap v663: 129 pgs: 129 active+clean; 914 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.0 MiB/s wr, 285 op/s 2024-03-30T01:10:18.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:18 smithi130 ceph-mon[47626]: pgmap v664: 129 pgs: 129 active+clean; 903 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.0 MiB/s wr, 223 op/s 2024-03-30T01:10:18.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:18 smithi116 ceph-mon[38086]: pgmap v664: 129 pgs: 129 active+clean; 903 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.0 MiB/s wr, 223 op/s 2024-03-30T01:10:19.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:18 smithi181 ceph-mon[47622]: pgmap v664: 129 pgs: 129 active+clean; 903 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.0 MiB/s wr, 223 op/s 2024-03-30T01:10:20.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:20 smithi130 ceph-mon[47626]: pgmap v665: 129 pgs: 129 active+clean; 885 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 817 KiB/s wr, 280 op/s 2024-03-30T01:10:20.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:20 smithi116 ceph-mon[38086]: pgmap v665: 129 pgs: 129 active+clean; 885 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 817 KiB/s wr, 280 op/s 2024-03-30T01:10:21.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:20 smithi181 ceph-mon[47622]: pgmap v665: 129 pgs: 129 active+clean; 885 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 817 KiB/s wr, 280 op/s 2024-03-30T01:10:22.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:22 smithi116 ceph-mon[38086]: pgmap v666: 129 pgs: 129 active+clean; 877 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 301 op/s 2024-03-30T01:10:23.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:22 smithi181 ceph-mon[47622]: pgmap v666: 129 pgs: 129 active+clean; 877 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 301 op/s 2024-03-30T01:10:23.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:22 smithi130 ceph-mon[47626]: pgmap v666: 129 pgs: 129 active+clean; 877 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 301 op/s 2024-03-30T01:10:24.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:24 smithi116 ceph-mon[38086]: pgmap v667: 129 pgs: 129 active+clean; 869 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 335 op/s 2024-03-30T01:10:25.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:24 smithi181 ceph-mon[47622]: pgmap v667: 129 pgs: 129 active+clean; 869 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 335 op/s 2024-03-30T01:10:25.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:24 smithi130 ceph-mon[47626]: pgmap v667: 129 pgs: 129 active+clean; 869 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 335 op/s 2024-03-30T01:10:26.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:26 smithi116 ceph-mon[38086]: pgmap v668: 129 pgs: 129 active+clean; 851 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 370 op/s 2024-03-30T01:10:27.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:26 smithi181 ceph-mon[47622]: pgmap v668: 129 pgs: 129 active+clean; 851 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 370 op/s 2024-03-30T01:10:27.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:26 smithi130 ceph-mon[47626]: pgmap v668: 129 pgs: 129 active+clean; 851 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 370 op/s 2024-03-30T01:10:28.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:28 smithi116 ceph-mon[38086]: pgmap v669: 129 pgs: 129 active+clean; 848 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 977 KiB/s wr, 318 op/s 2024-03-30T01:10:29.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:28 smithi181 ceph-mon[47622]: pgmap v669: 129 pgs: 129 active+clean; 848 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 977 KiB/s wr, 318 op/s 2024-03-30T01:10:29.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:28 smithi130 ceph-mon[47626]: pgmap v669: 129 pgs: 129 active+clean; 848 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 977 KiB/s wr, 318 op/s 2024-03-30T01:10:30.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:30 smithi116 ceph-mon[38086]: pgmap v670: 129 pgs: 129 active+clean; 832 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 366 op/s 2024-03-30T01:10:31.006 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:30 smithi181 ceph-mon[47622]: pgmap v670: 129 pgs: 129 active+clean; 832 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 366 op/s 2024-03-30T01:10:31.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:30 smithi130 ceph-mon[47626]: pgmap v670: 129 pgs: 129 active+clean; 832 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 366 op/s 2024-03-30T01:10:32.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:32 smithi116 ceph-mon[38086]: pgmap v671: 129 pgs: 129 active+clean; 821 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 338 op/s 2024-03-30T01:10:33.010 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:32 smithi181 ceph-mon[47622]: pgmap v671: 129 pgs: 129 active+clean; 821 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 338 op/s 2024-03-30T01:10:33.098 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:32 smithi130 ceph-mon[47626]: pgmap v671: 129 pgs: 129 active+clean; 821 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 338 op/s 2024-03-30T01:10:34.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:34 smithi116 ceph-mon[38086]: pgmap v672: 129 pgs: 129 active+clean; 812 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 323 op/s 2024-03-30T01:10:35.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:34 smithi181 ceph-mon[47622]: pgmap v672: 129 pgs: 129 active+clean; 812 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 323 op/s 2024-03-30T01:10:35.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:34 smithi130 ceph-mon[47626]: pgmap v672: 129 pgs: 129 active+clean; 812 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 323 op/s 2024-03-30T01:10:36.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:36 smithi116 ceph-mon[38086]: pgmap v673: 129 pgs: 129 active+clean; 796 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 323 op/s 2024-03-30T01:10:37.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:36 smithi181 ceph-mon[47622]: pgmap v673: 129 pgs: 129 active+clean; 796 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 323 op/s 2024-03-30T01:10:37.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:36 smithi130 ceph-mon[47626]: pgmap v673: 129 pgs: 129 active+clean; 796 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 323 op/s 2024-03-30T01:10:38.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:38 smithi116 ceph-mon[38086]: pgmap v674: 129 pgs: 129 active+clean; 795 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1018 KiB/s wr, 255 op/s 2024-03-30T01:10:39.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:38 smithi181 ceph-mon[47622]: pgmap v674: 129 pgs: 129 active+clean; 795 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1018 KiB/s wr, 255 op/s 2024-03-30T01:10:39.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:38 smithi130 ceph-mon[47626]: pgmap v674: 129 pgs: 129 active+clean; 795 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1018 KiB/s wr, 255 op/s 2024-03-30T01:10:40.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:40 smithi116 ceph-mon[38086]: pgmap v675: 129 pgs: 129 active+clean; 793 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1020 KiB/s wr, 248 op/s 2024-03-30T01:10:40.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:40 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:10:41.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:40 smithi181 ceph-mon[47622]: pgmap v675: 129 pgs: 129 active+clean; 793 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1020 KiB/s wr, 248 op/s 2024-03-30T01:10:41.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:40 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:10:41.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:40 smithi130 ceph-mon[47626]: pgmap v675: 129 pgs: 129 active+clean; 793 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1020 KiB/s wr, 248 op/s 2024-03-30T01:10:41.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:40 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T01:10:42.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:42 smithi116 ceph-mon[38086]: pgmap v676: 129 pgs: 129 active+clean; 796 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 201 op/s 2024-03-30T01:10:42.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:42 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:10:42.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:42 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:10:42.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:42 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:10:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:42 smithi181 ceph-mon[47622]: pgmap v676: 129 pgs: 129 active+clean; 796 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 201 op/s 2024-03-30T01:10:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:42 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:10:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:42 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:10:43.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:42 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:10:43.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:42 smithi130 ceph-mon[47626]: pgmap v676: 129 pgs: 129 active+clean; 796 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 201 op/s 2024-03-30T01:10:43.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:42 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T01:10:43.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:42 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T01:10:43.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:42 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' 2024-03-30T01:10:44.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:44 smithi116 ceph-mon[38086]: pgmap v677: 129 pgs: 129 active+clean; 785 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 936 KiB/s wr, 188 op/s 2024-03-30T01:10:45.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:44 smithi181 ceph-mon[47622]: pgmap v677: 129 pgs: 129 active+clean; 785 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 936 KiB/s wr, 188 op/s 2024-03-30T01:10:45.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:44 smithi130 ceph-mon[47626]: pgmap v677: 129 pgs: 129 active+clean; 785 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 936 KiB/s wr, 188 op/s 2024-03-30T01:10:46.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:46 smithi116 ceph-mon[38086]: pgmap v678: 129 pgs: 129 active+clean; 778 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 208 op/s 2024-03-30T01:10:47.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:46 smithi181 ceph-mon[47622]: pgmap v678: 129 pgs: 129 active+clean; 778 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 208 op/s 2024-03-30T01:10:47.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:46 smithi130 ceph-mon[47626]: pgmap v678: 129 pgs: 129 active+clean; 778 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 208 op/s 2024-03-30T01:10:49.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:48 smithi181 ceph-mon[47622]: pgmap v679: 129 pgs: 129 active+clean; 771 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 1.0 MiB/s wr, 169 op/s 2024-03-30T01:10:49.099 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:48 smithi130 ceph-mon[47626]: pgmap v679: 129 pgs: 129 active+clean; 771 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 1.0 MiB/s wr, 169 op/s 2024-03-30T01:10:49.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:48 smithi116 ceph-mon[38086]: pgmap v679: 129 pgs: 129 active+clean; 771 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 1.0 MiB/s wr, 169 op/s 2024-03-30T01:10:51.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:50 smithi181 ceph-mon[47622]: pgmap v680: 129 pgs: 129 active+clean; 755 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 242 op/s 2024-03-30T01:10:51.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:50 smithi130 ceph-mon[47626]: pgmap v680: 129 pgs: 129 active+clean; 755 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 242 op/s 2024-03-30T01:10:51.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:50 smithi116 ceph-mon[38086]: pgmap v680: 129 pgs: 129 active+clean; 755 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 242 op/s 2024-03-30T01:10:53.005 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:52 smithi181 ceph-mon[47622]: pgmap v681: 129 pgs: 129 active+clean; 738 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 303 op/s 2024-03-30T01:10:53.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:52 smithi130 ceph-mon[47626]: pgmap v681: 129 pgs: 129 active+clean; 738 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 303 op/s 2024-03-30T01:10:53.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:52 smithi116 ceph-mon[38086]: pgmap v681: 129 pgs: 129 active+clean; 738 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 303 op/s 2024-03-30T01:10:55.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:54 smithi181 ceph-mon[47622]: pgmap v682: 129 pgs: 129 active+clean; 731 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 1.1 MiB/s wr, 322 op/s 2024-03-30T01:10:55.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:54 smithi130 ceph-mon[47626]: pgmap v682: 129 pgs: 129 active+clean; 731 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 1.1 MiB/s wr, 322 op/s 2024-03-30T01:10:55.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:54 smithi116 ceph-mon[38086]: pgmap v682: 129 pgs: 129 active+clean; 731 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 1.1 MiB/s wr, 322 op/s 2024-03-30T01:10:57.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:56 smithi181 ceph-mon[47622]: pgmap v683: 129 pgs: 129 active+clean; 712 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 352 op/s 2024-03-30T01:10:57.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:56 smithi130 ceph-mon[47626]: pgmap v683: 129 pgs: 129 active+clean; 712 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 352 op/s 2024-03-30T01:10:57.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:56 smithi116 ceph-mon[38086]: pgmap v683: 129 pgs: 129 active+clean; 712 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 352 op/s 2024-03-30T01:10:58.226 INFO:tasks.workunit:Stopping ['suites/blogbench.sh'] on client.0... 2024-03-30T01:10:58.227 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-30T01:10:59.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:10:58 smithi181 ceph-mon[47622]: pgmap v684: 129 pgs: 129 active+clean; 711 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 307 op/s 2024-03-30T01:10:59.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:10:58 smithi130 ceph-mon[47626]: pgmap v684: 129 pgs: 129 active+clean; 711 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 307 op/s 2024-03-30T01:10:59.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:10:58 smithi116 ceph-mon[38086]: pgmap v684: 129 pgs: 129 active+clean; 711 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 307 op/s 2024-03-30T01:10:59.594 DEBUG:teuthology.parallel:result is None 2024-03-30T01:10:59.595 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-30T01:10:59.710 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-30T01:10:59.710 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-30T01:10:59.721 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-30T01:10:59.722 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-03-30T01:10:59.723 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-30T01:10:59.732 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T01:10:59.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.922+0000 7f1c3b600700 1 -- 172.21.15.116:0/3252605956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c34102810 msgr2=0x7f1c341090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:10:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.922+0000 7f1c3b600700 1 --2- 172.21.15.116:0/3252605956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c34102810 0x7f1c341090c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c00b0d0 tx=0x7f1c2c00b3e0 comp rx=0 tx=0).stop 2024-03-30T01:10:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.922+0000 7f1c3b600700 1 -- 172.21.15.116:0/3252605956 shutdown_connections 2024-03-30T01:10:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.922+0000 7f1c3b600700 1 --2- 172.21.15.116:0/3252605956 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c34100c80 0x7f1c341010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:10:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.922+0000 7f1c3b600700 1 --2- 172.21.15.116:0/3252605956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c34102810 0x7f1c341090c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:10:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.922+0000 7f1c3b600700 1 --2- 172.21.15.116:0/3252605956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c34101e70 0x7f1c341022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:10:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.922+0000 7f1c3b600700 1 -- 172.21.15.116:0/3252605956 >> 172.21.15.116:0/3252605956 conn(0x7f1c340fc1f0 msgr2=0x7f1c340fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:10:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.922+0000 7f1c3b600700 1 -- 172.21.15.116:0/3252605956 shutdown_connections 2024-03-30T01:10:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.922+0000 7f1c3b600700 1 -- 172.21.15.116:0/3252605956 wait complete. 2024-03-30T01:10:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.923+0000 7f1c3b600700 1 Processor -- start 2024-03-30T01:10:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.923+0000 7f1c3b600700 1 -- start start 2024-03-30T01:10:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c3b600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c34100c80 0x7f1c34198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:10:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c3b600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c34101e70 0x7f1c34199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:10:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c3b600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c34102810 0x7f1c341a04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:10:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c3b600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c341a0a80 con 0x7f1c34101e70 2024-03-30T01:10:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c3b600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c341a0bc0 con 0x7f1c34100c80 2024-03-30T01:10:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c3b600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c341a0d50 con 0x7f1c34102810 2024-03-30T01:10:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c33e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c34101e70 0x7f1c34199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:10:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c38c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c34100c80 0x7f1c34198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:10:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c33e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c34101e70 0x7f1c34199490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51720/0 (socket says 172.21.15.116:51720) 2024-03-30T01:10:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c38c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c34100c80 0x7f1c34198f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:34152/0 (socket says 172.21.15.116:34152) 2024-03-30T01:10:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c33e00700 1 -- 172.21.15.116:0/1719728203 learned_addr learned my addr 172.21.15.116:0/1719728203 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:10:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c39600700 1 --2- 172.21.15.116:0/1719728203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c34102810 0x7f1c341a04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:10:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c33e00700 1 -- 172.21.15.116:0/1719728203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c34102810 msgr2=0x7f1c341a04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:10:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c33e00700 1 --2- 172.21.15.116:0/1719728203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c34102810 0x7f1c341a04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:10:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c33e00700 1 -- 172.21.15.116:0/1719728203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c34100c80 msgr2=0x7f1c34198f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:10:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c33e00700 1 --2- 172.21.15.116:0/1719728203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c34100c80 0x7f1c34198f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:10:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.924+0000 7f1c33e00700 1 -- 172.21.15.116:0/1719728203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c28009520 con 0x7f1c34101e70 2024-03-30T01:10:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.925+0000 7f1c33e00700 1 --2- 172.21.15.116:0/1719728203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c34101e70 0x7f1c34199490 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000e9a0 tx=0x7f1c2000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:10:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.925+0000 7f1c31600700 1 -- 172.21.15.116:0/1719728203 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20009790 con 0x7f1c34101e70 2024-03-30T01:10:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.925+0000 7f1c31600700 1 -- 172.21.15.116:0/1719728203 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c20010430 con 0x7f1c34101e70 2024-03-30T01:10:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.925+0000 7f1c3b600700 1 -- 172.21.15.116:0/1719728203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2c009d00 con 0x7f1c34101e70 2024-03-30T01:10:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.925+0000 7f1c31600700 1 -- 172.21.15.116:0/1719728203 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20018680 con 0x7f1c34101e70 2024-03-30T01:10:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.925+0000 7f1c3b600700 1 -- 172.21.15.116:0/1719728203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c341a1320 con 0x7f1c34101e70 2024-03-30T01:10:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.926+0000 7f1c3b600700 1 -- 172.21.15.116:0/1719728203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c34067a10 con 0x7f1c34101e70 2024-03-30T01:10:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.928+0000 7f1c31600700 1 -- 172.21.15.116:0/1719728203 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f1c200105a0 con 0x7f1c34101e70 2024-03-30T01:10:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.930+0000 7f1c31600700 1 --2- 172.21.15.116:0/1719728203 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1c1c0a1120 0x7f1c1c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:10:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.930+0000 7f1c31600700 1 -- 172.21.15.116:0/1719728203 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1c20014070 con 0x7f1c34101e70 2024-03-30T01:10:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.930+0000 7f1c38c00700 1 --2- 172.21.15.116:0/1719728203 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1c1c0a1120 0x7f1c1c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:10:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.931+0000 7f1c38c00700 1 --2- 172.21.15.116:0/1719728203 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1c1c0a1120 0x7f1c1c0a35e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1c34101cd0 tx=0x7f1c28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:10:59.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:10:59.935+0000 7f1c31600700 1 -- 172.21.15.116:0/1719728203 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1c200529f0 con 0x7f1c34101e70 2024-03-30T01:11:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.169+0000 7f1c3b600700 1 -- 172.21.15.116:0/1719728203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c3410c4f0 con 0x7f1c34101e70 2024-03-30T01:11:00.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.170+0000 7f1c31600700 1 -- 172.21.15.116:0/1719728203 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+24556 (secure 0 0 0) 0x7f1c20096220 con 0x7f1c34101e70 2024-03-30T01:11:00.177 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T01:11:00.177 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:56:36.456069+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T01:11:00.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.173+0000 7f1c3b600700 1 -- 172.21.15.116:0/1719728203 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1c1c0a1120 msgr2=0x7f1c1c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:00.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.173+0000 7f1c3b600700 1 --2- 172.21.15.116:0/1719728203 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1c1c0a1120 0x7f1c1c0a35e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1c34101cd0 tx=0x7f1c28009260 comp rx=0 tx=0).stop 2024-03-30T01:11:00.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.173+0000 7f1c3b600700 1 -- 172.21.15.116:0/1719728203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c34101e70 msgr2=0x7f1c34199490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:00.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.173+0000 7f1c3b600700 1 --2- 172.21.15.116:0/1719728203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c34101e70 0x7f1c34199490 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000e9a0 tx=0x7f1c2000ecb0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.173+0000 7f1c3b600700 1 -- 172.21.15.116:0/1719728203 shutdown_connections 2024-03-30T01:11:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.173+0000 7f1c3b600700 1 --2- 172.21.15.116:0/1719728203 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f1c1c0a1120 0x7f1c1c0a35e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.173+0000 7f1c3b600700 1 --2- 172.21.15.116:0/1719728203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c34102810 0x7f1c341a04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.173+0000 7f1c3b600700 1 --2- 172.21.15.116:0/1719728203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c34100c80 0x7f1c34198f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.173+0000 7f1c3b600700 1 --2- 172.21.15.116:0/1719728203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c34101e70 0x7f1c34199490 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.173+0000 7f1c3b600700 1 -- 172.21.15.116:0/1719728203 >> 172.21.15.116:0/1719728203 conn(0x7f1c340fc1f0 msgr2=0x7f1c34072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.173+0000 7f1c3b600700 1 -- 172.21.15.116:0/1719728203 shutdown_connections 2024-03-30T01:11:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.173+0000 7f1c3b600700 1 -- 172.21.15.116:0/1719728203 wait complete. 2024-03-30T01:11:00.198 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T00:52:29.602428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T00:55:50.220543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T00:55:51.274498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T00:55:54.301591+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T01:11:00.199 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T01:11:00.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.388+0000 7f444b600700 1 -- 172.21.15.116:0/1384313670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4444102910 msgr2=0x7f44441091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:00.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.388+0000 7f444b600700 1 --2- 172.21.15.116:0/1384313670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4444102910 0x7f44441091c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f443c009f90 tx=0x7f443c00b330 comp rx=0 tx=0).stop 2024-03-30T01:11:00.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.388+0000 7f444b600700 1 -- 172.21.15.116:0/1384313670 shutdown_connections 2024-03-30T01:11:00.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.388+0000 7f444b600700 1 --2- 172.21.15.116:0/1384313670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4444102910 0x7f44441091c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.388+0000 7f444b600700 1 --2- 172.21.15.116:0/1384313670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4444100d80 0x7f44441011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.388+0000 7f444b600700 1 --2- 172.21.15.116:0/1384313670 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4444101f70 0x7f44441023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.388+0000 7f444b600700 1 -- 172.21.15.116:0/1384313670 >> 172.21.15.116:0/1384313670 conn(0x7f44440fc2f0 msgr2=0x7f44440fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:00.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.388+0000 7f444b600700 1 -- 172.21.15.116:0/1384313670 shutdown_connections 2024-03-30T01:11:00.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.388+0000 7f444b600700 1 -- 172.21.15.116:0/1384313670 wait complete. 2024-03-30T01:11:00.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.389+0000 7f444b600700 1 Processor -- start 2024-03-30T01:11:00.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.389+0000 7f444b600700 1 -- start start 2024-03-30T01:11:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.390+0000 7f444b600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4444100d80 0x7f4444199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.390+0000 7f444b600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4444101f70 0x7f4444199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.390+0000 7f444b600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4444102910 0x7f44441a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.390+0000 7f444b600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44441a0b50 con 0x7f4444102910 2024-03-30T01:11:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.390+0000 7f444b600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44441a0c90 con 0x7f4444100d80 2024-03-30T01:11:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.390+0000 7f444b600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44441a0e20 con 0x7f4444101f70 2024-03-30T01:11:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.390+0000 7f4449600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4444102910 0x7f44441a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.390+0000 7f4449600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4444102910 0x7f44441a0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51732/0 (socket says 172.21.15.116:51732) 2024-03-30T01:11:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.390+0000 7f4449600700 1 -- 172.21.15.116:0/227004002 learned_addr learned my addr 172.21.15.116:0/227004002 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.390+0000 7f4448c00700 1 --2- 172.21.15.116:0/227004002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4444100d80 0x7f4444199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.390+0000 7f4443e00700 1 --2- 172.21.15.116:0/227004002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4444101f70 0x7f4444199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.390+0000 7f4449600700 1 -- 172.21.15.116:0/227004002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4444101f70 msgr2=0x7f4444199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.390+0000 7f4449600700 1 --2- 172.21.15.116:0/227004002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4444101f70 0x7f4444199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.391+0000 7f4449600700 1 -- 172.21.15.116:0/227004002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4444100d80 msgr2=0x7f4444199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.391+0000 7f4449600700 1 --2- 172.21.15.116:0/227004002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4444100d80 0x7f4444199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.391+0000 7f4449600700 1 -- 172.21.15.116:0/227004002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44380095f0 con 0x7f4444102910 2024-03-30T01:11:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.391+0000 7f4449600700 1 --2- 172.21.15.116:0/227004002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4444102910 0x7f44441a0580 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f443c009f60 tx=0x7f443c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.391+0000 7f4441600700 1 -- 172.21.15.116:0/227004002 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c00e040 con 0x7f4444102910 2024-03-30T01:11:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.391+0000 7f444b600700 1 -- 172.21.15.116:0/227004002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443c009c40 con 0x7f4444102910 2024-03-30T01:11:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.391+0000 7f444b600700 1 -- 172.21.15.116:0/227004002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44441a1390 con 0x7f4444102910 2024-03-30T01:11:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.391+0000 7f4441600700 1 -- 172.21.15.116:0/227004002 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f443c004e70 con 0x7f4444102910 2024-03-30T01:11:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.391+0000 7f4441600700 1 -- 172.21.15.116:0/227004002 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c0052a0 con 0x7f4444102910 2024-03-30T01:11:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.392+0000 7f444b600700 1 -- 172.21.15.116:0/227004002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4424005320 con 0x7f4444102910 2024-03-30T01:11:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.394+0000 7f4441600700 1 -- 172.21.15.116:0/227004002 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f443c01d470 con 0x7f4444102910 2024-03-30T01:11:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.395+0000 7f4441600700 1 --2- 172.21.15.116:0/227004002 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f442c0a11a0 0x7f442c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.395+0000 7f4441600700 1 -- 172.21.15.116:0/227004002 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f443c0cd9c0 con 0x7f4444102910 2024-03-30T01:11:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.395+0000 7f4448c00700 1 --2- 172.21.15.116:0/227004002 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f442c0a11a0 0x7f442c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.396+0000 7f4448c00700 1 --2- 172.21.15.116:0/227004002 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f442c0a11a0 0x7f442c0a3660 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4444101dd0 tx=0x7f4438009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:00.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.400+0000 7f4441600700 1 -- 172.21.15.116:0/227004002 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f443c096e70 con 0x7f4444102910 2024-03-30T01:11:00.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.609+0000 7f444b600700 1 -- 172.21.15.116:0/227004002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4424005b50 con 0x7f4444102910 2024-03-30T01:11:00.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.609+0000 7f4441600700 1 -- 172.21.15.116:0/227004002 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f443c09a490 con 0x7f4444102910 2024-03-30T01:11:00.616 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-30T01:11:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.612+0000 7f444b600700 1 -- 172.21.15.116:0/227004002 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f442c0a11a0 msgr2=0x7f442c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.612+0000 7f444b600700 1 --2- 172.21.15.116:0/227004002 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f442c0a11a0 0x7f442c0a3660 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4444101dd0 tx=0x7f4438009310 comp rx=0 tx=0).stop 2024-03-30T01:11:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.612+0000 7f444b600700 1 -- 172.21.15.116:0/227004002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4444102910 msgr2=0x7f44441a0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.612+0000 7f444b600700 1 --2- 172.21.15.116:0/227004002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4444102910 0x7f44441a0580 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f443c009f60 tx=0x7f443c004c80 comp rx=0 tx=0).stop 2024-03-30T01:11:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.612+0000 7f444b600700 1 -- 172.21.15.116:0/227004002 shutdown_connections 2024-03-30T01:11:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.612+0000 7f444b600700 1 --2- 172.21.15.116:0/227004002 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f442c0a11a0 0x7f442c0a3660 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.612+0000 7f444b600700 1 --2- 172.21.15.116:0/227004002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4444102910 0x7f44441a0580 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.612+0000 7f444b600700 1 --2- 172.21.15.116:0/227004002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4444100d80 0x7f4444199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.612+0000 7f444b600700 1 --2- 172.21.15.116:0/227004002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4444101f70 0x7f4444199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.612+0000 7f444b600700 1 -- 172.21.15.116:0/227004002 >> 172.21.15.116:0/227004002 conn(0x7f44440fc2f0 msgr2=0x7f4444072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.612+0000 7f444b600700 1 -- 172.21.15.116:0/227004002 shutdown_connections 2024-03-30T01:11:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.612+0000 7f444b600700 1 -- 172.21.15.116:0/227004002 wait complete. 2024-03-30T01:11:00.637 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T01:11:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.827+0000 7f138fe00700 1 -- 172.21.15.116:0/4083268630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388101f70 msgr2=0x7f13881023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.827+0000 7f138fe00700 1 --2- 172.21.15.116:0/4083268630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388101f70 0x7f13881023d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f13740099c0 tx=0x7f1374009cd0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.827+0000 7f138fe00700 1 -- 172.21.15.116:0/4083268630 shutdown_connections 2024-03-30T01:11:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.827+0000 7f138fe00700 1 --2- 172.21.15.116:0/4083268630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1388102910 0x7f13881091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.827+0000 7f138fe00700 1 --2- 172.21.15.116:0/4083268630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1388100d80 0x7f13881011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.827+0000 7f138fe00700 1 --2- 172.21.15.116:0/4083268630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388101f70 0x7f13881023d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.827+0000 7f138fe00700 1 -- 172.21.15.116:0/4083268630 >> 172.21.15.116:0/4083268630 conn(0x7f13880fc2f0 msgr2=0x7f13880fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.827+0000 7f138fe00700 1 -- 172.21.15.116:0/4083268630 shutdown_connections 2024-03-30T01:11:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.827+0000 7f138fe00700 1 -- 172.21.15.116:0/4083268630 wait complete. 2024-03-30T01:11:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.828+0000 7f138fe00700 1 Processor -- start 2024-03-30T01:11:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.829+0000 7f138fe00700 1 -- start start 2024-03-30T01:11:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.829+0000 7f138fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388100d80 0x7f1388198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.829+0000 7f138fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1388101f70 0x7f13881994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.829+0000 7f138fe00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1388102910 0x7f13881a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.829+0000 7f138fe00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13881a0a60 con 0x7f1388100d80 2024-03-30T01:11:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.829+0000 7f138fe00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13881a0ba0 con 0x7f1388101f70 2024-03-30T01:11:00.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.829+0000 7f138fe00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13881a0d30 con 0x7f1388102910 2024-03-30T01:11:00.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.829+0000 7f138d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388100d80 0x7f1388198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:00.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.829+0000 7f138d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388100d80 0x7f1388198f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51750/0 (socket says 172.21.15.116:51750) 2024-03-30T01:11:00.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.829+0000 7f138d400700 1 -- 172.21.15.116:0/39881450 learned_addr learned my addr 172.21.15.116:0/39881450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:00.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.830+0000 7f138de00700 1 --2- 172.21.15.116:0/39881450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1388102910 0x7f13881a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:00.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.830+0000 7f138ca00700 1 --2- 172.21.15.116:0/39881450 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1388101f70 0x7f13881994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:00.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.830+0000 7f138d400700 1 -- 172.21.15.116:0/39881450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1388102910 msgr2=0x7f13881a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:00.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.830+0000 7f138d400700 1 --2- 172.21.15.116:0/39881450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1388102910 0x7f13881a0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.830+0000 7f138d400700 1 -- 172.21.15.116:0/39881450 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1388101f70 msgr2=0x7f13881994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:00.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.830+0000 7f138d400700 1 --2- 172.21.15.116:0/39881450 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1388101f70 0x7f13881994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:00.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.830+0000 7f138d400700 1 -- 172.21.15.116:0/39881450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13740095f0 con 0x7f1388100d80 2024-03-30T01:11:00.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.830+0000 7f138d400700 1 --2- 172.21.15.116:0/39881450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388100d80 0x7f1388198f80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f137c00e970 tx=0x7f137c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:00.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.830+0000 7f1386000700 1 -- 172.21.15.116:0/39881450 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c009790 con 0x7f1388100d80 2024-03-30T01:11:00.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.830+0000 7f1386000700 1 -- 172.21.15.116:0/39881450 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f137c00c7e0 con 0x7f1388100d80 2024-03-30T01:11:00.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.830+0000 7f138fe00700 1 -- 172.21.15.116:0/39881450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13881a0fb0 con 0x7f1388100d80 2024-03-30T01:11:00.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.830+0000 7f1386000700 1 -- 172.21.15.116:0/39881450 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c003c70 con 0x7f1388100d80 2024-03-30T01:11:00.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.830+0000 7f138fe00700 1 -- 172.21.15.116:0/39881450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13881a1430 con 0x7f1388100d80 2024-03-30T01:11:00.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.831+0000 7f138fe00700 1 -- 172.21.15.116:0/39881450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1388067af0 con 0x7f1388100d80 2024-03-30T01:11:00.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.833+0000 7f1386000700 1 -- 172.21.15.116:0/39881450 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f137c00cc50 con 0x7f1388100d80 2024-03-30T01:11:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.835+0000 7f1386000700 1 --2- 172.21.15.116:0/39881450 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13700a1120 0x7f13700a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.835+0000 7f1386000700 1 -- 172.21.15.116:0/39881450 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f137c014070 con 0x7f1388100d80 2024-03-30T01:11:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.835+0000 7f138ca00700 1 --2- 172.21.15.116:0/39881450 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13700a1120 0x7f13700a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.835+0000 7f138ca00700 1 --2- 172.21.15.116:0/39881450 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13700a1120 0x7f13700a35e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1374003730 tx=0x7f137400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:00.836+0000 7f1386000700 1 -- 172.21.15.116:0/39881450 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f137c096310 con 0x7f1388100d80 2024-03-30T01:11:00.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:00 smithi116 ceph-mon[38086]: pgmap v685: 129 pgs: 129 active+clean; 687 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 373 op/s 2024-03-30T01:11:00.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:00 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1719728203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T01:11:00.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:00 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/227004002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T01:11:01.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:00 smithi181 ceph-mon[47622]: pgmap v685: 129 pgs: 129 active+clean; 687 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 373 op/s 2024-03-30T01:11:01.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:00 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1719728203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T01:11:01.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:00 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/227004002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T01:11:01.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.042+0000 7f138fe00700 1 -- 172.21.15.116:0/39881450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f138810c790 con 0x7f1388100d80 2024-03-30T01:11:01.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.043+0000 7f1386000700 1 -- 172.21.15.116:0/39881450 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f137c003dd0 con 0x7f1388100d80 2024-03-30T01:11:01.049 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T01:11:01.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.045+0000 7f138fe00700 1 -- 172.21.15.116:0/39881450 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13700a1120 msgr2=0x7f13700a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:01.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.045+0000 7f138fe00700 1 --2- 172.21.15.116:0/39881450 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13700a1120 0x7f13700a35e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1374003730 tx=0x7f137400b540 comp rx=0 tx=0).stop 2024-03-30T01:11:01.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.045+0000 7f138fe00700 1 -- 172.21.15.116:0/39881450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388100d80 msgr2=0x7f1388198f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:01.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.045+0000 7f138fe00700 1 --2- 172.21.15.116:0/39881450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388100d80 0x7f1388198f80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f137c00e970 tx=0x7f137c00ec80 comp rx=0 tx=0).stop 2024-03-30T01:11:01.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.045+0000 7f138fe00700 1 -- 172.21.15.116:0/39881450 shutdown_connections 2024-03-30T01:11:01.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.045+0000 7f138fe00700 1 --2- 172.21.15.116:0/39881450 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f13700a1120 0x7f13700a35e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.045+0000 7f138fe00700 1 --2- 172.21.15.116:0/39881450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1388102910 0x7f13881a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.045+0000 7f138fe00700 1 --2- 172.21.15.116:0/39881450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388100d80 0x7f1388198f80 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.045+0000 7f138fe00700 1 --2- 172.21.15.116:0/39881450 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1388101f70 0x7f13881994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.045+0000 7f138fe00700 1 -- 172.21.15.116:0/39881450 >> 172.21.15.116:0/39881450 conn(0x7f13880fc2f0 msgr2=0x7f1388072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:01.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.046+0000 7f138fe00700 1 -- 172.21.15.116:0/39881450 shutdown_connections 2024-03-30T01:11:01.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.046+0000 7f138fe00700 1 -- 172.21.15.116:0/39881450 wait complete. 2024-03-30T01:11:01.071 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T01:11:01.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:00 smithi130 ceph-mon[47626]: pgmap v685: 129 pgs: 129 active+clean; 687 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 373 op/s 2024-03-30T01:11:01.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:00 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1719728203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T01:11:01.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:00 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/227004002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T01:11:01.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.260+0000 7f2b43200700 1 -- 172.21.15.116:0/1735778057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b3c100d80 msgr2=0x7f2b3c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:01.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.260+0000 7f2b43200700 1 --2- 172.21.15.116:0/1735778057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b3c100d80 0x7f2b3c1011a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c0099c0 tx=0x7f2b2c009cd0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.260+0000 7f2b43200700 1 -- 172.21.15.116:0/1735778057 shutdown_connections 2024-03-30T01:11:01.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.261+0000 7f2b43200700 1 --2- 172.21.15.116:0/1735778057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b3c102910 0x7f2b3c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.261+0000 7f2b43200700 1 --2- 172.21.15.116:0/1735778057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b3c100d80 0x7f2b3c1011a0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.261+0000 7f2b43200700 1 --2- 172.21.15.116:0/1735778057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b3c101f70 0x7f2b3c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.261+0000 7f2b43200700 1 -- 172.21.15.116:0/1735778057 >> 172.21.15.116:0/1735778057 conn(0x7f2b3c0fc2f0 msgr2=0x7f2b3c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:01.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.261+0000 7f2b43200700 1 -- 172.21.15.116:0/1735778057 shutdown_connections 2024-03-30T01:11:01.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.261+0000 7f2b43200700 1 -- 172.21.15.116:0/1735778057 wait complete. 2024-03-30T01:11:01.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.262+0000 7f2b43200700 1 Processor -- start 2024-03-30T01:11:01.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.262+0000 7f2b43200700 1 -- start start 2024-03-30T01:11:01.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.262+0000 7f2b43200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b3c100d80 0x7f2b3c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:01.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.262+0000 7f2b43200700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b3c101f70 0x7f2b3c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:01.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.262+0000 7f2b43200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b3c102910 0x7f2b3c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:01.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.262+0000 7f2b43200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c073e80 con 0x7f2b3c100d80 2024-03-30T01:11:01.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.262+0000 7f2b43200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c073ff0 con 0x7f2b3c102910 2024-03-30T01:11:01.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.262+0000 7f2b43200700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c1a0d80 con 0x7f2b3c101f70 2024-03-30T01:11:01.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.262+0000 7f2b3be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b3c100d80 0x7f2b3c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:01.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.262+0000 7f2b3be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b3c100d80 0x7f2b3c074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51756/0 (socket says 172.21.15.116:51756) 2024-03-30T01:11:01.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.262+0000 7f2b3be00700 1 -- 172.21.15.116:0/3713678899 learned_addr learned my addr 172.21.15.116:0/3713678899 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:01.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.262+0000 7f2b41200700 1 --2- 172.21.15.116:0/3713678899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b3c102910 0x7f2b3c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:01.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.262+0000 7f2b3b400700 1 --2- 172.21.15.116:0/3713678899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b3c101f70 0x7f2b3c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:01.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.263+0000 7f2b3be00700 1 -- 172.21.15.116:0/3713678899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b3c101f70 msgr2=0x7f2b3c0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:01.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.263+0000 7f2b3be00700 1 --2- 172.21.15.116:0/3713678899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b3c101f70 0x7f2b3c0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.263+0000 7f2b3be00700 1 -- 172.21.15.116:0/3713678899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b3c102910 msgr2=0x7f2b3c0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:01.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.263+0000 7f2b3be00700 1 --2- 172.21.15.116:0/3713678899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b3c102910 0x7f2b3c0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.263+0000 7f2b3be00700 1 -- 172.21.15.116:0/3713678899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2c0095f0 con 0x7f2b3c100d80 2024-03-30T01:11:01.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.263+0000 7f2b3be00700 1 --2- 172.21.15.116:0/3713678899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b3c100d80 0x7f2b3c074b10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c003730 tx=0x7f2b2c00fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:01.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.263+0000 7f2b38c00700 1 -- 172.21.15.116:0/3713678899 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c01d070 con 0x7f2b3c100d80 2024-03-30T01:11:01.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.263+0000 7f2b38c00700 1 -- 172.21.15.116:0/3713678899 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b2c00bda0 con 0x7f2b3c100d80 2024-03-30T01:11:01.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.263+0000 7f2b43200700 1 -- 172.21.15.116:0/3713678899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3c1a0fb0 con 0x7f2b3c100d80 2024-03-30T01:11:01.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.263+0000 7f2b38c00700 1 -- 172.21.15.116:0/3713678899 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c004b30 con 0x7f2b3c100d80 2024-03-30T01:11:01.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.263+0000 7f2b43200700 1 -- 172.21.15.116:0/3713678899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3c1a1320 con 0x7f2b3c100d80 2024-03-30T01:11:01.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.264+0000 7f2b43200700 1 -- 172.21.15.116:0/3713678899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3c108200 con 0x7f2b3c100d80 2024-03-30T01:11:01.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.266+0000 7f2b38c00700 1 -- 172.21.15.116:0/3713678899 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f2b2c004c90 con 0x7f2b3c100d80 2024-03-30T01:11:01.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.268+0000 7f2b38c00700 1 --2- 172.21.15.116:0/3713678899 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2b200a11a0 0x7f2b200a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.269+0000 7f2b38c00700 1 -- 172.21.15.116:0/3713678899 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2b2c0cda20 con 0x7f2b3c100d80 2024-03-30T01:11:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.269+0000 7f2b3b400700 1 --2- 172.21.15.116:0/3713678899 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2b200a11a0 0x7f2b200a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.269+0000 7f2b3b400700 1 --2- 172.21.15.116:0/3713678899 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2b200a11a0 0x7f2b200a3660 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2b28007c60 tx=0x7f2b280073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:01.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.272+0000 7f2b38c00700 1 -- 172.21.15.116:0/3713678899 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2b2c096ed0 con 0x7f2b3c100d80 2024-03-30T01:11:01.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.480+0000 7f2b43200700 1 -- 172.21.15.116:0/3713678899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2b3c107be0 con 0x7f2b3c100d80 2024-03-30T01:11:01.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.480+0000 7f2b38c00700 1 -- 172.21.15.116:0/3713678899 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2b2c09a4f0 con 0x7f2b3c100d80 2024-03-30T01:11:01.487 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-30T01:11:01.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.482+0000 7f2b43200700 1 -- 172.21.15.116:0/3713678899 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2b200a11a0 msgr2=0x7f2b200a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:01.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.482+0000 7f2b43200700 1 --2- 172.21.15.116:0/3713678899 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2b200a11a0 0x7f2b200a3660 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2b28007c60 tx=0x7f2b280073d0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.482+0000 7f2b43200700 1 -- 172.21.15.116:0/3713678899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b3c100d80 msgr2=0x7f2b3c074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:01.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.482+0000 7f2b43200700 1 --2- 172.21.15.116:0/3713678899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b3c100d80 0x7f2b3c074b10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c003730 tx=0x7f2b2c00fa00 comp rx=0 tx=0).stop 2024-03-30T01:11:01.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.482+0000 7f2b43200700 1 -- 172.21.15.116:0/3713678899 shutdown_connections 2024-03-30T01:11:01.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.482+0000 7f2b43200700 1 --2- 172.21.15.116:0/3713678899 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2b200a11a0 0x7f2b200a3660 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.482+0000 7f2b43200700 1 --2- 172.21.15.116:0/3713678899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b3c102910 0x7f2b3c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.482+0000 7f2b43200700 1 --2- 172.21.15.116:0/3713678899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b3c100d80 0x7f2b3c074b10 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.482+0000 7f2b43200700 1 --2- 172.21.15.116:0/3713678899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b3c101f70 0x7f2b3c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.482+0000 7f2b43200700 1 -- 172.21.15.116:0/3713678899 >> 172.21.15.116:0/3713678899 conn(0x7f2b3c0fc2f0 msgr2=0x7f2b3c105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:01.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.483+0000 7f2b43200700 1 -- 172.21.15.116:0/3713678899 shutdown_connections 2024-03-30T01:11:01.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.483+0000 7f2b43200700 1 -- 172.21.15.116:0/3713678899 wait complete. 2024-03-30T01:11:01.507 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T01:11:01.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.696+0000 7f9ed8400700 1 -- 172.21.15.116:0/2386950686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ed0101f70 msgr2=0x7f9ed01023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:01.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.696+0000 7f9ed8400700 1 --2- 172.21.15.116:0/2386950686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ed0101f70 0x7f9ed01023d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc009920 tx=0x7f9ebc009c30 comp rx=0 tx=0).stop 2024-03-30T01:11:01.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.696+0000 7f9ed8400700 1 -- 172.21.15.116:0/2386950686 shutdown_connections 2024-03-30T01:11:01.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.696+0000 7f9ed8400700 1 --2- 172.21.15.116:0/2386950686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ed0102910 0x7f9ed01091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.696+0000 7f9ed8400700 1 --2- 172.21.15.116:0/2386950686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ed0100d80 0x7f9ed01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.696+0000 7f9ed8400700 1 --2- 172.21.15.116:0/2386950686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ed0101f70 0x7f9ed01023d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.696+0000 7f9ed8400700 1 -- 172.21.15.116:0/2386950686 >> 172.21.15.116:0/2386950686 conn(0x7f9ed00fc2f0 msgr2=0x7f9ed00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:01.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.696+0000 7f9ed8400700 1 -- 172.21.15.116:0/2386950686 shutdown_connections 2024-03-30T01:11:01.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.696+0000 7f9ed8400700 1 -- 172.21.15.116:0/2386950686 wait complete. 2024-03-30T01:11:01.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.697+0000 7f9ed8400700 1 Processor -- start 2024-03-30T01:11:01.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed8400700 1 -- start start 2024-03-30T01:11:01.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed8400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ed0100d80 0x7f9ed0198fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:01.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed8400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ed0101f70 0x7f9ed0199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:01.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed8400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ed0102910 0x7f9ed01a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:01.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed8400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed01a0af0 con 0x7f9ed0101f70 2024-03-30T01:11:01.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed8400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed01a0c30 con 0x7f9ed0102910 2024-03-30T01:11:01.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed8400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed01a0dc0 con 0x7f9ed0100d80 2024-03-30T01:11:01.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed5000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ed0101f70 0x7f9ed0199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:01.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed5000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ed0101f70 0x7f9ed0199500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51770/0 (socket says 172.21.15.116:51770) 2024-03-30T01:11:01.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed5000700 1 -- 172.21.15.116:0/4206625356 learned_addr learned my addr 172.21.15.116:0/4206625356 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:01.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed5a00700 1 --2- 172.21.15.116:0/4206625356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ed0100d80 0x7f9ed0198fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:01.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed6400700 1 --2- 172.21.15.116:0/4206625356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ed0102910 0x7f9ed01a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:01.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed5000700 1 -- 172.21.15.116:0/4206625356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ed0100d80 msgr2=0x7f9ed0198fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:01.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed5000700 1 --2- 172.21.15.116:0/4206625356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ed0100d80 0x7f9ed0198fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed5000700 1 -- 172.21.15.116:0/4206625356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ed0102910 msgr2=0x7f9ed01a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:01.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed5000700 1 --2- 172.21.15.116:0/4206625356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ed0102910 0x7f9ed01a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.698+0000 7f9ed5000700 1 -- 172.21.15.116:0/4206625356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec40095f0 con 0x7f9ed0101f70 2024-03-30T01:11:01.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.699+0000 7f9ed5000700 1 --2- 172.21.15.116:0/4206625356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ed0101f70 0x7f9ed0199500 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc005e90 tx=0x7f9ebc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:01.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.699+0000 7f9ece000700 1 -- 172.21.15.116:0/4206625356 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc01c070 con 0x7f9ed0101f70 2024-03-30T01:11:01.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.699+0000 7f9ece000700 1 -- 172.21.15.116:0/4206625356 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ebc00f7f0 con 0x7f9ed0101f70 2024-03-30T01:11:01.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.699+0000 7f9ece000700 1 -- 172.21.15.116:0/4206625356 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc003da0 con 0x7f9ed0101f70 2024-03-30T01:11:01.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.699+0000 7f9ed8400700 1 -- 172.21.15.116:0/4206625356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ebc009520 con 0x7f9ed0101f70 2024-03-30T01:11:01.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.699+0000 7f9ed8400700 1 -- 172.21.15.116:0/4206625356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed01a1280 con 0x7f9ed0101f70 2024-03-30T01:11:01.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.700+0000 7f9ed8400700 1 -- 172.21.15.116:0/4206625356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed0067af0 con 0x7f9ed0101f70 2024-03-30T01:11:01.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.702+0000 7f9ece000700 1 -- 172.21.15.116:0/4206625356 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f9ebc005040 con 0x7f9ed0101f70 2024-03-30T01:11:01.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.704+0000 7f9ece000700 1 --2- 172.21.15.116:0/4206625356 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9eb80a1120 0x7f9eb80a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:01.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.704+0000 7f9ece000700 1 -- 172.21.15.116:0/4206625356 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9ebc0cda30 con 0x7f9ed0101f70 2024-03-30T01:11:01.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.704+0000 7f9ed5a00700 1 --2- 172.21.15.116:0/4206625356 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9eb80a1120 0x7f9eb80a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:01.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.705+0000 7f9ed5a00700 1 --2- 172.21.15.116:0/4206625356 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9eb80a1120 0x7f9eb80a35e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0101dd0 tx=0x7f9ec4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:01.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.708+0000 7f9ece000700 1 -- 172.21.15.116:0/4206625356 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9ebc096ee0 con 0x7f9ed0101f70 2024-03-30T01:11:01.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.916+0000 7f9ed8400700 1 -- 172.21.15.116:0/4206625356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9ed010c7b0 con 0x7f9ed0101f70 2024-03-30T01:11:01.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.917+0000 7f9ece000700 1 -- 172.21.15.116:0/4206625356 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v83) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9ebc09a500 con 0x7f9ed0101f70 2024-03-30T01:11:01.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:01 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/39881450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T01:11:01.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:01 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3713678899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T01:11:01.924 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T01:11:01.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.919+0000 7f9ed8400700 1 -- 172.21.15.116:0/4206625356 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9eb80a1120 msgr2=0x7f9eb80a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:01.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.919+0000 7f9ed8400700 1 --2- 172.21.15.116:0/4206625356 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9eb80a1120 0x7f9eb80a35e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0101dd0 tx=0x7f9ec4009310 comp rx=0 tx=0).stop 2024-03-30T01:11:01.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.920+0000 7f9ed8400700 1 -- 172.21.15.116:0/4206625356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ed0101f70 msgr2=0x7f9ed0199500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:01.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.920+0000 7f9ed8400700 1 --2- 172.21.15.116:0/4206625356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ed0101f70 0x7f9ed0199500 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc005e90 tx=0x7f9ebc00b840 comp rx=0 tx=0).stop 2024-03-30T01:11:01.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.920+0000 7f9ed8400700 1 -- 172.21.15.116:0/4206625356 shutdown_connections 2024-03-30T01:11:01.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.920+0000 7f9ed8400700 1 --2- 172.21.15.116:0/4206625356 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f9eb80a1120 0x7f9eb80a35e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.920+0000 7f9ed8400700 1 --2- 172.21.15.116:0/4206625356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ed0100d80 0x7f9ed0198fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.920+0000 7f9ed8400700 1 --2- 172.21.15.116:0/4206625356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ed0102910 0x7f9ed01a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.920+0000 7f9ed8400700 1 --2- 172.21.15.116:0/4206625356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ed0101f70 0x7f9ed0199500 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:01.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.920+0000 7f9ed8400700 1 -- 172.21.15.116:0/4206625356 >> 172.21.15.116:0/4206625356 conn(0x7f9ed00fc2f0 msgr2=0x7f9ed01077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:01.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.920+0000 7f9ed8400700 1 -- 172.21.15.116:0/4206625356 shutdown_connections 2024-03-30T01:11:01.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:01.920+0000 7f9ed8400700 1 -- 172.21.15.116:0/4206625356 wait complete. 2024-03-30T01:11:01.945 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-30T01:11:02.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:01 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/39881450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T01:11:02.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:01 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3713678899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T01:11:02.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:01 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/39881450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T01:11:02.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:01 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3713678899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T01:11:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.131+0000 7ff140600700 1 -- 172.21.15.116:0/339980734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff138102810 msgr2=0x7ff1381090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.131+0000 7ff140600700 1 --2- 172.21.15.116:0/339980734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff138102810 0x7ff1381090c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff13000b0d0 tx=0x7ff13000b3e0 comp rx=0 tx=0).stop 2024-03-30T01:11:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.132+0000 7ff140600700 1 -- 172.21.15.116:0/339980734 shutdown_connections 2024-03-30T01:11:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.132+0000 7ff140600700 1 --2- 172.21.15.116:0/339980734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff138100c80 0x7ff1381010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.132+0000 7ff140600700 1 --2- 172.21.15.116:0/339980734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff138102810 0x7ff1381090c0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.132+0000 7ff140600700 1 --2- 172.21.15.116:0/339980734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff138101e70 0x7ff1381022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.132+0000 7ff140600700 1 -- 172.21.15.116:0/339980734 >> 172.21.15.116:0/339980734 conn(0x7ff1380fc1f0 msgr2=0x7ff1380fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.132+0000 7ff140600700 1 -- 172.21.15.116:0/339980734 shutdown_connections 2024-03-30T01:11:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.132+0000 7ff140600700 1 -- 172.21.15.116:0/339980734 wait complete. 2024-03-30T01:11:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.133+0000 7ff140600700 1 Processor -- start 2024-03-30T01:11:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.133+0000 7ff140600700 1 -- start start 2024-03-30T01:11:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.133+0000 7ff140600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff138100c80 0x7ff138196cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.133+0000 7ff140600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff138101e70 0x7ff138197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.133+0000 7ff140600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff138102810 0x7ff13819e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.133+0000 7ff140600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13819e810 con 0x7ff138102810 2024-03-30T01:11:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.133+0000 7ff140600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13819e980 con 0x7ff138100c80 2024-03-30T01:11:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.133+0000 7ff140600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13819eb40 con 0x7ff138101e70 2024-03-30T01:11:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.133+0000 7ff13e600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff138102810 0x7ff13819e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.133+0000 7ff13e600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff138102810 0x7ff13819e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51780/0 (socket says 172.21.15.116:51780) 2024-03-30T01:11:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.133+0000 7ff13e600700 1 -- 172.21.15.116:0/593021763 learned_addr learned my addr 172.21.15.116:0/593021763 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.133+0000 7ff13d200700 1 --2- 172.21.15.116:0/593021763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff138101e70 0x7ff138197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.133+0000 7ff13dc00700 1 --2- 172.21.15.116:0/593021763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff138100c80 0x7ff138196cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.134+0000 7ff13e600700 1 -- 172.21.15.116:0/593021763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff138101e70 msgr2=0x7ff138197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.134+0000 7ff13e600700 1 --2- 172.21.15.116:0/593021763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff138101e70 0x7ff138197230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.134+0000 7ff13e600700 1 -- 172.21.15.116:0/593021763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff138100c80 msgr2=0x7ff138196cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.134+0000 7ff13e600700 1 --2- 172.21.15.116:0/593021763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff138100c80 0x7ff138196cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.134+0000 7ff13e600700 1 -- 172.21.15.116:0/593021763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff130009d00 con 0x7ff138102810 2024-03-30T01:11:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.134+0000 7ff13e600700 1 --2- 172.21.15.116:0/593021763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff138102810 0x7ff13819e240 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff130007b60 tx=0x7ff130008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.134+0000 7ff136000700 1 -- 172.21.15.116:0/593021763 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13000e040 con 0x7ff138102810 2024-03-30T01:11:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.134+0000 7ff136000700 1 -- 172.21.15.116:0/593021763 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff13001d930 con 0x7ff138102810 2024-03-30T01:11:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.134+0000 7ff140600700 1 -- 172.21.15.116:0/593021763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13819edc0 con 0x7ff138102810 2024-03-30T01:11:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.134+0000 7ff136000700 1 -- 172.21.15.116:0/593021763 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13001b9f0 con 0x7ff138102810 2024-03-30T01:11:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.135+0000 7ff140600700 1 -- 172.21.15.116:0/593021763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13819f190 con 0x7ff138102810 2024-03-30T01:11:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.135+0000 7ff140600700 1 -- 172.21.15.116:0/593021763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff138108030 con 0x7ff138102810 2024-03-30T01:11:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.137+0000 7ff136000700 1 -- 172.21.15.116:0/593021763 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7ff13001daa0 con 0x7ff138102810 2024-03-30T01:11:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.139+0000 7ff136000700 1 --2- 172.21.15.116:0/593021763 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1200a1120 0x7ff1200a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.140+0000 7ff136000700 1 -- 172.21.15.116:0/593021763 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff1300ce170 con 0x7ff138102810 2024-03-30T01:11:02.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.140+0000 7ff13dc00700 1 --2- 172.21.15.116:0/593021763 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1200a1120 0x7ff1200a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:02.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.140+0000 7ff13dc00700 1 --2- 172.21.15.116:0/593021763 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1200a1120 0x7ff1200a35e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff1380fe580 tx=0x7ff12c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:02.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.143+0000 7ff136000700 1 -- 172.21.15.116:0/593021763 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff130097620 con 0x7ff138102810 2024-03-30T01:11:02.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.408+0000 7ff140600700 1 -- 172.21.15.116:0/593021763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7ff13810c4f0 con 0x7ff138102810 2024-03-30T01:11:02.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.419+0000 7ff136000700 1 -- 172.21.15.116:0/593021763 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) v1 ==== 81+0+733401 (secure 0 0 0) 0x7ff13009ac40 con 0x7ff138102810 2024-03-30T01:11:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T01:11:02.442 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:11:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 18, 2024-03-30T01:11:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "active_gid": 14152, 2024-03-30T01:11:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "active_name": "x", 2024-03-30T01:11:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "active_addrs": { 2024-03-30T01:11:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-30T01:11:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-30T01:11:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6800", 2024-03-30T01:11:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 3072243062 2024-03-30T01:11:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-30T01:11:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6801", 2024-03-30T01:11:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 3072243062 2024-03-30T01:11:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-30T01:11:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "active_addr": "172.21.15.116:6801/3072243062", 2024-03-30T01:11:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "active_change": "2024-03-30T00:50:35.029597+0000", 2024-03-30T01:11:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "active_mgr_features": 4540138322906710015, 2024-03-30T01:11:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "available": true, 2024-03-30T01:11:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "standbys": [ 2024-03-30T01:11:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "gid": 24112, 2024-03-30T01:11:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "z", 2024-03-30T01:11:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr_features": 4540138322906710015, 2024-03-30T01:11:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-30T01:11:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-30T01:11:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-30T01:11:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-30T01:11:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-30T01:11:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-30T01:11:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-30T01:11:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-30T01:11:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-30T01:11:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-30T01:11:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-30T01:11:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-30T01:11:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-30T01:11:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-30T01:11:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-30T01:11:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-30T01:11:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-30T01:11:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-30T01:11:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-30T01:11:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-30T01:11:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-30T01:11:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-30T01:11:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-30T01:11:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-30T01:11:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-30T01:11:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-30T01:11:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-30T01:11:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-30T01:11:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-30T01:11:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-30T01:11:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-30T01:11:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-30T01:11:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-30T01:11:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-30T01:11:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-30T01:11:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-30T01:11:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-30T01:11:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-30T01:11:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap" 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-30T01:11:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-30T01:11:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-30T01:11:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-30T01:11:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-30T01:11:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-30T01:11:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-30T01:11:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-30T01:11:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-30T01:11:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-30T01:11:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-30T01:11:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-30T01:11:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-30T01:11:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-30T01:11:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-30T01:11:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-30T01:11:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-30T01:11:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-30T01:11:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-30T01:11:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-30T01:11:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-30T01:11:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-30T01:11:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-30T01:11:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-30T01:11:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-30T01:11:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-30T01:11:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-30T01:11:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-30T01:11:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-30T01:11:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-30T01:11:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-30T01:11:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-30T01:11:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-30T01:11:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-30T01:11:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-30T01:11:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-30T01:11:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-30T01:11:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-30T01:11:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-30T01:11:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-30T01:11:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-30T01:11:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-30T01:11:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-30T01:11:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-30T01:11:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-30T01:11:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-30T01:11:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-30T01:11:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-30T01:11:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-30T01:11:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-30T01:11:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-30T01:11:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-30T01:11:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-30T01:11:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-30T01:11:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-30T01:11:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-30T01:11:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-30T01:11:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-30T01:11:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-30T01:11:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-30T01:11:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-30T01:11:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-30T01:11:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-30T01:11:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-30T01:11:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-30T01:11:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-30T01:11:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-30T01:11:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-30T01:11:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-30T01:11:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-30T01:11:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-30T01:11:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-30T01:11:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-30T01:11:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-30T01:11:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-30T01:11:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-30T01:11:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-30T01:11:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-30T01:11:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-30T01:11:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-30T01:11:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-30T01:11:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-30T01:11:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-30T01:11:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-30T01:11:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-30T01:11:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-30T01:11:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-30T01:11:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-30T01:11:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-30T01:11:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-30T01:11:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-30T01:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-30T01:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-30T01:11:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-30T01:11:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-30T01:11:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-30T01:11:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-30T01:11:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-30T01:11:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-30T01:11:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-30T01:11:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-30T01:11:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-30T01:11:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-30T01:11:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-30T01:11:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-30T01:11:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-30T01:11:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-30T01:11:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-30T01:11:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-30T01:11:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-30T01:11:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-30T01:11:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-30T01:11:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-30T01:11:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-30T01:11:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-30T01:11:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-30T01:11:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-30T01:11:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-30T01:11:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-30T01:11:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-30T01:11:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-30T01:11:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-30T01:11:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-30T01:11:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-30T01:11:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-30T01:11:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-30T01:11:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-30T01:11:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-30T01:11:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-30T01:11:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-30T01:11:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-30T01:11:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-30T01:11:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-30T01:11:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-30T01:11:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-30T01:11:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-30T01:11:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-30T01:11:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-30T01:11:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-30T01:11:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "UNSAFE_TLS_v1_2": { 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-30T01:11:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-30T01:11:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-30T01:11:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-30T01:11:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-30T01:11:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-30T01:11:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-30T01:11:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-30T01:11:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-30T01:11:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-30T01:11:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-30T01:11:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-30T01:11:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-30T01:11:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-30T01:11:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-30T01:11:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-30T01:11:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-30T01:11:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-30T01:11:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-30T01:11:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-30T01:11:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-30T01:11:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-30T01:11:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-30T01:11:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-30T01:11:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-30T01:11:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-30T01:11:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-30T01:11:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-30T01:11:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-30T01:11:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-30T01:11:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-30T01:11:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-30T01:11:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-30T01:11:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-30T01:11:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-30T01:11:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-30T01:11:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-30T01:11:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-30T01:11:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-30T01:11:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-30T01:11:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-30T01:11:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-30T01:11:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-30T01:11:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-30T01:11:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-30T01:11:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-30T01:11:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-30T01:11:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-30T01:11:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-30T01:11:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-30T01:11:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-30T01:11:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-30T01:11:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-30T01:11:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-30T01:11:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-30T01:11:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-30T01:11:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-30T01:11:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-30T01:11:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-30T01:11:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-30T01:11:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-30T01:11:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-30T01:11:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-30T01:11:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-30T01:11:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-30T01:11:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-30T01:11:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-30T01:11:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-30T01:11:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-30T01:11:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-30T01:11:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-30T01:11:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-30T01:11:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-30T01:11:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-30T01:11:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-30T01:11:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-30T01:11:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-30T01:11:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-30T01:11:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-30T01:11:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-30T01:11:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-30T01:11:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-30T01:11:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-30T01:11:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-30T01:11:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-30T01:11:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-30T01:11:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-30T01:11:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-30T01:11:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-30T01:11:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-30T01:11:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-30T01:11:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-30T01:11:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-30T01:11:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-30T01:11:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-30T01:11:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-30T01:11:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-30T01:11:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-30T01:11:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-30T01:11:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-30T01:11:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-30T01:11:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-30T01:11:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-30T01:11:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-30T01:11:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-30T01:11:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-30T01:11:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-30T01:11:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-30T01:11:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-30T01:11:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-30T01:11:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-30T01:11:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-30T01:11:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-30T01:11:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-30T01:11:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-30T01:11:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-30T01:11:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-30T01:11:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-30T01:11:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-30T01:11:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-30T01:11:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-30T01:11:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-30T01:11:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-30T01:11:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-30T01:11:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-30T01:11:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-30T01:11:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-30T01:11:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-30T01:11:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-30T01:11:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-30T01:11:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-30T01:11:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-30T01:11:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-30T01:11:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-30T01:11:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-30T01:11:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-30T01:11:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-30T01:11:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-30T01:11:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-30T01:11:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-30T01:11:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-30T01:11:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-30T01:11:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-30T01:11:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-30T01:11:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-30T01:11:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-30T01:11:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-30T01:11:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-30T01:11:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-30T01:11:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-30T01:11:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-30T01:11:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-30T01:11:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-30T01:11:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-30T01:11:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-30T01:11:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-30T01:11:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-30T01:11:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-30T01:11:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-30T01:11:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-30T01:11:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-30T01:11:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-30T01:11:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-30T01:11:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-30T01:11:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-30T01:11:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-30T01:11:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-30T01:11:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-30T01:11:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-30T01:11:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-30T01:11:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-30T01:11:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-30T01:11:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-30T01:11:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-30T01:11:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-30T01:11:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-30T01:11:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-30T01:11:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-30T01:11:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-30T01:11:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-30T01:11:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-30T01:11:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-30T01:11:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-30T01:11:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-30T01:11:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-30T01:11:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-30T01:11:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-30T01:11:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-30T01:11:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-30T01:11:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-30T01:11:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-30T01:11:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-30T01:11:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-30T01:11:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-30T01:11:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-30T01:11:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-30T01:11:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-30T01:11:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-30T01:11:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-30T01:11:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-30T01:11:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-30T01:11:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-30T01:11:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-30T01:11:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "gid": 24116, 2024-03-30T01:11:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "y", 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr_features": 4540138322906710015, 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-30T01:11:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-30T01:11:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-30T01:11:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-30T01:11:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-30T01:11:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-30T01:11:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-30T01:11:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-30T01:11:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-30T01:11:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-30T01:11:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-30T01:11:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-30T01:11:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-30T01:11:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-30T01:11:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-30T01:11:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-30T01:11:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-30T01:11:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-30T01:11:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-30T01:11:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-30T01:11:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-30T01:11:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-30T01:11:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-30T01:11:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-30T01:11:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-30T01:11:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-30T01:11:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-30T01:11:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-30T01:11:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-30T01:11:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-30T01:11:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-30T01:11:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-30T01:11:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-30T01:11:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-30T01:11:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-30T01:11:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-30T01:11:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-30T01:11:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-30T01:11:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-30T01:11:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-30T01:11:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-30T01:11:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-30T01:11:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-30T01:11:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap" 2024-03-30T01:11:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-30T01:11:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-30T01:11:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-30T01:11:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-30T01:11:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-30T01:11:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-30T01:11:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-30T01:11:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-30T01:11:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-30T01:11:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-30T01:11:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-30T01:11:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-30T01:11:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-30T01:11:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-30T01:11:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-30T01:11:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-30T01:11:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-30T01:11:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-30T01:11:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-30T01:11:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-30T01:11:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-30T01:11:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-30T01:11:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-30T01:11:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-30T01:11:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-30T01:11:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-30T01:11:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-30T01:11:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-30T01:11:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-30T01:11:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-30T01:11:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-30T01:11:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-30T01:11:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-30T01:11:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-30T01:11:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-30T01:11:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-30T01:11:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-30T01:11:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-30T01:11:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-30T01:11:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-30T01:11:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-30T01:11:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-30T01:11:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-30T01:11:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-03-30T01:11:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-30T01:11:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-30T01:11:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-30T01:11:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-30T01:11:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-30T01:11:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-30T01:11:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-30T01:11:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-30T01:11:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-30T01:11:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-30T01:11:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-30T01:11:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-30T01:11:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-30T01:11:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-30T01:11:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-30T01:11:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-30T01:11:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-30T01:11:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-30T01:11:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-30T01:11:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-30T01:11:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-30T01:11:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-30T01:11:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-30T01:11:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-30T01:11:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-30T01:11:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-30T01:11:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-30T01:11:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-30T01:11:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-30T01:11:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-30T01:11:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-30T01:11:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-30T01:11:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-30T01:11:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-30T01:11:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-30T01:11:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-30T01:11:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-30T01:11:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-30T01:11:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-30T01:11:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-30T01:11:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-30T01:11:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-30T01:11:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-30T01:11:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-30T01:11:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-30T01:11:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-30T01:11:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-30T01:11:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-30T01:11:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-30T01:11:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-30T01:11:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-30T01:11:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-30T01:11:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-30T01:11:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-30T01:11:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-30T01:11:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-30T01:11:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-30T01:11:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-30T01:11:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-30T01:11:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-30T01:11:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-30T01:11:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-30T01:11:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-30T01:11:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-30T01:11:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-30T01:11:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-30T01:11:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-30T01:11:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-30T01:11:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-30T01:11:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-30T01:11:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-30T01:11:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-30T01:11:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-30T01:11:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-30T01:11:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-30T01:11:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-30T01:11:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-30T01:11:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-30T01:11:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-30T01:11:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-30T01:11:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-30T01:11:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-30T01:11:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-30T01:11:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-30T01:11:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-30T01:11:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-30T01:11:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-30T01:11:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-30T01:11:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-30T01:11:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-30T01:11:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-30T01:11:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-30T01:11:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-30T01:11:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-30T01:11:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-30T01:11:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-30T01:11:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-30T01:11:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-30T01:11:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-30T01:11:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-30T01:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-30T01:11:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-30T01:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-30T01:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-30T01:11:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-30T01:11:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-30T01:11:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-30T01:11:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-30T01:11:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-30T01:11:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-30T01:11:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-30T01:11:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-30T01:11:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-30T01:11:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-30T01:11:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-30T01:11:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-30T01:11:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-30T01:11:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-30T01:11:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-30T01:11:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-30T01:11:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-30T01:11:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-30T01:11:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-30T01:11:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-30T01:11:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-30T01:11:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-30T01:11:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-30T01:11:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-30T01:11:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-30T01:11:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-30T01:11:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-30T01:11:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-30T01:11:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "UNSAFE_TLS_v1_2": { 2024-03-30T01:11:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-30T01:11:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-30T01:11:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-30T01:11:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-30T01:11:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-30T01:11:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-30T01:11:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-30T01:11:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-30T01:11:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-30T01:11:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-30T01:11:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-30T01:11:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-30T01:11:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-30T01:11:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-30T01:11:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-30T01:11:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-30T01:11:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-30T01:11:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-30T01:11:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-30T01:11:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-30T01:11:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-30T01:11:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-30T01:11:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-30T01:11:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-30T01:11:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-30T01:11:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-30T01:11:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-30T01:11:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-30T01:11:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-30T01:11:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-30T01:11:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-30T01:11:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-30T01:11:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-30T01:11:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-30T01:11:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-30T01:11:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-30T01:11:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-30T01:11:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-30T01:11:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-30T01:11:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-30T01:11:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-30T01:11:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-30T01:11:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-30T01:11:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-30T01:11:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-30T01:11:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-30T01:11:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-30T01:11:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-30T01:11:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-30T01:11:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-30T01:11:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-30T01:11:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-30T01:11:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-30T01:11:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-30T01:11:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-30T01:11:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-30T01:11:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-30T01:11:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-30T01:11:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-30T01:11:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-30T01:11:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-30T01:11:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-30T01:11:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-30T01:11:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-30T01:11:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-30T01:11:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-30T01:11:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-30T01:11:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-30T01:11:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-30T01:11:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-30T01:11:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-30T01:11:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-30T01:11:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-30T01:11:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-30T01:11:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-30T01:11:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-30T01:11:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-30T01:11:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-30T01:11:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-30T01:11:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-30T01:11:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-30T01:11:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.094 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.094 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.094 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.094 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.094 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.094 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.094 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.094 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.094 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.094 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.094 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.094 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.095 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.096 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.096 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.096 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.096 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.096 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.096 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.096 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.096 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.096 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.096 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.096 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.096 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.096 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.096 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.098 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-30T01:11:03.098 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-30T01:11:03.098 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.098 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.098 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.098 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.098 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.098 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.098 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.098 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-30T01:11:03.098 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-30T01:11:03.098 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.098 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.099 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-30T01:11:03.099 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-30T01:11:03.099 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.099 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.099 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.099 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-30T01:11:03.099 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.099 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.099 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.099 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-30T01:11:03.099 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.099 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.099 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.100 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-30T01:11:03.101 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-30T01:11:03.101 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.101 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.101 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.101 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.101 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.101 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.101 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.101 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-30T01:11:03.101 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.101 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.101 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.101 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-30T01:11:03.102 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-30T01:11:03.102 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.102 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.102 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.102 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-30T01:11:03.102 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.102 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.102 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.102 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-30T01:11:03.102 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-30T01:11:03.102 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.102 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.103 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-30T01:11:03.103 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.103 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.103 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.103 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.103 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.103 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.103 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.103 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.103 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.103 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.104 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.104 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.104 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.104 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.104 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.104 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.104 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.104 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.104 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.104 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.104 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.104 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.104 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.105 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.105 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.105 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.105 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.105 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.105 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.105 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.105 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.105 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.105 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.105 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.105 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.105 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.106 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.106 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.106 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.106 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.106 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.106 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.106 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.106 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.106 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.106 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.106 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.106 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.106 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.107 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.107 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.107 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.107 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.107 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.107 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.107 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.107 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.107 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.107 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.108 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.108 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.108 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.108 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.108 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.108 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.108 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.108 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.108 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.108 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.108 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.108 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.109 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-30T01:11:03.109 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.109 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.109 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.109 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.109 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.109 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.109 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.109 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.109 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.109 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.109 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-30T01:11:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-30T01:11:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.125 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.125 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.125 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.125 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.125 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.125 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.125 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.125 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.125 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.125 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.125 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.125 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.126 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.126 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.126 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.126 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.126 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-30T01:11:03.126 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-30T01:11:03.126 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.126 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.126 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.126 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.126 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.126 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.127 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.128 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.129 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.130 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.130 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.130 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.130 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.130 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.130 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.130 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.130 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.130 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.130 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.130 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.130 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.130 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.130 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.131 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.132 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.133 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.134 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.134 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.134 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.134 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.134 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.134 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.134 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.134 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.134 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.134 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.134 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.134 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.134 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.135 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.135 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.135 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.135 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.135 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.135 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.135 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.135 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.135 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.135 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.135 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.135 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.135 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.136 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.136 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.136 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.136 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.136 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.136 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.136 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.136 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.136 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.136 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.136 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.136 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.136 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.136 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.137 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.137 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.137 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.137 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.137 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.137 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.137 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.137 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.137 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.137 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.137 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.137 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-30T01:11:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.144 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.145 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-30T01:11:03.145 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-30T01:11:03.145 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.145 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.145 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.145 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.145 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.145 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-30T01:11:03.145 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.145 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.145 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.145 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-30T01:11:03.145 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-30T01:11:03.146 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.146 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.146 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.146 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.146 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.146 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.146 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.146 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-30T01:11:03.146 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.146 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.146 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.146 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-30T01:11:03.146 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.147 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.148 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.149 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.149 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.149 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.149 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.149 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.149 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.149 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.149 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.149 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.149 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.149 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.149 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.149 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.150 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.151 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.151 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.151 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.151 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.151 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.151 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.151 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.151 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.151 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.151 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.151 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.151 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.151 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.152 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.152 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.152 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.152 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.152 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.152 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.152 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.152 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.152 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-30T01:11:03.152 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-30T01:11:03.152 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.152 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.152 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.153 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-30T01:11:03.154 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.154 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.154 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.154 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-30T01:11:03.154 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.154 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.154 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.154 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.154 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.154 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.154 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-30T01:11:03.154 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.154 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-30T01:11:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-30T01:11:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-30T01:11:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-30T01:11:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.162 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-30T01:11:03.162 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-30T01:11:03.162 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.162 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.162 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.162 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.162 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.162 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.162 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.162 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.162 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.162 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.162 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.163 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.163 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-30T01:11:03.163 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-30T01:11:03.163 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.163 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.163 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.163 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-30T01:11:03.163 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.163 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.163 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.163 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.163 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.163 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.163 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.164 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-30T01:11:03.164 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-30T01:11:03.164 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:03.164 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.164 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.164 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-30T01:11:03.164 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.164 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.164 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.164 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.164 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.164 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.164 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.165 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-30T01:11:03.165 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-30T01:11:03.165 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.165 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.165 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.165 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-30T01:11:03.165 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.165 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.165 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.165 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-30T01:11:03.165 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.165 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.165 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.166 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.166 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-30T01:11:03.166 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-30T01:11:03.166 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.166 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.166 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.166 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-30T01:11:03.166 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.166 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.166 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.166 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-30T01:11:03.166 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.166 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.167 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.167 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-30T01:11:03.167 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-30T01:11:03.167 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.167 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.169 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.169 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-30T01:11:03.169 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.169 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.169 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.169 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.169 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.170 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.170 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.170 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.170 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-30T01:11:03.170 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-30T01:11:03.170 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.170 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.170 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.170 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-30T01:11:03.170 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.170 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.170 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.170 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-30T01:11:03.170 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-30T01:11:03.171 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.172 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.172 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.172 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.172 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.172 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.172 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.172 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.172 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-30T01:11:03.172 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.172 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.172 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.172 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.172 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.173 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.174 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.174 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.174 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.174 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.174 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.174 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.174 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.174 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.174 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.174 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.174 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.174 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.174 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.175 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-30T01:11:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-30T01:11:03.178 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.178 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.178 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.178 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:03.178 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.178 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.178 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.178 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.178 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.178 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.178 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.178 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-30T01:11:03.178 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-30T01:11:03.179 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.179 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.179 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.179 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.179 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.179 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.179 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.179 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.179 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.179 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.179 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.179 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-30T01:11:03.180 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-30T01:11:03.180 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.180 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.180 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.180 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.180 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.180 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.180 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.180 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.180 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.180 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.180 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.181 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-30T01:11:03.181 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.181 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.181 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.181 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-30T01:11:03.181 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-30T01:11:03.181 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.181 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.181 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.181 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.181 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.181 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.182 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.182 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.182 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.182 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.182 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.182 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-30T01:11:03.182 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-30T01:11:03.182 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.182 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.182 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.182 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.182 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.182 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.183 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.183 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.183 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.183 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.183 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.183 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.183 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.183 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.183 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.183 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.183 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.183 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.183 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.185 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.186 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.186 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.186 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.186 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.186 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.186 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.186 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.186 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.186 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.186 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.186 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.186 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.186 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.187 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.188 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.188 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.188 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.188 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.188 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-30T01:11:03.188 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-30T01:11:03.188 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.188 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.188 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.188 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.188 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.188 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.188 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.189 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.190 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.190 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.190 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.190 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.190 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.190 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.190 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-30T01:11:03.190 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.190 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.190 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.190 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.190 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.190 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.191 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.191 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.191 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.191 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.191 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.191 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.191 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.191 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.191 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.191 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.191 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.191 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.191 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.192 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.192 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.192 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.192 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.192 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.192 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.192 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.192 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.192 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.192 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.192 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.192 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.192 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.193 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.194 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.195 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.195 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.195 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.195 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.195 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.195 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.195 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.195 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.195 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.195 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.195 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.195 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.196 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-30T01:11:03.196 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.196 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.196 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.196 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.196 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.196 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.196 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.196 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.196 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.196 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.196 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.196 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.196 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.197 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.198 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.198 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.198 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.198 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.198 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.198 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.198 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.198 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.198 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.198 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.198 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.198 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.198 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.198 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.199 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.200 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.200 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.200 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.200 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.200 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.200 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.200 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.200 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.200 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.200 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.200 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.200 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.200 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.201 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.202 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.203 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.203 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.203 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.203 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.203 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.203 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.203 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.203 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.203 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.203 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.203 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.203 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.203 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.204 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.204 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.204 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.204 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.204 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.204 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.204 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.204 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.204 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.204 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.204 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.204 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.204 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.205 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.205 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.205 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.205 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.205 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.205 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.205 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.205 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.205 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.205 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.205 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.205 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.205 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.205 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.206 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.206 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.206 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.206 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.206 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.206 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.206 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.206 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.206 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.206 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.206 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.206 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.207 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.207 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.207 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.207 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.207 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.207 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.207 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.207 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-30T01:11:03.207 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-30T01:11:03.207 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.207 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.207 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.207 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.208 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.208 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.208 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.208 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.208 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.208 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.208 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.208 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-30T01:11:03.208 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-30T01:11:03.208 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.208 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.208 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.208 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-30T01:11:03.209 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.209 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.209 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.209 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.209 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.209 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.209 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.209 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-30T01:11:03.209 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-30T01:11:03.209 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.209 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.209 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.209 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.210 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.210 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.210 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.210 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.210 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.210 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.210 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.210 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-30T01:11:03.210 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-30T01:11:03.210 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.210 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.210 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.210 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.211 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.212 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.212 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.212 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.212 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.212 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.212 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.212 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-30T01:11:03.212 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-30T01:11:03.212 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.212 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.212 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.212 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.212 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.213 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.214 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.214 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.214 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.214 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.214 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.214 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.214 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-30T01:11:03.214 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-30T01:11:03.214 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.214 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.214 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.214 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.214 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.214 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-30T01:11:03.215 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.216 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.216 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.216 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.216 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.216 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-30T01:11:03.216 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-30T01:11:03.216 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.216 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.216 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.216 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.216 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.216 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.216 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.217 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.218 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.218 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.218 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.218 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-30T01:11:03.218 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-30T01:11:03.218 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.218 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.218 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.218 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.218 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.218 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.218 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.219 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.219 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.219 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.219 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.219 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.219 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.219 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-30T01:11:03.219 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.219 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.219 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.219 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-30T01:11:03.219 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-30T01:11:03.220 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.221 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.221 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.221 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.221 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.221 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.221 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.221 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-30T01:11:03.221 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.221 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.221 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.221 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.221 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.221 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.222 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.222 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.222 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.222 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.222 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.222 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.222 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.222 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.222 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.222 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.222 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.222 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.222 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.222 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.223 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.223 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.223 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.223 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.223 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.223 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.223 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.223 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.223 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.223 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.223 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.223 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.224 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.224 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.224 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.224 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.224 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.224 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.224 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.224 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.224 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.224 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.224 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.224 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.224 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.225 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.225 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.225 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.225 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.225 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.225 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.225 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.225 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.225 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.225 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.225 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.225 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.225 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.226 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.226 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.226 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.226 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.226 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.226 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.226 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.226 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.226 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.226 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.226 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.226 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.227 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.227 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.227 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.227 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-30T01:11:03.227 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.227 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.227 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.227 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.227 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.227 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.227 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.227 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.227 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.227 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.229 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.230 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.230 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.230 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.230 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.230 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.230 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.230 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.230 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.230 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.230 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.230 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.230 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.230 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.230 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.231 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.232 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.232 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.232 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.232 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.232 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.232 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.232 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.232 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-30T01:11:03.232 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.232 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.232 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.232 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.233 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.233 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.233 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.233 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.233 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.233 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.233 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.233 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.233 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.233 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.233 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.233 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.233 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.233 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.234 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.235 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.235 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.235 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.235 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.235 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.235 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.235 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.235 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.235 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.235 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.235 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.235 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.235 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.236 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.237 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.237 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.237 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.237 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.237 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.237 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.237 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.237 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.237 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.237 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.237 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.237 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.237 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-30T01:11:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-30T01:11:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-30T01:11:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-30T01:11:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.239 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.239 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.239 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.239 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:03.239 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:03.239 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.239 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.239 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.239 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-30T01:11:03.239 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.239 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.239 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.239 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.240 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.240 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.240 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.240 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.240 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.240 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.240 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.240 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.240 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.240 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.240 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.240 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.240 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.241 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.242 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.243 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.244 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.245 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.246 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-30T01:11:03.246 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.246 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.246 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.246 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-30T01:11:03.246 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-30T01:11:03.246 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.246 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.246 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.246 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.246 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.247 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.247 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.247 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-30T01:11:03.247 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.247 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.247 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.247 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-30T01:11:03.247 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-30T01:11:03.247 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.247 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.247 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.247 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.247 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.248 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.248 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.248 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-30T01:11:03.248 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.248 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.248 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.248 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-30T01:11:03.248 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-30T01:11:03.248 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.248 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.248 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.248 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.248 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.249 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.250 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.250 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-30T01:11:03.250 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.250 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.250 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.250 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-30T01:11:03.250 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-30T01:11:03.250 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.250 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.250 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.250 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.250 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.250 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.251 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.252 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.253 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.254 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.254 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.254 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:03.254 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:03.254 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.254 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.254 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.254 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-30T01:11:03.254 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-30T01:11:03.254 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.254 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.254 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.254 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.254 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.256 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-30T01:11:03.256 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-30T01:11:03.256 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.256 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.256 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.256 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.256 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.256 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.256 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.256 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.256 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.256 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.256 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.257 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-30T01:11:03.257 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-30T01:11:03.257 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.257 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.257 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.257 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.257 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.257 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.257 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.257 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.257 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.257 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.257 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.257 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.258 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.258 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.258 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.258 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.258 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.258 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.258 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.258 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.258 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.258 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.258 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.258 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.258 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.259 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.260 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.261 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.261 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.261 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.261 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.261 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.261 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.261 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.261 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.261 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.261 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.261 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.261 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.261 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-30T01:11:03.262 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-30T01:11:03.263 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.263 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.263 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.263 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.263 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.263 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.263 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.263 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.263 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.263 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.263 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.263 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-30T01:11:03.263 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-30T01:11:03.264 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.265 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-30T01:11:03.266 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.266 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.266 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.266 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.266 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.266 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.266 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.266 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.266 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.266 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.266 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.267 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.268 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.268 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.268 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.268 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.268 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.268 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.268 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.268 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.268 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.268 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.268 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.268 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.268 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.268 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.269 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.270 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.270 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.270 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.270 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.270 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.270 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.270 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.270 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.270 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.270 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.270 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.271 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.271 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.271 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.271 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.271 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.271 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.271 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.272 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.272 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-30T01:11:03.272 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.272 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.272 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.273 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.273 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.273 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.273 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.273 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.273 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.273 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.274 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.274 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.274 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.274 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.274 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.274 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.275 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.275 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.275 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.275 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.275 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.275 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.275 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.276 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.276 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.276 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.276 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.276 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.276 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.277 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.277 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.277 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.277 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.277 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.277 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.278 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.278 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.278 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.278 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.278 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.278 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.279 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.279 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.279 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.279 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.279 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.279 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.280 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.280 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.280 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.280 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.280 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.280 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.280 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.280 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.280 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.280 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.280 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.280 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.280 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.281 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.281 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.281 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.281 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.281 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.281 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.281 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.281 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.281 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.281 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-30T01:11:03.281 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-30T01:11:03.281 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.281 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-30T01:11:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-30T01:11:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-30T01:11:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-30T01:11:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.283 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.284 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.284 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.284 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.284 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.284 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-30T01:11:03.284 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.284 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.284 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.284 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.284 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.284 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.284 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-30T01:11:03.284 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.285 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.286 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-30T01:11:03.286 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-30T01:11:03.286 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.286 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.286 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.286 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.286 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.286 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.286 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.286 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.286 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.286 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.287 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.287 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:03.287 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:03.287 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.287 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.287 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.287 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:03.287 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.287 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.287 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.287 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.287 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.287 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.288 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.288 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.288 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.288 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.288 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.288 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.288 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.288 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.288 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.288 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.288 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.288 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.288 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.289 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.290 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.290 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.290 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.290 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.290 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.290 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.290 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.290 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.290 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.290 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.290 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.290 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.290 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.290 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.291 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.291 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.291 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.291 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.291 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.291 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.291 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.291 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.291 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.291 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.291 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.291 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.291 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.291 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.293 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-30T01:11:03.293 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-30T01:11:03.293 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.293 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.293 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.293 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.293 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.293 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.293 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.293 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.293 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.293 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.293 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.293 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-30T01:11:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-30T01:11:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-30T01:11:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-30T01:11:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "modules": [ 2024-03-30T01:11:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-30T01:11:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "dashboard", 2024-03-30T01:11:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "iostat", 2024-03-30T01:11:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "nfs", 2024-03-30T01:11:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "restful" 2024-03-30T01:11:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-30T01:11:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-30T01:11:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.300 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.301 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.302 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-30T01:11:03.303 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-30T01:11:03.303 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.303 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.303 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.303 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.303 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.303 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.303 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.303 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-30T01:11:03.303 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.303 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.303 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.303 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.303 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-30T01:11:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-30T01:11:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-30T01:11:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-30T01:11:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-30T01:11:03.305 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-30T01:11:03.306 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.306 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.306 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.306 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.306 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.306 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.306 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.306 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-30T01:11:03.306 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.306 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.306 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.306 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.306 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-30T01:11:03.307 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-30T01:11:03.307 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.307 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.307 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.307 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-30T01:11:03.307 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.307 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.307 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.307 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-30T01:11:03.307 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.307 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.307 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.307 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.308 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-30T01:11:03.308 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-30T01:11:03.308 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.308 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.308 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.308 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.308 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.308 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.308 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.308 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-30T01:11:03.308 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.308 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.308 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.308 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-30T01:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-30T01:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-30T01:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-30T01:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-30T01:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-30T01:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-30T01:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-30T01:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-30T01:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-30T01:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-30T01:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-30T01:11:03.315 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.316 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.316 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-30T01:11:03.316 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-30T01:11:03.316 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:03.316 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.316 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.316 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-30T01:11:03.316 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-30T01:11:03.316 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-30T01:11:03.316 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.316 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-30T01:11:03.316 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-30T01:11:03.316 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.317 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.318 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-30T01:11:03.318 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-30T01:11:03.318 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:03.318 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.318 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.318 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:03.318 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-30T01:11:03.318 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-30T01:11:03.318 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.318 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-30T01:11:03.318 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-30T01:11:03.318 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.318 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.319 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.320 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.321 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.321 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.321 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.321 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.321 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.321 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.321 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.321 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.321 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.321 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.321 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.321 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.322 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.323 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.323 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.323 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.323 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.323 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.323 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.323 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.323 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.323 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.323 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.323 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.323 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-30T01:11:03.323 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-30T01:11:03.324 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:03.324 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.324 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.324 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:03.324 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.324 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.324 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.324 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-30T01:11:03.324 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.324 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.324 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.324 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-30T01:11:03.324 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap" 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.326 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.326 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-30T01:11:03.326 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-30T01:11:03.326 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.326 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.326 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.326 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.326 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.326 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.326 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.326 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-30T01:11:03.326 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.327 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.328 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.328 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-30T01:11:03.328 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-30T01:11:03.328 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.328 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.328 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.328 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-30T01:11:03.328 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-30T01:11:03.328 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.328 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.328 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-30T01:11:03.328 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-30T01:11:03.328 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.329 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.330 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.330 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.330 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.330 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.330 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-30T01:11:03.330 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.330 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.330 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.330 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-30T01:11:03.330 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-30T01:11:03.330 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:03.330 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.330 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.330 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-30T01:11:03.331 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.332 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.333 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-30T01:11:03.333 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.333 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.333 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.333 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-30T01:11:03.333 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-30T01:11:03.333 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.333 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.333 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.333 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.333 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.333 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.334 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.334 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-30T01:11:03.334 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-30T01:11:03.334 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.334 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.334 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.334 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-30T01:11:03.334 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-30T01:11:03.334 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.334 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.334 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.334 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:03.334 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.334 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.335 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.335 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-30T01:11:03.335 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.335 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.335 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.335 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.335 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-30T01:11:03.335 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-30T01:11:03.335 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:03.335 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.335 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.335 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-30T01:11:03.335 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.335 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.336 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.336 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-30T01:11:03.336 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.336 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.336 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.336 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-30T01:11:03.336 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-30T01:11:03.336 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.336 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.336 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.336 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.336 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.337 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.338 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.338 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-30T01:11:03.338 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.338 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.338 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.338 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-30T01:11:03.338 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-30T01:11:03.338 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.338 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.338 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.338 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.338 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.338 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-30T01:11:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-30T01:11:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-30T01:11:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-30T01:11:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-30T01:11:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-30T01:11:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-30T01:11:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-03-30T01:11:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-30T01:11:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-30T01:11:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-30T01:11:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-30T01:11:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-30T01:11:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-30T01:11:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-30T01:11:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-30T01:11:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-30T01:11:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-30T01:11:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-30T01:11:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-30T01:11:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-30T01:11:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-30T01:11:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-30T01:11:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-30T01:11:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-30T01:11:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-30T01:11:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-30T01:11:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-30T01:11:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-30T01:11:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-30T01:11:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-30T01:11:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-30T01:11:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-30T01:11:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.353 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.353 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-30T01:11:03.353 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-30T01:11:03.353 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.353 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.353 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.353 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-30T01:11:03.353 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.353 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.353 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.353 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-30T01:11:03.353 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.353 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.354 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-30T01:11:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-30T01:11:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-30T01:11:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.356 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-30T01:11:03.356 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-30T01:11:03.356 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.356 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.356 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.356 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-30T01:11:03.356 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.356 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.356 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.356 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-30T01:11:03.356 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.356 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.356 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.357 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-30T01:11:03.357 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-30T01:11:03.357 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.357 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.357 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.357 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-30T01:11:03.357 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.357 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.357 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.357 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-30T01:11:03.357 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.357 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.357 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.358 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.359 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-30T01:11:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-30T01:11:03.361 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-30T01:11:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-30T01:11:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-30T01:11:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.363 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.364 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-30T01:11:03.365 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.366 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-30T01:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.367 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.368 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-30T01:11:03.369 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.370 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.370 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.370 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.370 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.370 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.370 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.370 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-30T01:11:03.370 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.370 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.370 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.370 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.370 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-30T01:11:03.370 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-30T01:11:03.371 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.372 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.373 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.374 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.374 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.374 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.374 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.374 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-30T01:11:03.374 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.374 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.374 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.374 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.374 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-30T01:11:03.374 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-30T01:11:03.374 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.374 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.374 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.375 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.377 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.377 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-30T01:11:03.377 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.377 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.377 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.377 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-30T01:11:03.377 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.377 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.377 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.378 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-30T01:11:03.378 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-30T01:11:03.378 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.378 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.378 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.378 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-30T01:11:03.378 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.378 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.378 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.378 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-30T01:11:03.378 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.378 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.378 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.379 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-30T01:11:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-30T01:11:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-30T01:11:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-30T01:11:03.381 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-30T01:11:03.381 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.381 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.381 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.381 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.381 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.381 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.381 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.381 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-30T01:11:03.381 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.381 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.381 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.381 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-30T01:11:03.382 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-30T01:11:03.382 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.382 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.382 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.382 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.382 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.382 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.382 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.382 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-30T01:11:03.382 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.382 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.382 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.382 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-30T01:11:03.383 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-30T01:11:03.383 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.383 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.383 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.383 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.383 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.383 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.383 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.383 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-30T01:11:03.383 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.383 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.383 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.383 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-30T01:11:03.384 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-30T01:11:03.384 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.384 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.384 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.384 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-30T01:11:03.384 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.384 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.384 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.384 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-30T01:11:03.384 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.384 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.384 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.384 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-30T01:11:03.385 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-30T01:11:03.385 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.385 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.385 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.385 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.385 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.385 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.385 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.385 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-30T01:11:03.385 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.385 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.385 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.385 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-30T01:11:03.386 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-30T01:11:03.386 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.386 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.386 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.386 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.386 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.386 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.386 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.386 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-30T01:11:03.386 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.386 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.386 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.386 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-30T01:11:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-30T01:11:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-30T01:11:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.388 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-30T01:11:03.388 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-30T01:11:03.388 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.388 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.388 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.388 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.388 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.388 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.388 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.388 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-30T01:11:03.388 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.388 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.388 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.388 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.389 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-30T01:11:03.389 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-30T01:11:03.389 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.389 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.389 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.389 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.389 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.389 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.389 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.389 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-30T01:11:03.389 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.389 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.389 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.389 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.390 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.391 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.392 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.393 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.394 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.395 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.395 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.395 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.395 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.395 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.395 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.395 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.395 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.395 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.395 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.395 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.395 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.395 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.396 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.396 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.396 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-30T01:11:03.396 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-30T01:11:03.396 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.396 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.396 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.396 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-30T01:11:03.396 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.396 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.396 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.396 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-30T01:11:03.396 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.397 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.397 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.397 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-30T01:11:03.397 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-30T01:11:03.397 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.397 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.397 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.397 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-30T01:11:03.397 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.397 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.397 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.397 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-30T01:11:03.397 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-30T01:11:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-30T01:11:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-30T01:11:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-30T01:11:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-30T01:11:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-30T01:11:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-30T01:11:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.402 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.402 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.402 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.402 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.402 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.402 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.402 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.402 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.402 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-30T01:11:03.402 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-30T01:11:03.402 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.402 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.403 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.404 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.404 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.404 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.404 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.404 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.404 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.404 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.404 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.404 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-30T01:11:03.404 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-30T01:11:03.404 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.404 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.404 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-30T01:11:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-30T01:11:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-30T01:11:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-30T01:11:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-30T01:11:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-30T01:11:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.411 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.411 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.411 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.411 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.411 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.411 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.411 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-30T01:11:03.411 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-30T01:11:03.411 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.411 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.411 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.411 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.411 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.412 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.412 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.412 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.412 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.412 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.412 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.412 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-30T01:11:03.412 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-30T01:11:03.412 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.412 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.412 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.412 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-30T01:11:03.412 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.413 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.413 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.413 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.413 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.413 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.413 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.413 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-30T01:11:03.413 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-30T01:11:03.413 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.413 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.413 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.413 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.414 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.414 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.414 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.414 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.414 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.414 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.414 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.414 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-30T01:11:03.414 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-30T01:11:03.414 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.414 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.414 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.414 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.415 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.415 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.415 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.415 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.415 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.415 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.415 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.415 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-30T01:11:03.415 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-30T01:11:03.415 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.415 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.415 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.415 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-30T01:11:03.416 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.416 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.416 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.416 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.416 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.416 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.416 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.416 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-30T01:11:03.416 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-30T01:11:03.416 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.416 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.416 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.416 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.417 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.418 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.418 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.418 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.418 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.418 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.418 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.418 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-30T01:11:03.418 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-30T01:11:03.418 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.418 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.418 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.418 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.420 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.421 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.422 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.422 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.422 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.422 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.422 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-30T01:11:03.422 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-30T01:11:03.422 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.422 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.422 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.422 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.422 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.422 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.422 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-30T01:11:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-30T01:11:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.424 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.426 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.426 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-30T01:11:03.426 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-30T01:11:03.426 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.426 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.426 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.426 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.426 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.426 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.426 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.426 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.426 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.427 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.427 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.427 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-30T01:11:03.427 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-30T01:11:03.427 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.427 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.427 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.427 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.427 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.427 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.427 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.427 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.427 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.428 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.429 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.429 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-30T01:11:03.429 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-30T01:11:03.429 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.429 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.429 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.429 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.429 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.429 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.429 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.429 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.429 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.429 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.430 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.432 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-30T01:11:03.433 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-30T01:11:03.434 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.435 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.435 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.435 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-30T01:11:03.435 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.435 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.435 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.435 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.435 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.435 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.435 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.435 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-30T01:11:03.435 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-30T01:11:03.435 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.436 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.437 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.437 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.437 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.437 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.437 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.437 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.437 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.437 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.437 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.437 INFO:teuthology.orchestra.run.smithi116.stdout: "UNSAFE_TLS_v1_2": { 2024-03-30T01:11:03.437 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-30T01:11:03.437 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.437 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.438 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.439 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-30T01:11:03.440 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.441 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.442 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.442 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.442 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.442 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.442 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.442 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-30T01:11:03.442 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-30T01:11:03.442 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.442 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.442 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.442 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.442 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.443 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.444 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.444 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-30T01:11:03.444 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.444 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.444 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.444 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-30T01:11:03.444 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-30T01:11:03.444 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.444 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.444 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.444 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-30T01:11:03.444 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.444 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.445 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.446 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.447 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-30T01:11:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-30T01:11:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-30T01:11:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-30T01:11:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-30T01:11:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-30T01:11:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-30T01:11:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-30T01:11:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-30T01:11:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-30T01:11:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-30T01:11:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-30T01:11:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-30T01:11:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-30T01:11:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-30T01:11:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-30T01:11:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-30T01:11:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-30T01:11:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-30T01:11:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-30T01:11:03.461 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-30T01:11:03.461 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.461 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.461 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.461 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.461 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.461 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.461 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.461 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-30T01:11:03.461 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.461 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.461 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.461 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.463 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.464 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.465 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.465 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.465 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.465 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.465 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.465 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.465 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.465 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.465 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.465 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.465 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.465 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.465 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.466 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.466 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.466 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.466 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.466 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.466 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.466 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.466 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.466 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.466 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.466 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-30T01:11:03.466 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-30T01:11:03.467 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.467 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.467 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.467 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-30T01:11:03.467 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.467 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.467 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.467 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-30T01:11:03.467 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.467 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.467 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.468 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-30T01:11:03.468 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-30T01:11:03.468 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.468 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.468 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.468 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-30T01:11:03.468 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.468 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.468 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.468 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-30T01:11:03.469 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-30T01:11:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-30T01:11:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-30T01:11:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-30T01:11:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.471 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.471 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.471 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.471 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-30T01:11:03.471 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-30T01:11:03.471 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.471 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.471 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.471 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.471 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.471 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.471 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.471 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-30T01:11:03.471 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.472 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.473 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.473 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.473 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-30T01:11:03.473 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-30T01:11:03.473 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.473 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.473 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.473 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-30T01:11:03.473 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.473 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.473 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.473 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-30T01:11:03.473 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.473 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.474 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.475 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.475 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.475 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.475 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.475 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.475 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.475 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.475 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.475 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.475 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.475 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.475 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.475 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-30T01:11:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-30T01:11:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-30T01:11:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-30T01:11:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-30T01:11:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.481 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-30T01:11:03.482 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.482 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.482 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.482 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.482 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.482 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.482 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.482 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.482 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.482 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.482 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.482 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-30T01:11:03.482 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-30T01:11:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-30T01:11:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-30T01:11:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-30T01:11:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-30T01:11:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-30T01:11:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-30T01:11:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-30T01:11:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-30T01:11:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-30T01:11:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-30T01:11:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-30T01:11:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-30T01:11:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-30T01:11:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-30T01:11:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-30T01:11:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-30T01:11:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-30T01:11:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-30T01:11:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-30T01:11:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-30T01:11:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-30T01:11:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-30T01:11:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-30T01:11:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-30T01:11:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-30T01:11:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.497 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.498 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.498 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.498 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-30T01:11:03.498 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.498 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.498 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.498 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.498 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.498 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.498 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.498 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.498 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.498 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.499 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.499 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.499 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.499 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.499 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.499 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.499 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.499 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.499 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.499 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.499 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.499 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.499 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.499 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.500 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.501 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.502 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.502 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.502 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.502 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.502 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.502 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.502 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.502 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.502 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.502 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.502 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.502 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.502 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.502 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.503 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.505 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.507 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.507 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.507 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.507 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.507 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.507 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.507 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.507 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.507 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.507 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.507 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.507 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.507 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.508 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.508 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.508 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.508 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.508 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.508 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.508 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.508 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.508 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.508 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.508 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-30T01:11:03.508 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.509 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.509 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.509 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-30T01:11:03.509 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-30T01:11:03.509 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.509 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.509 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.509 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-30T01:11:03.509 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.509 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.509 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.509 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-30T01:11:03.509 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.509 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.510 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.510 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-30T01:11:03.510 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-30T01:11:03.510 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.510 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.510 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.510 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:03.510 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-30T01:11:03.510 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.510 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.510 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-30T01:11:03.510 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.511 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.512 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.514 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.514 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.514 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.514 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.514 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.514 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.514 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.514 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.514 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.514 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.514 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.514 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.514 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.514 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.515 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.516 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.517 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.518 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.518 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.518 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.518 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.518 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.518 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.518 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.518 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.518 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.518 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.518 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.518 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.518 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.519 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.520 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.520 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.520 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.520 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.520 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.520 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.520 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.520 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.520 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.520 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.520 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.520 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.520 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.520 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.522 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.522 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.522 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.522 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-30T01:11:03.522 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-30T01:11:03.522 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.522 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.522 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.522 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.522 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.522 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.522 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.522 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-30T01:11:03.523 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.524 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.524 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.524 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-30T01:11:03.524 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-30T01:11:03.524 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.524 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.524 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.524 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-30T01:11:03.524 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.524 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.524 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.524 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-30T01:11:03.524 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.525 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.525 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.525 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-30T01:11:03.525 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-30T01:11:03.525 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.525 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.525 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.525 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.525 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.525 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.525 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.525 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-30T01:11:03.525 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.526 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.526 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.526 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.526 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-30T01:11:03.526 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-30T01:11:03.526 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.526 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.526 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.526 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-30T01:11:03.526 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.526 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-30T01:11:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-30T01:11:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-30T01:11:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.528 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.528 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.528 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.528 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.528 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.528 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.528 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.528 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.528 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.528 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.528 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.528 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.528 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.528 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.529 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.530 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.530 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.530 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.530 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.530 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.530 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.530 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.530 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.530 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.530 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.530 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.530 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.530 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.531 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.531 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.531 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.531 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.531 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.531 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.531 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.531 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.531 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.531 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.531 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.531 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.533 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.534 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.534 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.534 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.534 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.534 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.534 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.534 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.534 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.534 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.534 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.534 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.534 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.534 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.535 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.536 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.536 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.536 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.536 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.536 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.536 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.536 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.536 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.536 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.536 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.536 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.536 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.536 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.536 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.537 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.537 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.537 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.537 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.537 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.537 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.537 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.537 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.537 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.537 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.537 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.537 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.537 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.538 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.538 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.538 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.538 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.538 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-30T01:11:03.538 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.538 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.538 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.538 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.538 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.538 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.538 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.539 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.540 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.541 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.541 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.541 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.541 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.541 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.541 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.541 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.541 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.541 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.541 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.541 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.541 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.541 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.542 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.542 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.542 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.542 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.542 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.542 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.542 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.542 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.542 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.542 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.542 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.542 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.542 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.543 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.543 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.543 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.543 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.543 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.543 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.543 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.543 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.543 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-30T01:11:03.543 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.543 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.545 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.545 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.545 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.545 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.545 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.545 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.545 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.545 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.545 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.545 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.545 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.545 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.545 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.545 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.546 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.546 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.546 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.546 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.546 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.546 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.546 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.546 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.546 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.546 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.546 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.546 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.547 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.547 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.547 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.547 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.547 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.547 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.547 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.547 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.547 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.547 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.547 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.547 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.547 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.548 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.548 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.548 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.548 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.548 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.548 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.548 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.548 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.548 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.548 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.548 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.548 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.548 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.548 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-30T01:11:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-30T01:11:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-30T01:11:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-30T01:11:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-30T01:11:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.553 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.554 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.554 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.554 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.554 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.554 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.554 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.554 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.554 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.554 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.554 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.554 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.554 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.554 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.554 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.555 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.556 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.556 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.556 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.556 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.556 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.556 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.556 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.556 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.556 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.556 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.556 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-30T01:11:03.556 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.556 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.557 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.557 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.557 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.557 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.557 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.557 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.557 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.557 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.557 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.557 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.557 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.557 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.557 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.557 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.558 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.558 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.558 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.558 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.558 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.558 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.558 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.558 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.558 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.558 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.558 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.558 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.558 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.559 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.560 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.560 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.560 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.560 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.560 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.560 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.560 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.560 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.560 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.560 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.560 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.560 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.560 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.561 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.561 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.561 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.561 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.561 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.561 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.561 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.561 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.561 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.561 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.561 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.561 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.561 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.562 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-30T01:11:03.562 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.562 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.562 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.562 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.562 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.562 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.562 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.562 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.562 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.562 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.563 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.564 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.564 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.564 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.564 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.564 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.564 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.564 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.564 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.564 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.564 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.564 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.564 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.566 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.566 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.566 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.566 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.566 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.566 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.566 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.566 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.566 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.566 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.566 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.566 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.567 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.567 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.567 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.567 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.567 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.567 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.567 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.567 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:03.567 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:03.567 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.567 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.567 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.568 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-30T01:11:03.569 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-30T01:11:03.569 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.569 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.569 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-30T01:11:03.569 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-30T01:11:03.569 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.569 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.569 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-30T01:11:03.569 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.570 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.570 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.570 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-30T01:11:03.570 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-30T01:11:03.570 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.570 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.570 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.570 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.570 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.570 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.570 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.570 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-30T01:11:03.570 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.570 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.571 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.572 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.572 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.572 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.572 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.572 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.572 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.572 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.572 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.572 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.572 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.572 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.572 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.572 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.573 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.573 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.573 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.573 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.573 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.573 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.573 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.573 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.573 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.573 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.573 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.573 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.574 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.574 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.574 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.574 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.574 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.574 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.574 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.574 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.574 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.574 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.574 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.574 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.574 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.575 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.575 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.575 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.575 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.575 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.575 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.575 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.575 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.575 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.575 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.575 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.575 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.575 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.576 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.576 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.576 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.576 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.576 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.576 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.576 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.576 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.576 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.576 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.576 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.576 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.576 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.577 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.577 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.577 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-30T01:11:03.577 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-30T01:11:03.577 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.577 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.577 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.577 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-30T01:11:03.577 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.577 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.577 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.577 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-30T01:11:03.578 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.578 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.578 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.578 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-30T01:11:03.578 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-30T01:11:03.578 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.578 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.578 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.578 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-30T01:11:03.578 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.578 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.578 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.578 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-30T01:11:03.579 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.581 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.581 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.581 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.581 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.581 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-30T01:11:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-30T01:11:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-30T01:11:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-30T01:11:03.583 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.584 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.584 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.584 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.584 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.584 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.584 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.584 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.584 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.584 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.584 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.584 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.584 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.584 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.585 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.586 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.586 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.586 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.586 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.586 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.586 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.586 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.586 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.586 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.586 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.586 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.586 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.587 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.589 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-30T01:11:03.590 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-30T01:11:03.590 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.590 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.590 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.590 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-30T01:11:03.590 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.590 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.590 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.590 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.590 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.590 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.590 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.590 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-30T01:11:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.592 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.592 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.592 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-30T01:11:03.592 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.592 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.592 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.592 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-30T01:11:03.592 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.592 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.592 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.592 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-30T01:11:03.592 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-30T01:11:03.592 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.593 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.593 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.593 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-30T01:11:03.593 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.593 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.593 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.593 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-30T01:11:03.593 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.593 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.593 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.593 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-30T01:11:03.593 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-30T01:11:03.594 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.594 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.594 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.594 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-30T01:11:03.594 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.594 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.594 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.594 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.594 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.594 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.594 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.594 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-30T01:11:03.594 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.596 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-30T01:11:03.596 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-30T01:11:03.596 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.596 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.596 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.596 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-30T01:11:03.596 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-30T01:11:03.596 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-30T01:11:03.596 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.596 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.596 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.596 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.596 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.596 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.597 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.597 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-30T01:11:03.597 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.597 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.597 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.597 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.597 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.597 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.597 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.597 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.597 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.597 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.598 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.598 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.598 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.598 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.598 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.598 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.598 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.598 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.598 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.598 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.598 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.598 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.598 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.600 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.601 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.601 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.601 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.601 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.601 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.601 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.601 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.601 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.601 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.601 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.601 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.601 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.601 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.602 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.602 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.602 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.602 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.602 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.602 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-30T01:11:03.602 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-30T01:11:03.602 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.602 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.602 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.602 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-30T01:11:03.602 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.602 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.603 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.604 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.604 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.604 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.604 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.604 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-30T01:11:03.604 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-30T01:11:03.604 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.604 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.604 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.604 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.604 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.604 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.604 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.605 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.605 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.605 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.605 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.605 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.605 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.605 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-30T01:11:03.605 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.605 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.605 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.605 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-30T01:11:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-30T01:11:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.607 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.608 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.608 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.608 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.608 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.608 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.608 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.608 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.608 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.608 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.608 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.608 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.608 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.608 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.608 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.609 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.609 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.609 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.609 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.609 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.609 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.609 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.609 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.609 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.609 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.609 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.609 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.610 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.610 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.610 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.610 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.610 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.610 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.610 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.610 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.610 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.610 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.610 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.610 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.610 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.611 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.611 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.611 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.611 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.611 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.611 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.611 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.611 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.611 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.611 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.611 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.611 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.611 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.612 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.612 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.612 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.612 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.612 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.612 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.612 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.612 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.612 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.612 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.612 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.612 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.612 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.613 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-30T01:11:03.613 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-30T01:11:03.613 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.613 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.613 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.613 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.613 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.613 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.613 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.613 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.613 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.613 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.613 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.614 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-30T01:11:03.614 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-30T01:11:03.614 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.614 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.614 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.614 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.614 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.614 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.614 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.614 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.614 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.614 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.614 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.615 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.616 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.616 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.616 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.616 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.616 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.616 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.616 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.616 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.616 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.616 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.616 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.616 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.616 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.617 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.617 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.617 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.617 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.617 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.617 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.617 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.617 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.617 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.617 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.617 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.617 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.617 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.618 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.620 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.621 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.622 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.622 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.622 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.622 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.622 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.622 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.622 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.622 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.622 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.622 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.622 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.622 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.622 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.624 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-30T01:11:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-30T01:11:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-30T01:11:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-30T01:11:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-30T01:11:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-30T01:11:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-30T01:11:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-30T01:11:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-30T01:11:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-30T01:11:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-30T01:11:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-30T01:11:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-30T01:11:03.638 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.639 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.640 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.641 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.642 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.642 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.642 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.642 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.642 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.642 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.642 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.642 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-30T01:11:03.642 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-30T01:11:03.642 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.642 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.642 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.642 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.644 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-30T01:11:03.644 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-30T01:11:03.644 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.644 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.644 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.644 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.644 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.644 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.644 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.644 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-30T01:11:03.644 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.644 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.644 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.645 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-30T01:11:03.645 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-30T01:11:03.645 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.645 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.645 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.645 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.645 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.645 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.645 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.645 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-30T01:11:03.645 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.645 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.645 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.646 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.646 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.646 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.646 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.646 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.646 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.646 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.646 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.646 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.646 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.646 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.647 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.655 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.656 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.656 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.656 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.656 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.656 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-30T01:11:03.656 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.656 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.656 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.656 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.656 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.656 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.656 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.656 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-30T01:11:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-30T01:11:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-30T01:11:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-30T01:11:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-30T01:11:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.666 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.666 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.666 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.666 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.666 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.666 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.666 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.666 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.666 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.666 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.666 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.666 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.667 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.667 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.667 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.667 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.667 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.667 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.667 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.667 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.667 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.668 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.668 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.668 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.668 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.668 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.668 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.668 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.669 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.669 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.669 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.669 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.669 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.669 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.669 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.670 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.670 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.670 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.670 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.670 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.670 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.671 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-30T01:11:03.671 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.671 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.671 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.671 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-30T01:11:03.671 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-30T01:11:03.671 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.672 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.672 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.672 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.672 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.672 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.672 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.672 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-30T01:11:03.673 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.673 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.673 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.673 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-30T01:11:03.673 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-30T01:11:03.674 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.674 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.674 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.674 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.674 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.674 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.674 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.675 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-30T01:11:03.675 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.675 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.675 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.675 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-30T01:11:03.675 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-30T01:11:03.675 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.675 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.675 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.675 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.675 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-30T01:11:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-30T01:11:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-30T01:11:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.677 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.678 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.678 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-30T01:11:03.678 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.678 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.678 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.678 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-30T01:11:03.678 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-30T01:11:03.678 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.678 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.678 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.678 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.678 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.678 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.679 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.679 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.679 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.679 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.679 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.679 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-30T01:11:03.679 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-30T01:11:03.679 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.679 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.679 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.679 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.679 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.679 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.680 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.680 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.680 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.680 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.680 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.680 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.680 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-30T01:11:03.680 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-30T01:11:03.680 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.680 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.680 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.680 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-30T01:11:03.680 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.681 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.681 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.681 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.681 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.681 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.681 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.681 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.681 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-30T01:11:03.681 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-30T01:11:03.681 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.681 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.681 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.681 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.681 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.682 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.682 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.682 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.682 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.682 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.682 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.682 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.682 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:03.682 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:03.682 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.682 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.682 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.682 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-30T01:11:03.683 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-30T01:11:03.683 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.683 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.683 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.683 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.683 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.683 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.683 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.683 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-30T01:11:03.683 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-30T01:11:03.683 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.683 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.683 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.683 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-30T01:11:03.684 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.684 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.684 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.684 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.684 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.684 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.684 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.684 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.684 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-30T01:11:03.684 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-30T01:11:03.684 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.684 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.684 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.684 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-30T01:11:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-30T01:11:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.686 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.686 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.686 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.686 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.686 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.686 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.686 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.686 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.686 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.686 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.686 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.686 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.686 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.687 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.687 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.687 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.687 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.687 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.687 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.687 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.687 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.687 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.687 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.687 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.687 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.687 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.688 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.689 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.690 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.690 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.690 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.690 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.690 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.690 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.690 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.690 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.690 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.690 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.690 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.690 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.690 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.690 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.691 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.691 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.691 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.691 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.691 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.691 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.691 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.691 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-30T01:11:03.691 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-30T01:11:03.691 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.691 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.691 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.691 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.692 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.693 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.693 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.693 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.693 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.693 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.693 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.693 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-30T01:11:03.693 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-30T01:11:03.693 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.693 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.693 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.693 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-30T01:11:03.693 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.694 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.694 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.694 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.694 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.694 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.694 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.694 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.694 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.694 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.694 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-30T01:11:03.694 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.694 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.694 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.695 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.696 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.696 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.696 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.696 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.696 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.696 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.696 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.696 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.696 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.696 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.696 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.696 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.697 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.698 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.699 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.699 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.699 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.699 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.699 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.699 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.699 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.699 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.699 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.699 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.699 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.699 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.699 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.700 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.701 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.701 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.701 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.701 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.701 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.701 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.701 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.701 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.701 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.701 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.701 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.701 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.701 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.702 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.702 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.702 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.702 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.702 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.702 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.702 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.702 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.702 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.702 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.702 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.702 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.702 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.703 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.703 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.703 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.703 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.703 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.703 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.703 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.703 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.703 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.703 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.703 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.703 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.703 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.703 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.704 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-30T01:11:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-30T01:11:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-30T01:11:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-30T01:11:03.706 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.706 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.706 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.706 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-30T01:11:03.706 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-30T01:11:03.706 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.706 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.706 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.706 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-30T01:11:03.706 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.706 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.706 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.707 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-30T01:11:03.707 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.707 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.707 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.707 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-30T01:11:03.707 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-30T01:11:03.707 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.707 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.707 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.707 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-30T01:11:03.707 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.707 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.707 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.708 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-30T01:11:03.708 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.708 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.708 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.708 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.708 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.708 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.708 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.708 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-30T01:11:03.708 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-30T01:11:03.708 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-30T01:11:03.708 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-30T01:11:03.708 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-30T01:11:03.708 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-30T01:11:03.709 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-30T01:11:03.709 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.709 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.709 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-30T01:11:03.709 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.709 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.709 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.709 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.709 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.709 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.709 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.709 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-30T01:11:03.709 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-30T01:11:03.710 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.710 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.710 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.710 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.710 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.710 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.710 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.710 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.710 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.710 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.710 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.710 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.710 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-30T01:11:03.710 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-30T01:11:03.711 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.712 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.713 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.713 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-30T01:11:03.713 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-30T01:11:03.713 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.713 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.713 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.713 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.713 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.713 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.713 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.713 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.713 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.713 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.713 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.714 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.714 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-30T01:11:03.714 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-30T01:11:03.714 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.714 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.714 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.714 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-30T01:11:03.714 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.714 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.714 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-30T01:11:03.714 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-30T01:11:03.714 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-30T01:11:03.714 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-30T01:11:03.714 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-30T01:11:03.715 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-30T01:11:03.715 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-30T01:11:03.715 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.715 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.715 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.715 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.715 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.715 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-30T01:11:03.715 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-30T01:11:03.715 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-30T01:11:03.715 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.715 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-30T01:11:03.715 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-30T01:11:03.716 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.716 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.716 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.716 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.716 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.716 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.716 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.716 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-30T01:11:03.716 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-30T01:11:03.716 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.716 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.716 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.717 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-30T01:11:03.717 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.717 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.717 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.717 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.717 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.717 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.717 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.717 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-30T01:11:03.717 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-30T01:11:03.717 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-30T01:11:03.717 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.717 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.718 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-30T01:11:03.718 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.718 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.718 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.718 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.718 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.718 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.718 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.718 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-30T01:11:03.718 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-30T01:11:03.718 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-30T01:11:03.719 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-30T01:11:03.719 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-30T01:11:03.719 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-30T01:11:03.719 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-30T01:11:03.719 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-30T01:11:03.719 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-30T01:11:03.719 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-30T01:11:03.719 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-30T01:11:03.719 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-30T01:11:03.719 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-30T01:11:03.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.719 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.720 INFO:teuthology.orchestra.run.smithi116.stdout: "services": { 2024-03-30T01:11:03.720 INFO:teuthology.orchestra.run.smithi116.stdout: "dashboard": "https://172.21.15.116:8443/" 2024-03-30T01:11:03.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.720 INFO:teuthology.orchestra.run.smithi116.stdout: "always_on_modules": { 2024-03-30T01:11:03.720 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus": [ 2024-03-30T01:11:03.720 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-30T01:11:03.720 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-30T01:11:03.720 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-30T01:11:03.720 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-30T01:11:03.720 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-30T01:11:03.720 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-30T01:11:03.720 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-30T01:11:03.720 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-30T01:11:03.720 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-30T01:11:03.721 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-30T01:11:03.721 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.721 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific": [ 2024-03-30T01:11:03.721 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-30T01:11:03.721 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-30T01:11:03.721 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-30T01:11:03.721 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-30T01:11:03.721 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-30T01:11:03.721 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-30T01:11:03.721 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-30T01:11:03.721 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-30T01:11:03.721 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-30T01:11:03.721 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-30T01:11:03.721 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.722 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy": [ 2024-03-30T01:11:03.722 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-30T01:11:03.722 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-30T01:11:03.722 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-30T01:11:03.722 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-30T01:11:03.722 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-30T01:11:03.722 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-30T01:11:03.722 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-30T01:11:03.722 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-30T01:11:03.722 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-30T01:11:03.722 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-30T01:11:03.722 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-30T01:11:03.722 INFO:teuthology.orchestra.run.smithi116.stdout: "reef": [ 2024-03-30T01:11:03.722 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-30T01:11:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-30T01:11:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-30T01:11:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-30T01:11:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-30T01:11:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-30T01:11:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-30T01:11:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-30T01:11:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-30T01:11:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-30T01:11:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-30T01:11:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: "last_failure_osd_epoch": 3, 2024-03-30T01:11:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: "active_clients": [ 2024-03-30T01:11:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1638454568 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "libcephsqlite", 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-30T01:11:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1330843846 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 721879338 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.726 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-30T01:11:03.726 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-30T01:11:03.726 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-30T01:11:03.726 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-30T01:11:03.726 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-30T01:11:03.726 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2494915144 2024-03-30T01:11:03.726 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.726 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-30T01:11:03.726 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:03.726 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-30T01:11:03.726 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:11:03.728 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:02 smithi130 ceph-mon[47626]: pgmap v686: 129 pgs: 129 active+clean; 674 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 352 op/s 2024-03-30T01:11:03.728 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:02 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/4206625356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T01:11:03.728 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:02 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/593021763' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-30T01:11:03.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:02 smithi181 ceph-mon[47622]: pgmap v686: 129 pgs: 129 active+clean; 674 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 352 op/s 2024-03-30T01:11:03.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:02 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/4206625356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T01:11:03.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:02 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/593021763' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-30T01:11:03.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.426+0000 7ff140600700 1 -- 172.21.15.116:0/593021763 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1200a1120 msgr2=0x7ff1200a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:03.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.426+0000 7ff140600700 1 --2- 172.21.15.116:0/593021763 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1200a1120 0x7ff1200a35e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff1380fe580 tx=0x7ff12c009260 comp rx=0 tx=0).stop 2024-03-30T01:11:03.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.426+0000 7ff140600700 1 -- 172.21.15.116:0/593021763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff138102810 msgr2=0x7ff13819e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:03.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.426+0000 7ff140600700 1 --2- 172.21.15.116:0/593021763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff138102810 0x7ff13819e240 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff130007b60 tx=0x7ff130008e40 comp rx=0 tx=0).stop 2024-03-30T01:11:03.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.426+0000 7ff140600700 1 -- 172.21.15.116:0/593021763 shutdown_connections 2024-03-30T01:11:03.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.426+0000 7ff140600700 1 --2- 172.21.15.116:0/593021763 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff1200a1120 0x7ff1200a35e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:03.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.426+0000 7ff140600700 1 --2- 172.21.15.116:0/593021763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff138102810 0x7ff13819e240 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:03.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.426+0000 7ff140600700 1 --2- 172.21.15.116:0/593021763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff138100c80 0x7ff138196cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:03.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.426+0000 7ff140600700 1 --2- 172.21.15.116:0/593021763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff138101e70 0x7ff138197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:03.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.426+0000 7ff140600700 1 -- 172.21.15.116:0/593021763 >> 172.21.15.116:0/593021763 conn(0x7ff1380fc1f0 msgr2=0x7ff138105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:03.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.426+0000 7ff140600700 1 -- 172.21.15.116:0/593021763 shutdown_connections 2024-03-30T01:11:03.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:02.426+0000 7ff140600700 1 -- 172.21.15.116:0/593021763 wait complete. 2024-03-30T01:11:03.736 DEBUG:tasks.check_counter:Getting stats from a 2024-03-30T01:11:03.736 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph daemon mds.a perf dump 2024-03-30T01:11:03.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:02 smithi116 ceph-mon[38086]: pgmap v686: 129 pgs: 129 active+clean; 674 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 352 op/s 2024-03-30T01:11:03.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:02 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/4206625356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T01:11:03.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:02 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/593021763' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-30T01:11:04.110 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T01:11:04.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:04 smithi116 ceph-mon[38086]: pgmap v687: 129 pgs: 129 active+clean; 671 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 991 KiB/s wr, 302 op/s 2024-03-30T01:11:05.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:04 smithi181 ceph-mon[47622]: pgmap v687: 129 pgs: 129 active+clean; 671 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 991 KiB/s wr, 302 op/s 2024-03-30T01:11:05.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:04 smithi130 ceph-mon[47626]: pgmap v687: 129 pgs: 129 active+clean; 671 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 991 KiB/s wr, 302 op/s 2024-03-30T01:11:05.289 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:11:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T01:11:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 249, 2024-03-30T01:11:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 246, 2024-03-30T01:11:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 481040, 2024-03-30T01:11:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 140384, 2024-03-30T01:11:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-30T01:11:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 0, 2024-03-30T01:11:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.068485491, 2024-03-30T01:11:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.035538917, 2024-03-30T01:11:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.021257882, 2024-03-30T01:11:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.007101226, 2024-03-30T01:11:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 245, 2024-03-30T01:11:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.032037881, 2024-03-30T01:11:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000130766 2024-03-30T01:11:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 481040, 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 140384 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 6, 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 190, 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 3312, 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 185920, 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 2, 2024-03-30T01:11:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.022297262, 2024-03-30T01:11:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.017290492, 2024-03-30T01:11:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.002750971, 2024-03-30T01:11:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T01:11:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 189, 2024-03-30T01:11:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.025522136, 2024-03-30T01:11:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000135037 2024-03-30T01:11:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-30T01:11:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 185920 2024-03-30T01:11:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T01:11:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 0, 2024-03-30T01:11:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2, 2024-03-30T01:11:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 0, 2024-03-30T01:11:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 0, 2024-03-30T01:11:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 2, 2024-03-30T01:11:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 1, 2024-03-30T01:11:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.000935895, 2024-03-30T01:11:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.000000000, 2024-03-30T01:11:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.000785650, 2024-03-30T01:11:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T01:11:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T01:11:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T01:11:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-30T01:11:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 0, 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T01:11:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-30T01:11:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T01:11:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T01:11:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-30T01:11:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-30T01:11:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-30T01:11:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-30T01:11:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T01:11:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-30T01:11:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T01:11:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-30T01:11:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 147920, 2024-03-30T01:11:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 27, 2024-03-30T01:11:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-30T01:11:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-30T01:11:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-30T01:11:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-30T01:11:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-30T01:11:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-30T01:11:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-30T01:11:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-30T01:11:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 0, 2024-03-30T01:11:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 0, 2024-03-30T01:11:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T01:11:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-30T01:11:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-30T01:11:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-30T01:11:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 928, 2024-03-30T01:11:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 2, 2024-03-30T01:11:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-30T01:11:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-30T01:11:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-30T01:11:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-30T01:11:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T01:11:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-30T01:11:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-30T01:11:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-30T01:11:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 251, 2024-03-30T01:11:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 445875, 2024-03-30T01:11:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 251, 2024-03-30T01:11:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-30T01:11:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-30T01:11:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 251, 2024-03-30T01:11:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 445875, 2024-03-30T01:11:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-30T01:11:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:05.302 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:11:05.678 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-30T01:11:05.678 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-30T01:11:05.678 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-03-30T01:11:05.679 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-03-30T01:11:05.679 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-03-30T01:11:05.679 DEBUG:tasks.check_counter:Getting stats from d 2024-03-30T01:11:05.679 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph daemon mds.d perf dump 2024-03-30T01:11:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T01:11:06.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:06 smithi116 ceph-mon[38086]: pgmap v688: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 980 KiB/s wr, 269 op/s 2024-03-30T01:11:07.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:06 smithi181 ceph-mon[47622]: pgmap v688: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 980 KiB/s wr, 269 op/s 2024-03-30T01:11:07.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:06 smithi130 ceph-mon[47626]: pgmap v688: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 980 KiB/s wr, 269 op/s 2024-03-30T01:11:07.241 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:11:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T01:11:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 0, 2024-03-30T01:11:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2, 2024-03-30T01:11:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 0, 2024-03-30T01:11:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 0, 2024-03-30T01:11:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-30T01:11:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 0, 2024-03-30T01:11:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.003407325, 2024-03-30T01:11:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.000000000, 2024-03-30T01:11:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.000808763, 2024-03-30T01:11:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T01:11:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 257, 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 252, 2024-03-30T01:11:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 485120, 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 143200, 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 2, 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 1, 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.061715368, 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.031728822, 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.023519669, 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.006975596, 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 252, 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.024773017, 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000098305 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 485120, 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 143200 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 2, 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 191, 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 320, 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 186000, 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 2, 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.021046766, 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.016587280, 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.002139396, 2024-03-30T01:11:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T01:11:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 189, 2024-03-30T01:11:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.025950387, 2024-03-30T01:11:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000137303 2024-03-30T01:11:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-30T01:11:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 186000 2024-03-30T01:11:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 0, 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T01:11:07.247 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-30T01:11:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 147937, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 28, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-30T01:11:07.249 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 0, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 0, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 928, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 2, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-30T01:11:07.250 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 255, 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 446395, 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 255, 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 255, 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 446395, 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:07.251 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:07.252 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:07.252 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:07.252 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:07.252 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:11:07.625 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-30T01:11:07.626 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-30T01:11:07.626 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-30T01:11:07.626 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-30T01:11:07.626 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-30T01:11:07.626 DEBUG:tasks.check_counter:Getting stats from g 2024-03-30T01:11:07.626 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph daemon mds.g perf dump 2024-03-30T01:11:07.992 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T01:11:08.911 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:08 smithi116 ceph-mon[38086]: pgmap v689: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 855 KiB/s wr, 199 op/s 2024-03-30T01:11:09.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:08 smithi181 ceph-mon[47622]: pgmap v689: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 855 KiB/s wr, 199 op/s 2024-03-30T01:11:09.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:08 smithi130 ceph-mon[47626]: pgmap v689: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 855 KiB/s wr, 199 op/s 2024-03-30T01:11:09.179 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:11:09.179 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T01:11:09.179 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 0, 2024-03-30T01:11:09.179 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2, 2024-03-30T01:11:09.179 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 0, 2024-03-30T01:11:09.180 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 0, 2024-03-30T01:11:09.180 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-30T01:11:09.180 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 0, 2024-03-30T01:11:09.180 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.003585938, 2024-03-30T01:11:09.180 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.000000000, 2024-03-30T01:11:09.180 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.000776554, 2024-03-30T01:11:09.180 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T01:11:09.180 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:09.181 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:09.181 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:09.181 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:09.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:09.181 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:09.182 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:09.182 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:09.182 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:09.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:09.182 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T01:11:09.182 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T01:11:09.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:09.182 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T01:11:09.183 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 255, 2024-03-30T01:11:09.183 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 250, 2024-03-30T01:11:09.183 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 484576, 2024-03-30T01:11:09.183 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 142016, 2024-03-30T01:11:09.183 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 2, 2024-03-30T01:11:09.183 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 1, 2024-03-30T01:11:09.183 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.057794355, 2024-03-30T01:11:09.183 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.025668074, 2024-03-30T01:11:09.183 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.026539911, 2024-03-30T01:11:09.184 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.006794687, 2024-03-30T01:11:09.184 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:09.184 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 250, 2024-03-30T01:11:09.184 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.052259807, 2024-03-30T01:11:09.184 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000209039 2024-03-30T01:11:09.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:09.184 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:09.184 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:09.185 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:09.185 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:09.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:09.185 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 484576, 2024-03-30T01:11:09.185 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 142016 2024-03-30T01:11:09.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:09.185 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T01:11:09.185 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 2, 2024-03-30T01:11:09.186 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 190, 2024-03-30T01:11:09.186 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 320, 2024-03-30T01:11:09.186 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 185824, 2024-03-30T01:11:09.186 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-30T01:11:09.186 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 2, 2024-03-30T01:11:09.186 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.025653251, 2024-03-30T01:11:09.186 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.017522684, 2024-03-30T01:11:09.187 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.005901667, 2024-03-30T01:11:09.187 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T01:11:09.187 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:09.187 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 188, 2024-03-30T01:11:09.187 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.027378982, 2024-03-30T01:11:09.187 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000145632 2024-03-30T01:11:09.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:09.187 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:09.187 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:09.188 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:09.188 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:09.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:09.188 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-30T01:11:09.188 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 185824 2024-03-30T01:11:09.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:09.188 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-30T01:11:09.188 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 0, 2024-03-30T01:11:09.189 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-30T01:11:09.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:09.189 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-30T01:11:09.189 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-30T01:11:09.189 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-30T01:11:09.189 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T01:11:09.190 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-30T01:11:09.190 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T01:11:09.190 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-30T01:11:09.190 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T01:11:09.190 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T01:11:09.190 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T01:11:09.191 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T01:11:09.191 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T01:11:09.191 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-30T01:11:09.191 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T01:11:09.191 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T01:11:09.191 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-30T01:11:09.191 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-30T01:11:09.192 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-30T01:11:09.192 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-30T01:11:09.192 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T01:11:09.192 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T01:11:09.192 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T01:11:09.192 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T01:11:09.192 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T01:11:09.192 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-30T01:11:09.192 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-30T01:11:09.193 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-30T01:11:09.193 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T01:11:09.193 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T01:11:09.193 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-30T01:11:09.193 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-30T01:11:09.193 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-30T01:11:09.193 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-30T01:11:09.193 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T01:11:09.194 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-30T01:11:09.194 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T01:11:09.194 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-30T01:11:09.194 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 147920, 2024-03-30T01:11:09.194 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 27, 2024-03-30T01:11:09.194 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-30T01:11:09.194 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-30T01:11:09.194 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-30T01:11:09.195 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-30T01:11:09.195 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-30T01:11:09.195 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-30T01:11:09.195 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-30T01:11:09.195 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-30T01:11:09.195 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 0, 2024-03-30T01:11:09.195 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 0, 2024-03-30T01:11:09.196 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T01:11:09.196 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-30T01:11:09.196 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-30T01:11:09.196 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-30T01:11:09.196 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 928, 2024-03-30T01:11:09.196 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 2, 2024-03-30T01:11:09.197 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-30T01:11:09.197 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-30T01:11:09.197 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-30T01:11:09.197 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-30T01:11:09.197 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:09.197 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T01:11:09.197 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-30T01:11:09.198 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-30T01:11:09.198 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-30T01:11:09.198 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 253, 2024-03-30T01:11:09.198 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 446135, 2024-03-30T01:11:09.198 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:09.198 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 253, 2024-03-30T01:11:09.198 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-30T01:11:09.199 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-30T01:11:09.199 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 253, 2024-03-30T01:11:09.199 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 446135, 2024-03-30T01:11:09.199 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-30T01:11:09.199 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:09.199 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:09.199 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:09.199 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:09.200 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:09.200 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:11:09.582 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-30T01:11:09.582 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-30T01:11:09.582 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-30T01:11:09.582 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-30T01:11:09.582 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-03-30T01:11:09.583 DEBUG:tasks.check_counter:Getting stats from j 2024-03-30T01:11:09.583 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph daemon mds.j perf dump 2024-03-30T01:11:09.947 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T01:11:11.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:10 smithi181 ceph-mon[47622]: pgmap v690: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 334 KiB/s wr, 177 op/s 2024-03-30T01:11:11.052 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-30T01:11:11.053 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T01:11:11.053 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 0, 2024-03-30T01:11:11.053 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2, 2024-03-30T01:11:11.053 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 0, 2024-03-30T01:11:11.053 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 0, 2024-03-30T01:11:11.053 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-30T01:11:11.053 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 0, 2024-03-30T01:11:11.053 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.003751496, 2024-03-30T01:11:11.053 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.000000000, 2024-03-30T01:11:11.053 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.000586619, 2024-03-30T01:11:11.053 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T01:11:11.053 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:11.053 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:11.053 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 259, 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 254, 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 485664, 2024-03-30T01:11:11.054 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 144384, 2024-03-30T01:11:11.055 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 2, 2024-03-30T01:11:11.055 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 1, 2024-03-30T01:11:11.055 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.050766676, 2024-03-30T01:11:11.055 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.014769238, 2024-03-30T01:11:11.055 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.021630011, 2024-03-30T01:11:11.055 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.012360488, 2024-03-30T01:11:11.055 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:11.055 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 254, 2024-03-30T01:11:11.055 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.047816637, 2024-03-30T01:11:11.055 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000188254 2024-03-30T01:11:11.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 485664, 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 144384 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 2, 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 193, 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 320, 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 186352, 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-30T01:11:11.056 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 2, 2024-03-30T01:11:11.057 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.022085722, 2024-03-30T01:11:11.057 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.016672563, 2024-03-30T01:11:11.057 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.002922835, 2024-03-30T01:11:11.057 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T01:11:11.057 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:11.057 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 191, 2024-03-30T01:11:11.057 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.029721199, 2024-03-30T01:11:11.059 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000155608 2024-03-30T01:11:11.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:11.059 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:11.059 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:11.059 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:11.059 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:11.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 186352 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 0, 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T01:11:11.060 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T01:11:11.061 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T01:11:11.062 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-30T01:11:11.062 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-30T01:11:11.062 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-30T01:11:11.062 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T01:11:11.062 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T01:11:11.062 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-30T01:11:11.063 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-30T01:11:11.063 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-30T01:11:11.063 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-30T01:11:11.063 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T01:11:11.063 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-30T01:11:11.063 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T01:11:11.064 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-30T01:11:11.064 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 148951, 2024-03-30T01:11:11.064 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 28, 2024-03-30T01:11:11.064 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-30T01:11:11.064 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-30T01:11:11.064 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-30T01:11:11.064 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-30T01:11:11.064 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-30T01:11:11.064 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-30T01:11:11.064 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-30T01:11:11.065 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-30T01:11:11.065 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 0, 2024-03-30T01:11:11.065 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 0, 2024-03-30T01:11:11.065 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T01:11:11.065 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-30T01:11:11.065 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-30T01:11:11.065 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-30T01:11:11.065 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 928, 2024-03-30T01:11:11.065 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 2, 2024-03-30T01:11:11.066 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-30T01:11:11.066 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-30T01:11:11.066 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-30T01:11:11.066 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-30T01:11:11.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-30T01:11:11.067 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T01:11:11.067 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-30T01:11:11.067 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-30T01:11:11.067 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-30T01:11:11.067 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 257, 2024-03-30T01:11:11.067 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 446655, 2024-03-30T01:11:11.067 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:11.067 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 257, 2024-03-30T01:11:11.068 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-30T01:11:11.068 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-30T01:11:11.068 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 257, 2024-03-30T01:11:11.068 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 446655, 2024-03-30T01:11:11.068 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-30T01:11:11.068 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-30T01:11:11.068 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-30T01:11:11.068 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-30T01:11:11.068 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:11.069 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-30T01:11:11.069 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-30T01:11:11.071 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:10 smithi116 ceph-mon[38086]: pgmap v690: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 334 KiB/s wr, 177 op/s 2024-03-30T01:11:11.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:10 smithi130 ceph-mon[47626]: pgmap v690: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 334 KiB/s wr, 177 op/s 2024-03-30T01:11:11.450 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-30T01:11:11.451 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-30T01:11:11.451 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-30T01:11:11.451 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-30T01:11:11.451 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-30T01:11:11.451 DEBUG:tasks.check_counter:Getting stats from b 2024-03-30T01:11:11.451 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph daemon mds.b perf dump 2024-03-30T01:11:11.847 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T01:11:12.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:12 smithi130 ceph-mon[47626]: pgmap v691: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 150 KiB/s wr, 95 op/s 2024-03-30T01:11:13.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:12 smithi181 ceph-mon[47622]: pgmap v691: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 150 KiB/s wr, 95 op/s 2024-03-30T01:11:13.076 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-03-30T01:11:13.076 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T01:11:13.076 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 95794, 2024-03-30T01:11:13.076 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 92468, 2024-03-30T01:11:13.076 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 32324244, 2024-03-30T01:11:13.076 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 349255059, 2024-03-30T01:11:13.076 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 82, 2024-03-30T01:11:13.076 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 6, 2024-03-30T01:11:13.077 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 5.951222889, 2024-03-30T01:11:13.077 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 2.198490358, 2024-03-30T01:11:13.077 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 2.295226281, 2024-03-30T01:11:13.077 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 1.697929288, 2024-03-30T01:11:13.077 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:13.077 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 92466, 2024-03-30T01:11:13.077 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 24.685623491, 2024-03-30T01:11:13.077 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000266969 2024-03-30T01:11:13.077 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.077 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:13.077 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 88736, 2024-03-30T01:11:13.077 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.012834282, 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000144 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 358191, 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 613586, 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 125177307, 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 474786715, 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 10, 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 6, 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 20.220533889, 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 8.527532439, 2024-03-30T01:11:13.078 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 45.437809279, 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 2.148550867, 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 613585, 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 139.825268894, 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000227882 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 674519, 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.129291163, 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000191 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 53456, 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 531440 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.079 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T01:11:13.080 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 122131, 2024-03-30T01:11:13.080 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 99689, 2024-03-30T01:11:13.080 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 81996207, 2024-03-30T01:11:13.080 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 171952213, 2024-03-30T01:11:13.080 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 14, 2024-03-30T01:11:13.080 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 6, 2024-03-30T01:11:13.080 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 6.819746953, 2024-03-30T01:11:13.080 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 2.455222365, 2024-03-30T01:11:13.080 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 3.153242511, 2024-03-30T01:11:13.080 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 1.959393659, 2024-03-30T01:11:13.080 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:13.080 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 99688, 2024-03-30T01:11:13.080 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 41.997946189, 2024-03-30T01:11:13.080 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000421293 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 94059, 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.011879340, 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000126 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 569360, 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 171808 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-03-30T01:11:13.081 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-03-30T01:11:13.082 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-03-30T01:11:13.082 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 60281, 2024-03-30T01:11:13.082 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 191.823464362, 2024-03-30T01:11:13.082 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.003182154 2024-03-30T01:11:13.082 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:13.082 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.082 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-03-30T01:11:13.082 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-03-30T01:11:13.082 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-03-30T01:11:13.082 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 54829, 2024-03-30T01:11:13.082 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 13.070922942, 2024-03-30T01:11:13.082 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000238394 2024-03-30T01:11:13.082 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:13.082 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.083 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-03-30T01:11:13.083 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 172520, 2024-03-30T01:11:13.083 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 161948, 2024-03-30T01:11:13.083 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-03-30T01:11:13.083 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 161948, 2024-03-30T01:11:13.083 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 1628.284462302, 2024-03-30T01:11:13.083 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.010054365 2024-03-30T01:11:13.083 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.083 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-03-30T01:11:13.083 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 10566, 2024-03-30T01:11:13.083 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 26, 2024-03-30T01:11:13.083 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 1, 2024-03-30T01:11:13.083 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 1785, 2024-03-30T01:11:13.083 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 174, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 7, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 44, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 0, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 5, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 39, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 42, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 126513, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 0, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 0, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 4, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 261172, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 234697, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 10439, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 1469, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2024-03-30T01:11:13.084 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 439, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 40, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 265, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 95798, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 257, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 75487, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 76386, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 4, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 234, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 0, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 76386, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-03-30T01:11:13.085 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 797, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 9, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 788, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 75589, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 149, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 34214, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 84110, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 6, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 101555, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 39742, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 2366, 2024-03-30T01:11:13.086 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 81584 2024-03-30T01:11:13.087 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.087 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-03-30T01:11:13.087 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 2323, 2024-03-30T01:11:13.087 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 4901, 2024-03-30T01:11:13.087 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 2687, 2024-03-30T01:11:13.087 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 4051, 2024-03-30T01:11:13.087 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 6025, 2024-03-30T01:11:13.087 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 0, 2024-03-30T01:11:13.087 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-03-30T01:11:13.087 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-03-30T01:11:13.087 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 32264, 2024-03-30T01:11:13.087 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 32264, 2024-03-30T01:11:13.087 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2024-03-30T01:11:13.087 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 9, 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 266, 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 181, 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 237063, 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 136253, 2024-03-30T01:11:13.088 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 136253, 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 100810, 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 257, 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 297, 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 169, 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 169, 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 129, 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 1, 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 356519575, 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 588905024, 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 0, 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4972, 2024-03-30T01:11:13.089 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 911.756942153, 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.183378306 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 0 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 20, 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 90780, 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 90760, 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 18, 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 6014, 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 5996, 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 44, 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 192982, 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 192938, 2024-03-30T01:11:13.090 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 0, 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 92038, 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 92038, 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 144804, 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 207156 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 188278, 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 4916, 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 172520, 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 483, 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 9129, 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 45678, 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 205.040132942, 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.004488815 2024-03-30T01:11:13.091 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4309, 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 176.085673417, 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.040864625 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.092 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 29091, 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 650.725031011, 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.022368603 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.093 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.094 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-03-30T01:11:13.095 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.095 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.095 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.095 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.095 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-03-30T01:11:13.095 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 310, 2024-03-30T01:11:13.095 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 6.861692830, 2024-03-30T01:11:13.095 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.022134493 2024-03-30T01:11:13.095 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.095 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-03-30T01:11:13.095 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 11086, 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 69.479610850, 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.006267329 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-03-30T01:11:13.096 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1145, 2024-03-30T01:11:13.097 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 2.769411857, 2024-03-30T01:11:13.097 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002418700 2024-03-30T01:11:13.097 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.097 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-03-30T01:11:13.097 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 45303, 2024-03-30T01:11:13.097 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 463.684782071, 2024-03-30T01:11:13.097 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.010235189 2024-03-30T01:11:13.097 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.097 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-03-30T01:11:13.097 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.097 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.097 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.097 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.097 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-03-30T01:11:13.098 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.098 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.098 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.098 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.098 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-03-30T01:11:13.098 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 178, 2024-03-30T01:11:13.098 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 19.502848013, 2024-03-30T01:11:13.098 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.109566561 2024-03-30T01:11:13.098 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.098 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-03-30T01:11:13.098 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.098 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.098 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.098 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.099 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-03-30T01:11:13.099 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.099 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.099 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.099 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.099 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-03-30T01:11:13.099 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 8, 2024-03-30T01:11:13.099 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.061200477, 2024-03-30T01:11:13.099 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.007650059 2024-03-30T01:11:13.099 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.099 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-03-30T01:11:13.099 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.099 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.099 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 8, 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.022125332, 2024-03-30T01:11:13.100 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002765666 2024-03-30T01:11:13.101 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.101 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-03-30T01:11:13.101 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 3, 2024-03-30T01:11:13.101 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.002911341, 2024-03-30T01:11:13.101 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000970447 2024-03-30T01:11:13.101 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.101 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-03-30T01:11:13.101 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 24829, 2024-03-30T01:11:13.101 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 34.049042161, 2024-03-30T01:11:13.101 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001371341 2024-03-30T01:11:13.102 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.102 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-03-30T01:11:13.102 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-03-30T01:11:13.102 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-03-30T01:11:13.102 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.102 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.102 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.102 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:13.102 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.102 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-03-30T01:11:13.102 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 1, 2024-03-30T01:11:13.102 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 3, 2024-03-30T01:11:13.102 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 2, 2024-03-30T01:11:13.102 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 1, 2024-03-30T01:11:13.103 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-03-30T01:11:13.103 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 19679, 2024-03-30T01:11:13.103 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 19679, 2024-03-30T01:11:13.103 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 813, 2024-03-30T01:11:13.103 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-03-30T01:11:13.103 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.103 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-03-30T01:11:13.103 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-03-30T01:11:13.103 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-03-30T01:11:13.103 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T01:11:13.103 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-03-30T01:11:13.103 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T01:11:13.103 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-03-30T01:11:13.103 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T01:11:13.104 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T01:11:13.105 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T01:11:13.105 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-03-30T01:11:13.105 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-03-30T01:11:13.105 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-03-30T01:11:13.105 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T01:11:13.105 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T01:11:13.105 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-03-30T01:11:13.105 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-03-30T01:11:13.105 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-03-30T01:11:13.105 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-03-30T01:11:13.105 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T01:11:13.105 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-03-30T01:11:13.105 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T01:11:13.105 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 308499, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 132, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 792, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 9, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 8696, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 166, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-03-30T01:11:13.106 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-03-30T01:11:13.107 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-03-30T01:11:13.107 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 103461, 2024-03-30T01:11:13.107 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 879, 2024-03-30T01:11:13.107 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-03-30T01:11:13.107 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-03-30T01:11:13.107 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-03-30T01:11:13.107 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-03-30T01:11:13.107 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.107 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-03-30T01:11:13.107 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-03-30T01:11:13.107 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-03-30T01:11:13.107 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 162058, 2024-03-30T01:11:13.107 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 630149793, 2024-03-30T01:11:13.107 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 37, 2024-03-30T01:11:13.108 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 162058, 2024-03-30T01:11:13.108 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-03-30T01:11:13.108 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 162058, 2024-03-30T01:11:13.108 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 7538.629155508, 2024-03-30T01:11:13.108 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.046518093 2024-03-30T01:11:13.108 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.108 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-03-30T01:11:13.108 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-03-30T01:11:13.108 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 162058, 2024-03-30T01:11:13.108 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 180006 2024-03-30T01:11:13.108 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.108 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 162058, 2024-03-30T01:11:13.108 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 77299, 2024-03-30T01:11:13.108 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 84759, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 185, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 10198, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 877, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 5845, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 459, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 174, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 66156, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-03-30T01:11:13.109 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-03-30T01:11:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 76400, 2024-03-30T01:11:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 15351, 2024-03-30T01:11:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-03-30T01:11:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-03-30T01:11:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-03-30T01:11:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-03-30T01:11:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-03-30T01:11:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-03-30T01:11:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T01:11:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-03-30T01:11:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-03-30T01:11:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 6, 2024-03-30T01:11:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-03-30T01:11:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-03-30T01:11:13.111 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-03-30T01:11:13.111 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-03-30T01:11:13.111 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-03-30T01:11:13.111 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-03-30T01:11:13.111 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-03-30T01:11:13.111 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-03-30T01:11:13.111 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-03-30T01:11:13.111 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-03-30T01:11:13.111 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-03-30T01:11:13.111 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-03-30T01:11:13.111 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-03-30T01:11:13.112 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-03-30T01:11:13.112 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-03-30T01:11:13.112 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-03-30T01:11:13.112 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 83, 2024-03-30T01:11:13.112 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-03-30T01:11:13.112 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 7, 2024-03-30T01:11:13.112 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 12, 2024-03-30T01:11:13.112 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 12, 2024-03-30T01:11:13.112 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2024-03-30T01:11:13.112 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-03-30T01:11:13.112 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 3807, 2024-03-30T01:11:13.112 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 54, 2024-03-30T01:11:13.112 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 494 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 1, 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 174563 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 24, 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-03-30T01:11:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 9, 2024-03-30T01:11:13.114 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 32264, 2024-03-30T01:11:13.114 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-03-30T01:11:13.114 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.114 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T01:11:13.114 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:13.114 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-30T01:11:13.114 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:13.114 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 576112, 2024-03-30T01:11:13.114 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 187621880, 2024-03-30T01:11:13.114 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:13.114 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 576112, 2024-03-30T01:11:13.114 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-30T01:11:13.114 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-30T01:11:13.114 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 576112, 2024-03-30T01:11:13.115 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 187621880, 2024-03-30T01:11:13.115 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:13.115 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.115 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.115 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.115 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:13.115 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.115 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-03-30T01:11:13.115 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:13.115 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-30T01:11:13.115 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:13.115 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-30T01:11:13.115 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-30T01:11:13.115 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 162058, 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 633665725, 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 162058, 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 633665725, 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-03-30T01:11:13.116 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 162058, 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 162058, 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 162058, 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 162058, 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.117 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 32264, 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 3516352, 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 32264, 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 871, 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 3516352, 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.118 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x5615a45f00a0": { 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 237063, 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 584710720, 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 237063, 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 4972, 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 584710720, 2024-03-30T01:11:13.119 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:13.120 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:13.120 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:13.120 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:13.120 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:13.120 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:13.120 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-03-30T01:11:13.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:12 smithi116 ceph-mon[38086]: pgmap v691: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 150 KiB/s wr, 95 op/s 2024-03-30T01:11:13.439 INFO:tasks.check_counter:Daemon mds.b mds.exported=265 2024-03-30T01:11:13.439 INFO:tasks.check_counter:Daemon mds.b mds.imported=257 2024-03-30T01:11:13.439 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=2323 2024-03-30T01:11:13.439 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=4901 2024-03-30T01:11:13.439 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=174 2024-03-30T01:11:13.439 DEBUG:tasks.check_counter:Getting stats from e 2024-03-30T01:11:13.439 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph daemon mds.e perf dump 2024-03-30T01:11:13.665 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T01:11:14.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:13 smithi130 ceph-mon[47626]: pgmap v692: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 90 KiB/s wr, 36 op/s 2024-03-30T01:11:14.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:13 smithi116 ceph-mon[38086]: pgmap v692: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 90 KiB/s wr, 36 op/s 2024-03-30T01:11:14.251 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:13 smithi181 ceph-mon[47622]: pgmap v692: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 90 KiB/s wr, 36 op/s 2024-03-30T01:11:14.624 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-03-30T01:11:14.625 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T01:11:14.625 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 1904, 2024-03-30T01:11:14.625 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 1907, 2024-03-30T01:11:14.625 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 237146865, 2024-03-30T01:11:14.625 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 592231, 2024-03-30T01:11:14.625 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 8, 2024-03-30T01:11:14.625 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 4, 2024-03-30T01:11:14.625 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.259720245, 2024-03-30T01:11:14.625 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.047472530, 2024-03-30T01:11:14.625 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.145204119, 2024-03-30T01:11:14.625 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.053277598, 2024-03-30T01:11:14.626 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:14.626 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1905, 2024-03-30T01:11:14.626 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.163882814, 2024-03-30T01:11:14.626 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000086027 2024-03-30T01:11:14.626 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.626 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:14.626 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.626 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.626 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.626 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.626 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T01:11:14.626 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T01:11:14.626 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.626 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T01:11:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 692, 2024-03-30T01:11:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 881, 2024-03-30T01:11:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 193124215, 2024-03-30T01:11:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 698772, 2024-03-30T01:11:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 8, 2024-03-30T01:11:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 6, 2024-03-30T01:11:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.159482743, 2024-03-30T01:11:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.018373255, 2024-03-30T01:11:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.119144778, 2024-03-30T01:11:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.015831965, 2024-03-30T01:11:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 880, 2024-03-30T01:11:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.079857363, 2024-03-30T01:11:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000090747 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 485360 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 1109, 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 1094, 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 155953751, 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 404655, 2024-03-30T01:11:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 5, 2024-03-30T01:11:14.629 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 4, 2024-03-30T01:11:14.629 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.191009683, 2024-03-30T01:11:14.629 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.028753264, 2024-03-30T01:11:14.629 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.127705010, 2024-03-30T01:11:14.629 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.028357025, 2024-03-30T01:11:14.629 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:14.629 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1093, 2024-03-30T01:11:14.629 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.077471243, 2024-03-30T01:11:14.629 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000070879 2024-03-30T01:11:14.629 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.629 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:14.629 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.629 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.630 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.631 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.631 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 544240, 2024-03-30T01:11:14.631 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 145280 2024-03-30T01:11:14.631 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.631 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-03-30T01:11:14.632 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-03-30T01:11:14.632 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-03-30T01:11:14.632 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.632 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-03-30T01:11:14.632 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-03-30T01:11:14.632 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-03-30T01:11:14.633 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 5431, 2024-03-30T01:11:14.633 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.185154018, 2024-03-30T01:11:14.633 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000034092 2024-03-30T01:11:14.633 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:14.633 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.633 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-03-30T01:11:14.633 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-03-30T01:11:14.634 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-03-30T01:11:14.634 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.634 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.640 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.640 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:14.640 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.641 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-03-30T01:11:14.641 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 0, 2024-03-30T01:11:14.641 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 0, 2024-03-30T01:11:14.641 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-03-30T01:11:14.641 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.641 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.642 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.642 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.642 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-03-30T01:11:14.642 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 0, 2024-03-30T01:11:14.642 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 0, 2024-03-30T01:11:14.643 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2024-03-30T01:11:14.643 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 0, 2024-03-30T01:11:14.643 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 0, 2024-03-30T01:11:14.643 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2024-03-30T01:11:14.643 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 32275, 2024-03-30T01:11:14.643 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 4605, 2024-03-30T01:11:14.644 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 27670, 2024-03-30T01:11:14.644 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 0, 2024-03-30T01:11:14.644 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 25696, 2024-03-30T01:11:14.644 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 0, 2024-03-30T01:11:14.644 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 0, 2024-03-30T01:11:14.645 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 0, 2024-03-30T01:11:14.645 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 3, 2024-03-30T01:11:14.645 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 0, 2024-03-30T01:11:14.645 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 0, 2024-03-30T01:11:14.645 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 0, 2024-03-30T01:11:14.646 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 0, 2024-03-30T01:11:14.646 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2024-03-30T01:11:14.646 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-03-30T01:11:14.646 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 0, 2024-03-30T01:11:14.646 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 0, 2024-03-30T01:11:14.647 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-03-30T01:11:14.647 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 0, 2024-03-30T01:11:14.647 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 0, 2024-03-30T01:11:14.647 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 0, 2024-03-30T01:11:14.647 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 0, 2024-03-30T01:11:14.648 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-03-30T01:11:14.648 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 0, 2024-03-30T01:11:14.648 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-03-30T01:11:14.648 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 4, 2024-03-30T01:11:14.648 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 234, 2024-03-30T01:11:14.649 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 0, 2024-03-30T01:11:14.649 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 0, 2024-03-30T01:11:14.649 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-03-30T01:11:14.649 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T01:11:14.649 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-03-30T01:11:14.649 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 0, 2024-03-30T01:11:14.650 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2024-03-30T01:11:14.650 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 0, 2024-03-30T01:11:14.651 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 0, 2024-03-30T01:11:14.651 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 0, 2024-03-30T01:11:14.652 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 0, 2024-03-30T01:11:14.652 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 0, 2024-03-30T01:11:14.652 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T01:11:14.652 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T01:11:14.653 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T01:11:14.653 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 0, 2024-03-30T01:11:14.653 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 0, 2024-03-30T01:11:14.653 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 0, 2024-03-30T01:11:14.654 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 0 2024-03-30T01:11:14.654 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.654 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-03-30T01:11:14.654 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 0, 2024-03-30T01:11:14.654 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 0, 2024-03-30T01:11:14.654 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 0, 2024-03-30T01:11:14.655 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 0, 2024-03-30T01:11:14.655 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 0, 2024-03-30T01:11:14.655 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 0, 2024-03-30T01:11:14.655 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-03-30T01:11:14.656 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-03-30T01:11:14.656 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 0, 2024-03-30T01:11:14.656 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 0, 2024-03-30T01:11:14.656 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2024-03-30T01:11:14.656 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-03-30T01:11:14.656 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-03-30T01:11:14.657 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-03-30T01:11:14.657 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-03-30T01:11:14.657 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-03-30T01:11:14.658 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-03-30T01:11:14.658 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2024-03-30T01:11:14.658 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 0, 2024-03-30T01:11:14.658 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-03-30T01:11:14.658 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 0, 2024-03-30T01:11:14.658 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-03-30T01:11:14.658 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-03-30T01:11:14.659 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.659 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-03-30T01:11:14.659 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 0, 2024-03-30T01:11:14.659 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2024-03-30T01:11:14.659 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2024-03-30T01:11:14.659 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 237063, 2024-03-30T01:11:14.659 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-03-30T01:11:14.659 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2024-03-30T01:11:14.659 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 0, 2024-03-30T01:11:14.660 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2024-03-30T01:11:14.660 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2024-03-30T01:11:14.660 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 129, 2024-03-30T01:11:14.660 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-03-30T01:11:14.660 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2024-03-30T01:11:14.661 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 356519575, 2024-03-30T01:11:14.661 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 588905024, 2024-03-30T01:11:14.662 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 588903372, 2024-03-30T01:11:14.662 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-03-30T01:11:14.662 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.662 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.663 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.663 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.663 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 237063 2024-03-30T01:11:14.663 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.663 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-03-30T01:11:14.663 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 14, 2024-03-30T01:11:14.663 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 89950, 2024-03-30T01:11:14.664 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 89936, 2024-03-30T01:11:14.664 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 13, 2024-03-30T01:11:14.664 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 3683, 2024-03-30T01:11:14.664 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 3670, 2024-03-30T01:11:14.665 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 32275, 2024-03-30T01:11:14.665 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 184394, 2024-03-30T01:11:14.665 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 152119, 2024-03-30T01:11:14.666 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 0, 2024-03-30T01:11:14.666 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 0, 2024-03-30T01:11:14.666 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 0, 2024-03-30T01:11:14.666 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 102784, 2024-03-30T01:11:14.667 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 207156 2024-03-30T01:11:14.667 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.667 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-03-30T01:11:14.667 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 0, 2024-03-30T01:11:14.667 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 0, 2024-03-30T01:11:14.668 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 0, 2024-03-30T01:11:14.668 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 0, 2024-03-30T01:11:14.668 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 0, 2024-03-30T01:11:14.668 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-03-30T01:11:14.668 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.668 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.668 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.668 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.669 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-03-30T01:11:14.669 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.669 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.669 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.669 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.669 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-03-30T01:11:14.670 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.670 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.670 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.670 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.670 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-03-30T01:11:14.670 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.670 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.671 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.671 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.671 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-03-30T01:11:14.671 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.672 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.672 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.672 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.673 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-03-30T01:11:14.673 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.673 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.673 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.673 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.673 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-03-30T01:11:14.674 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.674 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.674 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.674 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.674 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-03-30T01:11:14.674 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.674 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.675 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.675 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.675 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-03-30T01:11:14.675 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.675 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.675 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.675 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.675 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-03-30T01:11:14.676 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.676 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.676 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.676 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.676 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-03-30T01:11:14.676 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.676 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.676 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.676 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.676 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-03-30T01:11:14.676 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.676 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.676 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.676 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.677 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.678 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-03-30T01:11:14.678 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.678 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.678 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.678 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.678 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-03-30T01:11:14.678 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.678 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.678 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.678 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.679 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-03-30T01:11:14.679 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.679 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.679 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.679 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.679 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-03-30T01:11:14.679 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.679 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.679 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.679 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.679 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.680 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-03-30T01:11:14.681 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.681 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.681 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.681 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.681 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-03-30T01:11:14.681 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.681 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.681 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.682 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.682 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-03-30T01:11:14.682 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.682 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.682 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.682 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.682 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.683 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 1, 2024-03-30T01:11:14.684 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 3, 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 2, 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 1, 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 0, 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 0, 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 2599, 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T01:11:14.685 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-03-30T01:11:14.686 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T01:11:14.686 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T01:11:14.686 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T01:11:14.686 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T01:11:14.686 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T01:11:14.686 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-03-30T01:11:14.686 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T01:11:14.686 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T01:11:14.686 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-03-30T01:11:14.686 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-03-30T01:11:14.686 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-03-30T01:11:14.687 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 238609, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 75, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 0, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 0, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 8696, 2024-03-30T01:11:14.688 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 166, 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 22769248, 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 129248, 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-03-30T01:11:14.689 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 3424, 2024-03-30T01:11:14.690 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 26, 2024-03-30T01:11:14.690 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 4, 2024-03-30T01:11:14.690 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 3424, 2024-03-30T01:11:14.690 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-03-30T01:11:14.690 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 3424, 2024-03-30T01:11:14.690 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 49.108119898, 2024-03-30T01:11:14.690 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.014342324 2024-03-30T01:11:14.690 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.690 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 3424, 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 3425 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 3424, 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 3424, 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 0, 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 2000, 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 0, 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 1421, 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 0, 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 0, 2024-03-30T01:11:14.691 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-03-30T01:11:14.692 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-03-30T01:11:14.692 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-03-30T01:11:14.692 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2024-03-30T01:11:14.692 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 0, 2024-03-30T01:11:14.692 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-03-30T01:11:14.692 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-03-30T01:11:14.692 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-03-30T01:11:14.692 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 2, 2024-03-30T01:11:14.692 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 0, 2024-03-30T01:11:14.692 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-03-30T01:11:14.692 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-03-30T01:11:14.693 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-03-30T01:11:14.693 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-03-30T01:11:14.693 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-03-30T01:11:14.693 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-03-30T01:11:14.693 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T01:11:14.693 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-03-30T01:11:14.693 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-03-30T01:11:14.694 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2024-03-30T01:11:14.694 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-03-30T01:11:14.694 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-03-30T01:11:14.694 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-03-30T01:11:14.694 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-03-30T01:11:14.694 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-03-30T01:11:14.694 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-03-30T01:11:14.694 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-03-30T01:11:14.694 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-03-30T01:11:14.694 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-03-30T01:11:14.694 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-03-30T01:11:14.694 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-03-30T01:11:14.695 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-03-30T01:11:14.695 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-03-30T01:11:14.695 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-03-30T01:11:14.695 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-03-30T01:11:14.695 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-03-30T01:11:14.695 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 83, 2024-03-30T01:11:14.695 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-03-30T01:11:14.695 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 2, 2024-03-30T01:11:14.695 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 11, 2024-03-30T01:11:14.695 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 11, 2024-03-30T01:11:14.695 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2024-03-30T01:11:14.695 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-03-30T01:11:14.696 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 0, 2024-03-30T01:11:14.696 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 2, 2024-03-30T01:11:14.696 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 0 2024-03-30T01:11:14.696 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.696 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-03-30T01:11:14.696 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 0, 2024-03-30T01:11:14.696 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-03-30T01:11:14.696 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-03-30T01:11:14.696 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-03-30T01:11:14.696 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.696 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-03-30T01:11:14.696 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 0 2024-03-30T01:11:14.697 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.697 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-03-30T01:11:14.697 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-03-30T01:11:14.697 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 0, 2024-03-30T01:11:14.697 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-03-30T01:11:14.697 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 0, 2024-03-30T01:11:14.697 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 0, 2024-03-30T01:11:14.697 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-03-30T01:11:14.697 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.697 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T01:11:14.697 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:14.698 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-30T01:11:14.698 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:14.698 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 3701, 2024-03-30T01:11:14.698 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 585874591, 2024-03-30T01:11:14.698 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:14.698 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 3701, 2024-03-30T01:11:14.698 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-30T01:11:14.698 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-30T01:11:14.699 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 3701, 2024-03-30T01:11:14.699 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 585874591, 2024-03-30T01:11:14.699 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:14.699 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.699 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.699 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.699 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:14.699 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.699 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-03-30T01:11:14.699 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:14.700 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-30T01:11:14.700 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:14.700 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-30T01:11:14.701 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-30T01:11:14.701 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:14.701 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:14.701 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 3424, 2024-03-30T01:11:14.701 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 584736787, 2024-03-30T01:11:14.701 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 3424, 2024-03-30T01:11:14.701 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 584736787, 2024-03-30T01:11:14.701 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:14.701 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.701 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.701 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.701 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:14.701 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.702 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-03-30T01:11:14.702 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:14.702 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-03-30T01:11:14.702 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:14.702 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-30T01:11:14.702 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-30T01:11:14.702 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:14.702 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:14.702 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 3424, 2024-03-30T01:11:14.702 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 3424, 2024-03-30T01:11:14.702 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 3424, 2024-03-30T01:11:14.702 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 3424, 2024-03-30T01:11:14.702 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:14.702 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.703 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.703 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.703 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:14.703 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.703 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-03-30T01:11:14.703 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:14.703 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-03-30T01:11:14.703 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:14.703 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-30T01:11:14.703 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-30T01:11:14.703 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:14.703 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:14.703 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-30T01:11:14.703 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-30T01:11:14.704 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2024-03-30T01:11:14.704 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2024-03-30T01:11:14.704 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:14.704 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.704 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.704 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.704 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:14.704 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:14.704 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x56452ac78640": { 2024-03-30T01:11:14.704 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:14.704 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-03-30T01:11:14.705 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:14.705 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-30T01:11:14.705 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-30T01:11:14.705 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:14.705 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:14.705 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-30T01:11:14.705 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-30T01:11:14.705 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2024-03-30T01:11:14.705 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2024-03-30T01:11:14.705 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:14.705 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:14.705 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:14.705 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:14.706 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:14.706 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:14.706 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-03-30T01:11:14.973 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-30T01:11:14.974 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-30T01:11:14.974 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-30T01:11:14.974 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-30T01:11:14.974 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-30T01:11:14.974 DEBUG:tasks.check_counter:Getting stats from h 2024-03-30T01:11:14.974 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph daemon mds.h perf dump 2024-03-30T01:11:15.198 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T01:11:16.134 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-03-30T01:11:16.135 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T01:11:16.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 98512, 2024-03-30T01:11:16.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 100398, 2024-03-30T01:11:16.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 44692689, 2024-03-30T01:11:16.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 206902813, 2024-03-30T01:11:16.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 24, 2024-03-30T01:11:16.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 7, 2024-03-30T01:11:16.136 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 5.631818423, 2024-03-30T01:11:16.136 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 1.935786455, 2024-03-30T01:11:16.136 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 2.473626606, 2024-03-30T01:11:16.136 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 1.503421491, 2024-03-30T01:11:16.136 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:16.136 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 100395, 2024-03-30T01:11:16.136 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 21.911914632, 2024-03-30T01:11:16.136 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000218257 2024-03-30T01:11:16.136 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.136 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:16.137 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 98955, 2024-03-30T01:11:16.137 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.015467291, 2024-03-30T01:11:16.137 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000156 2024-03-30T01:11:16.137 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.137 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 557200, 2024-03-30T01:11:16.137 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 187936 2024-03-30T01:11:16.137 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.137 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T01:11:16.137 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 82446, 2024-03-30T01:11:16.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 105049, 2024-03-30T01:11:16.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 67080129, 2024-03-30T01:11:16.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 205862583, 2024-03-30T01:11:16.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 8, 2024-03-30T01:11:16.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 5, 2024-03-30T01:11:16.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 5.782476508, 2024-03-30T01:11:16.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 2.289274731, 2024-03-30T01:11:16.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 2.295224712, 2024-03-30T01:11:16.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 1.347744927, 2024-03-30T01:11:16.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:16.138 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 105050, 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 26.402417962, 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000251331 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 92857, 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.015240605, 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000164 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 400704 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 381952, 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 629030, 2024-03-30T01:11:16.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 131328970, 2024-03-30T01:11:16.140 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 477924037, 2024-03-30T01:11:16.140 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 17, 2024-03-30T01:11:16.140 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 5, 2024-03-30T01:11:16.140 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 20.969864707, 2024-03-30T01:11:16.140 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 8.872500451, 2024-03-30T01:11:16.140 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 21.772085007, 2024-03-30T01:11:16.140 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 2.290312843, 2024-03-30T01:11:16.140 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:16.140 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 629028, 2024-03-30T01:11:16.140 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 136.281657274, 2024-03-30T01:11:16.140 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000216654 2024-03-30T01:11:16.140 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.140 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:16.140 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 639395, 2024-03-30T01:11:16.141 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.126671142, 2024-03-30T01:11:16.141 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000198 2024-03-30T01:11:16.141 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.141 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T01:11:16.141 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T01:11:16.141 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.141 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-03-30T01:11:16.141 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-03-30T01:11:16.142 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-03-30T01:11:16.142 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.142 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-03-30T01:11:16.142 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-03-30T01:11:16.142 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-03-30T01:11:16.142 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 61369, 2024-03-30T01:11:16.142 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 141.297806460, 2024-03-30T01:11:16.142 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002302429 2024-03-30T01:11:16.142 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:16.142 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.142 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-03-30T01:11:16.142 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-03-30T01:11:16.142 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-03-30T01:11:16.143 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 66784, 2024-03-30T01:11:16.143 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 14.003228110, 2024-03-30T01:11:16.143 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000209679 2024-03-30T01:11:16.143 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:16.143 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.143 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-03-30T01:11:16.143 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 167679, 2024-03-30T01:11:16.143 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 164135, 2024-03-30T01:11:16.143 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-03-30T01:11:16.143 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 164135, 2024-03-30T01:11:16.143 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 1504.976050092, 2024-03-30T01:11:16.143 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.009169135 2024-03-30T01:11:16.143 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.143 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-03-30T01:11:16.144 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 3530, 2024-03-30T01:11:16.144 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 78, 2024-03-30T01:11:16.144 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2024-03-30T01:11:16.144 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 1375, 2024-03-30T01:11:16.144 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 133, 2024-03-30T01:11:16.145 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 4, 2024-03-30T01:11:16.145 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 68, 2024-03-30T01:11:16.145 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 33, 2024-03-30T01:11:16.145 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 23, 2024-03-30T01:11:16.145 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 12, 2024-03-30T01:11:16.145 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 20, 2024-03-30T01:11:16.145 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 93093, 2024-03-30T01:11:16.145 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 1, 2024-03-30T01:11:16.145 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 1, 2024-03-30T01:11:16.145 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 5, 2024-03-30T01:11:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 269636, 2024-03-30T01:11:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 240516, 2024-03-30T01:11:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 3229, 2024-03-30T01:11:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 1677, 2024-03-30T01:11:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 1, 2024-03-30T01:11:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-03-30T01:11:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 466, 2024-03-30T01:11:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 1633, 2024-03-30T01:11:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-03-30T01:11:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 182, 2024-03-30T01:11:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 78328, 2024-03-30T01:11:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 356, 2024-03-30T01:11:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 86770, 2024-03-30T01:11:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 77391, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 14289, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 296779860, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 0, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 77391, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 582, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 582, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 76809, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 321, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 33247, 2024-03-30T01:11:16.147 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 75951, 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 76197, 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 35498, 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 1320, 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 82112 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 2976, 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 6415, 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 3668, 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 6009, 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 4459, 2024-03-30T01:11:16.148 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 5, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 39523, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 39518, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 183, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 137, 2024-03-30T01:11:16.149 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 203983, 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 106226, 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 106226, 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 97757, 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 1024, 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 258, 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 130, 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 130, 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 129, 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-03-30T01:11:16.150 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 1, 2024-03-30T01:11:16.151 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 271228998, 2024-03-30T01:11:16.151 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 485457398, 2024-03-30T01:11:16.151 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 0, 2024-03-30T01:11:16.151 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-03-30T01:11:16.151 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 3637, 2024-03-30T01:11:16.151 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 685.787578427, 2024-03-30T01:11:16.151 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.188558586 2024-03-30T01:11:16.151 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.151 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 0 2024-03-30T01:11:16.151 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.151 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-03-30T01:11:16.152 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 33, 2024-03-30T01:11:16.152 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 86249, 2024-03-30T01:11:16.152 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 86216, 2024-03-30T01:11:16.152 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 52, 2024-03-30T01:11:16.152 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 6357, 2024-03-30T01:11:16.152 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 6305, 2024-03-30T01:11:16.152 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 68, 2024-03-30T01:11:16.152 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 181522, 2024-03-30T01:11:16.152 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 181454, 2024-03-30T01:11:16.152 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 1, 2024-03-30T01:11:16.152 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 85463, 2024-03-30T01:11:16.152 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 85462, 2024-03-30T01:11:16.152 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 130228, 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 198964 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 182701, 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 5253, 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 167679, 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 347, 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 9770, 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 31268, 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 141.637748413, 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.004529798 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-03-30T01:11:16.153 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2065, 2024-03-30T01:11:16.154 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 41.267410858, 2024-03-30T01:11:16.154 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.019984218 2024-03-30T01:11:16.154 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.154 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-03-30T01:11:16.154 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.154 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.154 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.154 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.154 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-03-30T01:11:16.154 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.155 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.155 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.155 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.155 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-03-30T01:11:16.155 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 56344, 2024-03-30T01:11:16.155 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 803.410408930, 2024-03-30T01:11:16.155 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.014259023 2024-03-30T01:11:16.155 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.155 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-03-30T01:11:16.155 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.155 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.155 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.156 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.156 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-03-30T01:11:16.156 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.156 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.156 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.156 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.156 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-03-30T01:11:16.156 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.156 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.156 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.157 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.157 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-03-30T01:11:16.157 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.157 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.157 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.157 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.157 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-03-30T01:11:16.157 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.157 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.157 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.157 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.157 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-03-30T01:11:16.157 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.157 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.158 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.158 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.158 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-03-30T01:11:16.158 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 162, 2024-03-30T01:11:16.158 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 20.739665726, 2024-03-30T01:11:16.158 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.128022627 2024-03-30T01:11:16.158 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.158 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-03-30T01:11:16.158 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.158 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.158 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.158 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.158 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-03-30T01:11:16.158 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 9132, 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 38.433508806, 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.004208662 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1434, 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 11.624810427, 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.008106562 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-03-30T01:11:16.159 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 31300, 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 354.002830045, 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.011309994 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 244, 2024-03-30T01:11:16.160 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 47.725636164, 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.195596869 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.161 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.162 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.162 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.162 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-03-30T01:11:16.162 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.162 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.162 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.162 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.162 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.163 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 32186, 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 46.134030723, 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001433357 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.165 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:16.165 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.165 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-03-30T01:11:16.165 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 1, 2024-03-30T01:11:16.165 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 2, 2024-03-30T01:11:16.165 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 1, 2024-03-30T01:11:16.165 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 1, 2024-03-30T01:11:16.166 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-03-30T01:11:16.166 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 18293, 2024-03-30T01:11:16.166 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 18293, 2024-03-30T01:11:16.166 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 708, 2024-03-30T01:11:16.166 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-03-30T01:11:16.166 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.166 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-03-30T01:11:16.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-03-30T01:11:16.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-03-30T01:11:16.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T01:11:16.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-03-30T01:11:16.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T01:11:16.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-03-30T01:11:16.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T01:11:16.167 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 295088, 2024-03-30T01:11:16.168 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 125, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 616, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 7, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 8696, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 166, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 197125, 2024-03-30T01:11:16.169 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 1839, 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 173946, 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 518091414, 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 27, 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 173946, 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 173946, 2024-03-30T01:11:16.170 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 6204.408023930, 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.035668586 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 173946, 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 188579 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 173946, 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 78446, 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 95500, 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 164, 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 8304, 2024-03-30T01:11:16.171 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 977, 2024-03-30T01:11:16.172 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 4619, 2024-03-30T01:11:16.172 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 487, 2024-03-30T01:11:16.172 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-03-30T01:11:16.172 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-03-30T01:11:16.172 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-03-30T01:11:16.172 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 133, 2024-03-30T01:11:16.172 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 80452, 2024-03-30T01:11:16.172 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-03-30T01:11:16.172 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 77393, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 12488, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-03-30T01:11:16.173 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 83, 2024-03-30T01:11:16.174 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-03-30T01:11:16.175 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 2, 2024-03-30T01:11:16.175 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 12, 2024-03-30T01:11:16.175 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 12, 2024-03-30T01:11:16.175 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2024-03-30T01:11:16.175 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-03-30T01:11:16.175 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 2986, 2024-03-30T01:11:16.175 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 156, 2024-03-30T01:11:16.175 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 420 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 1, 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 170358 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 24, 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-03-30T01:11:16.176 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 9, 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 39518, 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 562906, 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 192421109, 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 562906, 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 562906, 2024-03-30T01:11:16.177 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 192421109, 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:16.178 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 173946, 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 522397148, 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 173946, 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 522397148, 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-30T01:11:16.179 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 173946, 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 173946, 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 173946, 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 173946, 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-03-30T01:11:16.180 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 39518, 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 4306982, 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 39518, 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 981, 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 4306982, 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x561d8797e0a0": { 2024-03-30T01:11:16.181 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:16.182 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-03-30T01:11:16.182 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:16.182 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 203983, 2024-03-30T01:11:16.182 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 481263094, 2024-03-30T01:11:16.182 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:16.182 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 203983, 2024-03-30T01:11:16.182 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-30T01:11:16.182 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-30T01:11:16.182 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 3637, 2024-03-30T01:11:16.182 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 481263094, 2024-03-30T01:11:16.182 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:16.182 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:16.182 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:16.182 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:16.183 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:16.183 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:16.183 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-03-30T01:11:16.498 INFO:tasks.check_counter:Daemon mds.h mds.exported=182 2024-03-30T01:11:16.498 INFO:tasks.check_counter:Daemon mds.h mds.imported=356 2024-03-30T01:11:16.498 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=2976 2024-03-30T01:11:16.498 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=6415 2024-03-30T01:11:16.498 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=133 2024-03-30T01:11:16.498 DEBUG:tasks.check_counter:Getting stats from k 2024-03-30T01:11:16.498 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph daemon mds.k perf dump 2024-03-30T01:11:16.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:16 smithi130 ceph-mon[47626]: pgmap v693: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-03-30T01:11:16.724 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T01:11:16.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:16 smithi116 ceph-mon[38086]: pgmap v693: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-03-30T01:11:17.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:16 smithi181 ceph-mon[47622]: pgmap v693: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-03-30T01:11:17.600 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-03-30T01:11:17.600 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T01:11:17.601 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 257, 2024-03-30T01:11:17.601 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 254, 2024-03-30T01:11:17.601 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 483216, 2024-03-30T01:11:17.601 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 145120, 2024-03-30T01:11:17.601 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 4, 2024-03-30T01:11:17.601 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 0, 2024-03-30T01:11:17.601 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.034866347, 2024-03-30T01:11:17.601 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.012248808, 2024-03-30T01:11:17.601 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.015361902, 2024-03-30T01:11:17.601 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.006152151, 2024-03-30T01:11:17.601 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:17.601 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 253, 2024-03-30T01:11:17.602 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.036650102, 2024-03-30T01:11:17.602 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000144862 2024-03-30T01:11:17.602 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:17.602 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:17.602 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:17.602 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:17.602 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:17.602 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:17.603 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 483216, 2024-03-30T01:11:17.603 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 145120 2024-03-30T01:11:17.603 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:17.603 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T01:11:17.603 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 2, 2024-03-30T01:11:17.603 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 195, 2024-03-30T01:11:17.603 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 320, 2024-03-30T01:11:17.603 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 186704, 2024-03-30T01:11:17.603 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 4, 2024-03-30T01:11:17.603 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 2, 2024-03-30T01:11:17.604 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.010087800, 2024-03-30T01:11:17.604 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.007104008, 2024-03-30T01:11:17.604 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.001144661, 2024-03-30T01:11:17.604 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T01:11:17.604 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:17.604 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 193, 2024-03-30T01:11:17.604 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.021420948, 2024-03-30T01:11:17.604 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000110989 2024-03-30T01:11:17.604 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:17.604 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:17.604 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:17.605 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:17.605 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:17.605 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:17.605 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-30T01:11:17.605 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 186704 2024-03-30T01:11:17.605 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:17.606 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T01:11:17.606 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 4, 2024-03-30T01:11:17.606 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 4, 2024-03-30T01:11:17.606 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 2992, 2024-03-30T01:11:17.606 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 448, 2024-03-30T01:11:17.606 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 2, 2024-03-30T01:11:17.606 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 1, 2024-03-30T01:11:17.607 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.000783132, 2024-03-30T01:11:17.607 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.000038984, 2024-03-30T01:11:17.607 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.000724360, 2024-03-30T01:11:17.607 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T01:11:17.607 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:17.607 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 3, 2024-03-30T01:11:17.607 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000765812, 2024-03-30T01:11:17.607 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000255270 2024-03-30T01:11:17.608 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:17.608 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:17.608 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:17.608 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:17.608 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:17.609 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:17.609 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-30T01:11:17.609 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 448 2024-03-30T01:11:17.609 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:17.609 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-03-30T01:11:17.609 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 0, 2024-03-30T01:11:17.609 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-03-30T01:11:17.610 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:17.610 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-03-30T01:11:17.610 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-03-30T01:11:17.610 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-03-30T01:11:17.610 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T01:11:17.610 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-03-30T01:11:17.610 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T01:11:17.610 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-03-30T01:11:17.610 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T01:11:17.611 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T01:11:17.611 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T01:11:17.611 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T01:11:17.611 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T01:11:17.611 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-03-30T01:11:17.611 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T01:11:17.612 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T01:11:17.613 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-03-30T01:11:17.613 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-03-30T01:11:17.613 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-03-30T01:11:17.614 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-03-30T01:11:17.614 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T01:11:17.614 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T01:11:17.614 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T01:11:17.615 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T01:11:17.615 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T01:11:17.615 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-03-30T01:11:17.615 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-03-30T01:11:17.615 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-03-30T01:11:17.615 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T01:11:17.615 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T01:11:17.616 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-03-30T01:11:17.616 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-03-30T01:11:17.616 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-03-30T01:11:17.617 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-03-30T01:11:17.617 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T01:11:17.617 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-03-30T01:11:17.617 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T01:11:17.617 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-03-30T01:11:17.617 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 148761, 2024-03-30T01:11:17.617 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 31, 2024-03-30T01:11:17.617 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 0, 2024-03-30T01:11:17.618 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 0, 2024-03-30T01:11:17.618 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-03-30T01:11:17.618 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-03-30T01:11:17.619 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-03-30T01:11:17.619 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-03-30T01:11:17.619 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-03-30T01:11:17.619 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-03-30T01:11:17.619 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 0, 2024-03-30T01:11:17.620 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 0, 2024-03-30T01:11:17.620 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T01:11:17.620 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-03-30T01:11:17.620 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-03-30T01:11:17.620 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-03-30T01:11:17.621 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 928, 2024-03-30T01:11:17.621 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 2, 2024-03-30T01:11:17.621 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-03-30T01:11:17.621 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-03-30T01:11:17.621 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-03-30T01:11:17.621 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-03-30T01:11:17.621 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-30T01:11:17.622 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T01:11:17.622 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-30T01:11:17.622 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-30T01:11:17.622 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-30T01:11:17.622 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 259, 2024-03-30T01:11:17.622 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 446915, 2024-03-30T01:11:17.622 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:17.622 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 259, 2024-03-30T01:11:17.622 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-30T01:11:17.623 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-30T01:11:17.623 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 259, 2024-03-30T01:11:17.623 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 446915, 2024-03-30T01:11:17.623 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-30T01:11:17.623 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-30T01:11:17.623 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-30T01:11:17.623 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-30T01:11:17.623 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:17.624 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-30T01:11:17.624 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-03-30T01:11:17.962 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-30T01:11:17.962 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-30T01:11:17.963 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-03-30T01:11:17.963 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-03-30T01:11:17.963 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-03-30T01:11:17.963 DEBUG:tasks.check_counter:Getting stats from c 2024-03-30T01:11:17.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph daemon mds.c perf dump 2024-03-30T01:11:18.368 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T01:11:18.752 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:18 smithi181 ceph-mon[47622]: pgmap v694: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-30T01:11:18.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:18 smithi116 ceph-mon[38086]: pgmap v694: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-30T01:11:19.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:18 smithi130 ceph-mon[47626]: pgmap v694: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 853 B/s wr, 6 op/s 2024-03-30T01:11:19.464 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-30T01:11:19.464 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T01:11:19.464 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 1904, 2024-03-30T01:11:19.464 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 1907, 2024-03-30T01:11:19.464 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 168333808, 2024-03-30T01:11:19.464 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 592228, 2024-03-30T01:11:19.464 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 8, 2024-03-30T01:11:19.464 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 4, 2024-03-30T01:11:19.464 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.227148329, 2024-03-30T01:11:19.464 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.045895484, 2024-03-30T01:11:19.464 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.113761738, 2024-03-30T01:11:19.464 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.052371025, 2024-03-30T01:11:19.465 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:19.465 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 1905, 2024-03-30T01:11:19.465 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.134182746, 2024-03-30T01:11:19.465 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000070437 2024-03-30T01:11:19.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.465 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:19.465 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.465 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.465 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.465 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T01:11:19.465 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T01:11:19.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.465 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T01:11:19.466 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 825, 2024-03-30T01:11:19.466 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 810, 2024-03-30T01:11:19.466 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 129636914, 2024-03-30T01:11:19.466 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 316527, 2024-03-30T01:11:19.466 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 6, 2024-03-30T01:11:19.466 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 5, 2024-03-30T01:11:19.466 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.146744674, 2024-03-30T01:11:19.466 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.024199985, 2024-03-30T01:11:19.466 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.096875510, 2024-03-30T01:11:19.466 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.020470783, 2024-03-30T01:11:19.466 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:19.466 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 809, 2024-03-30T01:11:19.466 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.091499192, 2024-03-30T01:11:19.466 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000113101 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 544080, 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 145104 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 726, 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 915, 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 184750088, 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 710848, 2024-03-30T01:11:19.467 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 7, 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 5, 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.156343513, 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.020800057, 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.109221936, 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.019447043, 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 914, 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.062524514, 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000068407 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.468 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 486928 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: "cct": { 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: "total_workers": 1, 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: "unhealthy_workers": 0 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: "finisher-MDSRank": { 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: "queue_len": 0, 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: "complete_latency": { 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 5205, 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.163559147, 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000031423 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.469 INFO:teuthology.orchestra.run.smithi181.stdout: "finisher-PurgeQueue": { 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: "queue_len": 0, 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: "complete_latency": { 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: "mds": { 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: "request": 0, 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: "reply": 0, 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: "reply_latency": { 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "slow_reply": 0, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "forward": 0, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_fetch_complete": 0, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_fetch_keys": 0, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_commit": 0, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_split": 0, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_merge": 0, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes": 40498, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_top": 5002, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_bottom": 35496, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_pin_tail": 0, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_pinned": 33351, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_expired": 0, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_with_caps": 0, 2024-03-30T01:11:19.471 INFO:teuthology.orchestra.run.smithi181.stdout: "caps": 0, 2024-03-30T01:11:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: "subtrees": 3, 2024-03-30T01:11:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse": 0, 2024-03-30T01:11:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_hit": 0, 2024-03-30T01:11:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_forward": 0, 2024-03-30T01:11:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_discover": 0, 2024-03-30T01:11:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_dir_fetch": 0, 2024-03-30T01:11:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_remote_ino": 0, 2024-03-30T01:11:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_lock": 0, 2024-03-30T01:11:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: "load_cent": 0, 2024-03-30T01:11:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: "q": 0, 2024-03-30T01:11:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: "exported": 0, 2024-03-30T01:11:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: "exported_inodes": 0, 2024-03-30T01:11:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: "imported": 0, 2024-03-30T01:11:19.472 INFO:teuthology.orchestra.run.smithi181.stdout: "imported_inodes": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_dir_fetch": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_backtrace_fetch": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_peer_discover": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rfiles": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rbytes": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rsnaps": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_backtrace_fetch": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_set_tag": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_inotable_repaired": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dir_inodes": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dir_base_inodes": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dirfrag_rstats": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_file_inodes": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_inode_file_caps": 0, 2024-03-30T01:11:19.473 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_revoke": 0, 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_grant": 0, 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_caps": 0, 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_caps_dirty": 0, 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_cap_release": 0, 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: "process_request_cap_release": 0 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache": { 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_update": 0, 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_update_receipt": 0, 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_try_discover": 0, 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_send_discover": 0, 2024-03-30T01:11:19.474 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_handle_discover": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays_delayed": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays_enqueuing": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_created": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_enqueued": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_reintegrated": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_migrated": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_processing": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_enqueued": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_prioritized": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "recovery_started": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "recovery_completed": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_enqueue_scrub": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_exportdir": 0, 2024-03-30T01:11:19.475 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_flush": 0, 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_fragmentdir": 0, 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_fragstats": 0, 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_inodestats": 0 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log": { 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "evadd": 0, 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "evex": 0, 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "evtrm": 0, 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "ev": 203983, 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "evexg": 0, 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "evexd": 0, 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "segadd": 0, 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "segex": 0, 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "segtrm": 0, 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "seg": 131, 2024-03-30T01:11:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: "segexg": 0, 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: "segexd": 0, 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: "expos": 271228998, 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: "wrpos": 485457398, 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: "rdpos": 485456799, 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: "jlat": { 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: "replayed": 203983 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_mem": { 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: "ino": 23, 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: "ino+": 84881, 2024-03-30T01:11:19.477 INFO:teuthology.orchestra.run.smithi181.stdout: "ino-": 84858, 2024-03-30T01:11:19.478 INFO:teuthology.orchestra.run.smithi181.stdout: "dir": 42, 2024-03-30T01:11:19.478 INFO:teuthology.orchestra.run.smithi181.stdout: "dir+": 3637, 2024-03-30T01:11:19.478 INFO:teuthology.orchestra.run.smithi181.stdout: "dir-": 3595, 2024-03-30T01:11:19.478 INFO:teuthology.orchestra.run.smithi181.stdout: "dn": 40498, 2024-03-30T01:11:19.478 INFO:teuthology.orchestra.run.smithi181.stdout: "dn+": 175355, 2024-03-30T01:11:19.478 INFO:teuthology.orchestra.run.smithi181.stdout: "dn-": 134857, 2024-03-30T01:11:19.478 INFO:teuthology.orchestra.run.smithi181.stdout: "cap": 0, 2024-03-30T01:11:19.478 INFO:teuthology.orchestra.run.smithi181.stdout: "cap+": 0, 2024-03-30T01:11:19.478 INFO:teuthology.orchestra.run.smithi181.stdout: "cap-": 0, 2024-03-30T01:11:19.478 INFO:teuthology.orchestra.run.smithi181.stdout: "rss": 97236, 2024-03-30T01:11:19.478 INFO:teuthology.orchestra.run.smithi181.stdout: "heap": 207156 2024-03-30T01:11:19.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.478 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_server": { 2024-03-30T01:11:19.478 INFO:teuthology.orchestra.run.smithi181.stdout: "dispatch_client_request": 0, 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: "dispatch_server_request": 0, 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_request": 0, 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_session": 0, 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_peer_request": 0, 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: "req_create_latency": { 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getattr_latency": { 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getfilelock_latency": { 2024-03-30T01:11:19.479 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: "req_link_latency": { 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookup_latency": { 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookuphash_latency": { 2024-03-30T01:11:19.480 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupino_latency": { 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupname_latency": { 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupparent_latency": { 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.481 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupsnap_latency": { 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lssnap_latency": { 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mkdir_latency": { 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.482 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.483 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.483 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mknod_latency": { 2024-03-30T01:11:19.483 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.483 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.483 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.483 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mksnap_latency": { 2024-03-30T01:11:19.483 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.483 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.483 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.483 INFO:teuthology.orchestra.run.smithi181.stdout: "req_open_latency": { 2024-03-30T01:11:19.483 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: "req_readdir_latency": { 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rename_latency": { 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: "req_renamesnap_latency": { 2024-03-30T01:11:19.484 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.485 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.485 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.485 INFO:teuthology.orchestra.run.smithi181.stdout: "req_snapdiff_latency": { 2024-03-30T01:11:19.485 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.485 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.485 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.485 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmdir_latency": { 2024-03-30T01:11:19.486 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.486 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.486 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.486 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmsnap_latency": { 2024-03-30T01:11:19.486 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.486 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.486 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.487 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmxattr_latency": { 2024-03-30T01:11:19.487 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.487 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.487 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.487 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setattr_latency": { 2024-03-30T01:11:19.487 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.487 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.487 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.487 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setdirlayout_latency": { 2024-03-30T01:11:19.488 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.488 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.488 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.488 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setfilelock_latency": { 2024-03-30T01:11:19.488 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.488 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.488 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.488 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setlayout_latency": { 2024-03-30T01:11:19.488 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.488 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.488 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setxattr_latency": { 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: "req_symlink_latency": { 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: "req_unlink_latency": { 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.490 INFO:teuthology.orchestra.run.smithi181.stdout: "cap_revoke_eviction": 0, 2024-03-30T01:11:19.490 INFO:teuthology.orchestra.run.smithi181.stdout: "cap_acquisition_throttle": 0, 2024-03-30T01:11:19.490 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getvxattr_latency": { 2024-03-30T01:11:19.490 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.490 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.490 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.490 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:19.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.490 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_sessions": { 2024-03-30T01:11:19.490 INFO:teuthology.orchestra.run.smithi181.stdout: "session_count": 1, 2024-03-30T01:11:19.490 INFO:teuthology.orchestra.run.smithi181.stdout: "session_add": 2, 2024-03-30T01:11:19.490 INFO:teuthology.orchestra.run.smithi181.stdout: "session_remove": 1, 2024-03-30T01:11:19.490 INFO:teuthology.orchestra.run.smithi181.stdout: "sessions_open": 1, 2024-03-30T01:11:19.491 INFO:teuthology.orchestra.run.smithi181.stdout: "sessions_stale": 0, 2024-03-30T01:11:19.491 INFO:teuthology.orchestra.run.smithi181.stdout: "total_load": 0, 2024-03-30T01:11:19.491 INFO:teuthology.orchestra.run.smithi181.stdout: "average_load": 0, 2024-03-30T01:11:19.491 INFO:teuthology.orchestra.run.smithi181.stdout: "avg_session_uptime": 2609, 2024-03-30T01:11:19.491 INFO:teuthology.orchestra.run.smithi181.stdout: "mdthresh_evicted": 0 2024-03-30T01:11:19.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.491 INFO:teuthology.orchestra.run.smithi181.stdout: "mempool": { 2024-03-30T01:11:19.491 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_bytes": 0, 2024-03-30T01:11:19.491 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_items": 0, 2024-03-30T01:11:19.491 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T01:11:19.491 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_items": 0, 2024-03-30T01:11:19.491 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T01:11:19.491 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_items": 0, 2024-03-30T01:11:19.491 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T01:11:19.492 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T01:11:19.492 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T01:11:19.492 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T01:11:19.492 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T01:11:19.492 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_items": 0, 2024-03-30T01:11:19.492 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T01:11:19.492 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T01:11:19.492 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_bytes": 0, 2024-03-30T01:11:19.492 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_items": 0, 2024-03-30T01:11:19.492 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_bytes": 0, 2024-03-30T01:11:19.492 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_items": 0, 2024-03-30T01:11:19.493 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T01:11:19.493 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T01:11:19.493 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T01:11:19.493 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T01:11:19.493 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T01:11:19.493 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_items": 0, 2024-03-30T01:11:19.493 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_bytes": 0, 2024-03-30T01:11:19.493 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_items": 0, 2024-03-30T01:11:19.493 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T01:11:19.493 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T01:11:19.493 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_bytes": 0, 2024-03-30T01:11:19.493 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_items": 0, 2024-03-30T01:11:19.493 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_bytes": 0, 2024-03-30T01:11:19.493 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_items": 0, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_items": 0, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_items": 0, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_bytes": 239585, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_items": 82, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_bytes": 0, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_items": 0, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_bytes": 0, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_items": 0, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_bytes": 0, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_items": 0, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_bytes": 0, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_items": 0, 2024-03-30T01:11:19.494 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_bytes": 8696, 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_items": 166, 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_items": 0, 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_bytes": 0, 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_items": 0, 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_bytes": 28613685, 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_items": 163273, 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_bytes": 0, 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_items": 0, 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_bytes": 0, 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_items": 0 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter": { 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: "op_active": 0, 2024-03-30T01:11:19.495 INFO:teuthology.orchestra.run.smithi181.stdout: "op_laggy": 0, 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: "op_send": 3175, 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: "op_send_bytes": 21, 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: "op_resend": 3, 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: "op_reply": 3175, 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: "op_latency": { 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 3175, 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 54.020945742, 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.017014471 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: "op_inflight": 0, 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: "oplen_avg": { 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 3175, 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 3176 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.496 INFO:teuthology.orchestra.run.smithi181.stdout: "op": 3175, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "op_r": 3175, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "op_w": 0, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "op_rmw": 0, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "op_pg": 0, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_stat": 1810, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_create": 0, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_read": 1363, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_write": 0, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_writefull": 0, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_writesame": 0, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_append": 0, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_zero": 0, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_truncate": 0, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_delete": 0, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_mapext": 0, 2024-03-30T01:11:19.497 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_sparse_read": 0, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_clonerange": 0, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_getxattr": 1, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_setxattr": 0, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_cmpxattr": 0, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_rmxattr": 0, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_resetxattrs": 0, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_call": 0, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_watch": 0, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_notify": 0, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_pgls": 0, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_pgls_filter": 0, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_other": 0, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_active": 0, 2024-03-30T01:11:19.498 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_send": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_resend": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_ping": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_active": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_send": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_resend": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_active": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_send": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_resend": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_active": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_send": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_resend": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "command_active": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "command_send": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "command_resend": 0, 2024-03-30T01:11:19.499 INFO:teuthology.orchestra.run.smithi181.stdout: "map_epoch": 83, 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: "map_full": 0, 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: "map_inc": 2, 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_sessions": 11, 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_session_open": 11, 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_session_close": 0, 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_laggy": 0, 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_wr": 0, 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_rd": 2, 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_del": 0 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: "oft": { 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_objs": 0, 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_kv_pairs": 0, 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_updates": 0, 2024-03-30T01:11:19.500 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_removes": 0 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: "osd-slow-ops": { 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: "slow_ops_count": 0 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: "purge_queue": { 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_ops": 0, 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_ops_high_water": 0, 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing": 0, 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_high_water": 0, 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executed": 0, 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_item_in_journal": 0 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-03-30T01:11:19.501 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 3451, 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 482393116, 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 3451, 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 3451, 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 482393116, 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.502 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-objecter_bytes": { 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 3175, 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 481297006, 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 3175, 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 481297006, 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-objecter_ops": { 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 1024, 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 3175, 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 3175, 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 3175, 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 3175, 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:19.504 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-write_buf_throttle": { 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 3758096384, 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-03-30T01:11:19.505 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 0, 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 0, 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-write_buf_throttle-0x5626b87ba500": { 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 3758096384, 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:19.506 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:19.507 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-03-30T01:11:19.507 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-03-30T01:11:19.507 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 0, 2024-03-30T01:11:19.507 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 0, 2024-03-30T01:11:19.507 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:19.507 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:19.507 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:19.507 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:19.507 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:19.507 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:19.507 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-30T01:11:19.835 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-30T01:11:19.835 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-30T01:11:19.835 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-30T01:11:19.835 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-03-30T01:11:19.835 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-30T01:11:19.836 DEBUG:tasks.check_counter:Getting stats from f 2024-03-30T01:11:19.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph daemon mds.f perf dump 2024-03-30T01:11:20.061 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T01:11:20.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:20 smithi181 ceph-mon[47622]: pgmap v695: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-30T01:11:20.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:20 smithi116 ceph-mon[38086]: pgmap v695: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-30T01:11:21.011 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-30T01:11:21.011 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T01:11:21.011 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 1261, 2024-03-30T01:11:21.011 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 1263, 2024-03-30T01:11:21.011 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 132656935, 2024-03-30T01:11:21.011 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 392235, 2024-03-30T01:11:21.011 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 8, 2024-03-30T01:11:21.011 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 4, 2024-03-30T01:11:21.011 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.166111330, 2024-03-30T01:11:21.012 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.024186608, 2024-03-30T01:11:21.012 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.099020640, 2024-03-30T01:11:21.012 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.033903511, 2024-03-30T01:11:21.012 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:21.012 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 1261, 2024-03-30T01:11:21.012 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.140236341, 2024-03-30T01:11:21.012 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000111210 2024-03-30T01:11:21.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.012 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:21.012 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.012 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.012 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.012 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 849, 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 834, 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 97163857, 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 324215, 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 6, 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 5, 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.122683562, 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.026020908, 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.070334511, 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.021567283, 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:21.013 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 833, 2024-03-30T01:11:21.014 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.059388463, 2024-03-30T01:11:21.014 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000071294 2024-03-30T01:11:21.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.014 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:21.014 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.014 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.014 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.014 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 544352, 2024-03-30T01:11:21.014 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 145696 2024-03-30T01:11:21.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.014 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T01:11:21.014 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 1322, 2024-03-30T01:11:21.014 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 1512, 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 105267292, 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 811203, 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 6, 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 5, 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.178090136, 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.038558917, 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.086125377, 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.040558569, 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 1511, 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.136758551, 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000090508 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:21.015 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 401616 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: "cct": { 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: "total_workers": 1, 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: "unhealthy_workers": 0 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: "finisher-MDSRank": { 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: "queue_len": 0, 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: "complete_latency": { 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 5167, 2024-03-30T01:11:21.016 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.161240629, 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000031205 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: "finisher-PurgeQueue": { 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: "queue_len": 0, 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: "complete_latency": { 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: "mds": { 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: "request": 0, 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: "reply": 0, 2024-03-30T01:11:21.017 INFO:teuthology.orchestra.run.smithi181.stdout: "reply_latency": { 2024-03-30T01:11:21.018 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.018 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.018 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.018 INFO:teuthology.orchestra.run.smithi181.stdout: "slow_reply": 0, 2024-03-30T01:11:21.018 INFO:teuthology.orchestra.run.smithi181.stdout: "forward": 0, 2024-03-30T01:11:21.018 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_fetch_complete": 0, 2024-03-30T01:11:21.018 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_fetch_keys": 0, 2024-03-30T01:11:21.018 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_commit": 0, 2024-03-30T01:11:21.018 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_split": 0, 2024-03-30T01:11:21.018 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_merge": 0, 2024-03-30T01:11:21.018 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes": 26089, 2024-03-30T01:11:21.020 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_top": 2718, 2024-03-30T01:11:21.020 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_bottom": 23371, 2024-03-30T01:11:21.020 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_pin_tail": 0, 2024-03-30T01:11:21.020 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_pinned": 22206, 2024-03-30T01:11:21.020 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_expired": 0, 2024-03-30T01:11:21.020 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_with_caps": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "caps": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "subtrees": 1, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_hit": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_forward": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_discover": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_dir_fetch": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_remote_ino": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_lock": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "load_cent": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "q": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "exported": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "exported_inodes": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "imported": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "imported_inodes": 0, 2024-03-30T01:11:21.021 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_dir_fetch": 0, 2024-03-30T01:11:21.022 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_backtrace_fetch": 0, 2024-03-30T01:11:21.022 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_peer_discover": 0, 2024-03-30T01:11:21.022 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rfiles": 0, 2024-03-30T01:11:21.022 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rbytes": 0, 2024-03-30T01:11:21.022 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rsnaps": 0, 2024-03-30T01:11:21.022 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_backtrace_fetch": 0, 2024-03-30T01:11:21.022 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_set_tag": 0, 2024-03-30T01:11:21.022 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T01:11:21.022 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_inotable_repaired": 0, 2024-03-30T01:11:21.022 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dir_inodes": 0, 2024-03-30T01:11:21.022 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dir_base_inodes": 0, 2024-03-30T01:11:21.022 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dirfrag_rstats": 0, 2024-03-30T01:11:21.022 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_file_inodes": 0, 2024-03-30T01:11:21.022 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_inode_file_caps": 0, 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_revoke": 0, 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_grant": 0, 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_caps": 0, 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_caps_dirty": 0, 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_cap_release": 0, 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "process_request_cap_release": 0 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache": { 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_update": 0, 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_update_receipt": 0, 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_try_discover": 0, 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_send_discover": 0, 2024-03-30T01:11:21.023 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_handle_discover": 0, 2024-03-30T01:11:21.024 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays": 0, 2024-03-30T01:11:21.024 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays_delayed": 0, 2024-03-30T01:11:21.024 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays_enqueuing": 0, 2024-03-30T01:11:21.024 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_created": 0, 2024-03-30T01:11:21.024 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_enqueued": 0, 2024-03-30T01:11:21.024 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_reintegrated": 0, 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_migrated": 0, 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_processing": 0, 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_enqueued": 0, 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_prioritized": 0, 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: "recovery_started": 0, 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: "recovery_completed": 0, 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_enqueue_scrub": 0, 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_exportdir": 0, 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_flush": 0, 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_fragmentdir": 0, 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_fragstats": 0, 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_inodestats": 0 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log": { 2024-03-30T01:11:21.025 INFO:teuthology.orchestra.run.smithi181.stdout: "evadd": 0, 2024-03-30T01:11:21.026 INFO:teuthology.orchestra.run.smithi181.stdout: "evex": 0, 2024-03-30T01:11:21.026 INFO:teuthology.orchestra.run.smithi181.stdout: "evtrm": 0, 2024-03-30T01:11:21.026 INFO:teuthology.orchestra.run.smithi181.stdout: "ev": 148893, 2024-03-30T01:11:21.026 INFO:teuthology.orchestra.run.smithi181.stdout: "evexg": 0, 2024-03-30T01:11:21.026 INFO:teuthology.orchestra.run.smithi181.stdout: "evexd": 0, 2024-03-30T01:11:21.026 INFO:teuthology.orchestra.run.smithi181.stdout: "segadd": 0, 2024-03-30T01:11:21.026 INFO:teuthology.orchestra.run.smithi181.stdout: "segex": 0, 2024-03-30T01:11:21.026 INFO:teuthology.orchestra.run.smithi181.stdout: "segtrm": 0, 2024-03-30T01:11:21.026 INFO:teuthology.orchestra.run.smithi181.stdout: "seg": 128, 2024-03-30T01:11:21.026 INFO:teuthology.orchestra.run.smithi181.stdout: "segexg": 0, 2024-03-30T01:11:21.026 INFO:teuthology.orchestra.run.smithi181.stdout: "segexd": 0, 2024-03-30T01:11:21.026 INFO:teuthology.orchestra.run.smithi181.stdout: "expos": 92277658, 2024-03-30T01:11:21.026 INFO:teuthology.orchestra.run.smithi181.stdout: "wrpos": 337778582, 2024-03-30T01:11:21.026 INFO:teuthology.orchestra.run.smithi181.stdout: "rdpos": 337769079, 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: "jlat": { 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: "replayed": 148893 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_mem": { 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: "ino": 13, 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: "ino+": 43711, 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: "ino-": 43698, 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: "dir": 11, 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: "dir+": 1721, 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: "dir-": 1710, 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: "dn": 26089, 2024-03-30T01:11:21.027 INFO:teuthology.orchestra.run.smithi181.stdout: "dn+": 95112, 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: "dn-": 69023, 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: "cap": 0, 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: "cap+": 0, 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: "cap-": 0, 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: "rss": 106708, 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: "heap": 207156 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_server": { 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: "dispatch_client_request": 0, 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: "dispatch_server_request": 0, 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_request": 0, 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_session": 0, 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_peer_request": 0, 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: "req_create_latency": { 2024-03-30T01:11:21.028 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.029 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.029 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.029 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getattr_latency": { 2024-03-30T01:11:21.030 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.030 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.030 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.030 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getfilelock_latency": { 2024-03-30T01:11:21.030 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.030 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.030 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.030 INFO:teuthology.orchestra.run.smithi181.stdout: "req_link_latency": { 2024-03-30T01:11:21.031 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.031 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.031 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.031 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookup_latency": { 2024-03-30T01:11:21.031 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.031 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.031 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.031 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookuphash_latency": { 2024-03-30T01:11:21.031 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.031 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupino_latency": { 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupname_latency": { 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupparent_latency": { 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.032 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupsnap_latency": { 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lssnap_latency": { 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mkdir_latency": { 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.033 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.034 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.034 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mknod_latency": { 2024-03-30T01:11:21.034 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.034 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.034 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.034 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mksnap_latency": { 2024-03-30T01:11:21.034 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.035 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.035 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.035 INFO:teuthology.orchestra.run.smithi181.stdout: "req_open_latency": { 2024-03-30T01:11:21.035 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.035 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.035 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.035 INFO:teuthology.orchestra.run.smithi181.stdout: "req_readdir_latency": { 2024-03-30T01:11:21.035 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.035 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.035 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rename_latency": { 2024-03-30T01:11:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: "req_renamesnap_latency": { 2024-03-30T01:11:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: "req_snapdiff_latency": { 2024-03-30T01:11:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmdir_latency": { 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmsnap_latency": { 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmxattr_latency": { 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.037 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.038 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setattr_latency": { 2024-03-30T01:11:21.038 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.038 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.038 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.038 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setdirlayout_latency": { 2024-03-30T01:11:21.038 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.038 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.038 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.038 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setfilelock_latency": { 2024-03-30T01:11:21.038 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.038 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setlayout_latency": { 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setxattr_latency": { 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: "req_symlink_latency": { 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.039 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.040 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.040 INFO:teuthology.orchestra.run.smithi181.stdout: "req_unlink_latency": { 2024-03-30T01:11:21.040 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.040 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.040 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.040 INFO:teuthology.orchestra.run.smithi181.stdout: "cap_revoke_eviction": 0, 2024-03-30T01:11:21.040 INFO:teuthology.orchestra.run.smithi181.stdout: "cap_acquisition_throttle": 0, 2024-03-30T01:11:21.041 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getvxattr_latency": { 2024-03-30T01:11:21.041 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.041 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.041 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.041 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:21.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.041 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_sessions": { 2024-03-30T01:11:21.041 INFO:teuthology.orchestra.run.smithi181.stdout: "session_count": 1, 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: "session_add": 2, 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: "session_remove": 1, 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: "sessions_open": 1, 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: "sessions_stale": 0, 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: "total_load": 0, 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: "average_load": 0, 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: "avg_session_uptime": 2612, 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: "mdthresh_evicted": 0 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: "mempool": { 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_bytes": 0, 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_items": 0, 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_items": 0, 2024-03-30T01:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T01:11:21.043 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_items": 0, 2024-03-30T01:11:21.043 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T01:11:21.043 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T01:11:21.043 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T01:11:21.043 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T01:11:21.043 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T01:11:21.043 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_items": 0, 2024-03-30T01:11:21.043 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T01:11:21.043 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T01:11:21.043 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_bytes": 0, 2024-03-30T01:11:21.043 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_items": 0, 2024-03-30T01:11:21.043 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_bytes": 0, 2024-03-30T01:11:21.043 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_items": 0, 2024-03-30T01:11:21.043 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_items": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_bytes": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_items": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_bytes": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_items": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_bytes": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_items": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_items": 0, 2024-03-30T01:11:21.044 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T01:11:21.045 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_items": 0, 2024-03-30T01:11:21.045 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_bytes": 240215, 2024-03-30T01:11:21.045 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_items": 80, 2024-03-30T01:11:21.046 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_bytes": 0, 2024-03-30T01:11:21.046 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_items": 0, 2024-03-30T01:11:21.046 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_bytes": 0, 2024-03-30T01:11:21.046 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_items": 0, 2024-03-30T01:11:21.046 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_bytes": 0, 2024-03-30T01:11:21.046 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_items": 0, 2024-03-30T01:11:21.046 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_bytes": 0, 2024-03-30T01:11:21.046 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_items": 0, 2024-03-30T01:11:21.046 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_bytes": 8696, 2024-03-30T01:11:21.046 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_items": 166, 2024-03-30T01:11:21.046 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T01:11:21.046 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_items": 0, 2024-03-30T01:11:21.046 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_bytes": 0, 2024-03-30T01:11:21.046 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_items": 0, 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_bytes": 18409416, 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_items": 104484, 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_bytes": 0, 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_items": 0, 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_bytes": 0, 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_items": 0 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter": { 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "op_active": 0, 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "op_laggy": 0, 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "op_send": 3152, 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "op_send_bytes": 21, 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "op_resend": 3, 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "op_reply": 3152, 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "op_latency": { 2024-03-30T01:11:21.047 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 3152, 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 37.089276873, 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.011766902 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: "op_inflight": 0, 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: "oplen_avg": { 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 3152, 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 3153 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: "op": 3152, 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: "op_r": 3152, 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: "op_w": 0, 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: "op_rmw": 0, 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: "op_pg": 0, 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_stat": 1841, 2024-03-30T01:11:21.048 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_create": 0, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_read": 1309, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_write": 0, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_writefull": 0, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_writesame": 0, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_append": 0, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_zero": 0, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_truncate": 0, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_delete": 0, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_mapext": 0, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_sparse_read": 0, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_clonerange": 0, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_getxattr": 1, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_setxattr": 0, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_cmpxattr": 0, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_rmxattr": 0, 2024-03-30T01:11:21.049 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_resetxattrs": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_call": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_watch": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_notify": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_pgls": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_pgls_filter": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_other": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_active": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_send": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_resend": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_ping": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_active": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_send": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_resend": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_active": 0, 2024-03-30T01:11:21.050 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_send": 0, 2024-03-30T01:11:21.051 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_resend": 0, 2024-03-30T01:11:21.051 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_active": 0, 2024-03-30T01:11:21.051 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_send": 0, 2024-03-30T01:11:21.051 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_resend": 0, 2024-03-30T01:11:21.051 INFO:teuthology.orchestra.run.smithi181.stdout: "command_active": 0, 2024-03-30T01:11:21.051 INFO:teuthology.orchestra.run.smithi181.stdout: "command_send": 0, 2024-03-30T01:11:21.051 INFO:teuthology.orchestra.run.smithi181.stdout: "command_resend": 0, 2024-03-30T01:11:21.051 INFO:teuthology.orchestra.run.smithi181.stdout: "map_epoch": 83, 2024-03-30T01:11:21.051 INFO:teuthology.orchestra.run.smithi181.stdout: "map_full": 0, 2024-03-30T01:11:21.051 INFO:teuthology.orchestra.run.smithi181.stdout: "map_inc": 2, 2024-03-30T01:11:21.051 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_sessions": 11, 2024-03-30T01:11:21.051 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_session_open": 11, 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_session_close": 0, 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_laggy": 0, 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_wr": 0, 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_rd": 2, 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_del": 0 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: "oft": { 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_objs": 0, 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_kv_pairs": 0, 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_updates": 0, 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_removes": 0 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: "osd-slow-ops": { 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: "slow_ops_count": 0 2024-03-30T01:11:21.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: "purge_queue": { 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_ops": 0, 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_ops_high_water": 0, 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing": 0, 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_high_water": 0, 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executed": 0, 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_item_in_journal": 0 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 3428, 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 334762451, 2024-03-30T01:11:21.053 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:21.054 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 3428, 2024-03-30T01:11:21.054 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-03-30T01:11:21.054 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-03-30T01:11:21.054 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 3428, 2024-03-30T01:11:21.054 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 334762451, 2024-03-30T01:11:21.054 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:21.054 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.054 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.054 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.054 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:21.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.054 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-objecter_bytes": { 2024-03-30T01:11:21.054 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:21.054 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 3152, 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 333666872, 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 3152, 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 333666872, 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:21.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-objecter_ops": { 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 1024, 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 3152, 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 3152, 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 3152, 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 3152, 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.056 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.057 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.057 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:21.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.057 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-write_buf_throttle": { 2024-03-30T01:11:21.057 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:21.057 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 3758096384, 2024-03-30T01:11:21.057 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:21.057 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-03-30T01:11:21.057 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-03-30T01:11:21.057 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:21.057 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:21.057 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-03-30T01:11:21.057 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-03-30T01:11:21.058 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 0, 2024-03-30T01:11:21.058 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 0, 2024-03-30T01:11:21.058 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:21.058 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.058 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.058 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.058 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:21.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:21.058 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-write_buf_throttle-0x564013074500": { 2024-03-30T01:11:21.058 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:21.058 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 3758096384, 2024-03-30T01:11:21.058 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:21.058 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-03-30T01:11:21.059 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-03-30T01:11:21.059 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:21.059 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:21.059 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-03-30T01:11:21.059 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-03-30T01:11:21.059 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 0, 2024-03-30T01:11:21.059 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 0, 2024-03-30T01:11:21.059 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:21.059 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:21.059 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:21.059 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:21.059 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:21.059 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:21.059 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-30T01:11:21.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:20 smithi130 ceph-mon[47626]: pgmap v695: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-30T01:11:21.401 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-30T01:11:21.401 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-30T01:11:21.401 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-30T01:11:21.401 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-30T01:11:21.401 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-30T01:11:21.402 DEBUG:tasks.check_counter:Getting stats from i 2024-03-30T01:11:21.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph daemon mds.i perf dump 2024-03-30T01:11:21.625 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T01:11:22.569 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-30T01:11:22.569 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T01:11:22.569 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 310861, 2024-03-30T01:11:22.569 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 439749, 2024-03-30T01:11:22.569 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 107459092, 2024-03-30T01:11:22.569 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 284143264, 2024-03-30T01:11:22.570 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 21, 2024-03-30T01:11:22.570 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 5, 2024-03-30T01:11:22.570 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 14.161903535, 2024-03-30T01:11:22.570 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 5.939340458, 2024-03-30T01:11:22.570 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 13.749599193, 2024-03-30T01:11:22.570 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 1.334844831, 2024-03-30T01:11:22.570 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:22.570 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 439749, 2024-03-30T01:11:22.570 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 98.072546340, 2024-03-30T01:11:22.570 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000223019 2024-03-30T01:11:22.570 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.570 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:22.570 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 478621, 2024-03-30T01:11:22.570 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.082363626, 2024-03-30T01:11:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000172 2024-03-30T01:11:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 557792, 2024-03-30T01:11:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 371040 2024-03-30T01:11:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T01:11:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 64575, 2024-03-30T01:11:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 64769, 2024-03-30T01:11:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 17029051, 2024-03-30T01:11:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 185061883, 2024-03-30T01:11:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 20, 2024-03-30T01:11:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 7, 2024-03-30T01:11:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 4.156254218, 2024-03-30T01:11:22.571 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 1.366665906, 2024-03-30T01:11:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.849011582, 2024-03-30T01:11:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 1.696267072, 2024-03-30T01:11:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 64765, 2024-03-30T01:11:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 33.460842811, 2024-03-30T01:11:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000516650 2024-03-30T01:11:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 10, 2024-03-30T01:11:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000001188, 2024-03-30T01:11:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000118 2024-03-30T01:11:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.572 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-30T01:11:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 487584 2024-03-30T01:11:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T01:11:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 129084, 2024-03-30T01:11:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 130518, 2024-03-30T01:11:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 67609778, 2024-03-30T01:11:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 142432051, 2024-03-30T01:11:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 8, 2024-03-30T01:11:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 5, 2024-03-30T01:11:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 6.491226108, 2024-03-30T01:11:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 2.369025585, 2024-03-30T01:11:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 3.818596953, 2024-03-30T01:11:22.573 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 1.009331328, 2024-03-30T01:11:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 130518, 2024-03-30T01:11:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 32.497580592, 2024-03-30T01:11:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000248989 2024-03-30T01:11:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 184993, 2024-03-30T01:11:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.025147692, 2024-03-30T01:11:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000135 2024-03-30T01:11:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T01:11:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T01:11:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.574 INFO:teuthology.orchestra.run.smithi181.stdout: "cct": { 2024-03-30T01:11:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "total_workers": 1, 2024-03-30T01:11:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "unhealthy_workers": 0 2024-03-30T01:11:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "finisher-MDSRank": { 2024-03-30T01:11:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "queue_len": 0, 2024-03-30T01:11:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "complete_latency": { 2024-03-30T01:11:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 64578, 2024-03-30T01:11:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 91.033687717, 2024-03-30T01:11:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.001409670 2024-03-30T01:11:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "finisher-PurgeQueue": { 2024-03-30T01:11:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "queue_len": 0, 2024-03-30T01:11:22.575 INFO:teuthology.orchestra.run.smithi181.stdout: "complete_latency": { 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 45639, 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 8.321366981, 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000182330 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "mds": { 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "request": 108697, 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "reply": 106076, 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "reply_latency": { 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 106076, 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 1495.241888322, 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.014095949 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "slow_reply": 0, 2024-03-30T01:11:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: "forward": 2611, 2024-03-30T01:11:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_fetch_complete": 32, 2024-03-30T01:11:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_fetch_keys": 0, 2024-03-30T01:11:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_commit": 1152, 2024-03-30T01:11:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_split": 95, 2024-03-30T01:11:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_merge": 1, 2024-03-30T01:11:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes": 34, 2024-03-30T01:11:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_top": 13, 2024-03-30T01:11:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_bottom": 11, 2024-03-30T01:11:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_pin_tail": 10, 2024-03-30T01:11:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_pinned": 13, 2024-03-30T01:11:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_expired": 41747, 2024-03-30T01:11:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "inodes_with_caps": 2, 2024-03-30T01:11:22.577 INFO:teuthology.orchestra.run.smithi181.stdout: "caps": 2, 2024-03-30T01:11:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "subtrees": 5, 2024-03-30T01:11:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse": 182954, 2024-03-30T01:11:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_hit": 158378, 2024-03-30T01:11:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_forward": 2170, 2024-03-30T01:11:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_discover": 1190, 2024-03-30T01:11:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_dir_fetch": 0, 2024-03-30T01:11:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_remote_ino": 0, 2024-03-30T01:11:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "traverse_lock": 849, 2024-03-30T01:11:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "load_cent": 0, 2024-03-30T01:11:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "q": 0, 2024-03-30T01:11:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "exported": 418, 2024-03-30T01:11:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "exported_inodes": 31214, 2024-03-30T01:11:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "imported": 252, 2024-03-30T01:11:22.578 INFO:teuthology.orchestra.run.smithi181.stdout: "imported_inodes": 43083, 2024-03-30T01:11:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_dir_fetch": 0, 2024-03-30T01:11:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_backtrace_fetch": 71845, 2024-03-30T01:11:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "openino_peer_discover": 0, 2024-03-30T01:11:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rfiles": 14289, 2024-03-30T01:11:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rbytes": 296779860, 2024-03-30T01:11:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "root_rsnaps": 0, 2024-03-30T01:11:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_backtrace_fetch": 71845, 2024-03-30T01:11:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_set_tag": 0, 2024-03-30T01:11:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T01:11:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_inotable_repaired": 0, 2024-03-30T01:11:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dir_inodes": 293, 2024-03-30T01:11:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dir_base_inodes": 0, 2024-03-30T01:11:22.579 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_dirfrag_rstats": 293, 2024-03-30T01:11:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "scrub_file_inodes": 71552, 2024-03-30T01:11:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_inode_file_caps": 1223, 2024-03-30T01:11:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_revoke": 25564, 2024-03-30T01:11:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_grant": 51732, 2024-03-30T01:11:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T01:11:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T01:11:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T01:11:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_caps": 64648, 2024-03-30T01:11:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_caps_dirty": 25195, 2024-03-30T01:11:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_cap_release": 1399, 2024-03-30T01:11:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "process_request_cap_release": 62642 2024-03-30T01:11:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache": { 2024-03-30T01:11:22.580 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_update": 2448, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_update_receipt": 6647, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_try_discover": 3861, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_send_discover": 4863, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "dir_handle_discover": 5153, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays": 0, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays_delayed": 0, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "num_strays_enqueuing": 0, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_created": 26079, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_enqueued": 26079, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_reintegrated": 0, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "strays_migrated": 0, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_processing": 0, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_enqueued": 0, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "num_recovering_prioritized": 0, 2024-03-30T01:11:22.581 INFO:teuthology.orchestra.run.smithi181.stdout: "recovery_started": 0, 2024-03-30T01:11:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "recovery_completed": 0, 2024-03-30T01:11:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_enqueue_scrub": 0, 2024-03-30T01:11:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_exportdir": 418, 2024-03-30T01:11:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_flush": 0, 2024-03-30T01:11:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_fragmentdir": 96, 2024-03-30T01:11:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_fragstats": 0, 2024-03-30T01:11:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "ireq_inodestats": 0 2024-03-30T01:11:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log": { 2024-03-30T01:11:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "evadd": 148893, 2024-03-30T01:11:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "evex": 41119, 2024-03-30T01:11:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "evtrm": 41119, 2024-03-30T01:11:22.582 INFO:teuthology.orchestra.run.smithi181.stdout: "ev": 107774, 2024-03-30T01:11:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "evexg": 0, 2024-03-30T01:11:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "evexd": 3675, 2024-03-30T01:11:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "segadd": 176, 2024-03-30T01:11:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "segex": 48, 2024-03-30T01:11:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "segtrm": 48, 2024-03-30T01:11:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "seg": 132, 2024-03-30T01:11:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "segexg": 0, 2024-03-30T01:11:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "segexd": 4, 2024-03-30T01:11:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "expos": 92277658, 2024-03-30T01:11:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "wrpos": 337778582, 2024-03-30T01:11:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "rdpos": 0, 2024-03-30T01:11:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "jlat": { 2024-03-30T01:11:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 4626, 2024-03-30T01:11:22.583 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 213.546041985, 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.046162136 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "replayed": 0 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_mem": { 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "ino": 39, 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "ino+": 44658, 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "ino-": 44619, 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "dir": 38, 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "dir+": 4389, 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "dir-": 4351, 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "dn": 34, 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "dn+": 98567, 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "dn-": 98533, 2024-03-30T01:11:22.584 INFO:teuthology.orchestra.run.smithi181.stdout: "cap": 2, 2024-03-30T01:11:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "cap+": 47755, 2024-03-30T01:11:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "cap-": 47753, 2024-03-30T01:11:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "rss": 158672, 2024-03-30T01:11:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "heap": 198964 2024-03-30T01:11:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_server": { 2024-03-30T01:11:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "dispatch_client_request": 123397, 2024-03-30T01:11:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "dispatch_server_request": 8981, 2024-03-30T01:11:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_request": 108697, 2024-03-30T01:11:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_client_session": 661, 2024-03-30T01:11:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "handle_peer_request": 16895, 2024-03-30T01:11:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "req_create_latency": { 2024-03-30T01:11:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 20389, 2024-03-30T01:11:22.585 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 72.232506107, 2024-03-30T01:11:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.003542719 2024-03-30T01:11:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getattr_latency": { 2024-03-30T01:11:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 67, 2024-03-30T01:11:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 10.476238041, 2024-03-30T01:11:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.156361761 2024-03-30T01:11:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getfilelock_latency": { 2024-03-30T01:11:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "req_link_latency": { 2024-03-30T01:11:22.586 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookup_latency": { 2024-03-30T01:11:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 38538, 2024-03-30T01:11:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 848.782374064, 2024-03-30T01:11:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.022024556 2024-03-30T01:11:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookuphash_latency": { 2024-03-30T01:11:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.587 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupino_latency": { 2024-03-30T01:11:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 2, 2024-03-30T01:11:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000228476, 2024-03-30T01:11:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000114238 2024-03-30T01:11:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupname_latency": { 2024-03-30T01:11:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupparent_latency": { 2024-03-30T01:11:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lookupsnap_latency": { 2024-03-30T01:11:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "req_lssnap_latency": { 2024-03-30T01:11:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mkdir_latency": { 2024-03-30T01:11:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 66, 2024-03-30T01:11:22.589 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 7.818888643, 2024-03-30T01:11:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.118468009 2024-03-30T01:11:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mknod_latency": { 2024-03-30T01:11:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "req_mksnap_latency": { 2024-03-30T01:11:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.590 INFO:teuthology.orchestra.run.smithi181.stdout: "req_open_latency": { 2024-03-30T01:11:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 7298, 2024-03-30T01:11:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 22.132133272, 2024-03-30T01:11:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.003032629 2024-03-30T01:11:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: "req_readdir_latency": { 2024-03-30T01:11:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 771, 2024-03-30T01:11:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 1.698906750, 2024-03-30T01:11:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.002203510 2024-03-30T01:11:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rename_latency": { 2024-03-30T01:11:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 20340, 2024-03-30T01:11:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 493.266034763, 2024-03-30T01:11:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.024251034 2024-03-30T01:11:22.591 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "req_renamesnap_latency": { 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "req_snapdiff_latency": { 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmdir_latency": { 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 110, 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 12.985636928, 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.118051244 2024-03-30T01:11:22.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmsnap_latency": { 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "req_rmxattr_latency": { 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setattr_latency": { 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setdirlayout_latency": { 2024-03-30T01:11:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setfilelock_latency": { 2024-03-30T01:11:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setlayout_latency": { 2024-03-30T01:11:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.594 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "req_setxattr_latency": { 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "req_symlink_latency": { 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "req_unlink_latency": { 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 18495, 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 25.848941278, 2024-03-30T01:11:22.595 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.001397617 2024-03-30T01:11:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "cap_revoke_eviction": 0, 2024-03-30T01:11:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "cap_acquisition_throttle": 0, 2024-03-30T01:11:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "req_getvxattr_latency": { 2024-03-30T01:11:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_sessions": { 2024-03-30T01:11:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "session_count": 1, 2024-03-30T01:11:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "session_add": 2, 2024-03-30T01:11:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "session_remove": 1, 2024-03-30T01:11:22.596 INFO:teuthology.orchestra.run.smithi181.stdout: "sessions_open": 1, 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "sessions_stale": 0, 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "total_load": 15192, 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "average_load": 15192, 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "avg_session_uptime": 821, 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "mdthresh_evicted": 0 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "mempool": { 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_bytes": 0, 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_items": 0, 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_items": 0, 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_items": 0, 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T01:11:22.597 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T01:11:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T01:11:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T01:11:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T01:11:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_items": 0, 2024-03-30T01:11:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T01:11:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T01:11:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_bytes": 0, 2024-03-30T01:11:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_items": 0, 2024-03-30T01:11:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_bytes": 0, 2024-03-30T01:11:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_items": 0, 2024-03-30T01:11:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T01:11:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T01:11:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T01:11:22.598 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_items": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_bytes": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_items": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_bytes": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_items": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_bytes": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_items": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_items": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_items": 0, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_bytes": 290776, 2024-03-30T01:11:22.599 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_items": 111, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_bytes": 88, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_items": 1, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_bytes": 0, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_items": 0, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_bytes": 0, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_items": 0, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_bytes": 0, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_items": 0, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_bytes": 8696, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_items": 166, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_items": 0, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_bytes": 0, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_items": 0, 2024-03-30T01:11:22.600 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_bytes": 159277, 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_items": 679, 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_bytes": 0, 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_items": 0, 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_bytes": 0, 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_items": 0 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter": { 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "op_active": 0, 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "op_laggy": 0, 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "op_send": 147985, 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "op_send_bytes": 367329599, 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "op_resend": 27, 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "op_reply": 147985, 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "op_latency": { 2024-03-30T01:11:22.601 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 147985, 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 16569.063595149, 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.111964480 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "op_inflight": 0, 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "oplen_avg": { 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 147985, 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 172509 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "op": 147985, 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "op_r": 72811, 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "op_w": 75174, 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "op_rmw": 0, 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "op_pg": 0, 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_stat": 277, 2024-03-30T01:11:22.602 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_create": 14951, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_read": 934, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_write": 5567, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_writefull": 452, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_writesame": 0, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_append": 0, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_zero": 0, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_truncate": 95, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_delete": 52816, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_mapext": 0, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_sparse_read": 0, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_clonerange": 0, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_getxattr": 71845, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_setxattr": 22466, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_cmpxattr": 0, 2024-03-30T01:11:22.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_rmxattr": 0, 2024-03-30T01:11:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_resetxattrs": 0, 2024-03-30T01:11:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_call": 0, 2024-03-30T01:11:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_watch": 0, 2024-03-30T01:11:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_notify": 0, 2024-03-30T01:11:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T01:11:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_pgls": 0, 2024-03-30T01:11:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_pgls_filter": 0, 2024-03-30T01:11:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osdop_other": 0, 2024-03-30T01:11:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_active": 0, 2024-03-30T01:11:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_send": 0, 2024-03-30T01:11:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_resend": 0, 2024-03-30T01:11:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_ping": 0, 2024-03-30T01:11:22.604 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_active": 0, 2024-03-30T01:11:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_send": 0, 2024-03-30T01:11:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "poolop_resend": 0, 2024-03-30T01:11:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_active": 0, 2024-03-30T01:11:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_send": 0, 2024-03-30T01:11:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "poolstat_resend": 0, 2024-03-30T01:11:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_active": 0, 2024-03-30T01:11:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_send": 0, 2024-03-30T01:11:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_resend": 0, 2024-03-30T01:11:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "command_active": 0, 2024-03-30T01:11:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "command_send": 0, 2024-03-30T01:11:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "command_resend": 0, 2024-03-30T01:11:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "map_epoch": 83, 2024-03-30T01:11:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "map_full": 0, 2024-03-30T01:11:22.605 INFO:teuthology.orchestra.run.smithi181.stdout: "map_inc": 2, 2024-03-30T01:11:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_sessions": 12, 2024-03-30T01:11:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_session_open": 12, 2024-03-30T01:11:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_session_close": 0, 2024-03-30T01:11:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_laggy": 0, 2024-03-30T01:11:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_wr": 2547, 2024-03-30T01:11:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_rd": 64, 2024-03-30T01:11:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_del": 495 2024-03-30T01:11:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "oft": { 2024-03-30T01:11:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_objs": 1, 2024-03-30T01:11:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_kv_pairs": 0, 2024-03-30T01:11:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_updates": 0, 2024-03-30T01:11:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: "omap_total_removes": 0 2024-03-30T01:11:22.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "osd-slow-ops": { 2024-03-30T01:11:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "slow_ops_count": 113321 2024-03-30T01:11:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "purge_queue": { 2024-03-30T01:11:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_ops": 0, 2024-03-30T01:11:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_ops_high_water": 24, 2024-03-30T01:11:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing": 0, 2024-03-30T01:11:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executing_high_water": 9, 2024-03-30T01:11:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_executed": 26079, 2024-03-30T01:11:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "pq_item_in_journal": 0 2024-03-30T01:11:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T01:11:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:22.607 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-03-30T01:11:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 504516, 2024-03-30T01:11:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 146672125, 2024-03-30T01:11:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 504516, 2024-03-30T01:11:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-03-30T01:11:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-03-30T01:11:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 504516, 2024-03-30T01:11:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 146672125, 2024-03-30T01:11:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.608 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-objecter_bytes": { 2024-03-30T01:11:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-03-30T01:11:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-03-30T01:11:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-03-30T01:11:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 147985, 2024-03-30T01:11:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 370171466, 2024-03-30T01:11:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 147985, 2024-03-30T01:11:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 370171466, 2024-03-30T01:11:22.609 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:22.610 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.610 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.610 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.610 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:22.610 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.610 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-objecter_ops": { 2024-03-30T01:11:22.610 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:22.610 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 1024, 2024-03-30T01:11:22.610 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:22.610 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 0, 2024-03-30T01:11:22.610 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 0, 2024-03-30T01:11:22.610 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:22.610 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 0, 2024-03-30T01:11:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 147985, 2024-03-30T01:11:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 147985, 2024-03-30T01:11:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 147985, 2024-03-30T01:11:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 147985, 2024-03-30T01:11:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-write_buf_throttle": { 2024-03-30T01:11:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 3758096384, 2024-03-30T01:11:22.611 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 26079, 2024-03-30T01:11:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 2842379, 2024-03-30T01:11:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 26079, 2024-03-30T01:11:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-03-30T01:11:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-03-30T01:11:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 940, 2024-03-30T01:11:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 2842379, 2024-03-30T01:11:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:22.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-write_buf_throttle-0x55d55d6460a0": { 2024-03-30T01:11:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 3758096384, 2024-03-30T01:11:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 148893, 2024-03-30T01:11:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 333584278, 2024-03-30T01:11:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 148893, 2024-03-30T01:11:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-03-30T01:11:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-03-30T01:11:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 4626, 2024-03-30T01:11:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 333584278, 2024-03-30T01:11:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:22.613 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:22.614 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:22.614 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-30T01:11:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:22 smithi181 ceph-mon[47622]: pgmap v696: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-30T01:11:22.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:22 smithi116 ceph-mon[38086]: pgmap v696: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-30T01:11:22.935 INFO:tasks.check_counter:Daemon mds.i mds.exported=418 2024-03-30T01:11:22.935 INFO:tasks.check_counter:Daemon mds.i mds.imported=252 2024-03-30T01:11:22.935 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=2448 2024-03-30T01:11:22.935 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=6647 2024-03-30T01:11:22.936 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=95 2024-03-30T01:11:22.936 DEBUG:tasks.check_counter:Getting stats from l 2024-03-30T01:11:22.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 -- ceph daemon mds.l perf dump 2024-03-30T01:11:23.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:22 smithi130 ceph-mon[47626]: pgmap v696: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-30T01:11:23.157 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T01:11:24.076 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-30T01:11:24.077 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T01:11:24.077 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 4, 2024-03-30T01:11:24.077 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 4, 2024-03-30T01:11:24.077 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 2992, 2024-03-30T01:11:24.077 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 448, 2024-03-30T01:11:24.077 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 4, 2024-03-30T01:11:24.077 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 0, 2024-03-30T01:11:24.077 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.001321427, 2024-03-30T01:11:24.077 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.000039465, 2024-03-30T01:11:24.077 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.000461933, 2024-03-30T01:11:24.077 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T01:11:24.077 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:24.077 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 3, 2024-03-30T01:11:24.077 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000682488, 2024-03-30T01:11:24.078 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000227496 2024-03-30T01:11:24.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:24.078 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:24.078 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:24.078 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:24.078 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:24.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:24.078 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-30T01:11:24.078 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 448 2024-03-30T01:11:24.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:24.078 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T01:11:24.078 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 2, 2024-03-30T01:11:24.078 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 196, 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 320, 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 186880, 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 4, 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 2, 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.010185399, 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.007029771, 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.001149551, 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 194, 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.015503626, 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000079915 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:24.079 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 186880 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_messages": 259, 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages": 256, 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_bytes": 483760, 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_bytes": 146304, 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_created_connections": 2, 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_active_connections": 1, 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_total_time": 0.034730720, 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_send_time": 0.012524902, 2024-03-30T01:11:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_recv_time": 0.015375185, 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_running_fast_dispatch_time": 0.005945779, 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 255, 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.025767838, 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000101050 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_handle_ack_lat": { 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_recv_encrypted_bytes": 483760, 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: "msgr_send_encrypted_bytes": 146304 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:24.081 INFO:teuthology.orchestra.run.smithi181.stdout: "cct": { 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: "total_workers": 0, 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: "unhealthy_workers": 0 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: "mempool": { 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_bytes": 0, 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: "bloom_filter_items": 0, 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_items": 0, 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_data_items": 0, 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T01:11:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_other_items": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_bytes": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_items": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_bytes": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blob_items": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_items": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_bytes": 0, 2024-03-30T01:11:24.083 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_txc_items": 0, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_bytes": 0, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_writing_items": 0, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_bytes": 0, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_items": 0, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_reader_items": 0, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_file_writer_items": 0, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_bytes": 148722, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_anon_items": 30, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_bytes": 0, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "buffer_meta_items": 0, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_bytes": 0, 2024-03-30T01:11:24.084 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_items": 0, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_bytes": 0, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mapbl_items": 0, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_bytes": 0, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pglog_items": 0, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_bytes": 0, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_items": 0, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "osdmap_mapping_items": 0, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_bytes": 0, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "pgmap_items": 0, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_bytes": 928, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_co_items": 2, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_bytes": 0, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_1_items": 0, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_bytes": 0, 2024-03-30T01:11:24.085 INFO:teuthology.orchestra.run.smithi181.stdout: "unittest_2_items": 0 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: "val": 0, 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 104857600, 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: "get_started": 0, 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: "get": 261, 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: "get_sum": 447175, 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_fail": 0, 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: "get_or_fail_success": 261, 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: "take": 0, 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: "take_sum": 0, 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: "put": 261, 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: "put_sum": 447175, 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: "wait": { 2024-03-30T01:11:24.086 INFO:teuthology.orchestra.run.smithi181.stdout: "avgcount": 0, 2024-03-30T01:11:24.087 INFO:teuthology.orchestra.run.smithi181.stdout: "sum": 0.000000000, 2024-03-30T01:11:24.087 INFO:teuthology.orchestra.run.smithi181.stdout: "avgtime": 0.000000000 2024-03-30T01:11:24.087 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:24.087 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-30T01:11:24.087 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-30T01:11:24.450 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-30T01:11:24.451 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-30T01:11:24.451 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-03-30T01:11:24.451 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-03-30T01:11:24.451 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-03-30T01:11:24.451 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-30T01:11:24.461 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-30T01:11:24.462 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T01:11:24.462 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-30T01:11:24.512 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T01:11:24.512 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-30T01:11:24.597 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T01:11:24.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.829+0000 7fa6bf000700 1 -- 172.21.15.116:0/336461889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6b8101e70 msgr2=0x7fa6b81022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:24.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.829+0000 7fa6bf000700 1 --2- 172.21.15.116:0/336461889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6b8101e70 0x7fa6b81022d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40098f0 tx=0x7fa6a4009c00 comp rx=0 tx=0).stop 2024-03-30T01:11:24.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.829+0000 7fa6bf000700 1 -- 172.21.15.116:0/336461889 shutdown_connections 2024-03-30T01:11:24.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.829+0000 7fa6bf000700 1 --2- 172.21.15.116:0/336461889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6b8102810 0x7fa6b81090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:24.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.829+0000 7fa6bf000700 1 --2- 172.21.15.116:0/336461889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6b8100c80 0x7fa6b81010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:24.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.829+0000 7fa6bf000700 1 --2- 172.21.15.116:0/336461889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6b8101e70 0x7fa6b81022d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:24.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.829+0000 7fa6bf000700 1 -- 172.21.15.116:0/336461889 >> 172.21.15.116:0/336461889 conn(0x7fa6b80fc1f0 msgr2=0x7fa6b80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:24.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.830+0000 7fa6bf000700 1 -- 172.21.15.116:0/336461889 shutdown_connections 2024-03-30T01:11:24.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.830+0000 7fa6bf000700 1 -- 172.21.15.116:0/336461889 wait complete. 2024-03-30T01:11:24.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.831+0000 7fa6bf000700 1 Processor -- start 2024-03-30T01:11:24.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.831+0000 7fa6bf000700 1 -- start start 2024-03-30T01:11:24.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.831+0000 7fa6bf000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6b8100c80 0x7fa6b8198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:24.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.831+0000 7fa6bf000700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6b8101e70 0x7fa6b8199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:24.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.831+0000 7fa6bf000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6b8102810 0x7fa6b81a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:24.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.831+0000 7fa6bf000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b81a09d0 con 0x7fa6b8100c80 2024-03-30T01:11:24.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.831+0000 7fa6bf000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b81a0b10 con 0x7fa6b8102810 2024-03-30T01:11:24.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.831+0000 7fa6bf000700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b81a0ca0 con 0x7fa6b8101e70 2024-03-30T01:11:24.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.831+0000 7fa6b7e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6b8100c80 0x7fa6b8198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:24.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.831+0000 7fa6b7e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6b8100c80 0x7fa6b8198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49220/0 (socket says 172.21.15.116:49220) 2024-03-30T01:11:24.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.831+0000 7fa6b7e00700 1 -- 172.21.15.116:0/3518155039 learned_addr learned my addr 172.21.15.116:0/3518155039 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:24.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.832+0000 7fa6b7400700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6b8101e70 0x7fa6b8199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:24.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.832+0000 7fa6bd000700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6b8102810 0x7fa6b81a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:24.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.832+0000 7fa6b7e00700 1 -- 172.21.15.116:0/3518155039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6b8101e70 msgr2=0x7fa6b8199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:24.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.832+0000 7fa6b7e00700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6b8101e70 0x7fa6b8199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:24.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.832+0000 7fa6b7e00700 1 -- 172.21.15.116:0/3518155039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6b8102810 msgr2=0x7fa6b81a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:24.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.832+0000 7fa6b7e00700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6b8102810 0x7fa6b81a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:24.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.832+0000 7fa6b7e00700 1 -- 172.21.15.116:0/3518155039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b0009d00 con 0x7fa6b8100c80 2024-03-30T01:11:24.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.832+0000 7fa6bd000700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6b8102810 0x7fa6b81a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:11:24.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.832+0000 7fa6b7e00700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6b8100c80 0x7fa6b8198ef0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa6a800e970 tx=0x7fa6a800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:24.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.832+0000 7fa6b4c00700 1 -- 172.21.15.116:0/3518155039 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8009790 con 0x7fa6b8100c80 2024-03-30T01:11:24.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.832+0000 7fa6bf000700 1 -- 172.21.15.116:0/3518155039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a4009520 con 0x7fa6b8100c80 2024-03-30T01:11:24.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.832+0000 7fa6b4c00700 1 -- 172.21.15.116:0/3518155039 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6a800eec0 con 0x7fa6b8100c80 2024-03-30T01:11:24.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.832+0000 7fa6b4c00700 1 -- 172.21.15.116:0/3518155039 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8003cd0 con 0x7fa6b8100c80 2024-03-30T01:11:24.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.832+0000 7fa6bf000700 1 -- 172.21.15.116:0/3518155039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b81a1220 con 0x7fa6b8100c80 2024-03-30T01:11:24.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.834+0000 7fa6bf000700 1 -- 172.21.15.116:0/3518155039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b8067a80 con 0x7fa6b8100c80 2024-03-30T01:11:24.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.835+0000 7fa6b4c00700 1 -- 172.21.15.116:0/3518155039 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fa6a800ccd0 con 0x7fa6b8100c80 2024-03-30T01:11:24.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.836+0000 7fa6b4c00700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa69c0a11a0 0x7fa69c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:24.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.837+0000 7fa6b4c00700 1 -- 172.21.15.116:0/3518155039 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa6a8014070 con 0x7fa6b8100c80 2024-03-30T01:11:24.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.837+0000 7fa6b7400700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa69c0a11a0 0x7fa69c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:24.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.837+0000 7fa6b7400700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa69c0a11a0 0x7fa69c0a3660 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400fa70 tx=0x7fa6a4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:24.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:24.841+0000 7fa6b4c00700 1 -- 172.21.15.116:0/3518155039 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa6b8067a80 con 0x7fa6b8100c80 2024-03-30T01:11:24.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:24 smithi116 ceph-mon[38086]: pgmap v697: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T01:11:25.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:24 smithi181 ceph-mon[47622]: pgmap v697: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T01:11:25.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.084+0000 7fa6bf000700 1 -- 172.21.15.116:0/3518155039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6b810c4e0 con 0x7fa6b8100c80 2024-03-30T01:11:25.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.085+0000 7fa6b4c00700 1 -- 172.21.15.116:0/3518155039 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+24556 (secure 0 0 0) 0x7fa6a8096230 con 0x7fa6b8100c80 2024-03-30T01:11:25.094 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T01:11:25.095 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:56:36.456069+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T01:11:25.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.088+0000 7fa6bf000700 1 -- 172.21.15.116:0/3518155039 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa69c0a11a0 msgr2=0x7fa69c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.088+0000 7fa6bf000700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa69c0a11a0 0x7fa69c0a3660 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400fa70 tx=0x7fa6a4019040 comp rx=0 tx=0).stop 2024-03-30T01:11:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.088+0000 7fa6bf000700 1 -- 172.21.15.116:0/3518155039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6b8100c80 msgr2=0x7fa6b8198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.088+0000 7fa6bf000700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6b8100c80 0x7fa6b8198ef0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa6a800e970 tx=0x7fa6a800ec80 comp rx=0 tx=0).stop 2024-03-30T01:11:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.088+0000 7fa6bf000700 1 -- 172.21.15.116:0/3518155039 shutdown_connections 2024-03-30T01:11:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.088+0000 7fa6bf000700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa69c0a11a0 0x7fa69c0a3660 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.088+0000 7fa6bf000700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6b8102810 0x7fa6b81a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.088+0000 7fa6bf000700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6b8100c80 0x7fa6b8198ef0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.088+0000 7fa6bf000700 1 --2- 172.21.15.116:0/3518155039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6b8101e70 0x7fa6b8199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.088+0000 7fa6bf000700 1 -- 172.21.15.116:0/3518155039 >> 172.21.15.116:0/3518155039 conn(0x7fa6b80fc1f0 msgr2=0x7fa6b8072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.088+0000 7fa6bf000700 1 -- 172.21.15.116:0/3518155039 shutdown_connections 2024-03-30T01:11:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.088+0000 7fa6bf000700 1 -- 172.21.15.116:0/3518155039 wait complete. 2024-03-30T01:11:25.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:24 smithi130 ceph-mon[47626]: pgmap v697: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T01:11:25.113 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T00:52:29.602428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T00:55:50.220543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T00:55:51.274498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T00:55:54.301591+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T01:11:25.114 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T01:11:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.304+0000 7fc367600700 1 -- 172.21.15.116:0/3068327484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3600737b0 msgr2=0x7fc360073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.304+0000 7fc367600700 1 --2- 172.21.15.116:0/3068327484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3600737b0 0x7fc360073c30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc34c009990 tx=0x7fc34c009ca0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.304+0000 7fc367600700 1 -- 172.21.15.116:0/3068327484 shutdown_connections 2024-03-30T01:11:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.304+0000 7fc367600700 1 --2- 172.21.15.116:0/3068327484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc360074d90 0x7fc3600731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.304+0000 7fc367600700 1 --2- 172.21.15.116:0/3068327484 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc360108c50 0x7fc36010b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.304+0000 7fc367600700 1 --2- 172.21.15.116:0/3068327484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3600737b0 0x7fc360073c30 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.304+0000 7fc367600700 1 -- 172.21.15.116:0/3068327484 >> 172.21.15.116:0/3068327484 conn(0x7fc3600fc2f0 msgr2=0x7fc3600fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.305+0000 7fc367600700 1 -- 172.21.15.116:0/3068327484 shutdown_connections 2024-03-30T01:11:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.305+0000 7fc367600700 1 -- 172.21.15.116:0/3068327484 wait complete. 2024-03-30T01:11:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.306+0000 7fc367600700 1 Processor -- start 2024-03-30T01:11:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.306+0000 7fc367600700 1 -- start start 2024-03-30T01:11:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc367600700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3600737b0 0x7fc360198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc367600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc360074d90 0x7fc360199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc367600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc360108c50 0x7fc3601a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc367600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3601a0a40 con 0x7fc360074d90 2024-03-30T01:11:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc367600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3601a0b80 con 0x7fc360108c50 2024-03-30T01:11:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc367600700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3601a0d10 con 0x7fc3600737b0 2024-03-30T01:11:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc35fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc360074d90 0x7fc360199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc364c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3600737b0 0x7fc360198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc364c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3600737b0 0x7fc360198f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.116:59158/0 (socket says 172.21.15.116:59158) 2024-03-30T01:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc364c00700 1 -- 172.21.15.116:0/1891674742 learned_addr learned my addr 172.21.15.116:0/1891674742 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc365600700 1 --2- 172.21.15.116:0/1891674742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc360108c50 0x7fc3601a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc364c00700 1 -- 172.21.15.116:0/1891674742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc360108c50 msgr2=0x7fc3601a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc364c00700 1 --2- 172.21.15.116:0/1891674742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc360108c50 0x7fc3601a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc364c00700 1 -- 172.21.15.116:0/1891674742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc360074d90 msgr2=0x7fc360199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc364c00700 1 --2- 172.21.15.116:0/1891674742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc360074d90 0x7fc360199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc364c00700 1 -- 172.21.15.116:0/1891674742 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc34c0095f0 con 0x7fc3600737b0 2024-03-30T01:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.307+0000 7fc35fe00700 1 --2- 172.21.15.116:0/1891674742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc360074d90 0x7fc360199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.308+0000 7fc364c00700 1 --2- 172.21.15.116:0/1891674742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3600737b0 0x7fc360198f10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc35400e9a0 tx=0x7fc35400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.308+0000 7fc35d600700 1 -- 172.21.15.116:0/1891674742 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354009790 con 0x7fc3600737b0 2024-03-30T01:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.308+0000 7fc367600700 1 -- 172.21.15.116:0/1891674742 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3601a0ff0 con 0x7fc3600737b0 2024-03-30T01:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.308+0000 7fc35d600700 1 -- 172.21.15.116:0/1891674742 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc354010430 con 0x7fc3600737b0 2024-03-30T01:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.308+0000 7fc367600700 1 -- 172.21.15.116:0/1891674742 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3601a1420 con 0x7fc3600737b0 2024-03-30T01:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.308+0000 7fc35d600700 1 -- 172.21.15.116:0/1891674742 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354018680 con 0x7fc3600737b0 2024-03-30T01:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.309+0000 7fc367600700 1 -- 172.21.15.116:0/1891674742 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc360107a60 con 0x7fc3600737b0 2024-03-30T01:11:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.311+0000 7fc35d600700 1 -- 172.21.15.116:0/1891674742 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fc35400ca10 con 0x7fc3600737b0 2024-03-30T01:11:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.312+0000 7fc35d600700 1 --2- 172.21.15.116:0/1891674742 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc3480a1120 0x7fc3480a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.312+0000 7fc35d600700 1 -- 172.21.15.116:0/1891674742 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc354014070 con 0x7fc3600737b0 2024-03-30T01:11:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.313+0000 7fc35fe00700 1 --2- 172.21.15.116:0/1891674742 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc3480a1120 0x7fc3480a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.313+0000 7fc35fe00700 1 --2- 172.21.15.116:0/1891674742 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc3480a1120 0x7fc3480a35e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00b2e0 tx=0x7fc34c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:25.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.315+0000 7fc35d600700 1 -- 172.21.15.116:0/1891674742 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc3540529f0 con 0x7fc3600737b0 2024-03-30T01:11:25.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.535+0000 7fc367600700 1 -- 172.21.15.116:0/1891674742 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc360067af0 con 0x7fc3600737b0 2024-03-30T01:11:25.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.536+0000 7fc35d600700 1 -- 172.21.15.116:0/1891674742 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc354099be0 con 0x7fc3600737b0 2024-03-30T01:11:25.541 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-30T01:11:25.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.539+0000 7fc367600700 1 -- 172.21.15.116:0/1891674742 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc3480a1120 msgr2=0x7fc3480a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:25.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.539+0000 7fc367600700 1 --2- 172.21.15.116:0/1891674742 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc3480a1120 0x7fc3480a35e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00b2e0 tx=0x7fc34c005fb0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.539+0000 7fc367600700 1 -- 172.21.15.116:0/1891674742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3600737b0 msgr2=0x7fc360198f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:25.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.539+0000 7fc367600700 1 --2- 172.21.15.116:0/1891674742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3600737b0 0x7fc360198f10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc35400e9a0 tx=0x7fc35400ecb0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.539+0000 7fc367600700 1 -- 172.21.15.116:0/1891674742 shutdown_connections 2024-03-30T01:11:25.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.539+0000 7fc367600700 1 --2- 172.21.15.116:0/1891674742 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc3480a1120 0x7fc3480a35e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.539+0000 7fc367600700 1 --2- 172.21.15.116:0/1891674742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3600737b0 0x7fc360198f10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.539+0000 7fc367600700 1 --2- 172.21.15.116:0/1891674742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc360108c50 0x7fc3601a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.539+0000 7fc367600700 1 --2- 172.21.15.116:0/1891674742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc360074d90 0x7fc360199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.539+0000 7fc367600700 1 -- 172.21.15.116:0/1891674742 >> 172.21.15.116:0/1891674742 conn(0x7fc3600fc2f0 msgr2=0x7fc360104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:25.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.539+0000 7fc367600700 1 -- 172.21.15.116:0/1891674742 shutdown_connections 2024-03-30T01:11:25.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.539+0000 7fc367600700 1 -- 172.21.15.116:0/1891674742 wait complete. 2024-03-30T01:11:25.563 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T01:11:25.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.754+0000 7ff08a400700 1 -- 172.21.15.116:0/3635679826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff084101f70 msgr2=0x7ff0841023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:25.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.754+0000 7ff08a400700 1 --2- 172.21.15.116:0/3635679826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff084101f70 0x7ff0841023d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff0700099c0 tx=0x7ff070009cd0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.754+0000 7ff08a400700 1 -- 172.21.15.116:0/3635679826 shutdown_connections 2024-03-30T01:11:25.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.754+0000 7ff08a400700 1 --2- 172.21.15.116:0/3635679826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff084102910 0x7ff0841091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.754+0000 7ff08a400700 1 --2- 172.21.15.116:0/3635679826 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff084100d80 0x7ff0841011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.754+0000 7ff08a400700 1 --2- 172.21.15.116:0/3635679826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff084101f70 0x7ff0841023d0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.754+0000 7ff08a400700 1 -- 172.21.15.116:0/3635679826 >> 172.21.15.116:0/3635679826 conn(0x7ff0840fc2f0 msgr2=0x7ff0840fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:25.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.754+0000 7ff08a400700 1 -- 172.21.15.116:0/3635679826 shutdown_connections 2024-03-30T01:11:25.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.755+0000 7ff08a400700 1 -- 172.21.15.116:0/3635679826 wait complete. 2024-03-30T01:11:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.755+0000 7ff08a400700 1 Processor -- start 2024-03-30T01:11:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.756+0000 7ff08a400700 1 -- start start 2024-03-30T01:11:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.756+0000 7ff08a400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff084100d80 0x7ff084198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.756+0000 7ff08a400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff084101f70 0x7ff0841994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.756+0000 7ff08a400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff084102910 0x7ff0841a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.756+0000 7ff08a400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0841a0ab0 con 0x7ff084100d80 2024-03-30T01:11:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.756+0000 7ff08a400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0841a0bf0 con 0x7ff084101f70 2024-03-30T01:11:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.756+0000 7ff08a400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0841a0d80 con 0x7ff084102910 2024-03-30T01:11:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.756+0000 7ff083400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff084100d80 0x7ff084198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.756+0000 7ff083400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff084100d80 0x7ff084198f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49272/0 (socket says 172.21.15.116:49272) 2024-03-30T01:11:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.756+0000 7ff083400700 1 -- 172.21.15.116:0/1331373799 learned_addr learned my addr 172.21.15.116:0/1331373799 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:25.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.756+0000 7ff082a00700 1 --2- 172.21.15.116:0/1331373799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff084101f70 0x7ff0841994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:25.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.756+0000 7ff083e00700 1 --2- 172.21.15.116:0/1331373799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff084102910 0x7ff0841a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:25.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.757+0000 7ff083400700 1 -- 172.21.15.116:0/1331373799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff084102910 msgr2=0x7ff0841a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:25.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.757+0000 7ff083400700 1 --2- 172.21.15.116:0/1331373799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff084102910 0x7ff0841a04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.757+0000 7ff083400700 1 -- 172.21.15.116:0/1331373799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff084101f70 msgr2=0x7ff0841994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.757+0000 7ff083400700 1 --2- 172.21.15.116:0/1331373799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff084101f70 0x7ff0841994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.757+0000 7ff083400700 1 -- 172.21.15.116:0/1331373799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0700095f0 con 0x7ff084100d80 2024-03-30T01:11:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.757+0000 7ff083400700 1 --2- 172.21.15.116:0/1331373799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff084100d80 0x7ff084198f80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff07400e970 tx=0x7ff07400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.757+0000 7ff07be00700 1 -- 172.21.15.116:0/1331373799 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff074009790 con 0x7ff084100d80 2024-03-30T01:11:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.757+0000 7ff07be00700 1 -- 172.21.15.116:0/1331373799 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff07400c7e0 con 0x7ff084100d80 2024-03-30T01:11:25.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.757+0000 7ff07be00700 1 -- 172.21.15.116:0/1331373799 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff074003c70 con 0x7ff084100d80 2024-03-30T01:11:25.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.757+0000 7ff08a400700 1 -- 172.21.15.116:0/1331373799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0841a1060 con 0x7ff084100d80 2024-03-30T01:11:25.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.757+0000 7ff08a400700 1 -- 172.21.15.116:0/1331373799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0841a1570 con 0x7ff084100d80 2024-03-30T01:11:25.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.759+0000 7ff08a400700 1 -- 172.21.15.116:0/1331373799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff084067260 con 0x7ff084100d80 2024-03-30T01:11:25.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.760+0000 7ff07be00700 1 -- 172.21.15.116:0/1331373799 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7ff07400cc50 con 0x7ff084100d80 2024-03-30T01:11:25.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.762+0000 7ff07be00700 1 --2- 172.21.15.116:0/1331373799 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff0680a11a0 0x7ff0680a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:25.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.762+0000 7ff07be00700 1 -- 172.21.15.116:0/1331373799 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff074014070 con 0x7ff084100d80 2024-03-30T01:11:25.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.762+0000 7ff082a00700 1 --2- 172.21.15.116:0/1331373799 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff0680a11a0 0x7ff0680a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:25.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.762+0000 7ff082a00700 1 --2- 172.21.15.116:0/1331373799 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff0680a11a0 0x7ff0680a3660 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff070006010 tx=0x7ff07000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.765+0000 7ff07be00700 1 -- 172.21.15.116:0/1331373799 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff0740961c0 con 0x7ff084100d80 2024-03-30T01:11:25.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:25 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3518155039' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T01:11:25.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:25 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1891674742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T01:11:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.973+0000 7ff08a400700 1 -- 172.21.15.116:0/1331373799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff0841a18c0 con 0x7ff084100d80 2024-03-30T01:11:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.973+0000 7ff07be00700 1 -- 172.21.15.116:0/1331373799 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff074003dd0 con 0x7ff084100d80 2024-03-30T01:11:25.979 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T01:11:25.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.975+0000 7ff08a400700 1 -- 172.21.15.116:0/1331373799 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff0680a11a0 msgr2=0x7ff0680a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:25.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.975+0000 7ff08a400700 1 --2- 172.21.15.116:0/1331373799 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff0680a11a0 0x7ff0680a3660 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff070006010 tx=0x7ff07000b540 comp rx=0 tx=0).stop 2024-03-30T01:11:25.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.975+0000 7ff08a400700 1 -- 172.21.15.116:0/1331373799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff084100d80 msgr2=0x7ff084198f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:25.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.975+0000 7ff08a400700 1 --2- 172.21.15.116:0/1331373799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff084100d80 0x7ff084198f80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff07400e970 tx=0x7ff07400ec80 comp rx=0 tx=0).stop 2024-03-30T01:11:25.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.976+0000 7ff08a400700 1 -- 172.21.15.116:0/1331373799 shutdown_connections 2024-03-30T01:11:25.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.976+0000 7ff08a400700 1 --2- 172.21.15.116:0/1331373799 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7ff0680a11a0 0x7ff0680a3660 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.976+0000 7ff08a400700 1 --2- 172.21.15.116:0/1331373799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff084102910 0x7ff0841a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.976+0000 7ff08a400700 1 --2- 172.21.15.116:0/1331373799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff084100d80 0x7ff084198f80 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.976+0000 7ff08a400700 1 --2- 172.21.15.116:0/1331373799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff084101f70 0x7ff0841994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.976+0000 7ff08a400700 1 -- 172.21.15.116:0/1331373799 >> 172.21.15.116:0/1331373799 conn(0x7ff0840fc2f0 msgr2=0x7ff084072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.976+0000 7ff08a400700 1 -- 172.21.15.116:0/1331373799 shutdown_connections 2024-03-30T01:11:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:25.976+0000 7ff08a400700 1 -- 172.21.15.116:0/1331373799 wait complete. 2024-03-30T01:11:26.000 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T01:11:26.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:25 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3518155039' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T01:11:26.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:25 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1891674742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T01:11:26.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:25 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3518155039' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T01:11:26.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:25 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1891674742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T01:11:26.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.192+0000 7f147f400700 1 -- 172.21.15.116:0/2717810942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1478100d80 msgr2=0x7f14781011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:26.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.192+0000 7f147f400700 1 --2- 172.21.15.116:0/2717810942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1478100d80 0x7f14781011a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f146c0099c0 tx=0x7f146c009cd0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.192+0000 7f147f400700 1 -- 172.21.15.116:0/2717810942 shutdown_connections 2024-03-30T01:11:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.192+0000 7f147f400700 1 --2- 172.21.15.116:0/2717810942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1478102910 0x7f14781091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.192+0000 7f147f400700 1 --2- 172.21.15.116:0/2717810942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1478100d80 0x7f14781011a0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.192+0000 7f147f400700 1 --2- 172.21.15.116:0/2717810942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1478101f70 0x7f14781023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.192+0000 7f147f400700 1 -- 172.21.15.116:0/2717810942 >> 172.21.15.116:0/2717810942 conn(0x7f14780fc2f0 msgr2=0x7f14780fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.192+0000 7f147f400700 1 -- 172.21.15.116:0/2717810942 shutdown_connections 2024-03-30T01:11:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.192+0000 7f147f400700 1 -- 172.21.15.116:0/2717810942 wait complete. 2024-03-30T01:11:26.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.193+0000 7f147f400700 1 Processor -- start 2024-03-30T01:11:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.194+0000 7f147f400700 1 -- start start 2024-03-30T01:11:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.194+0000 7f147f400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1478100d80 0x7f1478074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.194+0000 7f147f400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1478101f70 0x7f14780731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.194+0000 7f147f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1478102910 0x7f14780736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.194+0000 7f147f400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1478073e80 con 0x7f1478102910 2024-03-30T01:11:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.194+0000 7f147f400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1478073ff0 con 0x7f1478101f70 2024-03-30T01:11:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.194+0000 7f147f400700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781a0d90 con 0x7f1478100d80 2024-03-30T01:11:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.194+0000 7f147d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1478102910 0x7f14780736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.194+0000 7f147d400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1478102910 0x7f14780736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49294/0 (socket says 172.21.15.116:49294) 2024-03-30T01:11:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.194+0000 7f147d400700 1 -- 172.21.15.116:0/2493264881 learned_addr learned my addr 172.21.15.116:0/2493264881 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.194+0000 7f1477e00700 1 --2- 172.21.15.116:0/2493264881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1478101f70 0x7f14780731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.194+0000 7f147ca00700 1 --2- 172.21.15.116:0/2493264881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1478100d80 0x7f1478074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.195+0000 7f147d400700 1 -- 172.21.15.116:0/2493264881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1478100d80 msgr2=0x7f1478074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:26.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.195+0000 7f147d400700 1 --2- 172.21.15.116:0/2493264881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1478100d80 0x7f1478074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.195+0000 7f147d400700 1 -- 172.21.15.116:0/2493264881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1478101f70 msgr2=0x7f14780731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:26.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.195+0000 7f147d400700 1 --2- 172.21.15.116:0/2493264881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1478101f70 0x7f14780731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.195+0000 7f147d400700 1 -- 172.21.15.116:0/2493264881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c0095f0 con 0x7f1478102910 2024-03-30T01:11:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.195+0000 7f147d400700 1 --2- 172.21.15.116:0/2493264881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1478102910 0x7f14780736e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1470007ee0 tx=0x7f147000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.195+0000 7f1475600700 1 -- 172.21.15.116:0/2493264881 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470017070 con 0x7f1478102910 2024-03-30T01:11:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.195+0000 7f1475600700 1 -- 172.21.15.116:0/2493264881 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f147000f040 con 0x7f1478102910 2024-03-30T01:11:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.195+0000 7f1475600700 1 -- 172.21.15.116:0/2493264881 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14700054b0 con 0x7f1478102910 2024-03-30T01:11:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.195+0000 7f147f400700 1 -- 172.21.15.116:0/2493264881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14781a0fc0 con 0x7f1478102910 2024-03-30T01:11:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.195+0000 7f147f400700 1 -- 172.21.15.116:0/2493264881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14781a1450 con 0x7f1478102910 2024-03-30T01:11:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.196+0000 7f147f400700 1 -- 172.21.15.116:0/2493264881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14781081a0 con 0x7f1478102910 2024-03-30T01:11:26.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.198+0000 7f1475600700 1 -- 172.21.15.116:0/2493264881 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f1470004ad0 con 0x7f1478102910 2024-03-30T01:11:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.200+0000 7f1475600700 1 --2- 172.21.15.116:0/2493264881 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14600a11a0 0x7f14600a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.200+0000 7f1475600700 1 -- 172.21.15.116:0/2493264881 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f14700cc260 con 0x7f1478102910 2024-03-30T01:11:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.200+0000 7f147ca00700 1 --2- 172.21.15.116:0/2493264881 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14600a11a0 0x7f14600a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:26.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.200+0000 7f147ca00700 1 --2- 172.21.15.116:0/2493264881 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14600a11a0 0x7f14600a3660 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f146c006010 tx=0x7f146c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:26.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.204+0000 7f1475600700 1 -- 172.21.15.116:0/2493264881 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1470095680 con 0x7f1478102910 2024-03-30T01:11:26.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.408+0000 7f147f400700 1 -- 172.21.15.116:0/2493264881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f147802cce0 con 0x7f1478102910 2024-03-30T01:11:26.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.409+0000 7f1475600700 1 -- 172.21.15.116:0/2493264881 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1470098ca0 con 0x7f1478102910 2024-03-30T01:11:26.414 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-30T01:11:26.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.411+0000 7f147f400700 1 -- 172.21.15.116:0/2493264881 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14600a11a0 msgr2=0x7f14600a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:26.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.411+0000 7f147f400700 1 --2- 172.21.15.116:0/2493264881 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14600a11a0 0x7f14600a3660 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f146c006010 tx=0x7f146c00b540 comp rx=0 tx=0).stop 2024-03-30T01:11:26.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.411+0000 7f147f400700 1 -- 172.21.15.116:0/2493264881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1478102910 msgr2=0x7f14780736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:26.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.411+0000 7f147f400700 1 --2- 172.21.15.116:0/2493264881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1478102910 0x7f14780736e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1470007ee0 tx=0x7f147000d3b0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.411+0000 7f147f400700 1 -- 172.21.15.116:0/2493264881 shutdown_connections 2024-03-30T01:11:26.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.411+0000 7f147f400700 1 --2- 172.21.15.116:0/2493264881 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f14600a11a0 0x7f14600a3660 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.411+0000 7f147f400700 1 --2- 172.21.15.116:0/2493264881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1478100d80 0x7f1478074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.411+0000 7f147f400700 1 --2- 172.21.15.116:0/2493264881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1478102910 0x7f14780736e0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.411+0000 7f147f400700 1 --2- 172.21.15.116:0/2493264881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1478101f70 0x7f14780731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.411+0000 7f147f400700 1 -- 172.21.15.116:0/2493264881 >> 172.21.15.116:0/2493264881 conn(0x7f14780fc2f0 msgr2=0x7f14781052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:26.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.411+0000 7f147f400700 1 -- 172.21.15.116:0/2493264881 shutdown_connections 2024-03-30T01:11:26.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.411+0000 7f147f400700 1 -- 172.21.15.116:0/2493264881 wait complete. 2024-03-30T01:11:26.436 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T01:11:26.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.627+0000 7fb6ce800700 1 -- 172.21.15.116:0/3076739097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8101f70 msgr2=0x7fb6c81023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:26.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.627+0000 7fb6ce800700 1 --2- 172.21.15.116:0/3076739097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8101f70 0x7fb6c81023d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40099c0 tx=0x7fb6b4009cd0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.627+0000 7fb6ce800700 1 -- 172.21.15.116:0/3076739097 shutdown_connections 2024-03-30T01:11:26.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.627+0000 7fb6ce800700 1 --2- 172.21.15.116:0/3076739097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6c8102910 0x7fb6c81091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.627+0000 7fb6ce800700 1 --2- 172.21.15.116:0/3076739097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6c8100d80 0x7fb6c81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.628+0000 7fb6ce800700 1 --2- 172.21.15.116:0/3076739097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8101f70 0x7fb6c81023d0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.628+0000 7fb6ce800700 1 -- 172.21.15.116:0/3076739097 >> 172.21.15.116:0/3076739097 conn(0x7fb6c80fc2f0 msgr2=0x7fb6c80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:26.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.628+0000 7fb6ce800700 1 -- 172.21.15.116:0/3076739097 shutdown_connections 2024-03-30T01:11:26.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.628+0000 7fb6ce800700 1 -- 172.21.15.116:0/3076739097 wait complete. 2024-03-30T01:11:26.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.629+0000 7fb6ce800700 1 Processor -- start 2024-03-30T01:11:26.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.629+0000 7fb6ce800700 1 -- start start 2024-03-30T01:11:26.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.629+0000 7fb6ce800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6c8100d80 0x7fb6c8074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:26.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.629+0000 7fb6ce800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8101f70 0x7fb6c80731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:26.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.629+0000 7fb6ce800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6c8102910 0x7fb6c80736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:26.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.629+0000 7fb6ce800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c8073e80 con 0x7fb6c8101f70 2024-03-30T01:11:26.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.629+0000 7fb6ce800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c8073ff0 con 0x7fb6c8100d80 2024-03-30T01:11:26.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.629+0000 7fb6ce800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c81a0d90 con 0x7fb6c8102910 2024-03-30T01:11:26.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.629+0000 7fb6c6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8101f70 0x7fb6c80731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:26.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.629+0000 7fb6c7400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6c8100d80 0x7fb6c8074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:26.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.629+0000 7fb6c6a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8101f70 0x7fb6c80731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49310/0 (socket says 172.21.15.116:49310) 2024-03-30T01:11:26.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.629+0000 7fb6c6a00700 1 -- 172.21.15.116:0/2550201152 learned_addr learned my addr 172.21.15.116:0/2550201152 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:26.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.629+0000 7fb6c7e00700 1 --2- 172.21.15.116:0/2550201152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6c8102910 0x7fb6c80736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:26.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.630+0000 7fb6c6a00700 1 -- 172.21.15.116:0/2550201152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6c8102910 msgr2=0x7fb6c80736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:26.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.630+0000 7fb6c6a00700 1 --2- 172.21.15.116:0/2550201152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6c8102910 0x7fb6c80736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.630+0000 7fb6c6a00700 1 -- 172.21.15.116:0/2550201152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6c8100d80 msgr2=0x7fb6c8074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:26.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.630+0000 7fb6c6a00700 1 --2- 172.21.15.116:0/2550201152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6c8100d80 0x7fb6c8074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.630+0000 7fb6c6a00700 1 -- 172.21.15.116:0/2550201152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b40095f0 con 0x7fb6c8101f70 2024-03-30T01:11:26.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.630+0000 7fb6c6a00700 1 --2- 172.21.15.116:0/2550201152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8101f70 0x7fb6c80731a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4006010 tx=0x7fb6b400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:26.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.630+0000 7fb6bfe00700 1 -- 172.21.15.116:0/2550201152 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b401d070 con 0x7fb6c8101f70 2024-03-30T01:11:26.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.630+0000 7fb6bfe00700 1 -- 172.21.15.116:0/2550201152 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6b4005030 con 0x7fb6c8101f70 2024-03-30T01:11:26.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.630+0000 7fb6ce800700 1 -- 172.21.15.116:0/2550201152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c81a0fc0 con 0x7fb6c8101f70 2024-03-30T01:11:26.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.630+0000 7fb6bfe00700 1 -- 172.21.15.116:0/2550201152 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4004b10 con 0x7fb6c8101f70 2024-03-30T01:11:26.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.630+0000 7fb6ce800700 1 -- 172.21.15.116:0/2550201152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c81a1380 con 0x7fb6c8101f70 2024-03-30T01:11:26.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.632+0000 7fb6ce800700 1 -- 172.21.15.116:0/2550201152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c8108200 con 0x7fb6c8101f70 2024-03-30T01:11:26.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.633+0000 7fb6bfe00700 1 -- 172.21.15.116:0/2550201152 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fb6b40051a0 con 0x7fb6c8101f70 2024-03-30T01:11:26.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.634+0000 7fb6bfe00700 1 --2- 172.21.15.116:0/2550201152 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb6ac0a11a0 0x7fb6ac0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:26.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.635+0000 7fb6bfe00700 1 -- 172.21.15.116:0/2550201152 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb6b40cd880 con 0x7fb6c8101f70 2024-03-30T01:11:26.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.635+0000 7fb6c7400700 1 --2- 172.21.15.116:0/2550201152 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb6ac0a11a0 0x7fb6ac0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:26.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.635+0000 7fb6c7400700 1 --2- 172.21.15.116:0/2550201152 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb6ac0a11a0 0x7fb6ac0a3660 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8101dd0 tx=0x7fb6b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:26.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.638+0000 7fb6bfe00700 1 -- 172.21.15.116:0/2550201152 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb6b4096d30 con 0x7fb6c8101f70 2024-03-30T01:11:26.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.845+0000 7fb6ce800700 1 -- 172.21.15.116:0/2550201152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb6c802cd00 con 0x7fb6c8101f70 2024-03-30T01:11:26.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.846+0000 7fb6bfe00700 1 -- 172.21.15.116:0/2550201152 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v83) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb6b409a350 con 0x7fb6c8101f70 2024-03-30T01:11:26.851 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-30T01:11:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.848+0000 7fb6ce800700 1 -- 172.21.15.116:0/2550201152 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb6ac0a11a0 msgr2=0x7fb6ac0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.848+0000 7fb6ce800700 1 --2- 172.21.15.116:0/2550201152 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb6ac0a11a0 0x7fb6ac0a3660 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8101dd0 tx=0x7fb6b8009260 comp rx=0 tx=0).stop 2024-03-30T01:11:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.848+0000 7fb6ce800700 1 -- 172.21.15.116:0/2550201152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8101f70 msgr2=0x7fb6c80731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.848+0000 7fb6ce800700 1 --2- 172.21.15.116:0/2550201152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8101f70 0x7fb6c80731a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4006010 tx=0x7fb6b400f690 comp rx=0 tx=0).stop 2024-03-30T01:11:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.848+0000 7fb6ce800700 1 -- 172.21.15.116:0/2550201152 shutdown_connections 2024-03-30T01:11:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.848+0000 7fb6ce800700 1 --2- 172.21.15.116:0/2550201152 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fb6ac0a11a0 0x7fb6ac0a3660 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.848+0000 7fb6ce800700 1 --2- 172.21.15.116:0/2550201152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6c8102910 0x7fb6c80736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.848+0000 7fb6ce800700 1 --2- 172.21.15.116:0/2550201152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6c8100d80 0x7fb6c8074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.848+0000 7fb6ce800700 1 --2- 172.21.15.116:0/2550201152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8101f70 0x7fb6c80731a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.848+0000 7fb6ce800700 1 -- 172.21.15.116:0/2550201152 >> 172.21.15.116:0/2550201152 conn(0x7fb6c80fc2f0 msgr2=0x7fb6c8105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.848+0000 7fb6ce800700 1 -- 172.21.15.116:0/2550201152 shutdown_connections 2024-03-30T01:11:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:26.848+0000 7fb6ce800700 1 -- 172.21.15.116:0/2550201152 wait complete. 2024-03-30T01:11:26.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:26 smithi116 ceph-mon[38086]: pgmap v698: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T01:11:26.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:26 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1331373799' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T01:11:26.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:26 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2493264881' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T01:11:26.872 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-30T01:11:27.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:26 smithi181 ceph-mon[47622]: pgmap v698: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T01:11:27.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:26 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1331373799' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T01:11:27.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:26 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2493264881' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T01:11:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.058+0000 7f387de00700 1 -- 172.21.15.116:0/3076455904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3878100d80 msgr2=0x7f38781011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.058+0000 7f387de00700 1 --2- 172.21.15.116:0/3076455904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3878100d80 0x7f38781011a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f38680099c0 tx=0x7f3868009cd0 comp rx=0 tx=0).stop 2024-03-30T01:11:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.058+0000 7f387de00700 1 -- 172.21.15.116:0/3076455904 shutdown_connections 2024-03-30T01:11:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.058+0000 7f387de00700 1 --2- 172.21.15.116:0/3076455904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3878102910 0x7f38781091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.058+0000 7f387de00700 1 --2- 172.21.15.116:0/3076455904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3878100d80 0x7f38781011a0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.058+0000 7f387de00700 1 --2- 172.21.15.116:0/3076455904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3878101f70 0x7f38781023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.058+0000 7f387de00700 1 -- 172.21.15.116:0/3076455904 >> 172.21.15.116:0/3076455904 conn(0x7f38780fc2f0 msgr2=0x7f38780fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.059+0000 7f387de00700 1 -- 172.21.15.116:0/3076455904 shutdown_connections 2024-03-30T01:11:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.059+0000 7f387de00700 1 -- 172.21.15.116:0/3076455904 wait complete. 2024-03-30T01:11:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.059+0000 7f387de00700 1 Processor -- start 2024-03-30T01:11:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.060+0000 7f387de00700 1 -- start start 2024-03-30T01:11:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.060+0000 7f387de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3878100d80 0x7f3878074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.060+0000 7f387de00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3878101f70 0x7f38780731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.060+0000 7f387de00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3878102910 0x7f38780736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.060+0000 7f387de00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3878073e80 con 0x7f3878100d80 2024-03-30T01:11:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.060+0000 7f387de00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3878073ff0 con 0x7f3878102910 2024-03-30T01:11:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.060+0000 7f387de00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38781a0dc0 con 0x7f3878101f70 2024-03-30T01:11:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.060+0000 7f3876a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3878100d80 0x7f3878074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.060+0000 7f3876a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3878100d80 0x7f3878074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49324/0 (socket says 172.21.15.116:49324) 2024-03-30T01:11:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.060+0000 7f3876a00700 1 -- 172.21.15.116:0/6290236 learned_addr learned my addr 172.21.15.116:0/6290236 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.060+0000 7f3876000700 1 --2- 172.21.15.116:0/6290236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3878101f70 0x7f38780731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.060+0000 7f3877400700 1 --2- 172.21.15.116:0/6290236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3878102910 0x7f38780736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.061+0000 7f3876a00700 1 -- 172.21.15.116:0/6290236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3878101f70 msgr2=0x7f38780731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.061+0000 7f3876a00700 1 --2- 172.21.15.116:0/6290236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3878101f70 0x7f38780731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.061+0000 7f3876a00700 1 -- 172.21.15.116:0/6290236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3878102910 msgr2=0x7f38780736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.061+0000 7f3876a00700 1 --2- 172.21.15.116:0/6290236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3878102910 0x7f38780736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.061+0000 7f3876a00700 1 -- 172.21.15.116:0/6290236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38680095f0 con 0x7f3878100d80 2024-03-30T01:11:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.061+0000 7f3876a00700 1 --2- 172.21.15.116:0/6290236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3878100d80 0x7f3878074b10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3868006010 tx=0x7f386800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.061+0000 7f387ca00700 1 -- 172.21.15.116:0/6290236 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386801d070 con 0x7f3878100d80 2024-03-30T01:11:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.061+0000 7f387ca00700 1 -- 172.21.15.116:0/6290236 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3868005020 con 0x7f3878100d80 2024-03-30T01:11:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.061+0000 7f387ca00700 1 -- 172.21.15.116:0/6290236 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3868004b10 con 0x7f3878100d80 2024-03-30T01:11:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.061+0000 7f387de00700 1 -- 172.21.15.116:0/6290236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38781a0ff0 con 0x7f3878100d80 2024-03-30T01:11:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.061+0000 7f387de00700 1 -- 172.21.15.116:0/6290236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38781a1360 con 0x7f3878100d80 2024-03-30T01:11:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.062+0000 7f387de00700 1 -- 172.21.15.116:0/6290236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3878108250 con 0x7f3878100d80 2024-03-30T01:11:27.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.064+0000 7f387ca00700 1 -- 172.21.15.116:0/6290236 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f3868005190 con 0x7f3878100d80 2024-03-30T01:11:27.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.066+0000 7f387ca00700 1 --2- 172.21.15.116:0/6290236 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f385c0a11a0 0x7f385c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:27.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.066+0000 7f387ca00700 1 -- 172.21.15.116:0/6290236 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f38680cd5f0 con 0x7f3878100d80 2024-03-30T01:11:27.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.066+0000 7f3876000700 1 --2- 172.21.15.116:0/6290236 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f385c0a11a0 0x7f385c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:27.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.067+0000 7f3876000700 1 --2- 172.21.15.116:0/6290236 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f385c0a11a0 0x7f385c0a3660 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3864009c60 tx=0x7f3864009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:27.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.070+0000 7f387ca00700 1 -- 172.21.15.116:0/6290236 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3868096aa0 con 0x7f3878100d80 2024-03-30T01:11:27.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:26 smithi130 ceph-mon[47626]: pgmap v698: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-30T01:11:27.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:26 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1331373799' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T01:11:27.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:26 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2493264881' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T01:11:27.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.332+0000 7f387de00700 1 -- 172.21.15.116:0/6290236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f387810c820 con 0x7f3878100d80 2024-03-30T01:11:27.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.332+0000 7f387ca00700 1 -- 172.21.15.116:0/6290236 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v83) v1 ==== 81+0+598 (secure 0 0 0) 0x7f386809a0c0 con 0x7f3878100d80 2024-03-30T01:11:27.340 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/309656254 2024-03-31T00:50:35.029431+0000 2024-03-30T01:11:27.340 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/131150228 2024-03-31T00:50:35.029431+0000 2024-03-30T01:11:27.340 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/755518827 2024-03-31T00:50:35.029431+0000 2024-03-30T01:11:27.341 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/3183509903 2024-03-31T00:50:11.610313+0000 2024-03-30T01:11:27.341 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/4184421214 2024-03-31T00:50:11.610313+0000 2024-03-30T01:11:27.341 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/3183509903 2024-03-31T00:50:11.610313+0000 2024-03-30T01:11:27.341 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/131150228 2024-03-31T00:50:35.029431+0000 2024-03-30T01:11:27.341 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2567186791 2024-03-31T00:50:35.029431+0000 2024-03-30T01:11:27.341 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2997035221 2024-03-31T00:50:11.610313+0000 2024-03-30T01:11:27.341 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2367387359 2024-03-31T00:50:11.610313+0000 2024-03-30T01:11:27.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.334+0000 7f387de00700 1 -- 172.21.15.116:0/6290236 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f385c0a11a0 msgr2=0x7f385c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:27.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.334+0000 7f387de00700 1 --2- 172.21.15.116:0/6290236 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f385c0a11a0 0x7f385c0a3660 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3864009c60 tx=0x7f3864009260 comp rx=0 tx=0).stop 2024-03-30T01:11:27.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.334+0000 7f387de00700 1 -- 172.21.15.116:0/6290236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3878100d80 msgr2=0x7f3878074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:27.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.334+0000 7f387de00700 1 --2- 172.21.15.116:0/6290236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3878100d80 0x7f3878074b10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3868006010 tx=0x7f386800f690 comp rx=0 tx=0).stop 2024-03-30T01:11:27.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.334+0000 7f387de00700 1 -- 172.21.15.116:0/6290236 shutdown_connections 2024-03-30T01:11:27.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.334+0000 7f387de00700 1 --2- 172.21.15.116:0/6290236 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f385c0a11a0 0x7f385c0a3660 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:27.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.334+0000 7f387de00700 1 --2- 172.21.15.116:0/6290236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3878102910 0x7f38780736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:27.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.335+0000 7f387de00700 1 --2- 172.21.15.116:0/6290236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3878100d80 0x7f3878074b10 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:27.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.335+0000 7f387de00700 1 --2- 172.21.15.116:0/6290236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3878101f70 0x7f38780731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:27.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.335+0000 7f387de00700 1 -- 172.21.15.116:0/6290236 >> 172.21.15.116:0/6290236 conn(0x7f38780fc2f0 msgr2=0x7f3878105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:27.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.335+0000 7f387de00700 1 -- 172.21.15.116:0/6290236 shutdown_connections 2024-03-30T01:11:27.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:27.335+0000 7f387de00700 1 -- 172.21.15.116:0/6290236 wait complete. 2024-03-30T01:11:27.343 INFO:teuthology.orchestra.run.smithi116.stderr:listed 10 entries 2024-03-30T01:11:27.359 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-30T01:11:27.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T01:11:27.359 DEBUG:teuthology.orchestra.run.smithi116:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-30T01:11:27.478 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2024-03-30T01:11:27.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T01:11:27.479 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-30T01:11:27.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T01:11:27.566 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-30T01:11:27.651 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:27 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2550201152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T01:11:27.651 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:27 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/6290236' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T01:11:27.662 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-30T01:11:27.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T01:11:27.662 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-30T01:11:27.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T01:11:27.766 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-03-30T01:11:27.870 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-30T01:11:27.870 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-30T01:11:27.871 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-30T01:11:27.916 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345668k,nr_inodes=4086417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=38,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/4c19cdc598f875d8e6011e30c9235bf22072d827ddd854fc7391c1b8bb55a6fc/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/PAVWPUWKF2UV4NHCYWEEAJYR3U:/var/lib/containers/storage/overlay/l/YBYQ4PO3RLHILESCFFKIVCE47F,upperdir=/var/lib/containers/storage/overlay/4c19cdc598f875d8e6011e30c9235bf22072d827ddd854fc7391c1b8bb55a6fc/diff,workdir=/var/lib/containers/storage/overlay/4c19cdc598f875d8e6011e30c9235bf22072d827ddd854fc7391c1b8bb55a6fc/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/8be876e03a8153073033a429a873f9dc83ad027e8261ea8f51c044803e2479af/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/PAVWPUWKF2UV4NHCYWEEAJYR3U:/var/lib/containers/storage/overlay/l/YBYQ4PO3RLHILESCFFKIVCE47F,upperdir=/var/lib/containers/storage/overlay/8be876e03a8153073033a429a873f9dc83ad027e8261ea8f51c044803e2479af/diff,workdir=/var/lib/containers/storage/overlay/8be876e03a8153073033a429a873f9dc83ad027e8261ea8f51c044803e2479af/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275388k,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 overlay /var/lib/containers/storage/overlay/f05170105b334f13edd088e294502601bf403d96b764a3c0399a5b2361479c1b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/PAVWPUWKF2UV4NHCYWEEAJYR3U:/var/lib/containers/storage/overlay/l/YBYQ4PO3RLHILESCFFKIVCE47F,upperdir=/var/lib/containers/storage/overlay/f05170105b334f13edd088e294502601bf403d96b764a3c0399a5b2361479c1b/diff,workdir=/var/lib/containers/storage/overlay/f05170105b334f13edd088e294502601bf403d96b764a3c0399a5b2361479c1b/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/f89c9c957d795f73410ca3ee52cac4c2b2678ef9e12f125d9747f83834512491/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/PAVWPUWKF2UV4NHCYWEEAJYR3U:/var/lib/containers/storage/overlay/l/YBYQ4PO3RLHILESCFFKIVCE47F,upperdir=/var/lib/containers/storage/overlay/f89c9c957d795f73410ca3ee52cac4c2b2678ef9e12f125d9747f83834512491/diff,workdir=/var/lib/containers/storage/overlay/f89c9c957d795f73410ca3ee52cac4c2b2678ef9e12f125d9747f83834512491/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/8f5d17af3880569c245ffb4ee8dd403c457638e59479dc717d902ffaed36947b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/PAVWPUWKF2UV4NHCYWEEAJYR3U:/var/lib/containers/storage/overlay/l/YBYQ4PO3RLHILESCFFKIVCE47F,upperdir=/var/lib/containers/storage/overlay/8f5d17af3880569c245ffb4ee8dd403c457638e59479dc717d902ffaed36947b/diff,workdir=/var/lib/containers/storage/overlay/8f5d17af3880569c245ffb4ee8dd403c457638e59479dc717d902ffaed36947b/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/5d829687e0e9e3033f07c81e3e7fe69f72dfb373e35366aa6f40ad39fea8b215/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/PAVWPUWKF2UV4NHCYWEEAJYR3U:/var/lib/containers/storage/overlay/l/YBYQ4PO3RLHILESCFFKIVCE47F,upperdir=/var/lib/containers/storage/overlay/5d829687e0e9e3033f07c81e3e7fe69f72dfb373e35366aa6f40ad39fea8b215/diff,workdir=/var/lib/containers/storage/overlay/5d829687e0e9e3033f07c81e3e7fe69f72dfb373e35366aa6f40ad39fea8b215/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/697eb98293cd26518d10390c725e6695c17d78db97b9cd6c6bafe6d5b64c855b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/PAVWPUWKF2UV4NHCYWEEAJYR3U:/var/lib/containers/storage/overlay/l/YBYQ4PO3RLHILESCFFKIVCE47F,upperdir=/var/lib/containers/storage/overlay/697eb98293cd26518d10390c725e6695c17d78db97b9cd6c6bafe6d5b64c855b/diff,workdir=/var/lib/containers/storage/overlay/697eb98293cd26518d10390c725e6695c17d78db97b9cd6c6bafe6d5b64c855b/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/c7a86dfc07859f7dbf2dce1c5af6878291ab4776da370bcf78322983ae382ba3/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/PAVWPUWKF2UV4NHCYWEEAJYR3U:/var/lib/containers/storage/overlay/l/YBYQ4PO3RLHILESCFFKIVCE47F,upperdir=/var/lib/containers/storage/overlay/c7a86dfc07859f7dbf2dce1c5af6878291ab4776da370bcf78322983ae382ba3/diff,workdir=/var/lib/containers/storage/overlay/c7a86dfc07859f7dbf2dce1c5af6878291ab4776da370bcf78322983ae382ba3/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d01c205ab930226a701177a82df30baf5f3aed40ab23a5df9b551f3fc31c7eb8/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/PAVWPUWKF2UV4NHCYWEEAJYR3U:/var/lib/containers/storage/overlay/l/YBYQ4PO3RLHILESCFFKIVCE47F,upperdir=/var/lib/containers/storage/overlay/d01c205ab930226a701177a82df30baf5f3aed40ab23a5df9b551f3fc31c7eb8/diff,workdir=/var/lib/containers/storage/overlay/d01c205ab930226a701177a82df30baf5f3aed40ab23a5df9b551f3fc31c7eb8/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/e43a3748d0c415bdf59616ecbae605b1700b61fb725c79f797aba380f03f2534/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/PAVWPUWKF2UV4NHCYWEEAJYR3U:/var/lib/containers/storage/overlay/l/YBYQ4PO3RLHILESCFFKIVCE47F,upperdir=/var/lib/containers/storage/overlay/e43a3748d0c415bdf59616ecbae605b1700b61fb725c79f797aba380f03f2534/diff,workdir=/var/lib/containers/storage/overlay/e43a3748d0c415bdf59616ecbae605b1700b61fb725c79f797aba380f03f2534/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 2024-03-30T01:11:27.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T01:11:27.916 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-30T01:11:27.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T01:11:27.999 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-03-30T01:11:28.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:27 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2550201152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T01:11:28.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:27 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/6290236' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T01:11:28.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:27 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2550201152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T01:11:28.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:27 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/6290236' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T01:11:28.111 INFO:teuthology.orchestra.run.smithi116.stderr:Cannot find device "ceph-brx" 2024-03-30T01:11:28.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T01:11:28.114 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-30T01:11:28.129 INFO:tasks.cephadm:Teardown begin 2024-03-30T01:11:28.130 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-30T01:11:28.130 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T01:11:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.320+0000 7f8845e00700 1 -- 172.21.15.116:0/1994123288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8840102810 msgr2=0x7f88401090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.320+0000 7f8845e00700 1 --2- 172.21.15.116:0/1994123288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8840102810 0x7f88401090c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8838009fc0 tx=0x7f883800b330 comp rx=0 tx=0).stop 2024-03-30T01:11:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.320+0000 7f8845e00700 1 -- 172.21.15.116:0/1994123288 shutdown_connections 2024-03-30T01:11:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.320+0000 7f8845e00700 1 --2- 172.21.15.116:0/1994123288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8840102810 0x7f88401090c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.320+0000 7f8845e00700 1 --2- 172.21.15.116:0/1994123288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8840100c80 0x7f88401010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.320+0000 7f8845e00700 1 --2- 172.21.15.116:0/1994123288 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8840101e70 0x7f88401022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.320+0000 7f8845e00700 1 -- 172.21.15.116:0/1994123288 >> 172.21.15.116:0/1994123288 conn(0x7f88400fc1f0 msgr2=0x7f88400fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.320+0000 7f8845e00700 1 -- 172.21.15.116:0/1994123288 shutdown_connections 2024-03-30T01:11:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.320+0000 7f8845e00700 1 -- 172.21.15.116:0/1994123288 wait complete. 2024-03-30T01:11:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.321+0000 7f8845e00700 1 Processor -- start 2024-03-30T01:11:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.321+0000 7f8845e00700 1 -- start start 2024-03-30T01:11:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.321+0000 7f8845e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8840100c80 0x7f8840074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.321+0000 7f8845e00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8840101e70 0x7f88400731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f8845e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8840102810 0x7f88400736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f8845e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8840073e80 con 0x7f8840102810 2024-03-30T01:11:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f8845e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8840073ff0 con 0x7f8840100c80 2024-03-30T01:11:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f8845e00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88401a0d80 con 0x7f8840101e70 2024-03-30T01:11:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f883f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8840102810 0x7f88400736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f883ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8840100c80 0x7f8840074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f883f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8840102810 0x7f88400736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49346/0 (socket says 172.21.15.116:49346) 2024-03-30T01:11:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f883f400700 1 -- 172.21.15.116:0/744300413 learned_addr learned my addr 172.21.15.116:0/744300413 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f883e000700 1 --2- 172.21.15.116:0/744300413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8840101e70 0x7f88400731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f883ea00700 1 -- 172.21.15.116:0/744300413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8840101e70 msgr2=0x7f88400731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f883ea00700 1 --2- 172.21.15.116:0/744300413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8840101e70 0x7f88400731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f883ea00700 1 -- 172.21.15.116:0/744300413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8840102810 msgr2=0x7f88400736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f883ea00700 1 --2- 172.21.15.116:0/744300413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8840102810 0x7f88400736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f883ea00700 1 -- 172.21.15.116:0/744300413 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8838009c40 con 0x7f8840100c80 2024-03-30T01:11:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.322+0000 7f883ea00700 1 --2- 172.21.15.116:0/744300413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8840100c80 0x7f8840074b10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8830005fd0 tx=0x7f883000eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.323+0000 7f8844a00700 1 -- 172.21.15.116:0/744300413 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8830004220 con 0x7f8840100c80 2024-03-30T01:11:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.323+0000 7f8845e00700 1 -- 172.21.15.116:0/744300413 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88401a0fb0 con 0x7f8840100c80 2024-03-30T01:11:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.323+0000 7f8844a00700 1 -- 172.21.15.116:0/744300413 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8830004380 con 0x7f8840100c80 2024-03-30T01:11:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.323+0000 7f8844a00700 1 -- 172.21.15.116:0/744300413 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8830003be0 con 0x7f8840100c80 2024-03-30T01:11:28.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.323+0000 7f8845e00700 1 -- 172.21.15.116:0/744300413 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88401a1480 con 0x7f8840100c80 2024-03-30T01:11:28.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.324+0000 7f8845e00700 1 -- 172.21.15.116:0/744300413 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88401080b0 con 0x7f8840100c80 2024-03-30T01:11:28.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.326+0000 7f8844a00700 1 -- 172.21.15.116:0/744300413 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f8830003d40 con 0x7f8840100c80 2024-03-30T01:11:28.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.327+0000 7f8844a00700 1 --2- 172.21.15.116:0/744300413 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f88240a11a0 0x7f88240a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:28.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.327+0000 7f8844a00700 1 -- 172.21.15.116:0/744300413 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8830014070 con 0x7f8840100c80 2024-03-30T01:11:28.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.327+0000 7f883e000700 1 --2- 172.21.15.116:0/744300413 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f88240a11a0 0x7f88240a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:28.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.328+0000 7f883e000700 1 --2- 172.21.15.116:0/744300413 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f88240a11a0 0x7f88240a3660 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f882c009c60 tx=0x7f882c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:28.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.329+0000 7f8844a00700 1 -- 172.21.15.116:0/744300413 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8830096290 con 0x7f8840100c80 2024-03-30T01:11:28.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.583+0000 7f8845e00700 1 -- 172.21.15.116:0/744300413 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f884010c4f0 con 0x7f8840100c80 2024-03-30T01:11:28.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.584+0000 7f8844a00700 1 -- 172.21.15.116:0/744300413 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+10896 (secure 0 0 0) 0x7f88300998b0 con 0x7f8840100c80 2024-03-30T01:11:28.590 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T01:11:28.590 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T01:05:49.619541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":140,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T01:11:28.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.587+0000 7f8845e00700 1 -- 172.21.15.116:0/744300413 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f88240a11a0 msgr2=0x7f88240a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.587+0000 7f8845e00700 1 --2- 172.21.15.116:0/744300413 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f88240a11a0 0x7f88240a3660 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f882c009c60 tx=0x7f882c009260 comp rx=0 tx=0).stop 2024-03-30T01:11:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.587+0000 7f8845e00700 1 -- 172.21.15.116:0/744300413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8840100c80 msgr2=0x7f8840074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.587+0000 7f8845e00700 1 --2- 172.21.15.116:0/744300413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8840100c80 0x7f8840074b10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8830005fd0 tx=0x7f883000eca0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.587+0000 7f8845e00700 1 -- 172.21.15.116:0/744300413 shutdown_connections 2024-03-30T01:11:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.587+0000 7f8845e00700 1 --2- 172.21.15.116:0/744300413 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f88240a11a0 0x7f88240a3660 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.587+0000 7f8845e00700 1 --2- 172.21.15.116:0/744300413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8840102810 0x7f88400736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.587+0000 7f8845e00700 1 --2- 172.21.15.116:0/744300413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8840100c80 0x7f8840074b10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.587+0000 7f8845e00700 1 --2- 172.21.15.116:0/744300413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8840101e70 0x7f88400731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.587+0000 7f8845e00700 1 -- 172.21.15.116:0/744300413 >> 172.21.15.116:0/744300413 conn(0x7f88400fc1f0 msgr2=0x7f88401051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.587+0000 7f8845e00700 1 -- 172.21.15.116:0/744300413 shutdown_connections 2024-03-30T01:11:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.587+0000 7f8845e00700 1 -- 172.21.15.116:0/744300413 wait complete. 2024-03-30T01:11:28.594 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 34 2024-03-30T01:11:28.611 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T01:11:28.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.798+0000 7fd3c1800700 1 -- 172.21.15.116:0/3371307451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc1016e0 msgr2=0x7fd3bc103ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:28.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.798+0000 7fd3c1800700 1 --2- 172.21.15.116:0/3371307451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc1016e0 0x7fd3bc103ad0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009990 tx=0x7fd3ac009ca0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.798+0000 7fd3c1800700 1 -- 172.21.15.116:0/3371307451 shutdown_connections 2024-03-30T01:11:28.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.798+0000 7fd3c1800700 1 --2- 172.21.15.116:0/3371307451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3bc106940 0x7fd3bc108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.798+0000 7fd3c1800700 1 --2- 172.21.15.116:0/3371307451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc1016e0 0x7fd3bc103ad0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.798+0000 7fd3c1800700 1 --2- 172.21.15.116:0/3371307451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3bc104010 0x7fd3bc106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.798+0000 7fd3c1800700 1 -- 172.21.15.116:0/3371307451 >> 172.21.15.116:0/3371307451 conn(0x7fd3bc0fb0d0 msgr2=0x7fd3bc0fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:28.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.798+0000 7fd3c1800700 1 -- 172.21.15.116:0/3371307451 shutdown_connections 2024-03-30T01:11:28.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.798+0000 7fd3c1800700 1 -- 172.21.15.116:0/3371307451 wait complete. 2024-03-30T01:11:28.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.799+0000 7fd3c1800700 1 Processor -- start 2024-03-30T01:11:28.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.799+0000 7fd3c1800700 1 -- start start 2024-03-30T01:11:28.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.799+0000 7fd3c1800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3bc1016e0 0x7fd3bc198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:28.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.799+0000 7fd3c1800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3bc104010 0x7fd3bc1992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:28.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.799+0000 7fd3c1800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc106940 0x7fd3bc1a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:28.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.799+0000 7fd3c1800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc1a0870 con 0x7fd3bc106940 2024-03-30T01:11:28.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.799+0000 7fd3c1800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc1a09e0 con 0x7fd3bc104010 2024-03-30T01:11:28.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.799+0000 7fd3c1800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc1a0ba0 con 0x7fd3bc1016e0 2024-03-30T01:11:28.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.800+0000 7fd3bb400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc106940 0x7fd3bc1a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:28.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.800+0000 7fd3ba000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3bc104010 0x7fd3bc1992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:28.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.800+0000 7fd3ba000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3bc104010 0x7fd3bc1992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.116:53904/0 (socket says 172.21.15.116:53904) 2024-03-30T01:11:28.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.800+0000 7fd3ba000700 1 -- 172.21.15.116:0/3145108408 learned_addr learned my addr 172.21.15.116:0/3145108408 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:28.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.800+0000 7fd3baa00700 1 --2- 172.21.15.116:0/3145108408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3bc1016e0 0x7fd3bc198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:28.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.800+0000 7fd3bb400700 1 -- 172.21.15.116:0/3145108408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3bc1016e0 msgr2=0x7fd3bc198d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:28.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.800+0000 7fd3bb400700 1 --2- 172.21.15.116:0/3145108408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3bc1016e0 0x7fd3bc198d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.800+0000 7fd3bb400700 1 -- 172.21.15.116:0/3145108408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3bc104010 msgr2=0x7fd3bc1992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:28.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.800+0000 7fd3bb400700 1 --2- 172.21.15.116:0/3145108408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3bc104010 0x7fd3bc1992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:28.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.800+0000 7fd3bb400700 1 -- 172.21.15.116:0/3145108408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac0095f0 con 0x7fd3bc106940 2024-03-30T01:11:28.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.800+0000 7fd3ba000700 1 --2- 172.21.15.116:0/3145108408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3bc104010 0x7fd3bc1992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:11:28.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.800+0000 7fd3bb400700 1 --2- 172.21.15.116:0/3145108408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc106940 0x7fd3bc1a02a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400efc0 tx=0x7fd3b400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:28.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.800+0000 7fd3b3400700 1 -- 172.21.15.116:0/3145108408 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4018070 con 0x7fd3bc106940 2024-03-30T01:11:28.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.801+0000 7fd3b3400700 1 -- 172.21.15.116:0/3145108408 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd3b400f040 con 0x7fd3bc106940 2024-03-30T01:11:28.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.801+0000 7fd3c1800700 1 -- 172.21.15.116:0/3145108408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3bc1a0e20 con 0x7fd3bc106940 2024-03-30T01:11:28.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.801+0000 7fd3b3400700 1 -- 172.21.15.116:0/3145108408 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b400cd40 con 0x7fd3bc106940 2024-03-30T01:11:28.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.801+0000 7fd3c1800700 1 -- 172.21.15.116:0/3145108408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3bc1a11f0 con 0x7fd3bc106940 2024-03-30T01:11:28.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.802+0000 7fd3c1800700 1 -- 172.21.15.116:0/3145108408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3bc108510 con 0x7fd3bc106940 2024-03-30T01:11:28.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.804+0000 7fd3b3400700 1 -- 172.21.15.116:0/3145108408 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fd3b4003a00 con 0x7fd3bc106940 2024-03-30T01:11:28.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.805+0000 7fd3b3400700 1 --2- 172.21.15.116:0/3145108408 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd3a00a1120 0x7fd3a00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:28.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.805+0000 7fd3b3400700 1 -- 172.21.15.116:0/3145108408 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd3b40cc770 con 0x7fd3bc106940 2024-03-30T01:11:28.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.805+0000 7fd3baa00700 1 --2- 172.21.15.116:0/3145108408 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd3a00a1120 0x7fd3a00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:28.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.806+0000 7fd3baa00700 1 --2- 172.21.15.116:0/3145108408 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd3a00a1120 0x7fd3a00a35e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac000c00 tx=0x7fd3ac005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:28.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:28.810+0000 7fd3b3400700 1 -- 172.21.15.116:0/3145108408 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd3b4095c80 con 0x7fd3bc106940 2024-03-30T01:11:28.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:28 smithi116 ceph-mon[38086]: pgmap v699: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T01:11:28.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:28 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/744300413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:11:29.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:28 smithi181 ceph-mon[47622]: pgmap v699: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T01:11:29.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:28 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/744300413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:11:29.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.061+0000 7fd3c1800700 1 -- 172.21.15.116:0/3145108408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3bc067a10 con 0x7fd3bc106940 2024-03-30T01:11:29.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.062+0000 7fd3b3400700 1 -- 172.21.15.116:0/3145108408 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+10896 (secure 0 0 0) 0x7fd3b40992a0 con 0x7fd3bc106940 2024-03-30T01:11:29.067 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T01:11:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3445796607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3445796607},{"type":"v1","addr":"172.21.15.116:6837","nonce":3445796607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14505,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1448499059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1448499059},{"type":"v1","addr":"172.21.15.116:6839","nonce":1448499059}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14517,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/2469944136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":2469944136},{"type":"v1","addr":"172.21.15.116:6841","nonce":2469944136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/1642031781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":1642031781},{"type":"v1","addr":"172.21.15.181:6833","nonce":1642031781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/459507942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":459507942},{"type":"v1","addr":"172.21.15.116:6835","nonce":459507942}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24488,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/2156314548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":2156314548},{"type":"v1","addr":"172.21.15.130:6833","nonce":2156314548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T00:55:51.790326+0000","modified":"2024-03-30T01:05:49.619541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24548,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/702133015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":702133015},{"type":"v1","addr":"172.21.15.181:6835","nonce":702133015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/941347789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":941347789},{"type":"v1","addr":"172.21.15.181:6837","nonce":941347789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24524":{"gid":24524,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/3346365474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3346365474},{"type":"v1","addr":"172.21.15.130:6835","nonce":3346365474}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":140,"addr":"172.21.15.181:6839/3114781386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3114781386},{"type":"v1","addr":"172.21.15.181:6839","nonce":3114781386}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24548":{"gid":24548,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6837/3389690261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3389690261},{"type":"v1","addr":"172.21.15.130:6837","nonce":3389690261}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/1184545085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1184545085},{"type":"v1","addr":"172.21.15.130:6839","nonce":1184545085}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T01:11:29.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.064+0000 7fd3c1800700 1 -- 172.21.15.116:0/3145108408 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd3a00a1120 msgr2=0x7fd3a00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:29.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.064+0000 7fd3c1800700 1 --2- 172.21.15.116:0/3145108408 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd3a00a1120 0x7fd3a00a35e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac000c00 tx=0x7fd3ac005fb0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.064+0000 7fd3c1800700 1 -- 172.21.15.116:0/3145108408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc106940 msgr2=0x7fd3bc1a02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:29.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.064+0000 7fd3c1800700 1 --2- 172.21.15.116:0/3145108408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc106940 0x7fd3bc1a02a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400efc0 tx=0x7fd3b400c3a0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.064+0000 7fd3c1800700 1 -- 172.21.15.116:0/3145108408 shutdown_connections 2024-03-30T01:11:29.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.064+0000 7fd3c1800700 1 --2- 172.21.15.116:0/3145108408 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fd3a00a1120 0x7fd3a00a35e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.064+0000 7fd3c1800700 1 --2- 172.21.15.116:0/3145108408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3bc1016e0 0x7fd3bc198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.064+0000 7fd3c1800700 1 --2- 172.21.15.116:0/3145108408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3bc106940 0x7fd3bc1a02a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.064+0000 7fd3c1800700 1 --2- 172.21.15.116:0/3145108408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3bc104010 0x7fd3bc1992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.064+0000 7fd3c1800700 1 -- 172.21.15.116:0/3145108408 >> 172.21.15.116:0/3145108408 conn(0x7fd3bc0fb0d0 msgr2=0x7fd3bc100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:29.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.064+0000 7fd3c1800700 1 -- 172.21.15.116:0/3145108408 shutdown_connections 2024-03-30T01:11:29.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.064+0000 7fd3c1800700 1 -- 172.21.15.116:0/3145108408 wait complete. 2024-03-30T01:11:29.071 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 34 2024-03-30T01:11:29.088 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T01:11:29.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:28 smithi130 ceph-mon[47626]: pgmap v699: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T01:11:29.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:28 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/744300413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:11:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.275+0000 7fa8adc00700 1 -- 172.21.15.116:0/1102291151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8a8101e70 msgr2=0x7fa8a81022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.275+0000 7fa8adc00700 1 --2- 172.21.15.116:0/1102291151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8a8101e70 0x7fa8a81022d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa894009990 tx=0x7fa894009ca0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.275+0000 7fa8adc00700 1 -- 172.21.15.116:0/1102291151 shutdown_connections 2024-03-30T01:11:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.275+0000 7fa8adc00700 1 --2- 172.21.15.116:0/1102291151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8a8102810 0x7fa8a81090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.275+0000 7fa8adc00700 1 --2- 172.21.15.116:0/1102291151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8a8100c80 0x7fa8a81010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.275+0000 7fa8adc00700 1 --2- 172.21.15.116:0/1102291151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8a8101e70 0x7fa8a81022d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.275+0000 7fa8adc00700 1 -- 172.21.15.116:0/1102291151 >> 172.21.15.116:0/1102291151 conn(0x7fa8a80fc1f0 msgr2=0x7fa8a80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.275+0000 7fa8adc00700 1 -- 172.21.15.116:0/1102291151 shutdown_connections 2024-03-30T01:11:29.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.275+0000 7fa8adc00700 1 -- 172.21.15.116:0/1102291151 wait complete. 2024-03-30T01:11:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.276+0000 7fa8adc00700 1 Processor -- start 2024-03-30T01:11:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.276+0000 7fa8adc00700 1 -- start start 2024-03-30T01:11:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8adc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8a8100c80 0x7fa8a8198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8adc00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8a8101e70 0x7fa8a8199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8adc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8a8102810 0x7fa8a81a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8adc00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a81a08f0 con 0x7fa8a8102810 2024-03-30T01:11:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8adc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a81a0a60 con 0x7fa8a8100c80 2024-03-30T01:11:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8adc00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a81a0c20 con 0x7fa8a8101e70 2024-03-30T01:11:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8a7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8a8102810 0x7fa8a81a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8a7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8a8102810 0x7fa8a81a0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49390/0 (socket says 172.21.15.116:49390) 2024-03-30T01:11:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8a7400700 1 -- 172.21.15.116:0/1606394652 learned_addr learned my addr 172.21.15.116:0/1606394652 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8a6a00700 1 --2- 172.21.15.116:0/1606394652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8a8100c80 0x7fa8a8198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8a6000700 1 --2- 172.21.15.116:0/1606394652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8a8101e70 0x7fa8a8199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8a7400700 1 -- 172.21.15.116:0/1606394652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8a8101e70 msgr2=0x7fa8a8199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8a7400700 1 --2- 172.21.15.116:0/1606394652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8a8101e70 0x7fa8a8199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8a7400700 1 -- 172.21.15.116:0/1606394652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8a8100c80 msgr2=0x7fa8a8198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8a7400700 1 --2- 172.21.15.116:0/1606394652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8a8100c80 0x7fa8a8198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.277+0000 7fa8a7400700 1 -- 172.21.15.116:0/1606394652 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8940095f0 con 0x7fa8a8102810 2024-03-30T01:11:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.278+0000 7fa8a7400700 1 --2- 172.21.15.116:0/1606394652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8a8102810 0x7fa8a81a0320 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000ef30 tx=0x7fa8a000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.278+0000 7fa89f400700 1 -- 172.21.15.116:0/1606394652 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a0018070 con 0x7fa8a8102810 2024-03-30T01:11:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.278+0000 7fa89f400700 1 -- 172.21.15.116:0/1606394652 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa8a000f040 con 0x7fa8a8102810 2024-03-30T01:11:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.278+0000 7fa89f400700 1 -- 172.21.15.116:0/1606394652 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a000cdb0 con 0x7fa8a8102810 2024-03-30T01:11:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.278+0000 7fa8adc00700 1 -- 172.21.15.116:0/1606394652 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a81a0ea0 con 0x7fa8a8102810 2024-03-30T01:11:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.278+0000 7fa8adc00700 1 -- 172.21.15.116:0/1606394652 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a81a1270 con 0x7fa8a8102810 2024-03-30T01:11:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.279+0000 7fa8adc00700 1 -- 172.21.15.116:0/1606394652 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a8067a10 con 0x7fa8a8102810 2024-03-30T01:11:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.281+0000 7fa89f400700 1 -- 172.21.15.116:0/1606394652 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fa8a0003a00 con 0x7fa8a8102810 2024-03-30T01:11:29.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.282+0000 7fa89f400700 1 --2- 172.21.15.116:0/1606394652 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa88c0a11a0 0x7fa88c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:29.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.282+0000 7fa89f400700 1 -- 172.21.15.116:0/1606394652 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa8a00cc780 con 0x7fa8a8102810 2024-03-30T01:11:29.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.282+0000 7fa8a6a00700 1 --2- 172.21.15.116:0/1606394652 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa88c0a11a0 0x7fa88c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:29.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.283+0000 7fa8a6a00700 1 --2- 172.21.15.116:0/1606394652 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa88c0a11a0 0x7fa88c0a3660 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa8a819a280 tx=0x7fa898009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.286+0000 7fa89f400700 1 -- 172.21.15.116:0/1606394652 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa8a0095c30 con 0x7fa8a8102810 2024-03-30T01:11:29.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.526+0000 7fa8adc00700 1 -- 172.21.15.116:0/1606394652 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8a810c490 con 0x7fa8a8102810 2024-03-30T01:11:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.528+0000 7fa89f400700 1 -- 172.21.15.116:0/1606394652 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+24556 (secure 0 0 0) 0x7fa8a0099250 con 0x7fa8a8102810 2024-03-30T01:11:29.534 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-30T01:11:29.534 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"fsid":"4909f004-ee2f-11ee-b647-cb9ed24678a4","created":"2024-03-30T00:49:38.164101+0000","modified":"2024-03-30T00:56:36.456069+0000","last_up_change":"2024-03-30T00:55:16.185660+0000","last_in_change":"2024-03-30T00:55:00.474866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T00:52:29.602428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T00:55:50.220543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T00:55:51.274498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T00:55:54.301591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec18ae5d-60d8-48ae-85ab-06e1c3294a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6803","nonce":3901884018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6805","nonce":3901884018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6809","nonce":3901884018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3901884018},{"type":"v1","addr":"172.21.15.116:6807","nonce":3901884018}]},"public_addr":"172.21.15.116:6803/3901884018","cluster_addr":"172.21.15.116:6805/3901884018","heartbeat_back_addr":"172.21.15.116:6809/3901884018","heartbeat_front_addr":"172.21.15.116:6807/3901884018","state":["exists","up"]},{"osd":1,"uuid":"e0047501-7c23-42fe-9e48-a949d3523427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6801","nonce":3819551112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6803","nonce":3819551112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6807","nonce":3819551112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3819551112},{"type":"v1","addr":"172.21.15.130:6805","nonce":3819551112}]},"public_addr":"172.21.15.130:6801/3819551112","cluster_addr":"172.21.15.130:6803/3819551112","heartbeat_back_addr":"172.21.15.130:6807/3819551112","heartbeat_front_addr":"172.21.15.130:6805/3819551112","state":["exists","up"]},{"osd":2,"uuid":"d7679f78-2053-4395-952b-efbdabfc12fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6801","nonce":3331770060}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6803","nonce":3331770060}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6807","nonce":3331770060}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3331770060},{"type":"v1","addr":"172.21.15.181:6805","nonce":3331770060}]},"public_addr":"172.21.15.181:6801/3331770060","cluster_addr":"172.21.15.181:6803/3331770060","heartbeat_back_addr":"172.21.15.181:6807/3331770060","heartbeat_front_addr":"172.21.15.181:6805/3331770060","state":["exists","up"]},{"osd":3,"uuid":"ec0571fc-3fb7-436a-82e6-7544e2491baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6811","nonce":2931358072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6813","nonce":2931358072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6817","nonce":2931358072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2931358072},{"type":"v1","addr":"172.21.15.116:6815","nonce":2931358072}]},"public_addr":"172.21.15.116:6811/2931358072","cluster_addr":"172.21.15.116:6813/2931358072","heartbeat_back_addr":"172.21.15.116:6817/2931358072","heartbeat_front_addr":"172.21.15.116:6815/2931358072","state":["exists","up"]},{"osd":4,"uuid":"3e7952d5-9d10-42ba-bea0-93dda26eff04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6809","nonce":81879957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6811","nonce":81879957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6815","nonce":81879957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":81879957},{"type":"v1","addr":"172.21.15.130:6813","nonce":81879957}]},"public_addr":"172.21.15.130:6809/81879957","cluster_addr":"172.21.15.130:6811/81879957","heartbeat_back_addr":"172.21.15.130:6815/81879957","heartbeat_front_addr":"172.21.15.130:6813/81879957","state":["exists","up"]},{"osd":5,"uuid":"d94364f4-bdb6-4c7f-9ed0-293887f07a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6809","nonce":1074863027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6811","nonce":1074863027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6815","nonce":1074863027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":1074863027},{"type":"v1","addr":"172.21.15.181:6813","nonce":1074863027}]},"public_addr":"172.21.15.181:6809/1074863027","cluster_addr":"172.21.15.181:6811/1074863027","heartbeat_back_addr":"172.21.15.181:6815/1074863027","heartbeat_front_addr":"172.21.15.181:6813/1074863027","state":["exists","up"]},{"osd":6,"uuid":"9866aade-0505-4465-87bc-91fa24eba9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6819","nonce":1426239130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6821","nonce":1426239130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6825","nonce":1426239130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1426239130},{"type":"v1","addr":"172.21.15.116:6823","nonce":1426239130}]},"public_addr":"172.21.15.116:6819/1426239130","cluster_addr":"172.21.15.116:6821/1426239130","heartbeat_back_addr":"172.21.15.116:6825/1426239130","heartbeat_front_addr":"172.21.15.116:6823/1426239130","state":["exists","up"]},{"osd":7,"uuid":"11c9be8d-f9e3-4e20-8557-cb9153a894ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6817","nonce":2137401701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6819","nonce":2137401701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6823","nonce":2137401701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2137401701},{"type":"v1","addr":"172.21.15.130:6821","nonce":2137401701}]},"public_addr":"172.21.15.130:6817/2137401701","cluster_addr":"172.21.15.130:6819/2137401701","heartbeat_back_addr":"172.21.15.130:6823/2137401701","heartbeat_front_addr":"172.21.15.130:6821/2137401701","state":["exists","up"]},{"osd":8,"uuid":"78423339-0c27-4d28-bde2-440127f4730d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6817","nonce":3650979389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6819","nonce":3650979389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6823","nonce":3650979389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3650979389},{"type":"v1","addr":"172.21.15.181:6821","nonce":3650979389}]},"public_addr":"172.21.15.181:6817/3650979389","cluster_addr":"172.21.15.181:6819/3650979389","heartbeat_back_addr":"172.21.15.181:6823/3650979389","heartbeat_front_addr":"172.21.15.181:6821/3650979389","state":["exists","up"]},{"osd":9,"uuid":"ce69b948-3a4c-4e7e-905f-109698ca78e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6827","nonce":3758826178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6829","nonce":3758826178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6833","nonce":3758826178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3758826178},{"type":"v1","addr":"172.21.15.116:6831","nonce":3758826178}]},"public_addr":"172.21.15.116:6827/3758826178","cluster_addr":"172.21.15.116:6829/3758826178","heartbeat_back_addr":"172.21.15.116:6833/3758826178","heartbeat_front_addr":"172.21.15.116:6831/3758826178","state":["exists","up"]},{"osd":10,"uuid":"575cf370-93a6-4985-b490-b962f2870aa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6825","nonce":3569352262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6827","nonce":3569352262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6831","nonce":3569352262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3569352262},{"type":"v1","addr":"172.21.15.130:6829","nonce":3569352262}]},"public_addr":"172.21.15.130:6825/3569352262","cluster_addr":"172.21.15.130:6827/3569352262","heartbeat_back_addr":"172.21.15.130:6831/3569352262","heartbeat_front_addr":"172.21.15.130:6829/3569352262","state":["exists","up"]},{"osd":11,"uuid":"09ffa044-3256-41a3-98d1-ddb3ed32ecc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6825","nonce":2545451272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6827","nonce":2545451272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6831","nonce":2545451272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2545451272},{"type":"v1","addr":"172.21.15.181:6829","nonce":2545451272}]},"public_addr":"172.21.15.181:6825/2545451272","cluster_addr":"172.21.15.181:6827/2545451272","heartbeat_back_addr":"172.21.15.181:6831/2545451272","heartbeat_front_addr":"172.21.15.181:6829/2545451272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:51:52.234526+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:10.542408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:27.343526+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:52:48.316453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:06.166571+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:23.341031+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:53:43.636364+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:01.695602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:19.233462+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:39.876005+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:54:57.905248+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T00:55:14.698639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/309656254":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6801/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/755518827":"2024-03-31T00:50:35.029431+0000","172.21.15.116:6800/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/4184421214":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6801/3183509903":"2024-03-31T00:50:11.610313+0000","172.21.15.116:6800/131150228":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2567186791":"2024-03-31T00:50:35.029431+0000","172.21.15.116:0/2997035221":"2024-03-31T00:50:11.610313+0000","172.21.15.116:0/2367387359":"2024-03-31T00:50:11.610313+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T01:11:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.531+0000 7fa8adc00700 1 -- 172.21.15.116:0/1606394652 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa88c0a11a0 msgr2=0x7fa88c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.531+0000 7fa8adc00700 1 --2- 172.21.15.116:0/1606394652 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa88c0a11a0 0x7fa88c0a3660 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa8a819a280 tx=0x7fa898009260 comp rx=0 tx=0).stop 2024-03-30T01:11:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.531+0000 7fa8adc00700 1 -- 172.21.15.116:0/1606394652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8a8102810 msgr2=0x7fa8a81a0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.531+0000 7fa8adc00700 1 --2- 172.21.15.116:0/1606394652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8a8102810 0x7fa8a81a0320 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000ef30 tx=0x7fa8a000c3a0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.531+0000 7fa8adc00700 1 -- 172.21.15.116:0/1606394652 shutdown_connections 2024-03-30T01:11:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.531+0000 7fa8adc00700 1 --2- 172.21.15.116:0/1606394652 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa88c0a11a0 0x7fa88c0a3660 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.531+0000 7fa8adc00700 1 --2- 172.21.15.116:0/1606394652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8a8102810 0x7fa8a81a0320 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.531+0000 7fa8adc00700 1 --2- 172.21.15.116:0/1606394652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8a8100c80 0x7fa8a8198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.531+0000 7fa8adc00700 1 --2- 172.21.15.116:0/1606394652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa8a8101e70 0x7fa8a8199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.531+0000 7fa8adc00700 1 -- 172.21.15.116:0/1606394652 >> 172.21.15.116:0/1606394652 conn(0x7fa8a80fc1f0 msgr2=0x7fa8a8072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.531+0000 7fa8adc00700 1 -- 172.21.15.116:0/1606394652 shutdown_connections 2024-03-30T01:11:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.531+0000 7fa8adc00700 1 -- 172.21.15.116:0/1606394652 wait complete. 2024-03-30T01:11:29.556 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-30T01:11:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.748+0000 7f2fb1c00700 1 -- 172.21.15.116:0/2060045740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fac102910 msgr2=0x7f2fac1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.748+0000 7f2fb1c00700 1 --2- 172.21.15.116:0/2060045740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fac102910 0x7f2fac1091c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400b0d0 tx=0x7f2fa400b3e0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.748+0000 7f2fb1c00700 1 -- 172.21.15.116:0/2060045740 shutdown_connections 2024-03-30T01:11:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.748+0000 7f2fb1c00700 1 --2- 172.21.15.116:0/2060045740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fac100d80 0x7f2fac1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.748+0000 7f2fb1c00700 1 --2- 172.21.15.116:0/2060045740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fac102910 0x7f2fac1091c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.748+0000 7f2fb1c00700 1 --2- 172.21.15.116:0/2060045740 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2fac101f70 0x7f2fac1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.748+0000 7f2fb1c00700 1 -- 172.21.15.116:0/2060045740 >> 172.21.15.116:0/2060045740 conn(0x7f2fac0fc2f0 msgr2=0x7f2fac0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.748+0000 7f2fb1c00700 1 -- 172.21.15.116:0/2060045740 shutdown_connections 2024-03-30T01:11:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.748+0000 7f2fb1c00700 1 -- 172.21.15.116:0/2060045740 wait complete. 2024-03-30T01:11:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.749+0000 7f2fb1c00700 1 Processor -- start 2024-03-30T01:11:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.749+0000 7f2fb1c00700 1 -- start start 2024-03-30T01:11:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2fb1c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fac100d80 0x7f2fac199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2fb1c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fac101f70 0x7f2fac199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2fb1c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2fac102910 0x7f2fac1a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2fb1c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac1a0b60 con 0x7f2fac100d80 2024-03-30T01:11:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2fb1c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac1a0ca0 con 0x7f2fac102910 2024-03-30T01:11:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2fb1c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac1a0e30 con 0x7f2fac101f70 2024-03-30T01:11:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2faaa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fac100d80 0x7f2fac199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2faaa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fac100d80 0x7f2fac199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49406/0 (socket says 172.21.15.116:49406) 2024-03-30T01:11:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2faaa00700 1 -- 172.21.15.116:0/3287220095 learned_addr learned my addr 172.21.15.116:0/3287220095 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2faa000700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fac101f70 0x7f2fac199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2fab400700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2fac102910 0x7f2fac1a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2faaa00700 1 -- 172.21.15.116:0/3287220095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fac101f70 msgr2=0x7f2fac199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2faaa00700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fac101f70 0x7f2fac199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2faaa00700 1 -- 172.21.15.116:0/3287220095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2fac102910 msgr2=0x7f2fac1a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2faaa00700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2fac102910 0x7f2fac1a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2faaa00700 1 -- 172.21.15.116:0/3287220095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa4009d00 con 0x7f2fac100d80 2024-03-30T01:11:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.750+0000 7f2fab400700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2fac102910 0x7f2fac1a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T01:11:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.751+0000 7f2faaa00700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fac100d80 0x7f2fac199030 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00e970 tx=0x7f2f9c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.751+0000 7f2fa3400700 1 -- 172.21.15.116:0/3287220095 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c009790 con 0x7f2fac100d80 2024-03-30T01:11:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.751+0000 7f2fb1c00700 1 -- 172.21.15.116:0/3287220095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fac1a1110 con 0x7f2fac100d80 2024-03-30T01:11:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.751+0000 7f2fa3400700 1 -- 172.21.15.116:0/3287220095 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f9c00eec0 con 0x7f2fac100d80 2024-03-30T01:11:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.751+0000 7f2fa3400700 1 -- 172.21.15.116:0/3287220095 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c003cf0 con 0x7f2fac100d80 2024-03-30T01:11:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.751+0000 7f2fb1c00700 1 -- 172.21.15.116:0/3287220095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fac1a1670 con 0x7f2fac100d80 2024-03-30T01:11:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.752+0000 7f2fb1c00700 1 -- 172.21.15.116:0/3287220095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fac067af0 con 0x7f2fac100d80 2024-03-30T01:11:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.754+0000 7f2fa3400700 1 -- 172.21.15.116:0/3287220095 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f2f9c00ccd0 con 0x7f2fac100d80 2024-03-30T01:11:29.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.755+0000 7f2fa3400700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2f900a1120 0x7f2f900a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:29.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.755+0000 7f2fa3400700 1 -- 172.21.15.116:0/3287220095 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2f9c014070 con 0x7f2fac100d80 2024-03-30T01:11:29.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.755+0000 7f2faa000700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2f900a1120 0x7f2f900a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:29.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.756+0000 7f2faa000700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2f900a1120 0x7f2f900a35e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2f98009c60 tx=0x7f2f98009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:29.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:29.759+0000 7f2fa3400700 1 -- 172.21.15.116:0/3287220095 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2f9c0962e0 con 0x7f2fac100d80 2024-03-30T01:11:29.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:29 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3145108408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:11:29.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:29 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/1606394652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T01:11:30.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:29 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3145108408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:11:30.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:29 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/1606394652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T01:11:30.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.008+0000 7f2fb1c00700 1 -- 172.21.15.116:0/3287220095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2fac10c5f0 con 0x7f2fac100d80 2024-03-30T01:11:30.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.022+0000 7f2fa3400700 1 -- 172.21.15.116:0/3287220095 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v35) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2f9c099900 con 0x7f2fac100d80 2024-03-30T01:11:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.025+0000 7f2fb1c00700 1 -- 172.21.15.116:0/3287220095 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2f900a1120 msgr2=0x7f2f900a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.025+0000 7f2fb1c00700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2f900a1120 0x7f2f900a35e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2f98009c60 tx=0x7f2f98009260 comp rx=0 tx=0).stop 2024-03-30T01:11:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.025+0000 7f2fb1c00700 1 -- 172.21.15.116:0/3287220095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fac100d80 msgr2=0x7f2fac199030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.025+0000 7f2fb1c00700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fac100d80 0x7f2fac199030 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00e970 tx=0x7f2f9c00ec80 comp rx=0 tx=0).stop 2024-03-30T01:11:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.025+0000 7f2fb1c00700 1 -- 172.21.15.116:0/3287220095 shutdown_connections 2024-03-30T01:11:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.025+0000 7f2fb1c00700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f2f900a1120 0x7f2f900a35e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.025+0000 7f2fb1c00700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2fac102910 0x7f2fac1a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.025+0000 7f2fb1c00700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fac100d80 0x7f2fac199030 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:30.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.025+0000 7f2fb1c00700 1 --2- 172.21.15.116:0/3287220095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fac101f70 0x7f2fac199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:30.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.025+0000 7f2fb1c00700 1 -- 172.21.15.116:0/3287220095 >> 172.21.15.116:0/3287220095 conn(0x7f2fac0fc2f0 msgr2=0x7f2fac072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:30.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.025+0000 7f2fb1c00700 1 -- 172.21.15.116:0/3287220095 shutdown_connections 2024-03-30T01:11:30.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.025+0000 7f2fb1c00700 1 -- 172.21.15.116:0/3287220095 wait complete. 2024-03-30T01:11:30.032 INFO:teuthology.orchestra.run.smithi116.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-30T01:11:30.053 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-30T01:11:30.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:29 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3145108408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T01:11:30.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:29 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/1606394652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T01:11:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.245+0000 7f4666c00700 1 -- 172.21.15.116:0/4101989108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4660100d80 msgr2=0x7f46601011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.245+0000 7f4666c00700 1 --2- 172.21.15.116:0/4101989108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4660100d80 0x7f46601011a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f46500099c0 tx=0x7f4650009cd0 comp rx=0 tx=0).stop 2024-03-30T01:11:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.245+0000 7f4666c00700 1 -- 172.21.15.116:0/4101989108 shutdown_connections 2024-03-30T01:11:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.245+0000 7f4666c00700 1 --2- 172.21.15.116:0/4101989108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4660102910 0x7f46601091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.245+0000 7f4666c00700 1 --2- 172.21.15.116:0/4101989108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4660100d80 0x7f46601011a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.245+0000 7f4666c00700 1 --2- 172.21.15.116:0/4101989108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4660101f70 0x7f46601023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.245+0000 7f4666c00700 1 -- 172.21.15.116:0/4101989108 >> 172.21.15.116:0/4101989108 conn(0x7f46600fc2f0 msgr2=0x7f46600fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.245+0000 7f4666c00700 1 -- 172.21.15.116:0/4101989108 shutdown_connections 2024-03-30T01:11:30.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.245+0000 7f4666c00700 1 -- 172.21.15.116:0/4101989108 wait complete. 2024-03-30T01:11:30.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.246+0000 7f4666c00700 1 Processor -- start 2024-03-30T01:11:30.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.246+0000 7f4666c00700 1 -- start start 2024-03-30T01:11:30.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f4666c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4660100d80 0x7f4660198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:30.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f4666c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4660101f70 0x7f46601994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:30.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f4666c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4660102910 0x7f46601a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:30.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f4666c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46601a0a90 con 0x7f4660100d80 2024-03-30T01:11:30.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f4666c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46601a0bd0 con 0x7f4660102910 2024-03-30T01:11:30.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f4666c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46601a0d60 con 0x7f4660101f70 2024-03-30T01:11:30.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f465fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4660100d80 0x7f4660198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:30.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f465fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4660100d80 0x7f4660198fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49420/0 (socket says 172.21.15.116:49420) 2024-03-30T01:11:30.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f465fe00700 1 -- 172.21.15.116:0/695872713 learned_addr learned my addr 172.21.15.116:0/695872713 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:30.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f4664c00700 1 --2- 172.21.15.116:0/695872713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4660102910 0x7f46601a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:30.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f465f400700 1 --2- 172.21.15.116:0/695872713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4660101f70 0x7f46601994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:30.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f465fe00700 1 -- 172.21.15.116:0/695872713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4660101f70 msgr2=0x7f46601994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:30.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f465fe00700 1 --2- 172.21.15.116:0/695872713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4660101f70 0x7f46601994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:30.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f465fe00700 1 -- 172.21.15.116:0/695872713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4660102910 msgr2=0x7f46601a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:30.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f465fe00700 1 --2- 172.21.15.116:0/695872713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4660102910 0x7f46601a04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:30.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.247+0000 7f465fe00700 1 -- 172.21.15.116:0/695872713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46500095f0 con 0x7f4660100d80 2024-03-30T01:11:30.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.248+0000 7f465fe00700 1 --2- 172.21.15.116:0/695872713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4660100d80 0x7f4660198fb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4650006010 tx=0x7f465000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:30.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.248+0000 7f465cc00700 1 -- 172.21.15.116:0/695872713 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465001d070 con 0x7f4660100d80 2024-03-30T01:11:30.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.248+0000 7f465cc00700 1 -- 172.21.15.116:0/695872713 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4650005020 con 0x7f4660100d80 2024-03-30T01:11:30.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.248+0000 7f4666c00700 1 -- 172.21.15.116:0/695872713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46601a0fe0 con 0x7f4660100d80 2024-03-30T01:11:30.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.248+0000 7f465cc00700 1 -- 172.21.15.116:0/695872713 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4650004b10 con 0x7f4660100d80 2024-03-30T01:11:30.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.248+0000 7f4666c00700 1 -- 172.21.15.116:0/695872713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46601a1350 con 0x7f4660100d80 2024-03-30T01:11:30.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.249+0000 7f4666c00700 1 -- 172.21.15.116:0/695872713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4660067af0 con 0x7f4660100d80 2024-03-30T01:11:30.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.251+0000 7f465cc00700 1 -- 172.21.15.116:0/695872713 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7f4650005190 con 0x7f4660100d80 2024-03-30T01:11:30.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.252+0000 7f465cc00700 1 --2- 172.21.15.116:0/695872713 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46440a1120 0x7f46440a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:30.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.252+0000 7f465cc00700 1 -- 172.21.15.116:0/695872713 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f46500cd920 con 0x7f4660100d80 2024-03-30T01:11:30.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.252+0000 7f465f400700 1 --2- 172.21.15.116:0/695872713 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46440a1120 0x7f46440a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:30.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.253+0000 7f465f400700 1 --2- 172.21.15.116:0/695872713 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46440a1120 0x7f46440a35e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f464c009c60 tx=0x7f464c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:30.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.255+0000 7f465cc00700 1 -- 172.21.15.116:0/695872713 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4650096cd0 con 0x7f4660100d80 2024-03-30T01:11:30.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:30 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-a[38063]: 2024-03-30T01:11:30.010+0000 7f3669e00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T01:11:30.501 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-f[85084]: ignoring --setuser ceph since I am not root 2024-03-30T01:11:30.501 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-f[85084]: ignoring --setgroup ceph since I am not root 2024-03-30T01:11:30.502 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-mds[85107]: -- 172.21.15.181:0/2190514876 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e7a09821a0 con 0x55e7a0959000 2024-03-30T01:11:30.502 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-f[85084]: starting mds.f at 2024-03-30T01:11:30.502 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-i[85770]: ignoring --setuser ceph since I am not root 2024-03-30T01:11:30.502 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-i[85770]: ignoring --setgroup ceph since I am not root 2024-03-30T01:11:30.502 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-mds[85793]: -- 172.21.15.181:0/3249601090 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5618d97d41a0 con 0x5618d97aa800 2024-03-30T01:11:30.502 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-i[85770]: starting mds.i at 2024-03-30T01:11:30.502 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-c[85430]: ignoring --setuser ceph since I am not root 2024-03-30T01:11:30.503 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-c[85430]: ignoring --setgroup ceph since I am not root 2024-03-30T01:11:30.503 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-mds[85453]: -- 172.21.15.181:0/1508209363 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56491b68c1a0 con 0x56491b662c00 2024-03-30T01:11:30.503 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-c[85430]: starting mds.c at 2024-03-30T01:11:30.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:30.507+0000 7f4666c00700 1 -- 172.21.15.116:0/695872713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f466010c800 con 0x7f4660100d80 2024-03-30T01:11:30.594 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-b[84904]: ignoring --setuser ceph since I am not root 2024-03-30T01:11:30.595 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-b[84904]: ignoring --setgroup ceph since I am not root 2024-03-30T01:11:30.595 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-mds[84927]: -- 172.21.15.130:0/2156066475 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a6678041a0 con 0x55a6677dac00 2024-03-30T01:11:30.595 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-b[84904]: starting mds.b at 2024-03-30T01:11:30.595 INFO:journalctl@ceph.mds.e.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-e[84221]: ignoring --setuser ceph since I am not root 2024-03-30T01:11:30.595 INFO:journalctl@ceph.mds.e.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-e[84221]: ignoring --setgroup ceph since I am not root 2024-03-30T01:11:30.595 INFO:journalctl@ceph.mds.e.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-e[84221]: starting mds.e at 2024-03-30T01:11:30.595 INFO:journalctl@ceph.mds.h.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-h[84562]: ignoring --setuser ceph since I am not root 2024-03-30T01:11:30.596 INFO:journalctl@ceph.mds.h.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-h[84562]: ignoring --setgroup ceph since I am not root 2024-03-30T01:11:30.596 INFO:journalctl@ceph.mds.h.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-mds[84585]: -- 172.21.15.130:0/2578582324 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558b380821a0 con 0x558b38058800 2024-03-30T01:11:30.596 INFO:journalctl@ceph.mds.h.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-h[84562]: starting mds.h at 2024-03-30T01:11:30.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:30 smithi116 ceph-mon[38086]: pgmap v700: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T01:11:30.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:30 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3287220095' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-30T01:11:30.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:30 smithi116 ceph-mon[38086]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-30T01:11:30.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:30 smithi116 ceph-mon[38086]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T01:11:30.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:30 smithi116 ceph-mon[38086]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-03-30T01:11:30.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:30 smithi116 ceph-mon[38086]: osdmap e84: 12 total, 12 up, 12 in 2024-03-30T01:11:30.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:30 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3287220095' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-30T01:11:30.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:30 smithi116 ceph-mon[38086]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-30T01:11:30.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:30 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/695872713' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T01:11:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-mon[47622]: pgmap v700: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T01:11:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3287220095' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-30T01:11:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-mon[47622]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-30T01:11:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-mon[47622]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T01:11:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-mon[47622]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-03-30T01:11:31.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-mon[47622]: osdmap e84: 12 total, 12 up, 12 in 2024-03-30T01:11:31.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3287220095' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-30T01:11:31.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-mon[47622]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-30T01:11:31.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:30 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/695872713' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T01:11:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.025+0000 7f465cc00700 1 -- 172.21.15.116:0/695872713 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v36) v1 ==== 104+0+0 (secure 0 0 0) 0x7f465009a2f0 con 0x7f4660100d80 2024-03-30T01:11:31.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.028+0000 7f4666c00700 1 -- 172.21.15.116:0/695872713 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46440a1120 msgr2=0x7f46440a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:31.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.029+0000 7f4666c00700 1 --2- 172.21.15.116:0/695872713 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46440a1120 0x7f46440a35e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f464c009c60 tx=0x7f464c009260 comp rx=0 tx=0).stop 2024-03-30T01:11:31.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.029+0000 7f4666c00700 1 -- 172.21.15.116:0/695872713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4660100d80 msgr2=0x7f4660198fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:31.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.029+0000 7f4666c00700 1 --2- 172.21.15.116:0/695872713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4660100d80 0x7f4660198fb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4650006010 tx=0x7f465000f690 comp rx=0 tx=0).stop 2024-03-30T01:11:31.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.029+0000 7f4666c00700 1 -- 172.21.15.116:0/695872713 shutdown_connections 2024-03-30T01:11:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.029+0000 7f4666c00700 1 --2- 172.21.15.116:0/695872713 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7f46440a1120 0x7f46440a35e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.029+0000 7f4666c00700 1 --2- 172.21.15.116:0/695872713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4660102910 0x7f46601a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.029+0000 7f4666c00700 1 --2- 172.21.15.116:0/695872713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4660100d80 0x7f4660198fb0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.029+0000 7f4666c00700 1 --2- 172.21.15.116:0/695872713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4660101f70 0x7f46601994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.029+0000 7f4666c00700 1 -- 172.21.15.116:0/695872713 >> 172.21.15.116:0/695872713 conn(0x7f46600fc2f0 msgr2=0x7f4660107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.029+0000 7f4666c00700 1 -- 172.21.15.116:0/695872713 shutdown_connections 2024-03-30T01:11:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.029+0000 7f4666c00700 1 -- 172.21.15.116:0/695872713 wait complete. 2024-03-30T01:11:31.057 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-30T01:11:31.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-mon[47626]: pgmap v700: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-30T01:11:31.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3287220095' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-30T01:11:31.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-mon[47626]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-30T01:11:31.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-mon[47626]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T01:11:31.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-mon[47626]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-03-30T01:11:31.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-mon[47626]: osdmap e84: 12 total, 12 up, 12 in 2024-03-30T01:11:31.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3287220095' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-30T01:11:31.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-mon[47626]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-30T01:11:31.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:30 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/695872713' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T01:11:31.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.251+0000 7efc2ca00700 1 -- 172.21.15.116:0/2421435098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc28108c40 msgr2=0x7efc2810b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:31.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.251+0000 7efc2ca00700 1 --2- 172.21.15.116:0/2421435098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc28108c40 0x7efc2810b0d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7efc2000b0a0 tx=0x7efc2000b3b0 comp rx=0 tx=0).stop 2024-03-30T01:11:31.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.251+0000 7efc2ca00700 1 -- 172.21.15.116:0/2421435098 shutdown_connections 2024-03-30T01:11:31.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.251+0000 7efc2ca00700 1 --2- 172.21.15.116:0/2421435098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc28108c40 0x7efc2810b0d0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:31.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.251+0000 7efc2ca00700 1 --2- 172.21.15.116:0/2421435098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc28074df0 0x7efc28073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:31.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.251+0000 7efc2ca00700 1 --2- 172.21.15.116:0/2421435098 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc28073810 0x7efc28073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:31.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.251+0000 7efc2ca00700 1 -- 172.21.15.116:0/2421435098 >> 172.21.15.116:0/2421435098 conn(0x7efc280fc2f0 msgr2=0x7efc280fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:31.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.251+0000 7efc2ca00700 1 -- 172.21.15.116:0/2421435098 shutdown_connections 2024-03-30T01:11:31.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.251+0000 7efc2ca00700 1 -- 172.21.15.116:0/2421435098 wait complete. 2024-03-30T01:11:31.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.252+0000 7efc2ca00700 1 Processor -- start 2024-03-30T01:11:31.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.252+0000 7efc2ca00700 1 -- start start 2024-03-30T01:11:31.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.252+0000 7efc2ca00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc28073810 0x7efc28199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.252+0000 7efc2ca00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc28074df0 0x7efc281995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.252+0000 7efc2ca00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc28108c40 0x7efc281a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.252+0000 7efc2ca00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc281a0b50 con 0x7efc28073810 2024-03-30T01:11:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.252+0000 7efc2ca00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc281a0c90 con 0x7efc28108c40 2024-03-30T01:11:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.252+0000 7efc2ca00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc281a0e20 con 0x7efc28074df0 2024-03-30T01:11:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc25e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc28073810 0x7efc28199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc25e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc28073810 0x7efc28199070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49448/0 (socket says 172.21.15.116:49448) 2024-03-30T01:11:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc25e00700 1 -- 172.21.15.116:0/2382449723 learned_addr learned my addr 172.21.15.116:0/2382449723 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc25400700 1 --2- 172.21.15.116:0/2382449723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc28074df0 0x7efc281995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc26800700 1 --2- 172.21.15.116:0/2382449723 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc28108c40 0x7efc281a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc25e00700 1 -- 172.21.15.116:0/2382449723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc28074df0 msgr2=0x7efc281995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc25e00700 1 --2- 172.21.15.116:0/2382449723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc28074df0 0x7efc281995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc25e00700 1 -- 172.21.15.116:0/2382449723 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc28108c40 msgr2=0x7efc281a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc25e00700 1 --2- 172.21.15.116:0/2382449723 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc28108c40 0x7efc281a0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc25e00700 1 -- 172.21.15.116:0/2382449723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc20009d00 con 0x7efc28073810 2024-03-30T01:11:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc25e00700 1 --2- 172.21.15.116:0/2382449723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc28073810 0x7efc28199070 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7efc1800e9a0 tx=0x7efc1800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc1ea00700 1 -- 172.21.15.116:0/2382449723 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc18009790 con 0x7efc28073810 2024-03-30T01:11:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc2ca00700 1 -- 172.21.15.116:0/2382449723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc281a1100 con 0x7efc28073810 2024-03-30T01:11:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc1ea00700 1 -- 172.21.15.116:0/2382449723 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc18010430 con 0x7efc28073810 2024-03-30T01:11:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.253+0000 7efc1ea00700 1 -- 172.21.15.116:0/2382449723 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc18018680 con 0x7efc28073810 2024-03-30T01:11:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.254+0000 7efc2ca00700 1 -- 172.21.15.116:0/2382449723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc281a1610 con 0x7efc28073810 2024-03-30T01:11:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.254+0000 7efc2ca00700 1 -- 172.21.15.116:0/2382449723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc28107a00 con 0x7efc28073810 2024-03-30T01:11:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.256+0000 7efc1ea00700 1 -- 172.21.15.116:0/2382449723 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7efc180105a0 con 0x7efc28073810 2024-03-30T01:11:31.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.258+0000 7efc1ea00700 1 --2- 172.21.15.116:0/2382449723 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efc0c0a1120 0x7efc0c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:31.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.258+0000 7efc1ea00700 1 -- 172.21.15.116:0/2382449723 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9278+0+0 (secure 0 0 0) 0x7efc18014070 con 0x7efc28073810 2024-03-30T01:11:31.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.258+0000 7efc25400700 1 --2- 172.21.15.116:0/2382449723 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efc0c0a1120 0x7efc0c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:31.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.259+0000 7efc25400700 1 --2- 172.21.15.116:0/2382449723 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efc0c0a1120 0x7efc0c0a35e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7efc28074b10 tx=0x7efc14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:31.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.260+0000 7efc1ea00700 1 -- 172.21.15.116:0/2382449723 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efc180963f0 con 0x7efc28073810 2024-03-30T01:11:31.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:31.470+0000 7efc2ca00700 1 -- 172.21.15.116:0/2382449723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efc2810e9f0 con 0x7efc28073810 2024-03-30T01:11:31.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-30T01:11:31.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T01:11:31.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: Cluster is now healthy 2024-03-30T01:11:31.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.181:6834/3348657935,v1:172.21.15.181:6835/3348657935] up:boot 2024-03-30T01:11:31.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.181:6838/900580167,v1:172.21.15.181:6839/900580167] up:boot 2024-03-30T01:11:31.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.130:6835/487580367,v1:172.21.15.130:6838/487580367] up:boot 2024-03-30T01:11:31.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.181:6836/2307511449,v1:172.21.15.181:6837/2307511449] up:boot 2024-03-30T01:11:31.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.130:6836/772060216,v1:172.21.15.130:6839/772060216] up:boot 2024-03-30T01:11:31.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: mds.? [v2:172.21.15.130:6834/951765417,v1:172.21.15.130:6837/951765417] up:boot 2024-03-30T01:11:31.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/695872713' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-30T01:11:31.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: fsmap 12 up:standby 2024-03-30T01:11:31.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T01:11:31.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T01:11:31.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T01:11:31.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T01:11:31.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T01:11:31.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T01:11:31.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:31 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2382449723' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T01:11:32.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-30T01:11:32.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T01:11:32.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: Cluster is now healthy 2024-03-30T01:11:32.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.181:6834/3348657935,v1:172.21.15.181:6835/3348657935] up:boot 2024-03-30T01:11:32.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.181:6838/900580167,v1:172.21.15.181:6839/900580167] up:boot 2024-03-30T01:11:32.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.130:6835/487580367,v1:172.21.15.130:6838/487580367] up:boot 2024-03-30T01:11:32.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.181:6836/2307511449,v1:172.21.15.181:6837/2307511449] up:boot 2024-03-30T01:11:32.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.130:6836/772060216,v1:172.21.15.130:6839/772060216] up:boot 2024-03-30T01:11:32.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: mds.? [v2:172.21.15.130:6834/951765417,v1:172.21.15.130:6837/951765417] up:boot 2024-03-30T01:11:32.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/695872713' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-30T01:11:32.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: fsmap 12 up:standby 2024-03-30T01:11:32.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T01:11:32.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T01:11:32.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T01:11:32.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T01:11:32.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T01:11:32.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T01:11:32.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:31 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2382449723' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T01:11:32.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.034+0000 7efc1ea00700 1 -- 172.21.15.116:0/2382449723 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v85) v1 ==== 181+0+0 (secure 0 0 0) 0x7efc180260d0 con 0x7efc28073810 2024-03-30T01:11:32.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.038+0000 7efc2ca00700 1 -- 172.21.15.116:0/2382449723 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efc0c0a1120 msgr2=0x7efc0c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:32.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.038+0000 7efc2ca00700 1 --2- 172.21.15.116:0/2382449723 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efc0c0a1120 0x7efc0c0a35e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7efc28074b10 tx=0x7efc14009260 comp rx=0 tx=0).stop 2024-03-30T01:11:32.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.038+0000 7efc2ca00700 1 -- 172.21.15.116:0/2382449723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc28073810 msgr2=0x7efc28199070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:32.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.038+0000 7efc2ca00700 1 --2- 172.21.15.116:0/2382449723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc28073810 0x7efc28199070 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7efc1800e9a0 tx=0x7efc1800ecb0 comp rx=0 tx=0).stop 2024-03-30T01:11:32.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.039+0000 7efc2ca00700 1 -- 172.21.15.116:0/2382449723 shutdown_connections 2024-03-30T01:11:32.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.039+0000 7efc2ca00700 1 --2- 172.21.15.116:0/2382449723 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7efc0c0a1120 0x7efc0c0a35e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:32.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.039+0000 7efc2ca00700 1 --2- 172.21.15.116:0/2382449723 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc28108c40 0x7efc281a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:32.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.039+0000 7efc2ca00700 1 --2- 172.21.15.116:0/2382449723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc28073810 0x7efc28199070 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:32.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.039+0000 7efc2ca00700 1 --2- 172.21.15.116:0/2382449723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc28074df0 0x7efc281995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:32.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.039+0000 7efc2ca00700 1 -- 172.21.15.116:0/2382449723 >> 172.21.15.116:0/2382449723 conn(0x7efc280fc2f0 msgr2=0x7efc28104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:32.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.039+0000 7efc2ca00700 1 -- 172.21.15.116:0/2382449723 shutdown_connections 2024-03-30T01:11:32.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.039+0000 7efc2ca00700 1 -- 172.21.15.116:0/2382449723 wait complete. 2024-03-30T01:11:32.047 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' removed 2024-03-30T01:11:32.073 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-30T01:11:32.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-30T01:11:32.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T01:11:32.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: Cluster is now healthy 2024-03-30T01:11:32.095 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.181:6834/3348657935,v1:172.21.15.181:6835/3348657935] up:boot 2024-03-30T01:11:32.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.181:6838/900580167,v1:172.21.15.181:6839/900580167] up:boot 2024-03-30T01:11:32.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.130:6835/487580367,v1:172.21.15.130:6838/487580367] up:boot 2024-03-30T01:11:32.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.181:6836/2307511449,v1:172.21.15.181:6837/2307511449] up:boot 2024-03-30T01:11:32.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.130:6836/772060216,v1:172.21.15.130:6839/772060216] up:boot 2024-03-30T01:11:32.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: mds.? [v2:172.21.15.130:6834/951765417,v1:172.21.15.130:6837/951765417] up:boot 2024-03-30T01:11:32.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/695872713' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-30T01:11:32.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: fsmap 12 up:standby 2024-03-30T01:11:32.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T01:11:32.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T01:11:32.096 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T01:11:32.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T01:11:32.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T01:11:32.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: from='mgr.14152 172.21.15.116:0/3895701107' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T01:11:32.097 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:31 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2382449723' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T01:11:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.301+0000 7fa546c00700 1 -- 172.21.15.116:0/4020435943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa540100d80 msgr2=0x7fa5401011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.301+0000 7fa546c00700 1 --2- 172.21.15.116:0/4020435943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa540100d80 0x7fa5401011a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa530009990 tx=0x7fa530009ca0 comp rx=0 tx=0).stop 2024-03-30T01:11:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.301+0000 7fa546c00700 1 -- 172.21.15.116:0/4020435943 shutdown_connections 2024-03-30T01:11:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.301+0000 7fa546c00700 1 --2- 172.21.15.116:0/4020435943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa540100d80 0x7fa5401011a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.301+0000 7fa546c00700 1 --2- 172.21.15.116:0/4020435943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa540102910 0x7fa5401091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.301+0000 7fa546c00700 1 --2- 172.21.15.116:0/4020435943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa540101f70 0x7fa5401023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.301+0000 7fa546c00700 1 -- 172.21.15.116:0/4020435943 >> 172.21.15.116:0/4020435943 conn(0x7fa5400fc2f0 msgr2=0x7fa5400fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.301+0000 7fa546c00700 1 -- 172.21.15.116:0/4020435943 shutdown_connections 2024-03-30T01:11:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.301+0000 7fa546c00700 1 -- 172.21.15.116:0/4020435943 wait complete. 2024-03-30T01:11:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.302+0000 7fa546c00700 1 Processor -- start 2024-03-30T01:11:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa546c00700 1 -- start start 2024-03-30T01:11:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa546c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa540100d80 0x7fa540196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa546c00700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa540101f70 0x7fa5401972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa546c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa540102910 0x7fa54019e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa546c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54019e870 con 0x7fa540100d80 2024-03-30T01:11:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa546c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54019e9e0 con 0x7fa540102910 2024-03-30T01:11:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa546c00700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54019eba0 con 0x7fa540101f70 2024-03-30T01:11:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa53fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa540100d80 0x7fa540196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa53fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa540100d80 0x7fa540196d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49470/0 (socket says 172.21.15.116:49470) 2024-03-30T01:11:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa53fe00700 1 -- 172.21.15.116:0/3738613159 learned_addr learned my addr 172.21.15.116:0/3738613159 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa53f400700 1 --2- 172.21.15.116:0/3738613159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa540101f70 0x7fa5401972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa544c00700 1 --2- 172.21.15.116:0/3738613159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa540102910 0x7fa54019e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa53fe00700 1 -- 172.21.15.116:0/3738613159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa540101f70 msgr2=0x7fa5401972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa53fe00700 1 --2- 172.21.15.116:0/3738613159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa540101f70 0x7fa5401972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa53fe00700 1 -- 172.21.15.116:0/3738613159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa540102910 msgr2=0x7fa54019e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa53fe00700 1 --2- 172.21.15.116:0/3738613159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa540102910 0x7fa54019e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.303+0000 7fa53fe00700 1 -- 172.21.15.116:0/3738613159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5300095f0 con 0x7fa540100d80 2024-03-30T01:11:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.304+0000 7fa53fe00700 1 --2- 172.21.15.116:0/3738613159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa540100d80 0x7fa540196d80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa530000c00 tx=0x7fa530009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.304+0000 7fa53cc00700 1 -- 172.21.15.116:0/3738613159 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53001d070 con 0x7fa540100d80 2024-03-30T01:11:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.304+0000 7fa546c00700 1 -- 172.21.15.116:0/3738613159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54019ee20 con 0x7fa540100d80 2024-03-30T01:11:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.304+0000 7fa53cc00700 1 -- 172.21.15.116:0/3738613159 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa530021410 con 0x7fa540100d80 2024-03-30T01:11:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.304+0000 7fa53cc00700 1 -- 172.21.15.116:0/3738613159 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530017650 con 0x7fa540100d80 2024-03-30T01:11:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.304+0000 7fa546c00700 1 -- 172.21.15.116:0/3738613159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54019f1e0 con 0x7fa540100d80 2024-03-30T01:11:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.305+0000 7fa546c00700 1 -- 172.21.15.116:0/3738613159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa540108220 con 0x7fa540100d80 2024-03-30T01:11:32.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.307+0000 7fa53cc00700 1 -- 172.21.15.116:0/3738613159 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fa530021580 con 0x7fa540100d80 2024-03-30T01:11:32.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.308+0000 7fa53cc00700 1 --2- 172.21.15.116:0/3738613159 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5240a1120 0x7fa5240a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:32.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.308+0000 7fa53cc00700 1 -- 172.21.15.116:0/3738613159 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8782+0+0 (secure 0 0 0) 0x7fa5300cdda0 con 0x7fa540100d80 2024-03-30T01:11:32.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.309+0000 7fa53f400700 1 --2- 172.21.15.116:0/3738613159 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5240a1120 0x7fa5240a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:32.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.309+0000 7fa53f400700 1 --2- 172.21.15.116:0/3738613159 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5240a1120 0x7fa5240a35e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa52c009c60 tx=0x7fa52c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:32.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.313+0000 7fa53cc00700 1 -- 172.21.15.116:0/3738613159 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa530097340 con 0x7fa540100d80 2024-03-30T01:11:32.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:32.521+0000 7fa546c00700 1 -- 172.21.15.116:0/3738613159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa540067af0 con 0x7fa540100d80 2024-03-30T01:11:32.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:32 smithi116 ceph-mon[38086]: pgmap v702: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 818 B/s wr, 7 op/s 2024-03-30T01:11:32.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:32 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/2382449723' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T01:11:32.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:32 smithi116 ceph-mon[38086]: osdmap e85: 12 total, 12 up, 12 in 2024-03-30T01:11:32.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:32 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3738613159' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T01:11:33.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:32 smithi181 ceph-mon[47622]: pgmap v702: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 818 B/s wr, 7 op/s 2024-03-30T01:11:33.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:32 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/2382449723' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T01:11:33.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:32 smithi181 ceph-mon[47622]: osdmap e85: 12 total, 12 up, 12 in 2024-03-30T01:11:33.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:32 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3738613159' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T01:11:33.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.038+0000 7fa53cc00700 1 -- 172.21.15.116:0/3738613159 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v86) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa530005740 con 0x7fa540100d80 2024-03-30T01:11:33.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.045+0000 7fa546c00700 1 -- 172.21.15.116:0/3738613159 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5240a1120 msgr2=0x7fa5240a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:33.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.045+0000 7fa546c00700 1 --2- 172.21.15.116:0/3738613159 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5240a1120 0x7fa5240a35e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa52c009c60 tx=0x7fa52c009260 comp rx=0 tx=0).stop 2024-03-30T01:11:33.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.045+0000 7fa546c00700 1 -- 172.21.15.116:0/3738613159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa540100d80 msgr2=0x7fa540196d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:33.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.045+0000 7fa546c00700 1 --2- 172.21.15.116:0/3738613159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa540100d80 0x7fa540196d80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa530000c00 tx=0x7fa530009580 comp rx=0 tx=0).stop 2024-03-30T01:11:33.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.046+0000 7fa546c00700 1 -- 172.21.15.116:0/3738613159 shutdown_connections 2024-03-30T01:11:33.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.046+0000 7fa546c00700 1 --2- 172.21.15.116:0/3738613159 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fa5240a1120 0x7fa5240a35e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:33.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.046+0000 7fa546c00700 1 --2- 172.21.15.116:0/3738613159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa540102910 0x7fa54019e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:33.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.046+0000 7fa546c00700 1 --2- 172.21.15.116:0/3738613159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa540100d80 0x7fa540196d80 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:33.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.046+0000 7fa546c00700 1 --2- 172.21.15.116:0/3738613159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa540101f70 0x7fa5401972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:33.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.046+0000 7fa546c00700 1 -- 172.21.15.116:0/3738613159 >> 172.21.15.116:0/3738613159 conn(0x7fa5400fc2f0 msgr2=0x7fa540105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:33.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.046+0000 7fa546c00700 1 -- 172.21.15.116:0/3738613159 shutdown_connections 2024-03-30T01:11:33.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.046+0000 7fa546c00700 1 -- 172.21.15.116:0/3738613159 wait complete. 2024-03-30T01:11:33.053 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' removed 2024-03-30T01:11:33.081 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-30T01:11:33.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:32 smithi130 ceph-mon[47626]: pgmap v702: 129 pgs: 129 active+clean; 669 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 818 B/s wr, 7 op/s 2024-03-30T01:11:33.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:32 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/2382449723' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T01:11:33.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:32 smithi130 ceph-mon[47626]: osdmap e85: 12 total, 12 up, 12 in 2024-03-30T01:11:33.094 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:32 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3738613159' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T01:11:33.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.293+0000 7fc8ec800700 1 -- 172.21.15.116:0/3533167411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e4104b20 msgr2=0x7fc8e410b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:33.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.293+0000 7fc8ec800700 1 --2- 172.21.15.116:0/3533167411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e4104b20 0x7fc8e410b3d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc8e000b0d0 tx=0x7fc8e000b3e0 comp rx=0 tx=0).stop 2024-03-30T01:11:33.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.293+0000 7fc8ec800700 1 -- 172.21.15.116:0/3533167411 shutdown_connections 2024-03-30T01:11:33.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.293+0000 7fc8ec800700 1 --2- 172.21.15.116:0/3533167411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e4104b20 0x7fc8e410b3d0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:33.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.293+0000 7fc8ec800700 1 --2- 172.21.15.116:0/3533167411 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8e4102f90 0x7fc8e41033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:33.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.293+0000 7fc8ec800700 1 --2- 172.21.15.116:0/3533167411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8e4104180 0x7fc8e41045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:33.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.293+0000 7fc8ec800700 1 -- 172.21.15.116:0/3533167411 >> 172.21.15.116:0/3533167411 conn(0x7fc8e40fe500 msgr2=0x7fc8e4100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:33.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.293+0000 7fc8ec800700 1 -- 172.21.15.116:0/3533167411 shutdown_connections 2024-03-30T01:11:33.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.293+0000 7fc8ec800700 1 -- 172.21.15.116:0/3533167411 wait complete. 2024-03-30T01:11:33.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.294+0000 7fc8ec800700 1 Processor -- start 2024-03-30T01:11:33.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.294+0000 7fc8ec800700 1 -- start start 2024-03-30T01:11:33.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8ec800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e4102f90 0x7fc8e4199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:33.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8ec800700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8e4104180 0x7fc8e4199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:33.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8ec800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8e4104b20 0x7fc8e41a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:33.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8ec800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e41a0b60 con 0x7fc8e4102f90 2024-03-30T01:11:33.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8ec800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e41a0ca0 con 0x7fc8e4104b20 2024-03-30T01:11:33.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8ec800700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e41a0e30 con 0x7fc8e4104180 2024-03-30T01:11:33.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8e9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e4102f90 0x7fc8e4199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:33.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8e9400700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8e4104180 0x7fc8e4199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8e9e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e4102f90 0x7fc8e4199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49488/0 (socket says 172.21.15.116:49488) 2024-03-30T01:11:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8e9e00700 1 -- 172.21.15.116:0/3312949347 learned_addr learned my addr 172.21.15.116:0/3312949347 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-30T01:11:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8ea800700 1 --2- 172.21.15.116:0/3312949347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8e4104b20 0x7fc8e41a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8e9e00700 1 -- 172.21.15.116:0/3312949347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8e4104180 msgr2=0x7fc8e4199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8e9e00700 1 --2- 172.21.15.116:0/3312949347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8e4104180 0x7fc8e4199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8e9e00700 1 -- 172.21.15.116:0/3312949347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8e4104b20 msgr2=0x7fc8e41a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8e9e00700 1 --2- 172.21.15.116:0/3312949347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8e4104b20 0x7fc8e41a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8e9e00700 1 -- 172.21.15.116:0/3312949347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e0009d00 con 0x7fc8e4102f90 2024-03-30T01:11:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.295+0000 7fc8e9400700 1 --2- 172.21.15.116:0/3312949347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8e4104180 0x7fc8e4199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T01:11:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.296+0000 7fc8e9e00700 1 --2- 172.21.15.116:0/3312949347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e4102f90 0x7fc8e4199030 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800e970 tx=0x7fc8d800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.296+0000 7fc8dea00700 1 -- 172.21.15.116:0/3312949347 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8009790 con 0x7fc8e4102f90 2024-03-30T01:11:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.296+0000 7fc8dea00700 1 -- 172.21.15.116:0/3312949347 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8d800c7e0 con 0x7fc8e4102f90 2024-03-30T01:11:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.296+0000 7fc8ec800700 1 -- 172.21.15.116:0/3312949347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e41a1110 con 0x7fc8e4102f90 2024-03-30T01:11:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.296+0000 7fc8dea00700 1 -- 172.21.15.116:0/3312949347 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8003c70 con 0x7fc8e4102f90 2024-03-30T01:11:33.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.296+0000 7fc8ec800700 1 -- 172.21.15.116:0/3312949347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e41a1620 con 0x7fc8e4102f90 2024-03-30T01:11:33.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.297+0000 7fc8ec800700 1 -- 172.21.15.116:0/3312949347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e410a390 con 0x7fc8e4102f90 2024-03-30T01:11:33.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.299+0000 7fc8dea00700 1 -- 172.21.15.116:0/3312949347 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 135866+0+0 (secure 0 0 0) 0x7fc8d800cc50 con 0x7fc8e4102f90 2024-03-30T01:11:33.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.300+0000 7fc8dea00700 1 --2- 172.21.15.116:0/3312949347 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc8cc0a1120 0x7fc8cc0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T01:11:33.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.300+0000 7fc8dea00700 1 -- 172.21.15.116:0/3312949347 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8252+0+0 (secure 0 0 0) 0x7fc8d8014070 con 0x7fc8e4102f90 2024-03-30T01:11:33.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.300+0000 7fc8e9400700 1 --2- 172.21.15.116:0/3312949347 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc8cc0a1120 0x7fc8cc0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T01:11:33.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.301+0000 7fc8e9400700 1 --2- 172.21.15.116:0/3312949347 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc8cc0a1120 0x7fc8cc0a35e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4011490 tx=0x7fc8d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T01:11:33.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.304+0000 7fc8dea00700 1 -- 172.21.15.116:0/3312949347 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc8d8096230 con 0x7fc8e4102f90 2024-03-30T01:11:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:33.512+0000 7fc8ec800700 1 -- 172.21.15.116:0/3312949347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc8e410ea30 con 0x7fc8e4102f90 2024-03-30T01:11:34.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:34.047+0000 7fc8dea00700 1 -- 172.21.15.116:0/3312949347 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v87) v1 ==== 178+0+0 (secure 0 0 0) 0x7fc8d8003dd0 con 0x7fc8e4102f90 2024-03-30T01:11:34.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:34.053+0000 7fc8ec800700 1 -- 172.21.15.116:0/3312949347 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc8cc0a1120 msgr2=0x7fc8cc0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:34.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:34.053+0000 7fc8ec800700 1 --2- 172.21.15.116:0/3312949347 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc8cc0a1120 0x7fc8cc0a35e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4011490 tx=0x7fc8d4009260 comp rx=0 tx=0).stop 2024-03-30T01:11:34.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:34.053+0000 7fc8ec800700 1 -- 172.21.15.116:0/3312949347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e4102f90 msgr2=0x7fc8e4199030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T01:11:34.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:34.053+0000 7fc8ec800700 1 --2- 172.21.15.116:0/3312949347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e4102f90 0x7fc8e4199030 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800e970 tx=0x7fc8d800ec80 comp rx=0 tx=0).stop 2024-03-30T01:11:34.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:34.054+0000 7fc8ec800700 1 -- 172.21.15.116:0/3312949347 shutdown_connections 2024-03-30T01:11:34.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:34.054+0000 7fc8ec800700 1 --2- 172.21.15.116:0/3312949347 >> [v2:172.21.15.116:6800/3072243062,v1:172.21.15.116:6801/3072243062] conn(0x7fc8cc0a1120 0x7fc8cc0a35e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:34.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:34.054+0000 7fc8ec800700 1 --2- 172.21.15.116:0/3312949347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8e4104b20 0x7fc8e41a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:34.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:34.054+0000 7fc8ec800700 1 --2- 172.21.15.116:0/3312949347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8e4102f90 0x7fc8e4199030 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:34.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:34.054+0000 7fc8ec800700 1 --2- 172.21.15.116:0/3312949347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8e4104180 0x7fc8e4199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T01:11:34.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:34.056+0000 7fc8ec800700 1 -- 172.21.15.116:0/3312949347 >> 172.21.15.116:0/3312949347 conn(0x7fc8e40fe500 msgr2=0x7fc8e41074d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T01:11:34.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:34.056+0000 7fc8ec800700 1 -- 172.21.15.116:0/3312949347 shutdown_connections 2024-03-30T01:11:34.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-30T01:11:34.056+0000 7fc8ec800700 1 -- 172.21.15.116:0/3312949347 wait complete. 2024-03-30T01:11:34.063 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data_ec' removed 2024-03-30T01:11:34.102 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T01:11:34.202 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T01:11:34.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T01:11:34.298 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:34 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3738613159' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T01:11:34.298 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:34 smithi181 ceph-mon[47622]: osdmap e86: 12 total, 12 up, 12 in 2024-03-30T01:11:34.298 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:34 smithi181 ceph-mon[47622]: pgmap v705: 33 pgs: 33 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T01:11:34.298 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:34 smithi181 ceph-mon[47622]: from='client.? 172.21.15.116:0/3312949347' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T01:11:34.300 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-30T01:11:34.300 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T01:11:34.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:34 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3738613159' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T01:11:34.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:34 smithi130 ceph-mon[47626]: osdmap e86: 12 total, 12 up, 12 in 2024-03-30T01:11:34.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:34 smithi130 ceph-mon[47626]: pgmap v705: 33 pgs: 33 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T01:11:34.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:34 smithi130 ceph-mon[47626]: from='client.? 172.21.15.116:0/3312949347' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T01:11:34.352 DEBUG:teuthology.orchestra.run.smithi130:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T01:11:34.370 DEBUG:teuthology.orchestra.run.smithi181:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T01:11:34.387 INFO:tasks.cephadm:Stopping all daemons... 2024-03-30T01:11:34.388 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-30T01:11:34.388 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mon.a 2024-03-30T01:11:34.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:34 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3738613159' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T01:11:34.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:34 smithi116 ceph-mon[38086]: osdmap e86: 12 total, 12 up, 12 in 2024-03-30T01:11:34.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:34 smithi116 ceph-mon[38086]: pgmap v705: 33 pgs: 33 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T01:11:34.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:34 smithi116 ceph-mon[38086]: from='client.? 172.21.15.116:0/3312949347' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T01:11:34.767 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:34 smithi116 systemd[1]: Stopping Ceph mon.a for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:11:35.050 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:34 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-a[38063]: 2024-03-30T01:11:34.760+0000 7f3670c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:11:35.050 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:34 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-a[38063]: 2024-03-30T01:11:34.760+0000 7f3670c00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-30T01:11:35.423 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 30 01:11:35 smithi116 bash[116359]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-a 2024-03-30T01:11:35.539 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mon.a.service' 2024-03-30T01:11:35.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:11:35.695 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-30T01:11:35.695 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-30T01:11:35.695 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mon.b 2024-03-30T01:11:35.977 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:35 smithi130 systemd[1]: Stopping Ceph mon.b for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:11:36.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:35 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-b[47603]: 2024-03-30T01:11:35.970+0000 7f3433c6f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:11:36.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:35 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-b[47603]: 2024-03-30T01:11:35.970+0000 7f3433c6f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-30T01:11:36.344 INFO:journalctl@ceph.mon.b.smithi130.stdout:Mar 30 01:11:36 smithi130 bash[90202]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-b 2024-03-30T01:11:36.542 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mon.b.service' 2024-03-30T01:11:36.614 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:11:36.615 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-30T01:11:36.615 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-30T01:11:36.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mon.c 2024-03-30T01:11:37.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:36 smithi181 systemd[1]: Stopping Ceph mon.c for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:11:37.001 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:36 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-c[47599]: 2024-03-30T01:11:36.863+0000 7f96f9c0f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:11:37.002 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:36 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-c[47599]: 2024-03-30T01:11:36.863+0000 7f96f9c0f700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-30T01:11:37.501 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 30 01:11:37 smithi181 bash[91406]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mon-c 2024-03-30T01:11:37.640 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mon.c.service' 2024-03-30T01:11:37.694 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:11:37.694 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-30T01:11:37.695 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-30T01:11:37.695 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.x 2024-03-30T01:11:38.053 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 01:11:37 smithi116 systemd[1]: Stopping Ceph mgr.x for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:11:38.318 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 01:11:38 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T01:11:38.046+0000 7fc7dbe00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:11:38.318 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 01:11:38 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x[38337]: 2024-03-30T01:11:38.046+0000 7fc7dbe00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-30T01:11:38.659 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 30 01:11:38 smithi116 bash[116580]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-x 2024-03-30T01:11:38.683 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.x.service' 2024-03-30T01:11:38.768 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:11:38.768 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-30T01:11:38.768 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-30T01:11:38.768 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.y 2024-03-30T01:11:39.094 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 01:11:38 smithi130 systemd[1]: Stopping Ceph mgr.y for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:11:39.493 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 01:11:39 smithi130 bash[90352]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-y 2024-03-30T01:11:39.493 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 30 01:11:39 smithi130 systemd[1]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-30T01:11:39.499 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.y.service' 2024-03-30T01:11:39.537 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:11:39.537 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-30T01:11:39.537 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-30T01:11:39.537 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.z 2024-03-30T01:11:39.969 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 01:11:39 smithi181 systemd[1]: Stopping Ceph mgr.z for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:11:40.252 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 01:11:40 smithi181 bash[91557]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mgr-z 2024-03-30T01:11:40.252 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 30 01:11:40 smithi181 systemd[1]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-30T01:11:40.340 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mgr.z.service' 2024-03-30T01:11:40.379 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:11:40.379 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-30T01:11:40.379 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-30T01:11:40.379 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.0 2024-03-30T01:11:40.735 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 01:11:40 smithi116 systemd[1]: Stopping Ceph osd.0 for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:11:41.173 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 01:11:40 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-0[55654]: 2024-03-30T01:11:40.729+0000 7f5eb0e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:11:41.174 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 01:11:40 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-0[55654]: 2024-03-30T01:11:40.729+0000 7f5eb0e00700 -1 osd.0 87 *** Got signal Terminated *** 2024-03-30T01:11:41.174 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 01:11:40 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-0[55654]: 2024-03-30T01:11:40.729+0000 7f5eb0e00700 -1 osd.0 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T01:11:46.673 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 30 01:11:46 smithi116 bash[116719]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-0 2024-03-30T01:11:47.886 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.0.service' 2024-03-30T01:11:47.971 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:11:47.972 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-30T01:11:47.972 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-30T01:11:47.972 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.1 2024-03-30T01:11:48.344 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 01:11:48 smithi130 systemd[1]: Stopping Ceph osd.1 for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:11:48.344 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 01:11:48 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-1[52636]: 2024-03-30T01:11:48.202+0000 7f501e163700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:11:48.344 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 01:11:48 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-1[52636]: 2024-03-30T01:11:48.202+0000 7f501e163700 -1 osd.1 87 *** Got signal Terminated *** 2024-03-30T01:11:48.344 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 01:11:48 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-1[52636]: 2024-03-30T01:11:48.202+0000 7f501e163700 -1 osd.1 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T01:11:54.094 INFO:journalctl@ceph.osd.1.smithi130.stdout:Mar 30 01:11:53 smithi130 bash[90493]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-1 2024-03-30T01:11:55.227 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.1.service' 2024-03-30T01:11:55.267 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:11:55.268 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-30T01:11:55.268 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-30T01:11:55.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.2 2024-03-30T01:11:55.751 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 01:11:55 smithi181 systemd[1]: Stopping Ceph osd.2 for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:11:55.751 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 01:11:55 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-2[53393]: 2024-03-30T01:11:55.497+0000 7f2f83b71700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:11:55.751 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 01:11:55 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-2[53393]: 2024-03-30T01:11:55.497+0000 7f2f83b71700 -1 osd.2 87 *** Got signal Terminated *** 2024-03-30T01:11:55.752 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 01:11:55 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-2[53393]: 2024-03-30T01:11:55.497+0000 7f2f83b71700 -1 osd.2 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T01:12:01.502 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 30 01:12:01 smithi181 bash[91701]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-2 2024-03-30T01:12:03.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.2.service' 2024-03-30T01:12:03.054 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:12:03.055 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-30T01:12:03.055 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-30T01:12:03.055 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.3 2024-03-30T01:12:03.423 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 30 01:12:03 smithi116 systemd[1]: Stopping Ceph osd.3 for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:12:03.923 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 30 01:12:03 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-3[63681]: 2024-03-30T01:12:03.416+0000 7fe1c5800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:12:03.923 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 30 01:12:03 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-3[63681]: 2024-03-30T01:12:03.416+0000 7fe1c5800700 -1 osd.3 87 *** Got signal Terminated *** 2024-03-30T01:12:03.923 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 30 01:12:03 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-3[63681]: 2024-03-30T01:12:03.416+0000 7fe1c5800700 -1 osd.3 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T01:12:09.424 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 30 01:12:09 smithi116 bash[117006]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-3 2024-03-30T01:12:09.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:09 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:09.034+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:10.173 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:10 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:10.047+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:10.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.3.service' 2024-03-30T01:12:10.751 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:10 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:10.292+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:10.771 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:12:10.771 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-30T01:12:10.771 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-30T01:12:10.772 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.4 2024-03-30T01:12:10.802 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:10 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:10.513+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:11.094 INFO:journalctl@ceph.osd.4.smithi130.stdout:Mar 30 01:12:10 smithi130 systemd[1]: Stopping Ceph osd.4 for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:12:11.094 INFO:journalctl@ceph.osd.4.smithi130.stdout:Mar 30 01:12:10 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-4[61134]: 2024-03-30T01:12:10.997+0000 7f44c8abd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:12:11.094 INFO:journalctl@ceph.osd.4.smithi130.stdout:Mar 30 01:12:10 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-4[61134]: 2024-03-30T01:12:10.997+0000 7f44c8abd700 -1 osd.4 87 *** Got signal Terminated *** 2024-03-30T01:12:11.094 INFO:journalctl@ceph.osd.4.smithi130.stdout:Mar 30 01:12:10 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-4[61134]: 2024-03-30T01:12:10.997+0000 7f44c8abd700 -1 osd.4 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T01:12:11.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:11 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:11.074+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:11.547 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:11 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:11.253+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:11.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:11 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:11.503+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:12.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:11 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:11.541+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:44.743847+0000 front 2024-03-30T01:11:44.743758+0000 (oldest deadline 2024-03-30T01:12:10.643807+0000) 2024-03-30T01:12:12.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:12 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:12.114+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:12.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:12 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:12.215+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:12.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:12 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:12.527+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:13.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:12 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:12.530+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:44.743847+0000 front 2024-03-30T01:11:44.743758+0000 (oldest deadline 2024-03-30T01:12:10.643807+0000) 2024-03-30T01:12:13.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:13 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:13.111+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:13.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:13 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:13.245+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:13.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:13 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:13.533+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:14.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:13 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:13.505+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:44.743847+0000 front 2024-03-30T01:11:44.743758+0000 (oldest deadline 2024-03-30T01:12:10.643807+0000) 2024-03-30T01:12:14.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:14 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:14.124+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:14.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:14 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:14.199+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:14.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:14 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:14.575+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:15.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:14 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:14.550+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:44.743847+0000 front 2024-03-30T01:11:44.743758+0000 (oldest deadline 2024-03-30T01:12:10.643807+0000) 2024-03-30T01:12:15.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:15.093+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:15.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:15 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:15.174+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:15.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:15 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:15.286+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:15.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:15 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:15.543+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:15.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:15 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:15.543+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:16.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:15 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:15.528+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:44.743847+0000 front 2024-03-30T01:11:44.743758+0000 (oldest deadline 2024-03-30T01:12:10.643807+0000) 2024-03-30T01:12:16.108 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:15 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:15.820+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:16.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:16 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:16.102+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:16.479 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:16 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:16.200+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:16.501 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:16 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:16.278+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:16.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:16 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:16.151+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:16.844 INFO:journalctl@ceph.osd.4.smithi130.stdout:Mar 30 01:12:16 smithi130 bash[90760]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-4 2024-03-30T01:12:16.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:16 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:16.555+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:16.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:16 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:16.555+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:17.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:16 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:16.524+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:44.743847+0000 front 2024-03-30T01:11:44.743758+0000 (oldest deadline 2024-03-30T01:12:10.643807+0000) 2024-03-30T01:12:17.133 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:16 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:16.805+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:17.344 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:17 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:17.190+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:17.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:17 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:17.127+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:17.501 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:17 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:17.324+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:17.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:17 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:17.169+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:17.674 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:17 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:17.505+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:17.674 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:17 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:17.505+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:17.681 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.4.service' 2024-03-30T01:12:17.764 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:12:17.764 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-30T01:12:17.764 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-30T01:12:17.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.5 2024-03-30T01:12:18.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:17 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:17.568+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:44.743847+0000 front 2024-03-30T01:11:44.743758+0000 (oldest deadline 2024-03-30T01:12:10.643807+0000) 2024-03-30T01:12:18.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:17 smithi181 systemd[1]: Stopping Ceph osd.5 for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:12:18.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:17 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:17.844+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:18.343 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:18 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:18.206+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:18.344 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:18 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:18.002+0000 7fe4ab42e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:12:18.344 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:18 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:18.002+0000 7fe4ab42e700 -1 osd.5 87 *** Got signal Terminated *** 2024-03-30T01:12:18.344 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:18 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:18.002+0000 7fe4ab42e700 -1 osd.5 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T01:12:18.486 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:18 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:18.179+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:18.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:18 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:18.173+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:18.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:18 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:18.173+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:18.751 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:18 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:18.338+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:18.752 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:18 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:18.544+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:44.743847+0000 front 2024-03-30T01:11:44.743758+0000 (oldest deadline 2024-03-30T01:12:10.643807+0000) 2024-03-30T01:12:18.752 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:18 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:18.544+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.944858+0000 front 2024-03-30T01:11:52.944733+0000 (oldest deadline 2024-03-30T01:12:17.644889+0000) 2024-03-30T01:12:18.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:18 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:18.481+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:18.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:18 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:18.481+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:19.159 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:18 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:18.809+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:19.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:19 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:19.153+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:19.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:19 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:19.153+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:19.516 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:19 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:19.202+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:19.543 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:19 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:19.359+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:19.543 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:19 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:19.246+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:19.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:19 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:19.510+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:19.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:19 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:19.510+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:20.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:19 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:19.536+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:44.743847+0000 front 2024-03-30T01:11:44.743758+0000 (oldest deadline 2024-03-30T01:12:10.643807+0000) 2024-03-30T01:12:20.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:19 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:19.536+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.944858+0000 front 2024-03-30T01:11:52.944733+0000 (oldest deadline 2024-03-30T01:12:17.644889+0000) 2024-03-30T01:12:20.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:19 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:19.852+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:20.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:20 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:20.375+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:20.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:20 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:20.241+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:20.594 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:20 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:20.228+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:20.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:20 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:20.477+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:20.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:20 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:20.477+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:20.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:20 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:20.190+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:20.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:20 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:20.190+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:21.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:20 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:20.529+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:44.743847+0000 front 2024-03-30T01:11:44.743758+0000 (oldest deadline 2024-03-30T01:12:10.643807+0000) 2024-03-30T01:12:21.002 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:20 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:20.529+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.944858+0000 front 2024-03-30T01:11:52.944733+0000 (oldest deadline 2024-03-30T01:12:17.644889+0000) 2024-03-30T01:12:21.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:20 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:20.823+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:21.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:21 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:21.230+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:21.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:21 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:21.406+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:21.502 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:21 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:21.234+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:21.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:21 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:21.231+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:21.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:21 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:21.231+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:21.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:21 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:21.495+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:21.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:21 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:21.495+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:22.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:21 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:21.540+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:44.743847+0000 front 2024-03-30T01:11:44.743758+0000 (oldest deadline 2024-03-30T01:12:10.643807+0000) 2024-03-30T01:12:22.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:21 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:21.540+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.944858+0000 front 2024-03-30T01:11:52.944733+0000 (oldest deadline 2024-03-30T01:12:17.644889+0000) 2024-03-30T01:12:22.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:21 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:21.824+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:22.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:21 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:21.824+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:55.233074+0000 front 2024-03-30T01:11:55.233119+0000 (oldest deadline 2024-03-30T01:12:21.132978+0000) 2024-03-30T01:12:22.501 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:22 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:22.366+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:22.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:22 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:22.227+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:22.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:22 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:22.227+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:22.551 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:22 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:22.236+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:22.551 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:22 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:22.236+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:22.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:22 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:22.203+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:22.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:22 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:22.203+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:22.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:22 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:22.545+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:22.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:22 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:22.545+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:23.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:22 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:22.540+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:44.743847+0000 front 2024-03-30T01:11:44.743758+0000 (oldest deadline 2024-03-30T01:12:10.643807+0000) 2024-03-30T01:12:23.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:22 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5[61840]: 2024-03-30T01:12:22.540+0000 7fe4a3a1c700 -1 osd.5 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.944858+0000 front 2024-03-30T01:11:52.944733+0000 (oldest deadline 2024-03-30T01:12:17.644889+0000) 2024-03-30T01:12:23.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:22 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:22.777+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:23.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:22 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:22.777+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:55.233074+0000 front 2024-03-30T01:11:55.233119+0000 (oldest deadline 2024-03-30T01:12:21.132978+0000) 2024-03-30T01:12:23.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:23 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:23.338+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:23.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:23 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:23.215+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:23.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:23 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:23.215+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:23.534 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:23 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:23.241+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:23.534 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:23 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:23.241+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:23.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:23 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:23.237+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:23.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:23 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:23.237+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:23.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:23 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:23.528+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:23.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:23 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:23.528+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:23.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:23 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:23.528+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:24.001 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 30 01:12:23 smithi181 bash[91956]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-5 2024-03-30T01:12:24.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:23 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:23.741+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:24.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:23 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:23.741+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:55.233074+0000 front 2024-03-30T01:11:55.233119+0000 (oldest deadline 2024-03-30T01:12:21.132978+0000) 2024-03-30T01:12:24.501 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:24 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:24.319+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:24.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:24 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:24.177+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:24.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:24 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:24.177+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:24.542 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:24 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:24.273+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:24.542 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:24 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:24.273+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:24.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:24 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:24.243+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:24.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:24 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:24.243+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:24.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:24 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:24.243+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:24.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:24 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:24.536+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:24.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:24 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:24.536+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:24.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:24 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:24.536+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:24.979 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.5.service' 2024-03-30T01:12:25.020 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:12:25.020 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-30T01:12:25.020 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-30T01:12:25.021 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.6 2024-03-30T01:12:25.086 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:24 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:24.692+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:25.087 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:24 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:24.692+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:55.233074+0000 front 2024-03-30T01:11:55.233119+0000 (oldest deadline 2024-03-30T01:12:21.132978+0000) 2024-03-30T01:12:25.251 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:25 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:25.175+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:25.251 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:25 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:25.175+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:25.387 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:25 smithi116 systemd[1]: Stopping Ceph osd.6 for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:12:25.387 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:25 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:25.271+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:25.387 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:25 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:25.271+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:25.387 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:25 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:25.271+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:25.573 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:25 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:25.311+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:25.574 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:25 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:25.311+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:25.663 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:25 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:25.381+0000 7f7465400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:12:25.663 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:25 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:25.381+0000 7f7465400700 -1 osd.6 87 *** Got signal Terminated *** 2024-03-30T01:12:25.663 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:25 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:25.381+0000 7f7465400700 -1 osd.6 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T01:12:25.751 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:25 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:25.367+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:25.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:25 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:25.568+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:25.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:25 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:25.568+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:25.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:25 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:25.568+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:25.924 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:25 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:25.658+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:25.924 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:25 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:25.658+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:55.233074+0000 front 2024-03-30T01:11:55.233119+0000 (oldest deadline 2024-03-30T01:12:21.132978+0000) 2024-03-30T01:12:26.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:26 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:26.398+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:26.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:26 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:26.398+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:26.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:26 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:26.188+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:26.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:26 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:26.188+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:26.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:26 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:26.313+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:26.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:26 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:26.313+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:26.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:26 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:26.313+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:26.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:26 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:26.358+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:26.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:26 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:26.358+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:26.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:26 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:26.586+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:26.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:26 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:26.586+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:26.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:26 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:26.586+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:27.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:26 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:26.682+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:27.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:26 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:26.682+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:55.233074+0000 front 2024-03-30T01:11:55.233119+0000 (oldest deadline 2024-03-30T01:12:21.132978+0000) 2024-03-30T01:12:27.501 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:27 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:27.416+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:27.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:27 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:27.416+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:27.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:27 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:27.234+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:27.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:27 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:27.234+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:27.620 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:27 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:27.354+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:27.620 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:27 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:27.354+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:27.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:27 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:27.358+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:27.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:27 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:27.358+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:27.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:27 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:27.358+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:28.094 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:27 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:27.614+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:28.094 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:27 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:27.614+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:28.094 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:27 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:27.614+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:28.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:27 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:27.704+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:28.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:27 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:27.704+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:55.233074+0000 front 2024-03-30T01:11:55.233119+0000 (oldest deadline 2024-03-30T01:12:21.132978+0000) 2024-03-30T01:12:28.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:28 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:28.316+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:28.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:28 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:28.316+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:28.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:28 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:28.316+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:28.751 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:28 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:28.369+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:28.751 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:28 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:28.369+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:28.752 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:28 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:28.267+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:28.752 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:28 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:28.267+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:28.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:28 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:28.355+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:28.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:28 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:28.355+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:28.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:28 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:28.585+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:28.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:28 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:28.585+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:28.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:28 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:28.585+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:29.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:28 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:28.727+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:29.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:28 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:28.727+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:55.233074+0000 front 2024-03-30T01:11:55.233119+0000 (oldest deadline 2024-03-30T01:12:21.132978+0000) 2024-03-30T01:12:29.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:29 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:29.304+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:29.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:29 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:29.304+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:29.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:29 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:29.304+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:29.751 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:29 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:29.331+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:29.752 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:29 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:29.331+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:29.752 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:29 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:29.268+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:29.752 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:29 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:29.268+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:29.752 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:29 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:29.268+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:29.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:29 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:29.362+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:29.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:29 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:29.362+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:29.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:29 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:29.545+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:29.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:29 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:29.545+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:29.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:29 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:29.545+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:30.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:29 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:29.753+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.932224+0000 front 2024-03-30T01:11:49.932274+0000 (oldest deadline 2024-03-30T01:12:15.232262+0000) 2024-03-30T01:12:30.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:29 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:29.753+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:55.233074+0000 front 2024-03-30T01:11:55.233119+0000 (oldest deadline 2024-03-30T01:12:21.132978+0000) 2024-03-30T01:12:30.173 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:29 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6[72190]: 2024-03-30T01:12:29.753+0000 7f745bc00700 -1 osd.6 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.035727+0000 front 2024-03-30T01:12:08.035666+0000 (oldest deadline 2024-03-30T01:12:29.735823+0000) 2024-03-30T01:12:30.501 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:30.292+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:30.501 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:30.292+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:30.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:30.233+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:30.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:30.233+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:30.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:30 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:30.233+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:30.594 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:30.338+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:30.594 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:30.338+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:30.594 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:30.338+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:05.457709+0000 front 2024-03-30T01:12:05.457734+0000 (oldest deadline 2024-03-30T01:12:30.058186+0000) 2024-03-30T01:12:30.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:30.516+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:30.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:30.516+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:30.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:30.516+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:30.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:30.516+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:30.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:30 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:30.289+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:30.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:30 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:30.289+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:30.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:30 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:30.289+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:31.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:31 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:31.280+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:31.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:31 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:31.280+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:31.503 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:31 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:31.200+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:31.503 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:31 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:31.200+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:31.503 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:31 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:31.200+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:31.594 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:31 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:31.330+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:31.594 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:31 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:31.330+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:31.594 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:31 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:31.330+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:05.457709+0000 front 2024-03-30T01:12:05.457734+0000 (oldest deadline 2024-03-30T01:12:30.058186+0000) 2024-03-30T01:12:31.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:31 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:31.544+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:31.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:31 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:31.544+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:31.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:31 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:31.544+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:31.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:31 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:31.544+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:31.673 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 30 01:12:31 smithi116 bash[117272]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-6 2024-03-30T01:12:31.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:31 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:31.246+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:31.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:31 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:31.247+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:31.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:31 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:31.247+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:32.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:32 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:32.292+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:32.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:32 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:32.292+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:32.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:32 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:32.292+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.192743+0000 front 2024-03-30T01:12:06.192653+0000 (oldest deadline 2024-03-30T01:12:32.092554+0000) 2024-03-30T01:12:32.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:32 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:32.243+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:32.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:32 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:32.243+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:32.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:32 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:32.243+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:32.546 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:32 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:32.199+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:32.546 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:32 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:32.199+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:32.547 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:32 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:32.199+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:32.723 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.6.service' 2024-03-30T01:12:32.809 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:12:32.809 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-30T01:12:32.809 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-30T01:12:32.809 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.7 2024-03-30T01:12:32.841 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:32 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:32.359+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:32.841 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:32 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:32.359+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:32.841 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:32 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:32.359+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:05.457709+0000 front 2024-03-30T01:12:05.457734+0000 (oldest deadline 2024-03-30T01:12:30.058186+0000) 2024-03-30T01:12:32.841 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:32 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:32.497+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:32.841 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:32 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:32.498+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:32.841 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:32 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:32.498+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:32.841 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:32 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:32.498+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:33.094 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:32 smithi130 systemd[1]: Stopping Ceph osd.7 for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:12:33.094 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:33 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:33.028+0000 7fd30f2ac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:12:33.095 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:33 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:33.028+0000 7fd30f2ac700 -1 osd.7 87 *** Got signal Terminated *** 2024-03-30T01:12:33.095 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:33 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:33.028+0000 7fd30f2ac700 -1 osd.7 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T01:12:33.501 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:33 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:33.274+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:33.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:33 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:33.274+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:33.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:33 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:33.274+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.192743+0000 front 2024-03-30T01:12:06.192653+0000 (oldest deadline 2024-03-30T01:12:32.092554+0000) 2024-03-30T01:12:33.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:33 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:33.207+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:33.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:33 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:33.207+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:33.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:33 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:33.207+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:33.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:33 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:33.180+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:33.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:33 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:33.180+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:33.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:33 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:33.180+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:33.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:33 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:33.180+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:33.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:33 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:33.380+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:33.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:33 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:33.380+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:33.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:33 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:33.380+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:05.457709+0000 front 2024-03-30T01:12:05.457734+0000 (oldest deadline 2024-03-30T01:12:30.058186+0000) 2024-03-30T01:12:33.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:33 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:33.466+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:33.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:33 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:33.466+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:33.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:33 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:33.466+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:33.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:33 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:33.466+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:34.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:34 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:34.134+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:34.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:34 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:34.134+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:34.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:34 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:34.134+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:34.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:34 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:34.134+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:34.501 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:34 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:34.313+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:34.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:34 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:34.313+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:34.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:34 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:34.313+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.192743+0000 front 2024-03-30T01:12:06.192653+0000 (oldest deadline 2024-03-30T01:12:32.092554+0000) 2024-03-30T01:12:34.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:34 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:34.217+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:34.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:34 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:34.217+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:34.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:34 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:34.217+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:34.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:34 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:34.400+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:34.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:34 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:34.400+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:34.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:34 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:34.400+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:05.457709+0000 front 2024-03-30T01:12:05.457734+0000 (oldest deadline 2024-03-30T01:12:30.058186+0000) 2024-03-30T01:12:34.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:34 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:34.458+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:34.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:34 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:34.458+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:34.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:34 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:34.458+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:34.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:34 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:34.459+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:35.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:35 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:35.293+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:35.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:35 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:35.293+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:35.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:35 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:35.293+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.192743+0000 front 2024-03-30T01:12:06.192653+0000 (oldest deadline 2024-03-30T01:12:32.092554+0000) 2024-03-30T01:12:35.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:35 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:35.238+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:35.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:35 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:35.238+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:35.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:35 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:35.238+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:35.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:35 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:35.182+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:35.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:35 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:35.182+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:35.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:35 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:35.182+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:35.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:35 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:35.182+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:35.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:35 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:35.366+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:35.845 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:35 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:35.366+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:35.845 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:35 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:35.366+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:05.457709+0000 front 2024-03-30T01:12:05.457734+0000 (oldest deadline 2024-03-30T01:12:30.058186+0000) 2024-03-30T01:12:35.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:35 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:35.464+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:35.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:35 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:35.464+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:35.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:35 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:35.464+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:35.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:35 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:35.464+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:36.501 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:36 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:36.305+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:36.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:36 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:36.305+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:36.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:36 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:36.305+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.192743+0000 front 2024-03-30T01:12:06.192653+0000 (oldest deadline 2024-03-30T01:12:32.092554+0000) 2024-03-30T01:12:36.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:36 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:36.229+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:36.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:36 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:36.229+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:36.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:36 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:36.229+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:36.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:36 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:36.205+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:36.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:36 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:36.205+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:36.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:36 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:36.205+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:36.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:36 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:36.205+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:36.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:36 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:36.379+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:36.845 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:36 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:36.379+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:36.845 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:36 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:36.379+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:05.457709+0000 front 2024-03-30T01:12:05.457734+0000 (oldest deadline 2024-03-30T01:12:30.058186+0000) 2024-03-30T01:12:36.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:36 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:36.461+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:36.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:36 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:36.461+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:36.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:36 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:36.461+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:36.846 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:36 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:36.461+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:37.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:37 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:37.159+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:37.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:37 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:37.159+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:37.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:37 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:37.159+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:37.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:37 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:37.159+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:37.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:37 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:37.159+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:37.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:37 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:37.221+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:37.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:37 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:37.221+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:37.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:37 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:37.221+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:37.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:37 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:37.221+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:37.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:37 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:37.316+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:37.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:37 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:37.316+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:37.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:37 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:37.316+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.192743+0000 front 2024-03-30T01:12:06.192653+0000 (oldest deadline 2024-03-30T01:12:32.092554+0000) 2024-03-30T01:12:37.845 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:37 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:37.417+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:49.754830+0000 front 2024-03-30T01:11:49.754865+0000 (oldest deadline 2024-03-30T01:12:15.654962+0000) 2024-03-30T01:12:37.845 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:37 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:37.417+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:56.155825+0000 front 2024-03-30T01:11:56.155895+0000 (oldest deadline 2024-03-30T01:12:22.055914+0000) 2024-03-30T01:12:37.845 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:37 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:37.417+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:05.457709+0000 front 2024-03-30T01:12:05.457734+0000 (oldest deadline 2024-03-30T01:12:30.058186+0000) 2024-03-30T01:12:37.846 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:37 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7[69876]: 2024-03-30T01:12:37.417+0000 7fd30b8a2700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.959133+0000 front 2024-03-30T01:12:15.959073+0000 (oldest deadline 2024-03-30T01:12:37.059206+0000) 2024-03-30T01:12:37.846 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:37 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:37.489+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:37.846 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:37 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:37.489+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:37.846 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:37 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:37.489+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:37.846 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:37 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:37.489+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:37.847 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:37 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:37.489+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:38.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:38 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:38.153+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:38.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:38 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:38.153+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:38.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:38 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:38.153+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:38.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:38 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:38.153+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:38.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:38 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:38.153+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:38.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:38 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:38.194+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:38.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:38 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:38.194+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:38.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:38 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:38.194+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:38.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:38 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:38.194+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:38.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:38 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:38.340+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:38.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:38 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:38.340+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:38.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:38 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:38.340+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.192743+0000 front 2024-03-30T01:12:06.192653+0000 (oldest deadline 2024-03-30T01:12:32.092554+0000) 2024-03-30T01:12:38.844 INFO:journalctl@ceph.osd.7.smithi130.stdout:Mar 30 01:12:38 smithi130 bash[91017]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-7 2024-03-30T01:12:38.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:38 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:38.461+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:38.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:38 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:38.461+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:38.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:38 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:38.461+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:38.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:38 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:38.461+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:38.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:38 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:38.461+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:39.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:39 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:39.108+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:39.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:39 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:39.108+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:39.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:39 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:39.108+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:39.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:39 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:39.108+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:39.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:39 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:39.108+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:39.501 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:39 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:39.380+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:39.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:39 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:39.380+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:39.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:39 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:39.380+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.192743+0000 front 2024-03-30T01:12:06.192653+0000 (oldest deadline 2024-03-30T01:12:32.092554+0000) 2024-03-30T01:12:39.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:39 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:39.380+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.995102+0000 front 2024-03-30T01:12:15.995133+0000 (oldest deadline 2024-03-30T01:12:38.895109+0000) 2024-03-30T01:12:39.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:39 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:39.152+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:39.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:39 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:39.152+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:39.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:39 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:39.152+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:39.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:39 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:39.152+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:39.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:39 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:39.419+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:39.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:39 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:39.419+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:39.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:39 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:39.419+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:39.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:39 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:39.419+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:39.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:39 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:39.419+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:39.861 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.7.service' 2024-03-30T01:12:39.943 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:12:39.944 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-30T01:12:39.944 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-30T01:12:39.944 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.8 2024-03-30T01:12:40.252 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:39 smithi181 systemd[1]: Stopping Ceph osd.8 for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:12:40.252 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:40 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:40.181+0000 7f32ef6da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:12:40.252 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:40 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:40.181+0000 7f32ef6da700 -1 osd.8 87 *** Got signal Terminated *** 2024-03-30T01:12:40.252 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:40 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:40.181+0000 7f32ef6da700 -1 osd.8 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T01:12:40.253 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:40 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:40.112+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:40.253 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:40 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:40.112+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:40.253 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:40 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:40.112+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:40.253 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:40 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:40.112+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:40.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:40 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:40.117+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:40.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:40 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:40.117+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:40.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:40 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:40.117+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:40.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:40 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:40.117+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:40.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:40 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:40.117+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:40.751 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:40 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:40.425+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:40.752 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:40 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:40.425+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:40.752 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:40 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:40.425+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.192743+0000 front 2024-03-30T01:12:06.192653+0000 (oldest deadline 2024-03-30T01:12:32.092554+0000) 2024-03-30T01:12:40.752 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:40 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:40.425+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.995102+0000 front 2024-03-30T01:12:15.995133+0000 (oldest deadline 2024-03-30T01:12:38.895109+0000) 2024-03-30T01:12:40.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:40 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:40.388+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:40.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:40 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:40.388+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:40.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:40 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:40.388+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:40.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:40 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:40.388+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:40.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:40 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:40.388+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:41.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:41 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:41.118+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:41.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:41 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:41.118+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:41.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:41 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:41.118+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:41.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:41 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:41.118+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:41.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:41 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:41.118+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:41.501 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:41 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:41.388+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:41.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:41 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:41.388+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:41.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:41 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:41.388+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.192743+0000 front 2024-03-30T01:12:06.192653+0000 (oldest deadline 2024-03-30T01:12:32.092554+0000) 2024-03-30T01:12:41.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:41 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:41.388+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.995102+0000 front 2024-03-30T01:12:15.995133+0000 (oldest deadline 2024-03-30T01:12:38.895109+0000) 2024-03-30T01:12:41.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:41 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:41.146+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:41.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:41 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:41.146+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:41.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:41 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:41.146+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:41.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:41 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:41.146+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:41.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:41 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:41.363+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:41.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:41 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:41.363+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:41.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:41 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:41.363+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:41.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:41 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:41.363+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:41.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:41 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:41.363+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:42.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:42 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:42.143+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:42.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:42 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:42.143+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:42.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:42 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:42.143+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:42.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:42 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:42.143+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:42.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:42 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:42.143+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:42.433 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:42 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:42.157+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:42.433 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:42 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:42.157+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:42.433 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:42 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:42.157+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:42.433 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:42 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:42.157+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:42.751 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:42 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:42.428+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:42.751 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:42 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:42.428+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:42.751 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:42 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:42.428+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.192743+0000 front 2024-03-30T01:12:06.192653+0000 (oldest deadline 2024-03-30T01:12:32.092554+0000) 2024-03-30T01:12:42.752 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:42 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:42.428+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.995102+0000 front 2024-03-30T01:12:15.995133+0000 (oldest deadline 2024-03-30T01:12:38.895109+0000) 2024-03-30T01:12:42.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:42 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:42.408+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:42.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:42 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:42.408+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:42.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:42 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:42.408+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:42.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:42 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:42.408+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:42.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:42 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:42.408+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:43.438 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:43 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:43.171+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:43.438 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:43 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:43.171+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:43.438 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:43 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:43.171+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:43.438 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:43 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:43.171+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:43.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:43 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:43.182+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:43.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:43 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:43.182+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:43.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:43 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:43.182+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:43.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:43 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:43.182+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:43.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:43 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:43.182+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:43.751 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:43 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:43.433+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:43.752 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:43 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:43.433+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:43.752 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:43 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:43.433+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.192743+0000 front 2024-03-30T01:12:06.192653+0000 (oldest deadline 2024-03-30T01:12:32.092554+0000) 2024-03-30T01:12:43.752 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:43 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:43.433+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.995102+0000 front 2024-03-30T01:12:15.995133+0000 (oldest deadline 2024-03-30T01:12:38.895109+0000) 2024-03-30T01:12:43.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:43 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:43.404+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:43.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:43 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:43.404+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:43.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:43 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:43.404+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:43.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:43 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:43.404+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:43.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:43 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:43.404+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:44.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:44 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:44.157+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:44.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:44 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:44.157+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:44.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:44 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:44.157+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:44.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:44 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:44.157+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:44.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:44 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:44.157+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:44.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:44 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:44.157+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:18.415691+0000 front 2024-03-30T01:12:18.415762+0000 (oldest deadline 2024-03-30T01:12:43.715802+0000) 2024-03-30T01:12:44.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:44 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:44.220+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:44.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:44 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:44.220+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:44.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:44 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:44.220+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:44.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:44 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:44.220+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:44.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:44 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:44.399+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.289642+0000 front 2024-03-30T01:11:52.289625+0000 (oldest deadline 2024-03-30T01:12:15.090362+0000) 2024-03-30T01:12:44.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:44 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:44.399+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:12:00.291789+0000 front 2024-03-30T01:12:00.291814+0000 (oldest deadline 2024-03-30T01:12:26.191974+0000) 2024-03-30T01:12:44.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:44 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:44.399+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.192743+0000 front 2024-03-30T01:12:06.192653+0000 (oldest deadline 2024-03-30T01:12:32.092554+0000) 2024-03-30T01:12:44.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:44 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:44.399+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.995102+0000 front 2024-03-30T01:12:15.995133+0000 (oldest deadline 2024-03-30T01:12:38.895109+0000) 2024-03-30T01:12:44.502 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:44 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8[69808]: 2024-03-30T01:12:44.399+0000 7f32ebcd0700 -1 osd.8 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:18.895625+0000 front 2024-03-30T01:12:18.895587+0000 (oldest deadline 2024-03-30T01:12:43.595581+0000) 2024-03-30T01:12:44.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:44 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:44.440+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:44.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:44 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:44.440+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:44.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:44 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:44.440+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:44.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:44 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:44.440+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:44.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:44 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:44.440+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:45.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:45 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:45.194+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:45.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:45 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:45.194+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:45.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:45 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:45.194+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:45.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:45 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:45.194+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:45.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:45 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:45.194+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:45.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:45 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:45.199+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:45.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:45 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:45.199+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:45.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:45 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:45.199+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:45.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:45 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:45.199+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:45.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:45 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:45.199+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:45.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:45 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:45.199+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:18.415691+0000 front 2024-03-30T01:12:18.415762+0000 (oldest deadline 2024-03-30T01:12:43.715802+0000) 2024-03-30T01:12:45.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:45 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:45.441+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:45.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:45 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:45.441+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:45.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:45 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:45.441+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:45.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:45 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:45.441+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:45.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:45 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:45.441+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:46.161 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 30 01:12:45 smithi181 bash[92222]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-8 2024-03-30T01:12:46.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:46 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:46.156+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:46.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:46 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:46.156+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:46.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:46 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:46.156+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:46.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:46 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:46.156+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:46.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:46 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:46.156+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:46.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:46 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:46.249+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:46.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:46 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:46.249+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:46.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:46 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:46.249+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:46.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:46 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:46.249+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:46.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:46 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:46.249+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:46.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:46 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:46.249+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:18.415691+0000 front 2024-03-30T01:12:18.415762+0000 (oldest deadline 2024-03-30T01:12:43.715802+0000) 2024-03-30T01:12:46.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:46 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:46.451+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:46.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:46 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:46.451+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:46.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:46 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:46.451+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:46.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:46 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:46.451+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:46.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:46 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:46.451+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:47.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.8.service' 2024-03-30T01:12:47.099 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:12:47.100 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-30T01:12:47.100 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-30T01:12:47.100 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.9 2024-03-30T01:12:47.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:47 smithi116 systemd[1]: Stopping Ceph osd.9 for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:12:47.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:47 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:47.264+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:47.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:47 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:47.264+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:47.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:47 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:47.264+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:47.423 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:47 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:47.264+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:47.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:47 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:47.264+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:47.424 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:47 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:47.264+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:18.415691+0000 front 2024-03-30T01:12:18.415762+0000 (oldest deadline 2024-03-30T01:12:43.715802+0000) 2024-03-30T01:12:47.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:47 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:47.190+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:47.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:47 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:47.190+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:47.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:47 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:47.190+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:47.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:47 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:47.190+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:47.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:47 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:47.190+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:47.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:47 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:47.406+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:47.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:47 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:47.406+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:47.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:47 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:47.406+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:47.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:47 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:47.406+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:47.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:47 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:47.406+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:47.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:47 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:47.406+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.853941+0000 front 2024-03-30T01:12:21.853782+0000 (oldest deadline 2024-03-30T01:12:47.153684+0000) 2024-03-30T01:12:47.923 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:47 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:47.461+0000 7fee6b600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:12:47.923 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:47 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:47.461+0000 7fee6b600700 -1 osd.9 87 *** Got signal Terminated *** 2024-03-30T01:12:47.923 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:47 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:47.461+0000 7fee6b600700 -1 osd.9 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T01:12:48.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:48 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:48.237+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:48.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:48 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:48.237+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:48.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:48 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:48.237+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:48.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:48 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:48.237+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:48.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:48 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:48.237+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:48.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:48 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:48.258+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:48.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:48 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:48.258+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:48.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:48 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:48.258+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:48.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:48 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:48.258+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:48.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:48 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:48.258+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:48.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:48 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:48.258+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:18.415691+0000 front 2024-03-30T01:12:18.415762+0000 (oldest deadline 2024-03-30T01:12:43.715802+0000) 2024-03-30T01:12:48.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:48 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:48.370+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:48.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:48 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:48.370+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:48.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:48 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:48.370+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:48.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:48 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:48.370+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:48.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:48 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:48.370+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:48.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:48 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:48.370+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.853941+0000 front 2024-03-30T01:12:21.853782+0000 (oldest deadline 2024-03-30T01:12:47.153684+0000) 2024-03-30T01:12:49.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:49 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:49.207+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:49.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:49 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:49.207+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:49.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:49 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:49.207+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:49.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:49 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:49.207+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:49.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:49 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:49.207+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:49.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:49 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:49.224+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:49.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:49 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:49.224+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:49.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:49 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:49.224+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:49.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:49 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:49.224+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:49.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:49 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:49.224+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:49.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:49 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:49.224+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:18.415691+0000 front 2024-03-30T01:12:18.415762+0000 (oldest deadline 2024-03-30T01:12:43.715802+0000) 2024-03-30T01:12:49.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:49 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:49.368+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:49.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:49 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:49.368+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:49.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:49 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:49.368+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:49.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:49 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:49.368+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:49.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:49 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:49.368+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:49.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:49 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:49.368+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.853941+0000 front 2024-03-30T01:12:21.853782+0000 (oldest deadline 2024-03-30T01:12:47.153684+0000) 2024-03-30T01:12:50.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:50 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:50.170+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:50.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:50 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:50.170+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:50.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:50 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:50.170+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:50.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:50 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:50.170+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:50.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:50 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:50.170+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:50.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:50 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:50.236+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:50.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:50 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:50.236+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:50.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:50 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:50.236+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:50.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:50 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:50.236+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:50.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:50 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:50.236+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:50.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:50 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:50.236+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:18.415691+0000 front 2024-03-30T01:12:18.415762+0000 (oldest deadline 2024-03-30T01:12:43.715802+0000) 2024-03-30T01:12:50.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:50 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:50.417+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:50.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:50 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:50.417+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:50.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:50 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:50.417+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:50.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:50 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:50.417+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:50.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:50 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:50.417+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:50.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:50 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:50.417+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.853941+0000 front 2024-03-30T01:12:21.853782+0000 (oldest deadline 2024-03-30T01:12:47.153684+0000) 2024-03-30T01:12:51.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:51 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:51.176+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:51.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:51 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:51.176+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:51.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:51 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:51.176+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:51.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:51 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:51.176+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:51.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:51 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:51.176+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:51.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:51 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:51.279+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:51.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:51 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:51.279+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:51.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:51 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:51.279+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:51.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:51 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:51.279+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:51.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:51 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:51.279+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:51.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:51 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:51.280+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:18.415691+0000 front 2024-03-30T01:12:18.415762+0000 (oldest deadline 2024-03-30T01:12:43.715802+0000) 2024-03-30T01:12:51.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:51 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:51.397+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:51.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:51 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:51.397+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:51.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:51 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:51.397+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:51.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:51 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:51.397+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:51.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:51 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:51.397+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:51.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:51 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:51.397+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.853941+0000 front 2024-03-30T01:12:21.853782+0000 (oldest deadline 2024-03-30T01:12:47.153684+0000) 2024-03-30T01:12:52.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:52 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:52.183+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:52.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:52 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:52.183+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:52.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:52 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:52.183+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:52.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:52 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:52.183+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:52.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:52 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:52.183+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:52.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:52 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:52.308+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:40.007336+0000 front 2024-03-30T01:11:40.007394+0000 (oldest deadline 2024-03-30T01:12:08.208186+0000) 2024-03-30T01:12:52.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:52 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:52.308+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:52.210406+0000 front 2024-03-30T01:11:52.210333+0000 (oldest deadline 2024-03-30T01:12:18.110566+0000) 2024-03-30T01:12:52.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:52 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:52.308+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:58.111213+0000 front 2024-03-30T01:11:58.111348+0000 (oldest deadline 2024-03-30T01:12:24.011356+0000) 2024-03-30T01:12:52.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:52 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:52.308+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:06.813168+0000 front 2024-03-30T01:12:06.813191+0000 (oldest deadline 2024-03-30T01:12:32.614036+0000) 2024-03-30T01:12:52.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:52 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:52.308+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:12.614780+0000 front 2024-03-30T01:12:12.614674+0000 (oldest deadline 2024-03-30T01:12:36.714669+0000) 2024-03-30T01:12:52.674 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:52 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9[80880]: 2024-03-30T01:12:52.308+0000 7fee66e00700 -1 osd.9 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:18.415691+0000 front 2024-03-30T01:12:18.415762+0000 (oldest deadline 2024-03-30T01:12:43.715802+0000) 2024-03-30T01:12:52.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:52 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:52.415+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:52.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:52 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:52.415+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:52.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:52 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:52.415+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:52.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:52 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:52.415+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:52.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:52 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:52.415+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:52.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:52 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:52.415+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.853941+0000 front 2024-03-30T01:12:21.853782+0000 (oldest deadline 2024-03-30T01:12:47.153684+0000) 2024-03-30T01:12:53.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:53 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:53.194+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:53.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:53 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:53.194+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:53.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:53 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:53.194+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:53.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:53 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:53.194+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:53.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:53 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:53.194+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:53.673 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 30 01:12:53 smithi116 bash[117527]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-9 2024-03-30T01:12:53.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:53 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:53.393+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:53.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:53 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:53.393+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:53.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:53 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:53.393+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:53.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:53 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:53.393+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:53.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:53 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:53.393+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:53.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:53 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:53.393+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.853941+0000 front 2024-03-30T01:12:21.853782+0000 (oldest deadline 2024-03-30T01:12:47.153684+0000) 2024-03-30T01:12:54.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:54 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:54.178+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:54.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:54 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:54.178+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:54.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:54 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:54.178+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:54.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:54 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:54.178+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:54.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:54 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:54.178+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:54.794 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.9.service' 2024-03-30T01:12:54.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:54 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:54.381+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:54.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:54 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:54.381+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:54.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:54 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:54.381+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:54.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:54 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:54.381+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:54.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:54 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:54.381+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:54.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:54 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:54.381+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.853941+0000 front 2024-03-30T01:12:21.853782+0000 (oldest deadline 2024-03-30T01:12:47.153684+0000) 2024-03-30T01:12:54.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:54 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:54.381+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:27.154334+0000 front 2024-03-30T01:12:27.154216+0000 (oldest deadline 2024-03-30T01:12:53.554495+0000) 2024-03-30T01:12:54.878 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:12:54.879 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-30T01:12:54.879 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-30T01:12:54.879 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.10 2024-03-30T01:12:55.338 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:54 smithi130 systemd[1]: Stopping Ceph osd.10 for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:12:55.338 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:55 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:55.107+0000 7f3237440700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:12:55.338 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:55 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:55.107+0000 7f3237440700 -1 osd.10 87 *** Got signal Terminated *** 2024-03-30T01:12:55.338 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:55 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:55.107+0000 7f3237440700 -1 osd.10 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T01:12:55.338 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:55 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:55.331+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:55.338 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:55 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:55.331+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:55.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:55 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:55.201+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:55.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:55 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:55.201+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:55.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:55 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:55.201+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:55.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:55 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:55.201+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:55.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:55 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:55.201+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:55.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:55 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:55.201+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:29.255141+0000 front 2024-03-30T01:12:29.255183+0000 (oldest deadline 2024-03-30T01:12:54.554899+0000) 2024-03-30T01:12:55.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:55 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:55.331+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:55.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:55 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:55.331+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:55.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:55 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:55.331+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:55.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:55 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:55.331+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.853941+0000 front 2024-03-30T01:12:21.853782+0000 (oldest deadline 2024-03-30T01:12:47.153684+0000) 2024-03-30T01:12:55.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:55 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:55.331+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:27.154334+0000 front 2024-03-30T01:12:27.154216+0000 (oldest deadline 2024-03-30T01:12:53.554495+0000) 2024-03-30T01:12:56.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:56 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:56.231+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:56.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:56 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:56.231+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:56.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:56 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:56.231+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:56.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:56 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:56.231+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:56.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:56 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:56.231+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:56.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:56 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:56.231+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:29.255141+0000 front 2024-03-30T01:12:29.255183+0000 (oldest deadline 2024-03-30T01:12:54.554899+0000) 2024-03-30T01:12:56.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:56 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:56.323+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:56.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:56 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:56.323+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:56.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:56 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:56.323+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:56.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:56 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:56.323+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:56.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:56 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:56.323+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:56.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:56 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:56.323+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.853941+0000 front 2024-03-30T01:12:21.853782+0000 (oldest deadline 2024-03-30T01:12:47.153684+0000) 2024-03-30T01:12:56.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:56 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:56.323+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:27.154334+0000 front 2024-03-30T01:12:27.154216+0000 (oldest deadline 2024-03-30T01:12:53.554495+0000) 2024-03-30T01:12:57.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:57 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:57.213+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:57.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:57 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:57.213+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:57.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:57 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:57.213+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:57.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:57 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:57.213+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:57.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:57 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:57.213+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:57.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:57 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:57.213+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:29.255141+0000 front 2024-03-30T01:12:29.255183+0000 (oldest deadline 2024-03-30T01:12:54.554899+0000) 2024-03-30T01:12:57.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:57 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:57.371+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:57.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:57 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:57.371+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:57.844 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:57 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:57.371+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:57.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:57 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:57.371+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:57.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:57 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:57.371+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:57.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:57 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:57.371+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.853941+0000 front 2024-03-30T01:12:21.853782+0000 (oldest deadline 2024-03-30T01:12:47.153684+0000) 2024-03-30T01:12:57.845 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:57 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:57.371+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:27.154334+0000 front 2024-03-30T01:12:27.154216+0000 (oldest deadline 2024-03-30T01:12:53.554495+0000) 2024-03-30T01:12:58.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:58 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:58.241+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:58.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:58 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:58.241+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:58.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:58 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:58.241+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:58.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:58 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:58.241+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:58.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:58 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:58.241+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:58.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:58 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:58.241+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:29.255141+0000 front 2024-03-30T01:12:29.255183+0000 (oldest deadline 2024-03-30T01:12:54.554899+0000) 2024-03-30T01:12:58.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:58 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:58.335+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:58.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:58 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:58.335+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:58.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:58 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:58.335+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:58.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:58 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:58.335+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:58.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:58 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:58.335+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:58.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:58 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:58.335+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.853941+0000 front 2024-03-30T01:12:21.853782+0000 (oldest deadline 2024-03-30T01:12:47.153684+0000) 2024-03-30T01:12:58.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:58 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:58.335+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:27.154334+0000 front 2024-03-30T01:12:27.154216+0000 (oldest deadline 2024-03-30T01:12:53.554495+0000) 2024-03-30T01:12:59.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:59 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:59.197+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:12:59.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:59 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:59.197+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:12:59.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:59 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:59.197+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:12:59.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:59 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:59.197+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:12:59.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:59 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:59.197+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:12:59.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:12:59 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:12:59.197+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:29.255141+0000 front 2024-03-30T01:12:29.255183+0000 (oldest deadline 2024-03-30T01:12:54.554899+0000) 2024-03-30T01:12:59.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:59 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:59.301+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:45.347543+0000 front 2024-03-30T01:11:45.347450+0000 (oldest deadline 2024-03-30T01:12:10.047384+0000) 2024-03-30T01:12:59.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:59 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:59.301+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2024-03-30T01:11:50.047856+0000 front 2024-03-30T01:11:50.047840+0000 (oldest deadline 2024-03-30T01:12:15.348068+0000) 2024-03-30T01:12:59.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:59 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:59.301+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:59.249931+0000 front 2024-03-30T01:11:59.249995+0000 (oldest deadline 2024-03-30T01:12:22.749885+0000) 2024-03-30T01:12:59.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:59 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:59.301+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:08.050745+0000 front 2024-03-30T01:12:08.050769+0000 (oldest deadline 2024-03-30T01:12:30.350678+0000) 2024-03-30T01:12:59.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:59 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:59.301+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:13.152022+0000 front 2024-03-30T01:12:13.152044+0000 (oldest deadline 2024-03-30T01:12:37.152683+0000) 2024-03-30T01:12:59.594 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:59 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:59.301+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.853941+0000 front 2024-03-30T01:12:21.853782+0000 (oldest deadline 2024-03-30T01:12:47.153684+0000) 2024-03-30T01:12:59.595 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:12:59 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10[77869]: 2024-03-30T01:12:59.301+0000 7f3233a36700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:27.154334+0000 front 2024-03-30T01:12:27.154216+0000 (oldest deadline 2024-03-30T01:12:53.554495+0000) 2024-03-30T01:13:00.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:00 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:00.216+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:13:00.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:00 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:00.216+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:13:00.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:00 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:00.216+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:13:00.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:00 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:00.216+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:13:00.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:00 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:00.216+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:13:00.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:00 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:00.216+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:29.255141+0000 front 2024-03-30T01:12:29.255183+0000 (oldest deadline 2024-03-30T01:12:54.554899+0000) 2024-03-30T01:13:00.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:00 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:00.216+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2024-03-30T01:12:36.255910+0000 front 2024-03-30T01:12:36.255996+0000 (oldest deadline 2024-03-30T01:12:59.755699+0000) 2024-03-30T01:13:01.094 INFO:journalctl@ceph.osd.10.smithi130.stdout:Mar 30 01:13:00 smithi130 bash[91271]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-10 2024-03-30T01:13:01.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:01 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:01.184+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:13:01.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:01 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:01.184+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:13:01.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:01 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:01.184+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:13:01.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:01 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:01.184+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:13:01.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:01 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:01.184+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:13:01.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:01 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:01.184+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:29.255141+0000 front 2024-03-30T01:12:29.255183+0000 (oldest deadline 2024-03-30T01:12:54.554899+0000) 2024-03-30T01:13:01.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:01 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:01.184+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2024-03-30T01:12:36.255910+0000 front 2024-03-30T01:12:36.255996+0000 (oldest deadline 2024-03-30T01:12:59.755699+0000) 2024-03-30T01:13:01.926 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.10.service' 2024-03-30T01:13:01.966 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:13:01.967 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-30T01:13:01.967 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-30T01:13:01.967 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.11 2024-03-30T01:13:02.251 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:02 smithi181 systemd[1]: Stopping Ceph osd.11 for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:13:02.252 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:02 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:02.212+0000 7f7bd781d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:13:02.252 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:02 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:02.212+0000 7f7bd781d700 -1 osd.11 87 *** Got signal Terminated *** 2024-03-30T01:13:02.252 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:02 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:02.212+0000 7f7bd781d700 -1 osd.11 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T01:13:02.252 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:02 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:02.233+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:13:02.252 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:02 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:02.233+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:13:02.252 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:02 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:02.233+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:13:02.252 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:02 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:02.233+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:13:02.252 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:02 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:02.233+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:13:02.252 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:02 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:02.233+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:29.255141+0000 front 2024-03-30T01:12:29.255183+0000 (oldest deadline 2024-03-30T01:12:54.554899+0000) 2024-03-30T01:13:02.252 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:02 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:02.233+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2024-03-30T01:12:36.255910+0000 front 2024-03-30T01:12:36.255996+0000 (oldest deadline 2024-03-30T01:12:59.755699+0000) 2024-03-30T01:13:03.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:03 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:03.222+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:13:03.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:03 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:03.222+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:13:03.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:03 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:03.222+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:13:03.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:03 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:03.222+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:13:03.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:03 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:03.222+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:13:03.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:03 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:03.222+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:29.255141+0000 front 2024-03-30T01:12:29.255183+0000 (oldest deadline 2024-03-30T01:12:54.554899+0000) 2024-03-30T01:13:03.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:03 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:03.222+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2024-03-30T01:12:36.255910+0000 front 2024-03-30T01:12:36.255996+0000 (oldest deadline 2024-03-30T01:12:59.755699+0000) 2024-03-30T01:13:04.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:04 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:04.185+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:13:04.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:04 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:04.185+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:13:04.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:04 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:04.185+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:13:04.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:04 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:04.185+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:13:04.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:04 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:04.185+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:13:04.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:04 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:04.185+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:29.255141+0000 front 2024-03-30T01:12:29.255183+0000 (oldest deadline 2024-03-30T01:12:54.554899+0000) 2024-03-30T01:13:04.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:04 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:04.185+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2024-03-30T01:12:36.255910+0000 front 2024-03-30T01:12:36.255996+0000 (oldest deadline 2024-03-30T01:12:59.755699+0000) 2024-03-30T01:13:05.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:05 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:05.163+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:13:05.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:05 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:05.163+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:13:05.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:05 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:05.163+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:13:05.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:05 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:05.163+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:13:05.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:05 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:05.163+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:13:05.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:05 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:05.163+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:29.255141+0000 front 2024-03-30T01:12:29.255183+0000 (oldest deadline 2024-03-30T01:12:54.554899+0000) 2024-03-30T01:13:05.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:05 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:05.163+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2024-03-30T01:12:36.255910+0000 front 2024-03-30T01:12:36.255996+0000 (oldest deadline 2024-03-30T01:12:59.755699+0000) 2024-03-30T01:13:06.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:06 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:06.203+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:13:06.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:06 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:06.203+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:13:06.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:06 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:06.203+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:13:06.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:06 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:06.203+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:13:06.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:06 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:06.203+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:13:06.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:06 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:06.203+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:29.255141+0000 front 2024-03-30T01:12:29.255183+0000 (oldest deadline 2024-03-30T01:12:54.554899+0000) 2024-03-30T01:13:06.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:06 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:06.203+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2024-03-30T01:12:36.255910+0000 front 2024-03-30T01:12:36.255996+0000 (oldest deadline 2024-03-30T01:12:59.755699+0000) 2024-03-30T01:13:07.501 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:07 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:07.185+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-30T01:11:43.946984+0000 front 2024-03-30T01:11:43.947064+0000 (oldest deadline 2024-03-30T01:12:09.846992+0000) 2024-03-30T01:13:07.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:07 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:07.185+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6804 osd.2 since back 2024-03-30T01:11:57.949205+0000 front 2024-03-30T01:11:57.949228+0000 (oldest deadline 2024-03-30T01:12:21.449346+0000) 2024-03-30T01:13:07.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:07 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:07.185+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-30T01:12:03.750432+0000 front 2024-03-30T01:12:03.750376+0000 (oldest deadline 2024-03-30T01:12:29.050237+0000) 2024-03-30T01:13:07.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:07 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:07.185+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2024-03-30T01:12:15.451623+0000 front 2024-03-30T01:12:15.451608+0000 (oldest deadline 2024-03-30T01:12:36.551734+0000) 2024-03-30T01:13:07.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:07 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:07.185+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.181:6812 osd.5 since back 2024-03-30T01:12:21.753603+0000 front 2024-03-30T01:12:21.753668+0000 (oldest deadline 2024-03-30T01:12:44.653711+0000) 2024-03-30T01:13:07.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:07 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:07.185+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-30T01:12:29.255141+0000 front 2024-03-30T01:12:29.255183+0000 (oldest deadline 2024-03-30T01:12:54.554899+0000) 2024-03-30T01:13:07.502 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:07 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11[78603]: 2024-03-30T01:13:07.185+0000 7f7bd3e13700 -1 osd.11 87 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2024-03-30T01:12:36.255910+0000 front 2024-03-30T01:12:36.255996+0000 (oldest deadline 2024-03-30T01:12:59.755699+0000) 2024-03-30T01:13:08.251 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 30 01:13:07 smithi181 bash[92476]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-osd-11 2024-03-30T01:13:09.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@osd.11.service' 2024-03-30T01:13:09.178 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:13:09.178 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-30T01:13:09.178 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-30T01:13:09.179 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.a 2024-03-30T01:13:09.538 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 30 01:13:09 smithi116 systemd[1]: Stopping Ceph mds.a for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:13:09.923 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 30 01:13:09 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-a[88590]: 2024-03-30T01:13:09.531+0000 7fcd78e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:13:09.923 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 30 01:13:09 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-a[88590]: 2024-03-30T01:13:09.531+0000 7fcd78e00700 -1 mds.a *** got signal Terminated *** 2024-03-30T01:13:13.923 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 30 01:13:13 smithi116 bash[117790]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-a 2024-03-30T01:13:14.023 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.a.service' 2024-03-30T01:13:14.105 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:13:14.105 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-30T01:13:14.106 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-30T01:13:14.106 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.d 2024-03-30T01:13:14.423 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 30 01:13:14 smithi116 systemd[1]: Stopping Ceph mds.d for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:13:14.923 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 30 01:13:14 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-d[87639]: 2024-03-30T01:13:14.463+0000 7f1710600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:13:14.923 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 30 01:13:14 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-d[87639]: 2024-03-30T01:13:14.463+0000 7f1710600700 -1 mds.d *** got signal Terminated *** 2024-03-30T01:13:16.414 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 30 01:13:16 smithi116 bash[117928]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-d 2024-03-30T01:13:16.442 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.d.service' 2024-03-30T01:13:16.521 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:13:16.521 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-30T01:13:16.521 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-30T01:13:16.521 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.g 2024-03-30T01:13:16.862 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 30 01:13:16 smithi116 systemd[1]: Stopping Ceph mds.g for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:13:17.173 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 30 01:13:16 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-g[88207]: 2024-03-30T01:13:16.855+0000 7fb8ae400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:13:17.173 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 30 01:13:16 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-g[88207]: 2024-03-30T01:13:16.855+0000 7fb8ae400700 -1 mds.g *** got signal Terminated *** 2024-03-30T01:13:19.423 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 30 01:13:19 smithi116 bash[118067]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-g 2024-03-30T01:13:19.554 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.g.service' 2024-03-30T01:13:19.633 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:13:19.633 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-30T01:13:19.633 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-30T01:13:19.633 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.j 2024-03-30T01:13:19.923 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 30 01:13:19 smithi116 systemd[1]: Stopping Ceph mds.j for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:13:20.423 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 30 01:13:20 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-j[87252]: 2024-03-30T01:13:20.013+0000 7f18ad800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:13:20.423 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 30 01:13:20 smithi116 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-j[87252]: 2024-03-30T01:13:20.013+0000 7f18ad800700 -1 mds.j *** got signal Terminated *** 2024-03-30T01:13:21.923 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 30 01:13:21 smithi116 bash[118204]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-j 2024-03-30T01:13:22.111 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.j.service' 2024-03-30T01:13:22.189 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:13:22.189 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-30T01:13:22.189 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-30T01:13:22.189 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.b 2024-03-30T01:13:22.518 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 01:13:22 smithi130 systemd[1]: Stopping Ceph mds.b for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:13:22.844 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 01:13:22 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-b[84904]: 2024-03-30T01:13:22.511+0000 7f363cb01700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:13:22.844 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 01:13:22 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-b[84904]: 2024-03-30T01:13:22.511+0000 7f363cb01700 -1 mds.b *** got signal Terminated *** 2024-03-30T01:13:26.594 INFO:journalctl@ceph.mds.b.smithi130.stdout:Mar 30 01:13:26 smithi130 bash[91533]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-b 2024-03-30T01:13:26.652 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.b.service' 2024-03-30T01:13:26.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:13:26.690 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-30T01:13:26.691 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-30T01:13:26.691 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.e 2024-03-30T01:13:26.941 INFO:journalctl@ceph.mds.e.smithi130.stdout:Mar 30 01:13:26 smithi130 systemd[1]: Stopping Ceph mds.e for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:13:27.344 INFO:journalctl@ceph.mds.e.smithi130.stdout:Mar 30 01:13:26 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-e[84221]: 2024-03-30T01:13:26.935+0000 7fb2a5836700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:13:27.344 INFO:journalctl@ceph.mds.e.smithi130.stdout:Mar 30 01:13:26 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-e[84221]: 2024-03-30T01:13:26.935+0000 7fb2a5836700 -1 mds.e *** got signal Terminated *** 2024-03-30T01:13:30.579 INFO:journalctl@ceph.mds.e.smithi130.stdout:Mar 30 01:13:30 smithi130 bash[91674]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-e 2024-03-30T01:13:30.585 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.e.service' 2024-03-30T01:13:30.619 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:13:30.620 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-30T01:13:30.620 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-30T01:13:30.620 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.h 2024-03-30T01:13:30.844 INFO:journalctl@ceph.mds.h.smithi130.stdout:Mar 30 01:13:30 smithi130 systemd[1]: Stopping Ceph mds.h for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:13:31.344 INFO:journalctl@ceph.mds.h.smithi130.stdout:Mar 30 01:13:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-h[84562]: 2024-03-30T01:13:30.868+0000 7fae73182700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:13:31.344 INFO:journalctl@ceph.mds.h.smithi130.stdout:Mar 30 01:13:30 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-h[84562]: 2024-03-30T01:13:30.868+0000 7fae73182700 -1 mds.h *** got signal Terminated *** 2024-03-30T01:13:34.575 INFO:journalctl@ceph.mds.h.smithi130.stdout:Mar 30 01:13:34 smithi130 bash[91815]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-h 2024-03-30T01:13:34.580 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.h.service' 2024-03-30T01:13:34.614 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:13:34.614 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-30T01:13:34.614 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-30T01:13:34.614 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.k 2024-03-30T01:13:34.844 INFO:journalctl@ceph.mds.k.smithi130.stdout:Mar 30 01:13:34 smithi130 systemd[1]: Stopping Ceph mds.k for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:13:35.344 INFO:journalctl@ceph.mds.k.smithi130.stdout:Mar 30 01:13:34 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-k[83881]: 2024-03-30T01:13:34.859+0000 7ff03c945700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:13:35.344 INFO:journalctl@ceph.mds.k.smithi130.stdout:Mar 30 01:13:34 smithi130 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-k[83881]: 2024-03-30T01:13:34.859+0000 7ff03c945700 -1 mds.k *** got signal Terminated *** 2024-03-30T01:13:39.433 INFO:journalctl@ceph.mds.k.smithi130.stdout:Mar 30 01:13:39 smithi130 bash[91953]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-k 2024-03-30T01:13:39.439 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.k.service' 2024-03-30T01:13:39.473 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:13:39.473 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-30T01:13:39.473 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-30T01:13:39.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.c 2024-03-30T01:13:40.002 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 30 01:13:39 smithi181 systemd[1]: Stopping Ceph mds.c for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:13:40.002 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 30 01:13:39 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-c[85430]: 2024-03-30T01:13:39.715+0000 7f7b3c2cf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:13:40.002 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 30 01:13:39 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-c[85430]: 2024-03-30T01:13:39.715+0000 7f7b3c2cf700 -1 mds.c *** got signal Terminated *** 2024-03-30T01:13:42.501 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 30 01:13:42 smithi181 bash[92736]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-c 2024-03-30T01:13:42.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.c.service' 2024-03-30T01:13:42.632 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:13:42.632 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-30T01:13:42.632 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-30T01:13:42.633 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.f 2024-03-30T01:13:42.895 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 30 01:13:42 smithi181 systemd[1]: Stopping Ceph mds.f for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:13:43.251 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 30 01:13:42 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-f[85084]: 2024-03-30T01:13:42.888+0000 7fcc213df700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:13:43.251 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 30 01:13:42 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-f[85084]: 2024-03-30T01:13:42.888+0000 7fcc213df700 -1 mds.f *** got signal Terminated *** 2024-03-30T01:13:46.501 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 30 01:13:46 smithi181 bash[92875]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-f 2024-03-30T01:13:46.543 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.f.service' 2024-03-30T01:13:46.577 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:13:46.578 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-30T01:13:46.578 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-30T01:13:46.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.i 2024-03-30T01:13:46.835 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 01:13:46 smithi181 systemd[1]: Stopping Ceph mds.i for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:13:47.251 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 01:13:46 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-i[85770]: 2024-03-30T01:13:46.829+0000 7f2428dfd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:13:47.251 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 01:13:46 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-i[85770]: 2024-03-30T01:13:46.829+0000 7f2428dfd700 -1 mds.i *** got signal Terminated *** 2024-03-30T01:13:50.501 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 30 01:13:50 smithi181 bash[93016]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-i 2024-03-30T01:13:50.585 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.i.service' 2024-03-30T01:13:50.619 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:13:50.620 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-30T01:13:50.620 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-30T01:13:50.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.l 2024-03-30T01:13:50.885 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 30 01:13:50 smithi181 systemd[1]: Stopping Ceph mds.l for 4909f004-ee2f-11ee-b647-cb9ed24678a4... 2024-03-30T01:13:51.251 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 30 01:13:50 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-l[84743]: 2024-03-30T01:13:50.878+0000 7fef4732d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T01:13:51.251 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 30 01:13:50 smithi181 ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-l[84743]: 2024-03-30T01:13:50.878+0000 7fef4732d700 -1 mds.l *** got signal Terminated *** 2024-03-30T01:13:53.754 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 30 01:13:53 smithi181 bash[93154]: ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4-mds-l 2024-03-30T01:13:53.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4909f004-ee2f-11ee-b647-cb9ed24678a4@mds.l.service' 2024-03-30T01:13:53.828 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T01:13:53.829 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-30T01:13:53.829 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T01:13:54.088 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T01:13:56.907 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T01:13:57.095 INFO:teuthology.orchestra.run.smithi130.stdout:Deleting cluster with fsid: 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T01:13:58.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T01:13:58.849 INFO:teuthology.orchestra.run.smithi181.stdout:Deleting cluster with fsid: 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T01:14:00.314 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T01:14:00.391 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T01:14:00.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T01:14:00.458 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-30T01:14:00.460 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254/remote/smithi116/crash 2024-03-30T01:14:00.461 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T01:14:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:tar: /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T01:14:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:tar: Error is not recoverable: exiting now 2024-03-30T01:14:00.531 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254/remote/smithi130/crash 2024-03-30T01:14:00.532 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T01:14:00.681 INFO:teuthology.orchestra.run.smithi130.stderr:tar: /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T01:14:00.681 INFO:teuthology.orchestra.run.smithi130.stderr:tar: Error is not recoverable: exiting now 2024-03-30T01:14:00.683 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254/remote/smithi181/crash 2024-03-30T01:14:00.684 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T01:14:00.729 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T01:14:00.729 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now 2024-03-30T01:14:00.730 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-30T01:14:00.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-30T01:14:00.806 INFO:teuthology.orchestra.run.smithi116.stderr:grep: /var/log/ceph/4909f004-ee2f-11ee-b647-cb9ed24678a4/ceph.log: No such file or directory 2024-03-30T01:14:00.811 INFO:tasks.cephadm:Compressing logs... 2024-03-30T01:14:00.812 DEBUG:teuthology.orchestra.run.smithi116:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T01:14:00.815 DEBUG:teuthology.orchestra.run.smithi130:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T01:14:00.818 DEBUG:teuthology.orchestra.run.smithi181:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T01:14:00.851 INFO:teuthology.orchestra.run.smithi130.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-30T01:14:00.859 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-30T01:14:00.875 INFO:teuthology.orchestra.run.smithi116.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-30T01:23:30.356 INFO:tasks.cephadm:Archiving logs... 2024-03-30T01:23:30.357 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254/remote/smithi116/log 2024-03-30T01:23:30.358 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T01:23:54.259 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254/remote/smithi130/log 2024-03-30T01:23:54.260 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T01:25:06.132 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254/remote/smithi181/log 2024-03-30T01:25:06.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T01:25:48.364 INFO:tasks.cephadm:Removing cluster... 2024-03-30T01:25:48.365 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 --force 2024-03-30T01:25:48.625 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T01:25:49.592 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 --force 2024-03-30T01:25:50.037 INFO:teuthology.orchestra.run.smithi130.stdout:Deleting cluster with fsid: 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T01:25:50.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4909f004-ee2f-11ee-b647-cb9ed24678a4 --force 2024-03-30T01:25:50.947 INFO:teuthology.orchestra.run.smithi181.stdout:Deleting cluster with fsid: 4909f004-ee2f-11ee-b647-cb9ed24678a4 2024-03-30T01:25:51.514 INFO:tasks.cephadm:Removing cephadm ... 2024-03-30T01:25:51.514 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-30T01:25:51.561 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-30T01:25:51.593 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-30T01:25:51.625 INFO:tasks.cephadm:Teardown complete 2024-03-30T01:25:51.625 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-30T01:25:51.636 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-30T01:25:51.636 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T01:25:51.641 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T01:25:51.646 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T01:25:51.777 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-30T01:25:51.787 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-30T01:25:51.787 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T01:25:51.792 DEBUG:teuthology.orchestra.run.smithi130:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T01:25:51.794 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T01:25:51.810 INFO:teuthology.orchestra.run.smithi181.stderr:bash: ntpq: command not found 2024-03-30T01:25:51.810 INFO:teuthology.orchestra.run.smithi130.stderr:bash: ntpq: command not found 2024-03-30T01:25:51.828 INFO:teuthology.orchestra.run.smithi116.stderr:bash: ntpq: command not found 2024-03-30T01:25:51.829 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T01:25:51.829 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-03-30T01:25:51.829 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 +898us[ +898us] +/- 67ms 2024-03-30T01:25:51.829 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 +128us[ +291us] +/- 93ms 2024-03-30T01:25:51.829 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 -1243us[-1080us] +/- 91ms 2024-03-30T01:25:51.829 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T01:25:51.836 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T01:25:51.836 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-30T01:25:51.836 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +1029us[+1029us] +/- 67ms 2024-03-30T01:25:51.836 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 7 377 117 +289us[ +372us] +/- 92ms 2024-03-30T01:25:51.836 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 -1135us[-1135us] +/- 91ms 2024-03-30T01:25:51.836 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T01:25:51.850 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T01:25:51.850 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-03-30T01:25:51.850 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 +893us[ +893us] +/- 67ms 2024-03-30T01:25:51.850 INFO:teuthology.orchestra.run.smithi130.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 +123us[ +225us] +/- 93ms 2024-03-30T01:25:51.854 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 -1249us[-1147us] +/- 91ms 2024-03-30T01:25:51.854 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T01:25:51.855 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-30T01:25:51.866 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-30T01:25:51.867 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-30T01:25:51.876 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T01:25:52.012 DEBUG:teuthology.orchestra.run.smithi130:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T01:25:52.114 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T01:25:52.220 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T01:25:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T01:25:52.291 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T01:25:52.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T01:25:52.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T01:25:52.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T01:25:52.354 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-30T01:25:52.365 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-30T01:25:52.376 INFO:teuthology.task.internal:Duration was 3091.759667 seconds 2024-03-30T01:25:52.376 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-30T01:25:52.387 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-30T01:25:52.387 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T01:25:52.392 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T01:25:52.394 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T01:25:52.481 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T01:25:52.483 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T01:25:52.499 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T01:25:52.779 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-30T01:25:52.780 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-03-30T01:25:52.780 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T01:25:52.836 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi130.front.sepia.ceph.com 2024-03-30T01:25:52.836 DEBUG:teuthology.orchestra.run.smithi130:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T01:25:52.867 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-03-30T01:25:52.868 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T01:25:52.934 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-30T01:25:52.934 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T01:25:52.945 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T01:25:52.949 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T01:25:53.038 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-30T01:25:53.038 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T01:25:53.042 DEBUG:teuthology.orchestra.run.smithi130:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T01:25:53.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T01:25:53.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-30T01:25:53.221 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-30T01:25:53.221 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T01:25:53.290 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T01:25:53.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T01:25:53.402 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-30T01:25:53.414 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T01:25:53.418 DEBUG:teuthology.orchestra.run.smithi130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T01:25:53.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T01:25:53.454 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-03-30T01:25:53.457 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = core 2024-03-30T01:25:53.477 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-03-30T01:25:53.512 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T01:25:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T01:25:53.551 DEBUG:teuthology.orchestra.run.smithi130:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T01:25:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T01:25:53.568 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T01:25:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T01:25:53.584 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-30T01:25:53.596 INFO:teuthology.task.internal:Transferring archived files... 2024-03-30T01:25:53.598 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254/remote/smithi116 2024-03-30T01:25:53.598 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T01:25:53.679 DEBUG:teuthology.misc:Transferring archived files from smithi130:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254/remote/smithi130 2024-03-30T01:25:53.680 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T01:25:53.721 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254/remote/smithi181 2024-03-30T01:25:53.721 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T01:25:53.762 INFO:teuthology.task.internal:Removing archive directory... 2024-03-30T01:25:53.762 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T01:25:53.766 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T01:25:53.768 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T01:25:53.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-30T01:25:53.825 INFO:teuthology.task.internal:Not uploading archives. 2024-03-30T01:25:53.826 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-30T01:25:53.837 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-30T01:25:53.837 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T01:25:53.848 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T01:25:53.850 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T01:25:53.865 INFO:teuthology.orchestra.run.smithi130.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 01:25 /home/ubuntu/cephtest 2024-03-30T01:25:53.869 INFO:teuthology.orchestra.run.smithi181.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 01:25 /home/ubuntu/cephtest 2024-03-30T01:25:53.885 INFO:teuthology.orchestra.run.smithi116.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 01:25 /home/ubuntu/cephtest 2024-03-30T01:25:53.889 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-30T01:25:53.902 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-30T01:25:53.934 INFO:teuthology.nuke:Checking targets against current locks 2024-03-30T01:25:53.958 DEBUG:teuthology.nuke:shortname: smithi116 2024-03-30T01:25:53.959 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T01:25:53.980 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 00:23:46.500069', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA6XbZFTDc54XT+aHs3Y2K72HMrW5XLMk+7FAiTHcVxVgTja3iUR1A0aKnE602FFJd1crdGwHYrks5TRnOyHJ+8='} 2024-03-30T01:25:53.985 DEBUG:teuthology.nuke:shortname: smithi130 2024-03-30T01:25:53.985 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T01:25:54.005 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 00:23:46.498528', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMX4g0uFsn++8qv8DwFoauiwYRSbKZAeu6KCLqrAg8f6uAXA+IM9hl1MkMM1NlTwd9IdHWv/V9wA/DbTjf80MPY='} 2024-03-30T01:25:54.008 DEBUG:teuthology.nuke:shortname: smithi181 2024-03-30T01:25:54.008 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T01:25:54.029 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 00:23:46.501407', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNwPJjYQmBQoAIcUGr6zYEui9LBk9TBV9mpoOZxEWpIAvCfb6aY5bd5GXH+KO6A/5kNuWWClpHLNZKdGzdie1Jw='} 2024-03-30T01:25:54.038 INFO:teuthology.orchestra.console.smithi116:Power off 2024-03-30T01:25:54.039 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T01:25:54.056 INFO:teuthology.orchestra.console.smithi130:Power off 2024-03-30T01:25:54.056 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T01:25:54.071 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2024-03-30T01:25:54.071 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:25:54.083 DEBUG:teuthology.orchestra.console.smithi130:power off output: Chassis Power Control: Down/Off 2024-03-30T01:25:54.083 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:25:54.096 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-30T01:25:54.104 INFO:teuthology.orchestra.console.smithi181:Power off 2024-03-30T01:25:54.105 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T01:25:54.118 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-03-30T01:25:54.131 DEBUG:teuthology.orchestra.console.smithi181:power off output: Chassis Power Control: Down/Off 2024-03-30T01:25:54.132 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:25:54.156 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-03-30T01:25:58.096 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:25:58.119 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:25:58.158 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:25:58.209 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-30T01:25:58.232 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-03-30T01:25:58.273 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-03-30T01:26:02.214 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:26:02.233 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:26:02.274 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:26:02.326 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-30T01:26:02.343 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-03-30T01:26:02.384 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is off 2024-03-30T01:26:02.485 INFO:teuthology.orchestra.console.smithi181:Power off completed 2024-03-30T01:26:06.329 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:26:06.344 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:26:06.440 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2024-03-30T01:26:06.454 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-03-30T01:26:06.542 INFO:teuthology.orchestra.console.smithi116:Power off completed 2024-03-30T01:26:06.555 INFO:teuthology.orchestra.console.smithi130:Power off completed 2024-03-30T01:26:06.661 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} duration: 3091.7596673965454 failure_reason: 'error during scrub thrashing: rank damage found: {''backtrace''}' flavor: default owner: scheduled_rishabh@teuthology status: fail success: false 2024-03-30T01:26:06.661 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T01:26:06.746 INFO:teuthology.run:FAIL